DeepMind--a Google-owned artificial intelligence company--has decided to suspend use of its Streams app, co-designed with the National Health Service (NHS), until the software has been registered as a medical device with the Medicines & Healthcare products Regulatory Agency (MHRA), TechCrunch alleged in a report.
The interesting part of this story, TechCrunch explained, isn’t that the app needs approval, but rather that despite that fact it has been used in multiple NHS hospitals in what were called “user tests.” According to the Royal Free NHS Trust, the app was used by up to 6 clinicians in three tests in London hospitals. A spokesperson for DeepMind could not confirm how many patients were involved in the three tests.
For those not following along as easily, TechCrunch laid it out quite clearly: “Which, put another way, means a profit-driven commercial entity has been involved in a real-world test of an unregistered medical device on actual hospital patients.”
TechCrunch noted that in May, the Royal Free Trust had stopped using the app in the wake of discussions with the MHRA regarding if the app should be registered as a medical device. Regulatory compliance at that time could be helped by parties having “a discussion” before tests, though this did not happen in this case, an MHRA spokesperson said.
“MHRA understands that the Streams software is based on the NHS England AKI algorithm and is likely to be a class I medical device,” a MHRA spokesperson told TechCrunch. “As a class 1 device it is subject to self-declaration. The manufacturer of the device must hold all relevant test and validation data [including for software] to support the intended purpose for the device.”
DeepMind seems to have every intention to go through the necessary steps to gain the needed certifications. A spokesperson for DeepMind noted that the company is still developing the Steams app and intends to ensure it complies with all EU and U.K. medical device legislation. “We would only place the Streams app on the market as a medical device after it had been fully certified, CE-marked and registered,” the spokesperson said.
A Royal Free spokesperson confirmed that DeepMind is working with the MHRA on device compliance, though offered no word on when the device will be used again.
The relationship between DeepMind and the NHS has seen controversy from early on, particularly in the realm of data sharing. The data sharing agreement between DeepMind and the NHS offers DeepMind access to possible millions of identifiable medical records of NHS patients without the need for patient consent. The Stream app aims to speed up identification of acute kidney injury, and thus DeepMind claims to need a wide range of data to achieve this. However, TechCrunch noted that this way of targeting a condition affecting groups of patients would be considered “indirect care” and not “direct care,” which is what’s needed for implied consent to access identifiable data.
Additionally, Moorfields Eye Hospital in London has shared around one million non-identifiable pieces of data in the form of detailed biometric eye scans. While the scans themselves are not identified, scans with that much detailed data surely could be easily linked to individuals in the event of a data link, TechCrunch explained.
DeepMind co-founder Mustafa Suleyman noted that the focus is currently on developing tools, and the company will worry about monetizing them later. The aim is to monetize the tool based on outcomes. “Ultimately we want to get paid when we deliver concrete clinical benefits,” Suleyman told the BBC recently. “We want to get paid to change the system and improve patient outcomes.”
As TechCrunch explains, this essentially means that the publicly funded NHS is providing free health datasets to DeepMind which, in turn, is using those data to build up a machine learning model. If that tool proves successful, it could be used to bill the NHS in the future based on the tool’s effectiveness. No doubt this case is bound to bring up more than a few questions about data sharing and profiting in the future.
- here's the article