2 Nature studies illustrate the ability of AI to dive deeper into medical images and pathology

In two separate studies, researchers have used machine learning to train computers to spot and classify cells—as well as their hidden, internal structures and DNA—by using images alone.

For a specific example, NYU researchers using an open source, deep neural network developed by Google have taught an algorithm to differentiate between two types of lung cancer and underlying genetic mutations, by visually analyzing slide images of tumor samples. The technology could provide an important second opinion when deciding on a course of treatment.

From a slice of cancerous tissue, the algorithm was able to distinguish between adenocarcinoma and squamous cell carcinoma with 97% accuracy, two visually similar lung cancer types that can be difficult for pathologists to single out without additional tests.

The artificial intelligence program was also able to detect abnormal versions of six genes—including EGFR, KRAS and TP53, based on cell shape and interactions with its surroundings—with accuracies ranging from 73% to 86%, depending on the gene. The study’s results were published in Nature Medicine.

To effectively wield today’s targeted cancer therapies, one first needs the target—but genetic tests to confirm specific mutations may take weeks.   

"Delaying the start of cancer treatment is never good," said study author Aristotelis Tsirigos, Ph.D., an associate professor of pathology at the NYU School of Medicine and NYU Langone Health's Perlmutter Cancer Center. “Our study provides strong evidence that an AI approach will be able to instantly determine cancer subtype and mutational profile to get patients started on targeted therapies sooner.”

Using a database of images from The Cancer Genome Atlas with predetermined diagnoses, the study found that about half of the small percentage of images misclassified by the AI were also misclassified by pathologists. But at the same time, 45 of the 54 images misclassified by at least one pathologist in the study were assigned correctly by the program.

"In our study, we were excited to improve on pathologist-level accuracies, and to show that AI can discover previously unknown patterns in the visible features of cancer cells and the tissues around them," said co-corresponding author Narges Razavian, Ph.D., an assistant professor in the departments of Radiology and Population Health.

The NYU researchers plan to keep training its AI until it can determine genetic mutations with more than 90% accuracy, before seeking regulatory approvals as a diagnostic for several cancer types.

In a separate, broader application, scientists at the Allen Institute for Cell Science trained their programs to find proteins and structures inside living cells using black-and-white images from brightfield microscopy—a much cheaper option than tagging and following cell processes using fluorescent, glowing molecules, which only works with a few structures at a time.

"This technology lets us view a larger set of those structures than was possible before," said the Allen Institute’s Greg Johnson, Ph.D. "This means that we can explore the organization of the cell in ways that nobody has been able to do, especially in live cells."

Human cells have upwards of 20,000 proteins that, if observed together, could reveal important findings about both healthy and diseased cells at a lower cost, the researchers said. Their study was published in Nature Methods.

"This technique has huge potential ramifications for these and related fields," said Rick Horwitz, Ph.D., executive director of the Allen Institute for Cell Science. Researchers described applications for better understanding cancer treatment responses, mapping neuron connections and more.

To the human eye, cells in a brightfield microscope are rendered in shades of gray and offer little more than the edges of the cell and its nucleus. Using the same images, the AI was able to discern details such as the mitochondria.

"You can watch processes live as they are taking place—it's almost like magic,” Horwitz said. “This method allows us, in the most non-invasive way that we have so far, to obtain information about human cells that we were previously unable to get."