Cancer diagnosis and prognosis are largely based on two main approaches. In one, histopathologists examine the appearance of cancertissue under the microscope. In the other, cancer geneticists analyze the changes that occur in the genetic code of cancer cells. Both approaches are essential to understand and treat cancer, but they are rarely used together.

"Clinicians use microscopy slides for cancer diagnosis all the time. However, the full potential of these slides hasn't been unlocked yet. As computer vision advances, we can analyze digital images of these slides to understand what happens at a molecular level," says Yu Fu, Postdoctoral Fellow in the Gerstung Group at EMBL-EBI.

Computer vision algorithms are a form of artificial intelligence that can recognize certain features in images. Fu and colleagues repurposed such an algorithm developed by Google—originally used to classify everyday objects such as lemons, sunglasses and radiators—to distinguish various cancer types from healthy tissue. They showed that this algorithm can also be used to predict survival and even patterns of DNA and RNA changes from images of tumor tissue.

Teaching algorithms to detect molecular changes

Previous studies have used similar methods to analyze images from single or a few cancer types with selected molecular alterations. However, Fu and colleagues generalized the approach on an unprecedented scale: they trained the algorithm with more than 17,000 images from 28 cancer types collected for The Cancer Genome Atlas, and studied all known genomic alterations.

"What is quite remarkable is that our algorithm can automatically link the histological appearance of almost any tumor with a very broad set of molecular characteristics, and with patient survival," explains Moritz Gerstung, Group Leader at EMBL-EBI.

Overall, their algorithm was capable of detecting patterns of 167 different mutations and thousands of gene activity changes. These findings show in detail how genetic mutations alter the appearance of tumor cells and tissues.

Another research group has independently validated these results with a similar AI algorithm applied to images from eight cancer types. Their study was published in the same issue of Nature Cancer.

A potential tool for personalized medicine

The integration of molecular and histopathological data provides a clearer picture of a tumor's profile. Detecting the molecular features, cell composition, and survival associated with individual tumors would help clinicians tailor appropriate treatments to their patients' needs.

"From a clinician's point of view, these findings are incredibly exciting. Our work shows how artificial intelligence could be used in clinical practice," explains Luiza Moore, Clinician Scientist and Pathologist at the Wellcome Sanger Institute and Addenbrooke's Hospital. "While the number of cancer cases is increasing worldwide, the number of pathologists is declining. At the same time, we strive to move away from the 'one size fits all' approach and into personalized medicine. A combination of digital pathology and artificial intelligence can potentially alleviate those pressures and enhance our practice and patient care."

Sequencing technologies have propelled genomics to the forefront of cancer research, yet these technologies remain inaccessible to most clinics around the world. A possible alternative to direct sequencing would be to use AI to emulate a genomic analysis using data that is cheaper to collect, like microscopy slides.

"Getting all that information from standard tumor images in a completely automatic manner is revolutionary," says Alexander Jung, Ph.D. student at EMBL-EBI. "This study shows what might be possible in the coming years, but these algorithms will have to be refined before clinical implementation."