Treffer: Deep learning-based multimodal pathogenomics integration for precision cancer prognosis.

Title:
Deep learning-based multimodal pathogenomics integration for precision cancer prognosis.
Authors:
Feng X; College of Electrical and Information Engineering, Hunan University, Changsha, 410082, China.; Zhejiang Cancer Hospital, Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, 310022, China., Song G; Zhejiang Cancer Hospital, Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, 310022, China., Zhang Y; Zhejiang Cancer Hospital, Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, 310022, China.; Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, 310024, China., Guo L; Zhejiang Cancer Hospital, Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, 310022, China., Jiang Y; Zhejiang Cancer Hospital, Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, 310022, China., Gong W; Zhejiang Cancer Hospital, Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, 310022, China., Feng Y; Zhejiang Cancer Hospital, Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, 310022, China., Xu C; Zhejiang Cancer Hospital, Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, 310022, China.; Department of Respiratory Medicine, Affiliated Jinling Hospital, Medical School of Nanjing University, Nanjing, 210002, China., Yang Y; Zhejiang Cancer Hospital, Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, 310022, China. yangyang@zjcc.org.cn.; Zhejiang Key Laboratory of Radiation Oncology, Hangzhou, 310022, China. yangyang@zjcc.org.cn., He M; College of Electrical and Information Engineering, Hunan University, Changsha, 410082, China. hemin@him.cas.cn.; Zhejiang Cancer Hospital, Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, 310022, China. hemin@him.cas.cn.; Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, 310024, China. hemin@him.cas.cn.
Source:
Journal of translational medicine [J Transl Med] 2026 Jan 17. Date of Electronic Publication: 2026 Jan 17.
Publication Model:
Ahead of Print
Publication Type:
Journal Article
Language:
English
Journal Info:
Publisher: BioMed Central Country of Publication: England NLM ID: 101190741 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1479-5876 (Electronic) Linking ISSN: 14795876 NLM ISO Abbreviation: J Transl Med Subsets: MEDLINE
Imprint Name(s):
Original Publication: [London] : BioMed Central, 2003-
References:
Mobadersany P, Yousefi S, Amgad M, Gutman DA, Barnholtz-Sloan JS, Velázquez Vega JE, et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc Natl Acad Sci U S A. 2018;115:E2970–e 2979.
Bejnordi BE, Veta M, Van Diest PJ, Van Ginneken B, Karssemeijer N, Litjens G, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. Jama. 2017;318:2199–210.
Lu MY, Chen TY, Williamson DF, Zhao M, Shady M, Lipkova J, et al. AI-based pathology predicts origins for cancers of unknown primary. Nature. 2021;594:106–10.
Shmatko A, Ghaffari Laleh N, Gerstung M, Kather JN. Artificial intelligence in histopathology: enhancing cancer research and clinical oncology. Nat Cancer. 2022;3:1026–38.
Feng X, Shu W, Li M, Li J, Xu J, He M. Pathogenomics for accurate diagnosis, treatment, prognosis of oncology: a cutting edge overview. J Transl Med. 2024;22:131.
Lipkova J, Chen RJ, Chen B, Lu MY, Barbieri M, Shao D, et al. Artificial intelligence for multimodal data integration in oncology. Cancer Cell. 2022;40:1095–110.
Yang Y, Xu S, Hong Y, Cai Y, Tang W, Wang J, et al. Computational modeling for medical data: from data collection to knowledge discovery. The Innov Life. 2024;100079–100071–100079–100016.
Boehm KM, Aherne EA, Ellenson L, Nikolovski I, Alghamdi M, Vázquez-García I, et al. Multimodal data integration using machine learning improves risk stratification of high-grade serous ovarian cancer. Nat Cancer. 2022;3:723–33.
Xiong C, Chen H, Zheng H, Wei D, Zheng Y, Sung JJ, et al. MoME: mixture of multimodal experts for cancer survival prediction. arXiv preprint arXiv: 240609696 2024.
Perez-Lopez R, Ghaffari Laleh N, Mahmood F, Kather JN. A guide to artificial intelligence for cancer researchers. Nat Rev Cancer. 2024;24:427–41.
Chen Z, Chen Y, Sun Y, Tang L, Zhang L, Hu Y, et al. Predicting gastric cancer response to anti-HER2 therapy or anti-HER2 combined immunotherapy based on multi-modal data. Signal Transduct Target Ther. 2024;9:222.
Cui J, Li L, Liu N, Hou W, Dong Y, Yang F, et al. Model integrating CT-based radiomics and genomics for survival prediction in esophageal cancer patients receiving definitive chemoradiotherapy. Biomark Res. 2023;11:44.
Gong J, Zhang W, Huang W, Liao Y, Yin Y, Shi M, et al. CT-based radiomics nomogram may predict local recurrence-free survival in esophageal cancer patients receiving definitive chemoradiation or radiotherapy: a multicenter study. Radiother Oncol. 2022;174:8–15.
Liu J, Song J. 418P a PET/CT image-based deep learning approach for precise survival prognosis and clinical management of treatments in patients with esophageal carcinoma. Ann of Oncol. 2024;35:S169.
Niazi MKK, Parwani AV, Gurcan MN. Digital pathology and artificial intelligence. Lancet Oncol. 2019;20:e253–61.
Chen Y, Gao R, Jing D, Shi L, Kuang F, Jing R. Classification and prediction of chemoradiotherapy response and survival from esophageal carcinoma histopathology images. Spectrochim Acta A Mol Biomol Spectrosc. 2024;312:124030.
Li B, Qin W, Yang L, Li H, Jiang C, Yao Y, et al. From pixels to patient care: deep learning-enabled pathomics signature offers precise outcome predictions for immunotherapy in esophageal squamous cell cancer. J Transl Med. 2024;22:195.
Zhang Y, Yang Z, Chen R, Zhu Y, Liu L, Dong J, et al. Histopathology images-based deep learning prediction of prognosis and therapeutic response in small cell lung cancer. NPJ Digit Med. 2024;7:15.
Fan L, Sowmya A, Meijering E, Song Y. Cancer survival prediction from whole slide images with self-supervised learning and slide consistency. IEEE Trans Med Imag. 2023;42:1401–12.
Liu P, Ji L, Ye F, Fu B. AdvMIL: adversarial multiple instance learning for the survival analysis on whole-slide images. Med Image Anal. 2024;91:103020.
Li K, Lin Y, Zhou Y, Xiong X, Wang L, Li J, et al. Salivary extracellular MicroRNAs for early detection and prognostication of esophageal cancer: a clinical study. Gastroenterology. 2023;165:932–45.e939.
Cao K, Zhu J, Lu M, Zhang J, Yang Y, Ling X, et al. Analysis of multiple programmed cell death-related prognostic genes and functional validations of necroptosis-associated genes in oesophageal squamous cell carcinoma. EBioMedicine. 2024;99:104920.
Ilse M, Tomczak J, Welling M (Jennifer D, Andreas K, editors). Attention-based deep multiple instance learning. Proceedings of the 35th International Conference on Machine Learning. 2018. pp. 2127–36. Proceedings of Machine Learning Research: PMLR; 80.
Yao J, Zhu X, Jonnagaddala J, Hawkins N, Huang J. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Med Image Anal. 2020;65:101789.
Chen RJ, Lu MY, Weng W-H, Chen TY, Williamson DF, Manz T, et al. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. Proceedings of the IEEE/CVF international conference on computer vision. 2021: 4015–25.
Boehm KM, Khosravi P, Vanguri R, Gao J, Shah SP. Harnessing multimodal data integration to advance precision oncology. Nat Rev Cancer. 2022;22:114–26.
Jennings CN, Humphries MP, Wood S, Jadhav M, Chabra R, Brown C, et al. Bridging the gap with the UK genomics pathology imaging collection. Nat Med. 2022;28:1107–08.
Hoadley KA, Yau C, Hinoue T, Wolf DM, Lazar AJ, Drill E, et al. Cell-of-origin patterns dominate the molecular classification of 10, 000 tumors from 33 types of cancer. Cell. 2018;173:291–304.e296.
Chen RJ, Lu MY, Williamson DFK, Chen TY, Lipkova J, Noor Z, et al. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell. 2022;40:865–78.e866.
Yang Y, Feng T, Fan X, Wang C, Jiang Y, Zhou X, et al. Genomic and transcriptomic remodeling by Neoadjuvant chemoradiotherapy (nCRT) and the indicative role of acquired INDEL percentage for nCRT efficacy in esophageal squamous cell carcinoma. Int J Radiat Oncol Biol Phys. 2023;117:979–93.
Yang Y, Wang C, Jiang Y, Zhou X, Wang S, Su D, et al. Different Radiation dose of neoadjuvant chemoradiation for resectable thoracic esophageal squamous carcinoma: a randomized phase II clinical trial. Int J Radiat Oncol, Biol, Phys. 2023;117:S13.
Wang X, Du Y, Yang S, Zhang J, Wang M, Zhang J, et al. RetCCL: clustering-guided contrastive learning for whole-slide image retrieval. Med Image Anal. 2023;83:102645.
Han D, Ye T, Han Y, Xia Z, Song S, Huang G. Agent attention: on the integration of softmax and linear attention. arXiv preprint arXiv: 231208874 2023.
Tan K, Huang W, Liu X, Hu J, Dong S. A hierarchical graph convolution network for representation learning of gene expression data. IEEE J Biomed Health Inf. 2021;25:3219–29.
Wang Z, Li R, Wang M, Li A. GPDBN: deep bilinear network integrating both genomic data and pathological images for breast cancer prognosis prediction. Bioinformatics. 2021;37:2963–70.
Borgan Ø, Zhang Y. Using cumulative sums of martingale residuals for model checking in nested case-control studies. Biometrics. 2015;71:696–703.
Zadeh SG, Schmid M. Bias in cross-Entropy-based training of deep survival networks. IEEE Trans Pattern Anal Mach Intell. 2021;43:3126–37.
Xu Y, Chen H. Multimodal optimal transport-based co-attention transformer with global structure consistency for survival prediction. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 21241–51.
Zhou F, Chen H. Cross-modal translation and alignment for survival analysis. 2023 IEEE/CVF International Conference on Computer Vision (ICCV); 1-6: 21428–37Oct. 2023.
Liu Z, Zhao Y, Kong P, Liu Y, Huang J, Xu E, et al. Integrated multi-omics profiling yields a clinically relevant molecular classification for esophageal squamous cell carcinoma. Cancer Cell. 2023;41:181–95.e189.
Chen SB, Weng HR, Wang G, Yang JS, Yang WP, Liu DT, et al. Prognostic factors and outcome for patients with esophageal squamous cell carcinoma underwent surgical resection alone: evaluation of the seventh edition of the American joint Committee on cancer staging system for esophageal squamous cell carcinoma. J Thorac Oncol. 2013;8:495–501.
Chen W, Zhang P, Tran TN, Xiao Y, Li S, Vv S, et al. A visual-omics foundation model to bridge histopathology with spatial transcriptomics. Nat Methods. 2025;22:1568–82.
Pang M, Su K, Li M. Leveraging information in spatial transcriptomics to predict super-resolution gene expression from histology images in tumors. bioRxiv. 2021;2021. 2011.2028.470212.
Min W, Shi Z, Zhang J, Wan J, Wang C. Multimodal contrastive learning for spatial gene expression prediction using histology images. Briefings In Bioinf. 2024;25.
Xie R, Pang K, Chung SW, Perciani CT, MacParland SA, Wang B, et al. Spatially resolved gene expression prediction from H & amp;E histology images via bi-modal contrastive learning. Proceedings of the 37th International Conference on Neural Information Processing Systems. New Orleans, LA, USA: Curran Associates Inc; 2023. pp. Article 3095.
Zeng Y, Wei Z, Yu W, Yin R, Yuan Y, Li B, et al. Spatial transcriptomics prediction from histology jointly through transformer and graph neural networks. Briefings In Bioinf. 2022;23.
Kusumoto H, Hirohashi Y, Nishizawa S, Yamashita M, Yasuda K, Murai A, et al. Cellular stress induces cancer stem-like cells through expression of DNAJB8 by activation of heat shock factor 1. Cancer Sci. 2018;109:741–50.
Jiang YY, Jiang Y, Li CQ, Zhang Y, Dakle P, Kaur H, et al. TP63, SOX2, and KLF5 establish a core regulatory circuitry that controls epigenetic and transcription patterns in esophageal squamous cell carcinoma cell lines. Gastroenterology. 2020 1319;159:1311–27.e.
Wu Z, Zhou J, Zhang X, Zhang Z, Xie Y, Liu JB, et al. Reprogramming of the esophageal squamous carcinoma epigenome by SOX2 promotes ADAR1 dependence. Nat Genet. 2021;53:881–94.
Pan X, Wang J, Guo L, Na F, Du J, Chen X, et al. Identifying a confused cell identity for esophageal squamous cell carcinoma. Signal Transduct Target Ther. 2022;7:122.
Jia Y, Zhang B, Zhang C, Kwong DL, Chang Z, Li S, et al. Single-cell transcriptomic analysis of primary and metastatic tumor ecosystems in esophageal squamous cell carcinoma. Adv Sci (Weinh). 2023;10:e2204565.
Liu X, Zhao S, Wang K, Zhou L, Jiang M, Gao Y, et al. Spatial transcriptomics analysis of esophageal squamous precancerous lesions and their progression to esophageal cancer. Nat Commun. 2023;14:4779.
Contributed Indexing:
Keywords: Artificial intelligence; Multimodal learning; Pathogenomics; Prognosis; Whole slide image
Entry Date(s):
Date Created: 20260117 Latest Revision: 20260117
Update Code:
20260118
DOI:
10.1186/s12967-026-07682-5
PMID:
41547829
Database:
MEDLINE

Weitere Informationen

Declarations. Ethics approval and consent to participate: This study was conducted in accordance with the Declaration of Helsinki. Only retrospective data was used in this study. The authors had no role in the recruitment of participants. Ethics oversight of the TCGA study is described at https://www.cancer.gov/aboutnci/organization/ccg/research/structural-genomics/tcga/history/policies. Informed consent was obtained by all participants in the TCGA and CPTAC study. Informed consent was waived by the Committee due to the retrospective and anonymous nature. All patients in ZC-BRCA and ZC-ESCA cohort signed informed consent documents, and the study was approved by the Zhejiang Cancer Hospital Ethics Committee. Consent for publication: All authors agree to publish the final manuscript. Competing interests: The authors declare no competing interests.