Treffer: Deep learning-based multimodal pathogenomics integration for precision cancer prognosis.
Bejnordi BE, Veta M, Van Diest PJ, Van Ginneken B, Karssemeijer N, Litjens G, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. Jama. 2017;318:2199–210.
Lu MY, Chen TY, Williamson DF, Zhao M, Shady M, Lipkova J, et al. AI-based pathology predicts origins for cancers of unknown primary. Nature. 2021;594:106–10.
Shmatko A, Ghaffari Laleh N, Gerstung M, Kather JN. Artificial intelligence in histopathology: enhancing cancer research and clinical oncology. Nat Cancer. 2022;3:1026–38.
Feng X, Shu W, Li M, Li J, Xu J, He M. Pathogenomics for accurate diagnosis, treatment, prognosis of oncology: a cutting edge overview. J Transl Med. 2024;22:131.
Lipkova J, Chen RJ, Chen B, Lu MY, Barbieri M, Shao D, et al. Artificial intelligence for multimodal data integration in oncology. Cancer Cell. 2022;40:1095–110.
Yang Y, Xu S, Hong Y, Cai Y, Tang W, Wang J, et al. Computational modeling for medical data: from data collection to knowledge discovery. The Innov Life. 2024;100079–100071–100079–100016.
Boehm KM, Aherne EA, Ellenson L, Nikolovski I, Alghamdi M, Vázquez-García I, et al. Multimodal data integration using machine learning improves risk stratification of high-grade serous ovarian cancer. Nat Cancer. 2022;3:723–33.
Xiong C, Chen H, Zheng H, Wei D, Zheng Y, Sung JJ, et al. MoME: mixture of multimodal experts for cancer survival prediction. arXiv preprint arXiv: 240609696 2024.
Perez-Lopez R, Ghaffari Laleh N, Mahmood F, Kather JN. A guide to artificial intelligence for cancer researchers. Nat Rev Cancer. 2024;24:427–41.
Chen Z, Chen Y, Sun Y, Tang L, Zhang L, Hu Y, et al. Predicting gastric cancer response to anti-HER2 therapy or anti-HER2 combined immunotherapy based on multi-modal data. Signal Transduct Target Ther. 2024;9:222.
Cui J, Li L, Liu N, Hou W, Dong Y, Yang F, et al. Model integrating CT-based radiomics and genomics for survival prediction in esophageal cancer patients receiving definitive chemoradiotherapy. Biomark Res. 2023;11:44.
Gong J, Zhang W, Huang W, Liao Y, Yin Y, Shi M, et al. CT-based radiomics nomogram may predict local recurrence-free survival in esophageal cancer patients receiving definitive chemoradiation or radiotherapy: a multicenter study. Radiother Oncol. 2022;174:8–15.
Liu J, Song J. 418P a PET/CT image-based deep learning approach for precise survival prognosis and clinical management of treatments in patients with esophageal carcinoma. Ann of Oncol. 2024;35:S169.
Niazi MKK, Parwani AV, Gurcan MN. Digital pathology and artificial intelligence. Lancet Oncol. 2019;20:e253–61.
Chen Y, Gao R, Jing D, Shi L, Kuang F, Jing R. Classification and prediction of chemoradiotherapy response and survival from esophageal carcinoma histopathology images. Spectrochim Acta A Mol Biomol Spectrosc. 2024;312:124030.
Li B, Qin W, Yang L, Li H, Jiang C, Yao Y, et al. From pixels to patient care: deep learning-enabled pathomics signature offers precise outcome predictions for immunotherapy in esophageal squamous cell cancer. J Transl Med. 2024;22:195.
Zhang Y, Yang Z, Chen R, Zhu Y, Liu L, Dong J, et al. Histopathology images-based deep learning prediction of prognosis and therapeutic response in small cell lung cancer. NPJ Digit Med. 2024;7:15.
Fan L, Sowmya A, Meijering E, Song Y. Cancer survival prediction from whole slide images with self-supervised learning and slide consistency. IEEE Trans Med Imag. 2023;42:1401–12.
Liu P, Ji L, Ye F, Fu B. AdvMIL: adversarial multiple instance learning for the survival analysis on whole-slide images. Med Image Anal. 2024;91:103020.
Li K, Lin Y, Zhou Y, Xiong X, Wang L, Li J, et al. Salivary extracellular MicroRNAs for early detection and prognostication of esophageal cancer: a clinical study. Gastroenterology. 2023;165:932–45.e939.
Cao K, Zhu J, Lu M, Zhang J, Yang Y, Ling X, et al. Analysis of multiple programmed cell death-related prognostic genes and functional validations of necroptosis-associated genes in oesophageal squamous cell carcinoma. EBioMedicine. 2024;99:104920.
Ilse M, Tomczak J, Welling M (Jennifer D, Andreas K, editors). Attention-based deep multiple instance learning. Proceedings of the 35th International Conference on Machine Learning. 2018. pp. 2127–36. Proceedings of Machine Learning Research: PMLR; 80.
Yao J, Zhu X, Jonnagaddala J, Hawkins N, Huang J. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Med Image Anal. 2020;65:101789.
Chen RJ, Lu MY, Weng W-H, Chen TY, Williamson DF, Manz T, et al. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. Proceedings of the IEEE/CVF international conference on computer vision. 2021: 4015–25.
Boehm KM, Khosravi P, Vanguri R, Gao J, Shah SP. Harnessing multimodal data integration to advance precision oncology. Nat Rev Cancer. 2022;22:114–26.
Jennings CN, Humphries MP, Wood S, Jadhav M, Chabra R, Brown C, et al. Bridging the gap with the UK genomics pathology imaging collection. Nat Med. 2022;28:1107–08.
Hoadley KA, Yau C, Hinoue T, Wolf DM, Lazar AJ, Drill E, et al. Cell-of-origin patterns dominate the molecular classification of 10, 000 tumors from 33 types of cancer. Cell. 2018;173:291–304.e296.
Chen RJ, Lu MY, Williamson DFK, Chen TY, Lipkova J, Noor Z, et al. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell. 2022;40:865–78.e866.
Yang Y, Feng T, Fan X, Wang C, Jiang Y, Zhou X, et al. Genomic and transcriptomic remodeling by Neoadjuvant chemoradiotherapy (nCRT) and the indicative role of acquired INDEL percentage for nCRT efficacy in esophageal squamous cell carcinoma. Int J Radiat Oncol Biol Phys. 2023;117:979–93.
Yang Y, Wang C, Jiang Y, Zhou X, Wang S, Su D, et al. Different Radiation dose of neoadjuvant chemoradiation for resectable thoracic esophageal squamous carcinoma: a randomized phase II clinical trial. Int J Radiat Oncol, Biol, Phys. 2023;117:S13.
Wang X, Du Y, Yang S, Zhang J, Wang M, Zhang J, et al. RetCCL: clustering-guided contrastive learning for whole-slide image retrieval. Med Image Anal. 2023;83:102645.
Han D, Ye T, Han Y, Xia Z, Song S, Huang G. Agent attention: on the integration of softmax and linear attention. arXiv preprint arXiv: 231208874 2023.
Tan K, Huang W, Liu X, Hu J, Dong S. A hierarchical graph convolution network for representation learning of gene expression data. IEEE J Biomed Health Inf. 2021;25:3219–29.
Wang Z, Li R, Wang M, Li A. GPDBN: deep bilinear network integrating both genomic data and pathological images for breast cancer prognosis prediction. Bioinformatics. 2021;37:2963–70.
Borgan Ø, Zhang Y. Using cumulative sums of martingale residuals for model checking in nested case-control studies. Biometrics. 2015;71:696–703.
Zadeh SG, Schmid M. Bias in cross-Entropy-based training of deep survival networks. IEEE Trans Pattern Anal Mach Intell. 2021;43:3126–37.
Xu Y, Chen H. Multimodal optimal transport-based co-attention transformer with global structure consistency for survival prediction. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 21241–51.
Zhou F, Chen H. Cross-modal translation and alignment for survival analysis. 2023 IEEE/CVF International Conference on Computer Vision (ICCV); 1-6: 21428–37Oct. 2023.
Liu Z, Zhao Y, Kong P, Liu Y, Huang J, Xu E, et al. Integrated multi-omics profiling yields a clinically relevant molecular classification for esophageal squamous cell carcinoma. Cancer Cell. 2023;41:181–95.e189.
Chen SB, Weng HR, Wang G, Yang JS, Yang WP, Liu DT, et al. Prognostic factors and outcome for patients with esophageal squamous cell carcinoma underwent surgical resection alone: evaluation of the seventh edition of the American joint Committee on cancer staging system for esophageal squamous cell carcinoma. J Thorac Oncol. 2013;8:495–501.
Chen W, Zhang P, Tran TN, Xiao Y, Li S, Vv S, et al. A visual-omics foundation model to bridge histopathology with spatial transcriptomics. Nat Methods. 2025;22:1568–82.
Pang M, Su K, Li M. Leveraging information in spatial transcriptomics to predict super-resolution gene expression from histology images in tumors. bioRxiv. 2021;2021. 2011.2028.470212.
Min W, Shi Z, Zhang J, Wan J, Wang C. Multimodal contrastive learning for spatial gene expression prediction using histology images. Briefings In Bioinf. 2024;25.
Xie R, Pang K, Chung SW, Perciani CT, MacParland SA, Wang B, et al. Spatially resolved gene expression prediction from H & amp;E histology images via bi-modal contrastive learning. Proceedings of the 37th International Conference on Neural Information Processing Systems. New Orleans, LA, USA: Curran Associates Inc; 2023. pp. Article 3095.
Zeng Y, Wei Z, Yu W, Yin R, Yuan Y, Li B, et al. Spatial transcriptomics prediction from histology jointly through transformer and graph neural networks. Briefings In Bioinf. 2022;23.
Kusumoto H, Hirohashi Y, Nishizawa S, Yamashita M, Yasuda K, Murai A, et al. Cellular stress induces cancer stem-like cells through expression of DNAJB8 by activation of heat shock factor 1. Cancer Sci. 2018;109:741–50.
Jiang YY, Jiang Y, Li CQ, Zhang Y, Dakle P, Kaur H, et al. TP63, SOX2, and KLF5 establish a core regulatory circuitry that controls epigenetic and transcription patterns in esophageal squamous cell carcinoma cell lines. Gastroenterology. 2020 1319;159:1311–27.e.
Wu Z, Zhou J, Zhang X, Zhang Z, Xie Y, Liu JB, et al. Reprogramming of the esophageal squamous carcinoma epigenome by SOX2 promotes ADAR1 dependence. Nat Genet. 2021;53:881–94.
Pan X, Wang J, Guo L, Na F, Du J, Chen X, et al. Identifying a confused cell identity for esophageal squamous cell carcinoma. Signal Transduct Target Ther. 2022;7:122.
Jia Y, Zhang B, Zhang C, Kwong DL, Chang Z, Li S, et al. Single-cell transcriptomic analysis of primary and metastatic tumor ecosystems in esophageal squamous cell carcinoma. Adv Sci (Weinh). 2023;10:e2204565.
Liu X, Zhao S, Wang K, Zhou L, Jiang M, Gao Y, et al. Spatial transcriptomics analysis of esophageal squamous precancerous lesions and their progression to esophageal cancer. Nat Commun. 2023;14:4779.
Weitere Informationen
Declarations. Ethics approval and consent to participate: This study was conducted in accordance with the Declaration of Helsinki. Only retrospective data was used in this study. The authors had no role in the recruitment of participants. Ethics oversight of the TCGA study is described at https://www.cancer.gov/aboutnci/organization/ccg/research/structural-genomics/tcga/history/policies. Informed consent was obtained by all participants in the TCGA and CPTAC study. Informed consent was waived by the Committee due to the retrospective and anonymous nature. All patients in ZC-BRCA and ZC-ESCA cohort signed informed consent documents, and the study was approved by the Zhejiang Cancer Hospital Ethics Committee. Consent for publication: All authors agree to publish the final manuscript. Competing interests: The authors declare no competing interests.