Treffer: A knowledge-driven deep learning framework for organoid morphological segmentation and characterization

Background: Organoids have great potential to revolutionize various aspects of biomedical research and healthcare. Researchers typically use the fluorescence-based approach to analyse their dynamics, which requires specialized equipment and may interfere with their growth. Therefore, it is an open challenge to develop a general framework to analyse organoid dynamics under non-invasive and low-resource settings. Results: In this paper, we present a knowledge-driven deep learning system named TransOrga-plus to automatically analyse organoid dynamics in a non-invasive manner. Given a bright-field microscopic image, TransOrga-plus detects organoids through a multi-modal transformer-based segmentation module. To provide customized and robust organoid analysis, a biological knowledge-driven branch is embedded into the segmentation module which integrates biological knowledge, e.g. the morphological characteristics of organoids, into the analysis process. Then, based on the detection results, a lightweight multi-object tracking module based on the decoupling of visual and identity features is introduced to track organoids over time. Finally, TransOrga-plus outputs the dynamics analysis to assist biologists for further research. To train and validate our framework, we curate a large-scale organoid dataset encompassing diverse tissue types and various microscopic imaging settings. Extensive experimental results demonstrate that our method outperforms all baselines in organoid analysis. The results show that TransOrga-plus provides comparable analytical results to biologists and significantly accelerates organoid work process. Conclusions: In conclusion, TransOrga-plus integrates the biological expertise with cutting-edge deep learning-based model and enables the non-invasive analysis of various organoids from complex, low-resource, and time-lapse situations.