Accuracy of automated machine learning in classifying ocular toxoplasmosis from fundus images

Authors: Daniel Milad , Fares Antaki, Allison Bernstein, Samir Touma, Renaud Duval.  Université de Montréal.

Author Disclosures: D. Milad:  None.  F. Antaki:    Funded grants or clinical trials; Name of for-profit or not-for-profit organization(s); Bayer. Funded grants or clinical trials; Description of relationship(s); Research grant.  A. Bernstein:  None.  S. Touma:   None.  R. Duval:   Funded grants or clinical trials; Name of for-profit or not-for-profit organization(s); Alcon, Bayer, Novartis, Roche. Funded grants or clinical trials; Description of relationship(s); Consulting and research grants.

Abstract Body:

​Purpose:  Ocular toxoplasmosis (OT) is the leading cause of posterior uveitis worldwide. Prompt diagnosis and treatment is fundamental to reducing vision loss from this disease. While machine learning (ML) models have been proposed to automate OT detection from fundus images, their development has been reserved to artificial intelligence (AI) experts. In this study, clinicians with no coding experience developed automated ML (AutoML) models to detect OT using a graphical user interface.  

Study Design:  Artificial intelligence diagnostic algorithm.  

Methods:  Ophthalmologists with no coding experience carried out AutoML model design using a public data set of fundus images from medical centers in Paraguay. The dataset was previously labelled by experts. Duplicate images and images not showing the optic nerve were excluded. Google Cloud AutoML Vision was used for training and testing. We designed one binary model to differentiate OT from healthy fundi and then compared it to bespoke OT detection models designed by AI experts using the same dataset (Parra et al.,  Stud Health Technol Inform.,  2021). The AutoML model performance is reported using the area under the precision-recall curve (AuPRC), sensitivity, specificity and accuracy. Saliency maps were generated to assess the interpretability of the models.   

Results:  A total of 304 fundus images were included (117 normal, 187 OT). The AutoML OT detection model showed high diagnostic properties that were comparable to bespoke deep-learning models. The model had an AuPRC of 0.945, sensitivity of 100%, specificity of 83% and accuracy of 93.5% (vs. 94%, 86% and 91% for the bespoke models).  

Conclusions:   Our study highlights the value of AutoML in the democratization of AI. We demonstrate the feasibility of using AutoML by ophthalmologists with no coding experience to create deep-learning models to detect toxoplasmosis from fundus images. The model can detect OT with performance similar or better than those designed by AI experts. Our results also demonstrate the potential value of open-access data and big data in improving public health issues worldwide.