Using AI to prognosticate COVID-19 patients

In this study, Zhicheng Jiao et al., attempt to prognosticate COVID-19 patients based on a combination of AI, chest X-rays, and clinical data. Note that the research is partially funded by a for-profit organisation. Although the wording in the article would suggest that the algorithm is accurate, the authors do cite this major limitation at the end:

This study has several limitations. First, the artificial intelligence model showed decreased performance on the external testing set relative to the internal testing set, indicating that generalisation might not be possible.

There’s also a basic design flaw, which is often seen in AI studies. The authors reduce the complicated medical conditions of the subjects into artificial binary oucomes: severe or not-severe outcome.

Disease outcome severity was defined as critical if the patient had any of the following outcomes: utilisation of mechanical ventilation, admission to the ICU, or death

One can argue that there’s some difference in severity between a patient receiving mechanical ventilation and one that dies.

Also, both sensitivy and specificity of the model, when tested on external data, are between 0.60 and 0.70, which is not very impressive. Use of the model would result in many both false positive and negative cases, and I’m not even sure it would fare any better than a clinician observing the patient.