The Foundations of AI are riddled with errors

Basically, “shit in - shit out”, but used to drive your car, identify a face or, in the future, help set a diagnosis.

In the competition, a method called deep learning, which involves feeding examples to a giant simulated neural network, proved dramatically better at identifying objects in images than other approaches. That kick-started interest in using AI to solve different problems. But research revealed this week shows that ImageNet and nine other key AI data sets contain many errors. Researchers at MIT compared how an AI algorithm trained on the data interprets an image with the label that was applied to it. If, for instance, an algorithm decides that an image is 70 percent likely to be a cat but the label says “spoon,” then it’s likely that the image is wrongly labeled and actually shows a cat. To check, where the algorithm and the label disagreed, researchers showed the image to more people.

Such errors might lead machine learning engineers down the wrong path when choosing among different AI models. “They might actually choose the model that has worse performance in the real world,” Northcutt says.