AI has a notorious history of biases, from facial recognition systems misidentifying Black people to chatbots admiring Hitler. In some settings, the outcomes could be deadly.

A case in point was exposed this week. According to new research, the pedestrian detection systems used in autonomous vehicle research have major age and race biases.

The study adds another obstacle to the rollout of driverless cars. It also uncovers an alarming potential addition to road safety.

The findings derive from a systematic review of eight popular pedestrian detection systems. Researchers from King’s College London (KCL) tested the software on over 8,000 images of pedestrians.

They found that the average detection accuracy was almost 20% higher for adults than it was for children. The systems were also 7.5% more accurate for light-skinned pedestrians than hey were for darker-skinned ones.

These discrepancies stem from a common cause of AI biases: unrepresentative training data.

“There’s an old saying when it comes to engineering and data science, ‘Rubbish in, rubbish out.’  AI systems need to be trained with a lot of training data, and inadequacies with that data are inevitably reflected in the AI,” Dr Jie Zhang, a computer science lecturer at KCL, told TNW.

“In this case, the open-source image galleries used to train these pedestrian detection systems are not representative of all pedestrians, and are skewed towards lighter-skinned adults. With less data to train on, the AI becomes less accurate when detecting under-represented groups.”

Another issue emerged in the lighting conditions. Under low contrast and low brightness, the biases against children and dark-skinned people were exacerbated. This suggests that both groups would be at increased risk during nighttime driving.

Although car manufacturers don’t publicise details on their pedestrian detection software, they’re typically based on the same open-source systems used in the research. Zhang is therefore confident that they experience the same issues.

To reduce the risks, he wants more transparency and tighter regulations on pedestrian detection systems.

“Developers should start by being more transparent when it comes to how their detection systems are trained, as well as how they perform, so they can be measured objectively — the consequences of not doing so could be dire,” he said.

“But in addition to this, manufacturers need to work to make sure that their AI systems are fair and representative, and part of the impetus for that will come from policymakers and tighter regulation around fairness in AI.”

You can read the study paper here.