The coronavirus pandemic has changed everything and one of its most unsuspected victims has been artificial intelligence (AI), unable to anticipate people’s decisions, something that has caused peculiar events.

Let’s start with the example of consumption since the pandemic began, the top 10 search terms on Amazon have completely changed. From looking for phone cases, chargers and Lego toys, it switched to toilet paper, masks and disinfectants. This affected artificial intelligence, causing an earthquake to behind-the-scenes algorithms in inventory management, fraud detection, or marketing. Machine learning models trained in normal human behaviour are now discovering that normalcy has changed and some are no longer functioning as they should.

According to Pactera Edge, a global artificial intelligence consultancy, “automation is in trouble.” Others note that they are cautiously monitoring automated systems and that they are being sustained by human manual correction when necessary. What is clear is that the pandemic has revealed how intertwined our lives are with artificial intelligence, exposing delicate codependency in which changes in our behaviour change the functioning of artificial intelligence and vice versa. And that should remind us that human participation in automated systems remains key.

Radical changes

Machine learning models are designed to respond to change. But most are also fragile; They work poorly when the input data differs too much from the data they were trained on. It’s wrong to assume that you can set up an AI system and walk away, says Rajeev Sharma, global vice president, Pactera Edge: “AI is a live, breathable engine.”

“A pandemic like this is a perfect trigger to build better machine learning models”

Sharma tells some examples to MIT Technology Review. A company supplying sauces and condiments to retailers in India needed help fixing its automated inventory management system when bulk orders broke its predictive algorithms. The system’s sales forecast that the company relied on to reorder shares no longer matched what was actually sold. “It was never trained on a peak like this, so AI system was out of control,” Sharma explains. Another firm uses AI to assess readers’ sentiment about news articles and provides daily advertising spending recommendations based on the results. But with more sad news than usual in this at the moment, the recommendation is going to be very biased, Sharma adds.

Another large streaming company, which has had a sudden influx of content-hungry subscribers, is also having trouble with its recommendation algorithms. The company uses machine learning to suggest relevant, personalized content to viewers to keep them coming back. But the sudden change in subscriber data was making their system recommendations less accurate.

Many of these problems with models arise because more companies are buying machine learning systems, but lack the internal knowledge necessary to maintain them. Retraining a model may require expert human intervention. Sharma thinks that more AI should be trained not only considering the ups and downs of recent years but also in rare events such as the Great Depression of the 1930s, the stock market crash of Black Monday in 1987 and the financial crisis of 2007-2008. “A pandemic like this is a perfect trigger to build better machine learning models,” he says.

Detect fraud

Still, you can’t prepare for everything. In general, if a machine learning system doesn’t see what it expects to see, it will have problems, says David Excell, founder of Featurespace, a behavioural analytics company that uses AI to detect credit card fraud. Perhaps surprisingly, Featurespace hasn’t seen its AI be affected too much. People are still buying things on Amazon and subscribing to Netflix like before, but they are not buying too expensive items or new sites, which are behaviours that can raise suspicions. The firm’s engineers only had to intervene to adjust to an increase in people buying garden equipment and power tools, practices that were considered suspicious.

With everything connected, the impact of a pandemic has been felt throughout the planet, touching mechanisms that in more normal times remain hidden. Now is a good time to take stock of those systems and ask how they could be better designed, more resilient. In order for the machines to be trusted, we have to take care of them.