Snip, Burn, Solder Blog

Yet another example of how absurdly easy it is to manipulate artificial intelligence—or even just accidentally make them into terrible bigots (and slightly above average antisemites). 

From University of Cambridge’s Ross Anderson (via security guru Bruce Schneier, “Manipulating Machine-Learning Systems through the Order of the Training Data”):

Most deep neural networks are trained by stochastic gradient descent. Now “stochastic” is a fancy Greek word for “random”; it means that the training data are fed into the model in random order.

So what happens if the bad guys can cause the order to be not random? You guessed it – all bets are off. Suppose for example a company or a country wanted to have a credit-scoring system that’s secretly sexist, but still be able to pretend that its training was actually fair. Well, they could assemble a set of financial data that was representative of the whole population, but start the model’s training on ten rich men and ten poor women drawn from that set – then let initialisation bias do the rest of the work.

Anderson concludes “It’s time for the machine-learning community to carefully examine their assumptions about randomness.”

I think that’s tangent to the real lesson, which is this:

All machines (including AIs) are created things, and created things bear the biases of their creators in unexpected, but ironclad ways: early color film was shite at photographing people of color simply because the folks who created the color film were all White and unintentionally selected techniques and chemical processes that worked better for their own paler skin tones than they did for darker ones. Similarly, male engineers built crash-test dummies that were roughly their own size and weight—and thus created “safety” features that killed women and children.