Weapons of Math Destruction

Cathy O’Neil of mathbabe.org released a book recently called Weapons of Math Destruction.

Read it. Read her blog.

In short, a Weapon of Math Destruction (WMD) is any statistical model or algorithm that

1) influences a lot of people

2) influences them negatively

3) is opaque in how it works

4) does not have any mechanism for feedback

5) introduces feedback loops that excerbate its damage

Consider the difference between Netflix and the recidivism model used in court sentencing. With Netflix, not only are the effects of a wrong recommendation not devastating, but the user has the ability to feed back into the machine that it is wrong (by giving movies ratings). The victim of a racist recidivism score (and predictive policing, among other algorithms) has no recourse to fix their rating, nor do they have any insight into how that score was generated. Since black people are disproportionately likely to get a high recidivism score, it feeds back into crime in black communities with a cycle of criminalization and joblessness.

This is a profound reminder, and it touches everything from college rankings to credit scores to political micro-targeting. I read it in two days. Do it.