The framing of algorithmic bias as a bug, an unintended error that technical refinement can fix, is itself ideologically significant. It locates the problem in the machine, not in the society that produced the machine. It suggests that if we just get the data right, or adjust the algorithm, we will achieve fairness. CRT suggests something more uncomfortable: that what appears as algorithmic bias is frequently algorithmic accuracy, an accurate reflection of a biased society.
Three cases illustrate this.
COMPAS, Correctional Offender Management Profiling for Alternative Sanctions
COMPAS is a risk assessment tool used in US courts to predict the likelihood that a defendant will reoffend. In 2016, the investigative journalism organisation ProPublica published an analysis showing that COMPAS was almost twice as likely to falsely flag Black defendants as higher risk than white defendants, and almost twice as likely to falsely flag white defendants as lower risk than they were.
The company that built COMPAS responded that the algorithm was accurate, its overall prediction rates were similar across racial groups. Both claims were true simultaneously, which is a mathematical fact rather than a contradiction. The algorithm performed equally well overall, but its errors were distributed unequally across racial groups. Black defendants paid a disproportionate price for the algorithm’s mistakes.
This case is important because it demonstrates that technical accuracy and racial fairness are not the same thing, and that they can be in direct tension. Optimising for one can worsen the other. The choice between different definitions of fairness is not a technical choice, it is a political and ethical one that reflects whose interests are prioritised.
Amazon Hiring Algorithm
In 2018, Reuters reported that Amazon had developed and then abandoned an AI recruiting tool after discovering it was systematically downgrading CVs from women. The algorithm had been trained on ten years of Amazon’s hiring decisions, a period in which the technology industry, and Amazon in particular, hired predominantly male candidates. The algorithm learned that male candidates were preferred and encoded that preference as a feature of what a good candidate looked like.
Amazon tried to adjust the algorithm to remove its gender bias. The attempts were unsuccessful. The algorithm found other ways to identify gender, through women’s colleges, through extracurricular activities coded as female, and continued to disadvantage women. Eventually Amazon abandoned the tool.
This case illustrates both the depth of the problem and the limits of the technical fix. The bias was not in one variable that could be removed. It was distributed throughout the structure of the training data, a reflection of a structurally sexist industry.
Credit Scoring and Wealth Extraction
Research has consistently shown that AI-based credit scoring systems produce racially unequal outcomes. Black and Hispanic applicants are more likely to be denied credit, or offered credit at higher rates, than white applicants with similar financial profiles. The explanations are structural: credit scoring models use variables, length of credit history, home ownership, savings, that reflect accumulated wealth differences produced by centuries of discriminatory policy. Redlining, discriminatory lending, exclusion from the GI Bill, these historical policies produced wealth gaps that contemporary algorithms treat as individual financial facts.
In each of these cases, the algorithm was doing something technically defensible. In each case, it was producing racially unjust outcomes. The connection between technical defensibility and racial injustice is not a bug. It is a feature of algorithms trained on data produced by unequal societies.
Reflection question: In each of these three cases, who bears the cost of the algorithm’s errors? Who benefits from it? What does the distribution of costs and benefits reveal?