COMPAS and AI in the courts: bias, due process and the limits of risk scores

COMPAS is a risk assessment tool used in US courts to estimate whether a defendant will reoffend. In 2016, ProPublica analyzed more than 7,000 people arrested in Broward County, Florida, and found Black defendants were 77% more likely than white defendants to get a higher violent recidivism score after controls. Its maker, Northpointe, disputed the finding.

EasWrk is a hosting platform, not a research institute. This essay summarizes published sources, which are listed at the end.

When the Algorithm Takes the Stand

A Wisconsin judge sentenced Eric Loomis to six years in prison and five years of extended supervision after a charge filed in early 2013, and the sentencing judge considered a risk score from a proprietary algorithm called COMPAS (Harvard Law Review, 2017). Loomis argued that using a secret algorithm violated his due process rights because neither he nor his attorney could examine the method behind the score. In 2016 the Wisconsin Supreme Court held that using the score at sentencing did not violate due process, with limits on how it may be used (State v. Loomis, 881 N.W.2d 749 (Wis. 2016)). The US Supreme Court declined to hear the case in 2017. The algorithm stayed secret.

The Loomis case is neither the beginning nor the end of the story of AI in the legal system. It is a moment of crystallization, a point at which a set of debates that had circulated in academic journals and advocacy reports suddenly had a name, a court docket number, and a human being sitting at the center of them. Since then, AI-assisted decision-making has expanded substantially throughout the American criminal justice system and legal profession, and the debates it has provoked have grown more urgent rather than less.

This paper examines three distinct but intersecting domains in which AI is reshaping legal and criminal justice processes: predictive policing and crime prevention; risk assessment in bail, sentencing, and parole decisions; and the use of AI tools in legal practice itself. Across all three domains, a common set of concerns emerges: the opacity of algorithmic decision-making, the systematic reproduction and amplification of historical biases, the erosion of constitutional due process protections, and the concentration of discretionary power in systems whose internal logic the affected individuals cannot scrutinize.

These are not theoretical concerns. They are playing out in real cases, affecting real people, and they are doing so faster than courts, legislatures, or bar associations have been able to respond. The gap between the deployment of AI in the justice system and the legal frameworks needed to govern that deployment is wide and growing.

Predictive Policing: The Technology and Its Deployment

Predictive policing refers to the use of data analysis, statistical modeling, and, increasingly, machine learning to forecast where crimes are likely to occur or to identify individuals deemed likely to commit crimes. The technology has been deployed in various forms by police departments in the United States and elsewhere.

Place-based predictive policing tools, such as PredPol (renamed Geolitica in March 2021), generate maps indicating areas with elevated predicted crime risk. Officers are directed to patrol these areas more intensively. The underlying assumption is that past crime patterns are predictive of future crime locations. PredPol's patented algorithm is based on a model used to predict earthquake aftershocks, which cluster near the site of a major quake; the assumption is that crime clusters in a similar way.

Person-based predictive policing goes further. Systems such as the Chicago Police Department's Strategic Subject List assigned individuals a numerical score indicating their estimated risk of becoming a victim or offender in a shooting. According to the city's Inspector General (2020), the models drew on attributes such as arrests, shooting and assault victimization and, in one version, gang affiliation. The department decommissioned the program on November 1, 2019, after $3.8 million in federal grants.

Richardson, Schultz, and Crawford (2019) studied thirteen jurisdictions that had used or developed predictive policing tools while under government investigation or court-monitored agreements, and argued that vendor tools can be trained on records produced by flawed and sometimes unlawful policing. The core problem is circular: predictive models trained on historical crime data reflect historical policing patterns, not the actual distribution of crime. If police have historically concentrated enforcement in predominantly Black and low-income neighborhoods, those neighborhoods will have disproportionately high rates of documented crime, not necessarily higher rates of actual crime. A model trained on documented crime data will predict more crime in those neighborhoods, directing more police there, generating more arrests, and feeding more data back into the model. The feedback loop produces predictions that appear statistically valid but are, in essence, a formalization of prior enforcement bias.

In March 2019 the Los Angeles Police Commission's Inspector General reviewed Operation LASER, PredPol and a third data-driven program. The review found significant inconsistencies in how people were selected for LASER's Chronic Offender program, which it tied to a lack of centralized oversight, and it found that the data made the programs' overall impact hard to assess (LAPD Office of the Inspector General, 2019). The department had already suspended the Chronic Offender database in August 2018. In June 2020 the Santa Cruz city council voted to ban predictive policing, a step press reports described as a first for a US city (Government Technology, 2020).

COMPAS Algorithm Bias: What ProPublica and Its Critics Found

The Correctional Offender Management Profiling for Alternative Sanctions, known as COMPAS, is one of the most widely studied and contested AI tools in the criminal justice system. Developed by Northpointe (renamed Equivant in January 2017), COMPAS generates risk scores across multiple dimensions, including general recidivism risk, violent recidivism risk, and pretrial failure risk. Courts and corrections officials in the United States use these scores.

Racial bias in COMPAS recidivism scores: what ProPublica measured

In 2016, Angwin, Larson, Mattu, and Kirchner published Machine Bias in ProPublica. They analyzed COMPAS scores for more than 7,000 people arrested in Broward County, Florida, in 2013 and 2014 and tracked new charges over the next two years. Among defendants who did not reoffend, 44.9% of Black defendants had been labeled higher risk, against 23.5% of white defendants. Among defendants who did reoffend, 47.7% of white defendants had been labeled lower risk, against 28.0% of Black defendants. After controlling for prior crimes, age and gender, Black defendants were 77% more likely to receive a higher score for violent recidivism (ProPublica methodology, 2016). These findings ignited a debate that continues today.

Northpointe disputed the ProPublica analysis and said it did not accurately reflect the model's outcomes. Chouldechova (2017) used the same Broward County data to show why both sides could point to numbers: when recidivism rates differ between groups, a score that is equally well calibrated for each group will generally produce different false positive and false negative rates between them. Kleinberg, Mullainathan, and Raghavan (2016) proved a related result. This revealed a genuine and unresolved problem in the field: different mathematical definitions of fairness cannot all be met at once when base rates differ. Any choice of optimization criterion privileges some version of fairness over others, and that choice is inherently a political and ethical decision, not a technical one.

Dressel and Farid (2018) published a finding in Science Advances that made the debate significantly sharper. They found that pooled predictions from small groups of online volunteers, who saw only seven facts about each defendant, reached 67.0% accuracy, and that COMPAS, which uses 137 features, was not significantly better. They also reported that a simple model using a handful of variables matched COMPAS. This raises a pointed question: if a simple model performs as well as a complex proprietary system, what is the value added by that system, and what is the cost of its opacity?

Mayson (2019) offered a searching theoretical critique of risk assessment in the criminal justice context. Writing in the Yale Law Journal, she argued that racial inequality in risk assessment comes from the nature of prediction itself, which projects past disparities forward, so fixes to input data or algorithm design alone are not enough. She proposed that criminal law define more clearly which risks matter, recognize that some risks cannot be measured without racial distortion and so cannot justify state coercion, and respond to measurable risk with support rather than restraint.

The Loomis decision did not resolve the underlying due process question. The Wisconsin court held that risk scores may not be used to decide whether an offender is incarcerated or how severe the sentence is, and it required that presentence reports carry written cautions about COMPAS. It permitted their use as one input among many, without requiring that the algorithm be disclosed. Defense attorneys in subsequent cases have continued to struggle with the fundamental asymmetry: the prosecution can present a risk score as evidence; the defense cannot examine the logic that produced it.

Critics of the decision, including a Harvard Law Review note (2017), argued that the written advisement is an ineffective safeguard when the method behind the score stays undisclosed. The prosecution can present a risk score as evidence; the defense cannot examine the logic that produced it.

Facial Recognition and Wrongful Arrest

Facial recognition technology represents a different but related application of AI in the legal system. Law enforcement agencies use facial recognition to identify suspects by matching images from surveillance cameras, social media, or police databases against a reference database of photographs. The technology has become more capable and more widely deployed over the past decade, with little consistent regulatory oversight.

The National Institute of Standards and Technology evaluated commercial facial recognition algorithms in 2019 and found that false positives were higher for Asian and African American faces than for Caucasian faces, by factors that often ranged from 10 to 100 times depending on the algorithm, in one-to-one matching. In one-to-many matching, it found higher false positive rates for African American females (NIST, 2019). These disparities are not marginal. They are the difference between a tool that is useful and one that generates systematic errors against specific populations.

Garvie, Bedoya, and Frankle's 2016 report, The Perpetual Line-Up, found that at least 26 states allowed law enforcement to run or request face recognition searches against driver's license and ID photo databases, and that law enforcement face recognition networks included more than 117 million American adults, about half of the adult population. Of the 52 agencies the authors examined, only four had a publicly available use policy.

The consequences of inaccurate facial recognition in a law enforcement context are severe. Several documented cases of wrongful arrest based on facial recognition errors involve Black people who were arrested and detained on a match that turned out to be incorrect. Robert Williams was arrested at his home in Farmington Hills, Michigan, in January 2020, in front of his family, after facial recognition flagged him as a suspect in a Detroit shoplifting case. Reports put his detention at nearly 30 hours. In 2024 a settlement of his lawsuit barred Detroit police from making arrests based solely on facial recognition results (Michigan Public, 2024).

Williams's case is not the only one. Porcha Woodruff's arrest in Detroit is another. These cases reflect a pattern produced by the interaction of algorithmic inaccuracy, sparse investigative follow-up, and a presumption that algorithmic outputs are reliable. When a facial recognition match is treated as probable cause rather than as a hypothesis to be investigated, the error rate of the underlying technology translates directly into wrongful arrests. The problem is compounded by the fact that defendants often do not know that facial recognition was used in their case, because law enforcement agencies do not consistently disclose the technology in discovery.

AI Lawyer Hallucinations: The Mata v. Avianca Sanctions

The concern about AI in the legal system extends beyond policing and courts to the legal profession itself. Lawyers have adopted AI tools for legal research, contract analysis, document drafting, and case strategy. These tools offer genuine efficiency gains but carry a risk that became publicly visible in 2023 when a case before the Southern District of New York brought the failure mode of generative AI into sharp relief.

In Mata v. Avianca, a personal injury suit against the airline, lawyers for the plaintiff, Roberto Mata, filed a brief citing court decisions that did not exist. Attorney Steven Schwartz later acknowledged that he had used ChatGPT to supplement his legal research. On June 22, 2023, Judge P. Kevin Castel wrote that six of the submitted cases appeared to be bogus judicial decisions with bogus quotes and bogus internal citations, and he fined the two lawyers and their firm $5,000 jointly (Opinion and Order on Sanctions, S.D.N.Y., 2023).

This phenomenon, which researchers call hallucination, is not a bug in the implementation of a particular tool. It is a structural feature of how large language models generate text. These models produce outputs that are statistically likely given their training data, not outputs that are verified against reality. A language model has no mechanism to distinguish between a real case it learned from training data and a case it invents that follows the same surface patterns. When asked to provide legal research, it will produce text that looks like legal research, because it has been trained on vast quantities of legal text, regardless of whether the specific citations are real.

Courts and bar bodies responded. Judge Brantley Starr of the Northern District of Texas issued a standing order in mid-2023 requiring attorneys to certify either that generative AI drafted no part of a filing or that a human checked any AI-drafted language against traditional legal sources (National Law Review, 2023). In July 2024 the American Bar Association issued Formal Opinion 512 on lawyers' use of generative AI tools, which addresses competence, confidentiality and verification of outputs.

These responses are reasonable first steps, but they do not address the underlying structural problem: lawyers face asymmetric incentive structures when it comes to AI adoption. The productivity gains from using AI tools for legal research are real and immediate. The risks are latent and realized only when something goes wrong. The temptation to use AI tools without adequate verification is strong, particularly for solo practitioners and small firms under cost pressure, and the consequences of getting it wrong can be severe for their clients.

The hallucination problem in legal AI is also not limited to case citations. AI tools have generated contract clauses that do not reflect the parties' actual agreement, legal opinions that mischaracterize the law of a jurisdiction, and due diligence reports that omit material information. In each case, the output looks authoritative. The attorney who does not read carefully, or who trusts the tool more than they should, may not catch the error before it causes harm.

Due Process and the Right to Understand Your Score

The due process implications of algorithmic decision-making in the legal system raise questions that existing constitutional doctrine is poorly equipped to answer. The Fifth and Fourteenth Amendments guarantee that the government may not deprive any person of life, liberty, or property without due process of law. What due process requires in any given context depends on the nature of the interest at stake, the risk of erroneous deprivation, and the governmental interest in the challenged procedure. The Supreme Court articulated this balancing test in Mathews v. Eldridge (1976), a case about Social Security disability benefits.

Applied to algorithmic sentencing or bail determinations, the Mathews balancing test produces a strong argument for disclosure. The liberty interest at stake is as significant as it gets: prison time, pretrial detention, conditions of release. The risk of erroneous deprivation is real and documented. The governmental interest in keeping the algorithm secret is primarily commercial, protecting the intellectual property of the tool's vendor, which is not the type of governmental interest the due process clause was designed to protect.

On that analysis, a defendant has a strong claim to meaningful access to the factors and logic behind an algorithmic score used against them. Courts have been reluctant to reach this conclusion, in part because doing so would effectively require vendors to disclose proprietary algorithms. The courts have instead engaged in a kind of procedural sleight of hand, holding that the score can be used as long as it is not the sole basis for the decision, without confronting the disclosure question directly.

The AI Now Institute (2018) argued that vendors of systems used by government should waive trade secrecy so the systems can be audited and tested for bias and error, and that people harmed by automated decisions must be able to contest them. A stricter rule is arguable: no algorithmic tool in a high-stakes proceeding unless it has been independently validated, tested for demographic disparities, and opened to adversarial examination. That is a higher bar than current practice requires, and it is the author's view of what the constitutional values at stake demand.

The Algorithmic Accountability Act was introduced by Senators Wyden and Booker as S. 3572 in the 117th Congress on February 3, 2022, and reintroduced by Senators Wyden and Booker and Representative Clarke as S. 2892 in the 118th Congress on September 21, 2023. The 2022 text would have required impact assessments of automated systems used in critical decisions, a category it defined to include legal services, education, employment, housing, health care and financial services (S. 3572, 117th Congress). It does not list criminal sentencing as a covered decision. Neither bill had become law when this essay was updated. The legislative response to algorithmic accountability in criminal justice has lagged behind the deployment of the technology.

Toward Accountable Algorithmic Justice

The path toward a legal system that uses AI responsibly is not a path that rejects AI entirely. Algorithmic tools can improve consistency in decision-making, reduce the idiosyncratic variation that produces sentencing disparities across judges, and help understaffed probation and parole offices allocate limited resources. The critique of algorithmic justice is not that prediction has no place in the system. The critique is that prediction is being used without adequate transparency, validation, or accountability, and that the populations bearing the costs of that inadequacy are those who were already most harmed by the pre-algorithmic system.

Several principles emerge from the literature that could form the basis of a more accountable framework. First, algorithmic tools used in high-stakes legal proceedings should be subject to mandatory independent validation before deployment, and that validation should include disaggregated accuracy testing across demographic groups. A tool that performs adequately on average but performs poorly on Black defendants is not an adequate tool.

Second, defendants and their counsel must have meaningful access to the factors and logic underlying any algorithmic score used against them. This does not necessarily require full disclosure of proprietary source code, but it does require, at minimum, disclosure of the variables used, the weighting applied, the demographic distribution of scores in the relevant population, and the validated error rates. Meaningful due process requires meaningful information.

Third, algorithmic outputs should be clearly labeled as statistical estimates with documented error rates, not as findings of fact. The presentation of a COMPAS score as a piece of evidence, without mandatory disclosure of its limitations and margins of error, invites misunderstanding by judges, juries, and defendants who may not have the statistical literacy to evaluate it appropriately.

Fourth, the feedback loops that allow biased historical data to perpetuate and amplify themselves in predictive models need to be broken by regular auditing. Models trained on data generated by a biased system will reproduce that bias. The only way to interrupt this cycle is to scrutinize training data, monitor outcomes, and adjust or discontinue tools that demonstrably produce disparate and unjust outcomes.

Fifth, the legal profession needs clearer, enforceable standards for the use of AI-generated content in legal practice. The Mata v. Avianca incident was a visible failure because it involved a federal court filing. Quieter failures, involving AI-generated contract language, research memos, or client advice that is not verified before it is acted upon, are likely more numerous and collectively more harmful.

The question animating all of these concerns is what justice means in a system where consequential decisions about liberty are increasingly influenced by algorithms that the affected individuals cannot see, challenge, or understand. The answer, in a society that claims to value due process and equal protection, should be that such a system is not acceptable as currently operated. The technology is not the problem. The lack of accountability around it is. That accountability deficit is a policy choice, and it can be changed.

The same proxy problem appears in health care risk scores, and the question of who is liable when a machine errs is covered in the autonomous vehicle essay. More essays are on the research index.