Algorithmic bias in healthcare: what the evidence shows

In a 2019 Science study, Obermeyer and colleagues found that a widely used US health risk algorithm assigned Black patients lower risk scores than equally sick White patients, because it used healthcare cost as a stand-in for health need. Reformulating it to predict need would have raised the share of Black patients flagged for extra care from 17.7% to 46.5%.

EasWrk is a hosting platform, not a research institute. This essay summarizes published sources, which are listed at the end.

The Promise of Precision Medicine

The convergence of artificial intelligence with biomedical data represents one of the most consequential technological developments in modern medicine. For decades, clinicians have recognised that medicine practised at the population level. Applying average outcomes from clinical trials to individual patients necessarily fails some portion of those patients, whose biology, genetics, social circumstances, and comorbidities differ from the trial population in ways that alter the risk-benefit calculus of treatment. Precision medicine, the project of tailoring diagnosis and treatment to the individual patient's specific characteristics, has been a long-standing aspiration. AI, operating on datasets of a scale and dimensionality that overwhelm traditional statistical methods, has made that aspiration seem newly achievable.

Topol (2019), in his review of "high-performance medicine," surveyed how deep learning was being applied across medicine, including detection of diabetic retinopathy, classification of skin lesions, and detection of arrhythmias from electrocardiograms. The benchmarks were real and meaningful. The limitations, in the author's reading of that literature, were frequently understated next to the achievements.

The fundamental promise of AI in medicine is that it can find signal in complexity that human cognition cannot efficiently process. A physician reviewing a patient's history, physical examination findings, laboratory results, and imaging studies is performing an impressive feat of synthesis, but one limited by memory, attention, time, and the cognitive biases that accompany human reasoning under uncertainty. An AI system trained on millions of patient records can identify correlations invisible to any individual clinician and apply those correlations consistently at scale, without fatigue, without the availability bias that makes recently-seen diagnoses disproportionately salient, and without the variation in clinical practice that makes outcomes geography-dependent in ways that are difficult to justify on clinical grounds.

These theoretical advantages are real. The empirical literature documenting AI systems that perform at or above specialist level in specific, well-defined tasks is now extensive. The path from these controlled demonstrations to routine clinical deployment is, however, far longer and more treacherous than early enthusiasm acknowledged. Performance on curated benchmark datasets does not straightforwardly predict performance on the messy, heterogeneous data of real clinical practice. The populations on which AI systems are trained are frequently not representative of the populations on which they will be deployed. The feedback loops that allow human clinicians to learn from errors do not exist in the same form for deployed AI systems. The structural features of healthcare, including the power dynamics, economic incentives, regulatory frameworks, and cultural norms, shape AI deployment in ways that the technical research literature largely does not address.

Perhaps most importantly, the aspiration of precision medicine contains within it an implicit claim about data: that more data, better integrated and more intelligently analysed, will yield better medicine. This claim is broadly correct. But it carries a corollary that is rarely examined with comparable rigour: that the benefits of data-intensive precision medicine will flow equitably to all patients. The evidence, examined carefully, does not support that corollary. Genetic research shows the pattern: about 78% of participants in genome-wide association studies were of European ancestry in GWAS Catalog data through January 2019 (Sirugo, Williams, and Tishkoff, 2019). This is a predictable consequence of historical patterns of health data collection and a structural challenge that technical solutions alone cannot adequately address.

Algorithmic Bias in Healthcare Examples: Clinical Decision Support

One landmark study of AI and healthcare equity was published in Science in 2019 by Obermeyer and colleagues. Their study examined a widely used commercial algorithm that US health systems used to guide health decisions, including who gets extra care, and found evidence of racial bias. Specifically, Black patients assigned the same level of risk by the algorithm were sicker than White patients. The authors estimated that this bias reduces the number of Black patients identified for extra care by more than half. The practical consequence was that Black patients had to be considerably sicker than White patients to receive the same risk score.

What causes algorithmic bias? Cost as a stand-in for need

The mechanism of the bias was not a flaw in the algorithm's programming. It was a consequence of what the algorithm was trained to predict. The algorithm used healthcare costs as a proxy for health need: a seemingly reasonable choice, given that costs are readily available in administrative claims data and would presumably track the severity of medical conditions. But costs, as a reflection of actual healthcare consumption, are also a reflection of systemic inequities in access to care. Less money is spent on Black patients with the same level of need (Obermeyer et al., 2019). Barriers to access and differences in how patients are treated are plausible reasons. An algorithm trained to predict costs therefore learned to predict healthcare utilisation, which is substantially determined by access, rather than healthcare need, which is substantially determined by biology and circumstance.

AI bias in healthcare statistics: the Obermeyer numbers

The Obermeyer study reported that reformulating the algorithm to predict health need rather than healthcare costs would have raised the share of Black patients flagged for extra care from 17.7% to 46.5%, an increase of more than 160%. This magnitude of effect was not a marginal calibration issue. It was a fundamental failure of the system to perform its claimed function equitably, with directly harmful consequences for the patients who most needed the services it was supposed to allocate.

AI racial bias in healthcare beyond one algorithm: dermatology

Adamson and Smith (2018), writing in JAMA Dermatology, warned that machine learning in dermatology could widen health care disparities if training images under-represent darker skin. Daneshjou and colleagues (2022) tested that concern in Science Advances. They built a pathologically confirmed image set with diverse skin tones and found that state-of-the-art dermatology AI models showed substantial limitations on it, particularly on dark skin tones and uncommon diseases. Dermatologists also performed worse on images of dark skin tones, and fine-tuning the models on the new images closed the performance gap between light and dark skin tones.

Mittelstadt and colleagues (2016) offered a framework for the ethical concerns raised by algorithms in general, not only in health care. Their map has six concerns: inconclusive evidence, inscrutable evidence and misguided evidence, which are epistemic; unfair outcomes and transformative effects, which are normative; and traceability. Each can be found in clinical AI, and in the author's view they compound one another.

Char, Shah, and Magnus (2018), writing in the New England Journal of Medicine, argued that implementing machine learning in health care raises ethical challenges that need attention early. They warned that algorithms could absorb human biases, that the goals of the people who build and buy these systems may diverge from those of clinicians and patients, and that treating them as black boxes may lead physicians to over-rely or under-rely on them.

The regulatory dimension was addressed by Wu and colleagues (2021) in their analysis of 130 FDA-approved AI medical devices cleared between January 2015 and December 2020. They found that 126 of the 130 were evaluated only in retrospective studies at submission, that none of the 54 high-risk devices had a prospective study, and that reporting of multi-site assessment and patient demographics was thin. In the author's view, an approval process designed for physical devices with stable performance is poorly adapted to software whose performance can vary across deployment contexts.

The same proxy problem appears in credit scoring, and the courts face a version of it with criminal risk scores. More essays are on the research index.

The Health Data Marketplace

Health data is among the most sensitive categories of personal information: a comprehensive record of vulnerabilities, behaviours, relationships, and risk factors that can affect employment, insurance, relationships, and life opportunities. It is also, by virtue of its informativeness, among the most commercially valuable. The intersection of these two characteristics has created an expanding and poorly-regulated marketplace in which health data flows far beyond the clinical relationships in which it was generated, often without the knowledge or meaningful consent of the individuals it describes.

Price and Cohen (2019), in their analysis of privacy in the age of medical big data, noted that the majority of health data is not covered by HIPAA at all. HIPAA regulates particular actors, such as health care providers and other covered entities, not the data themselves, so health information from sources such as apps and search histories falls largely outside it. Deidentified data may also become re-identifiable by combining it with other data sets.

The NHS and Google DeepMind case became a well-known example of the gap between stated purposes and patient expectations. In 2015 the Royal Free London NHS Foundation Trust gave DeepMind access to the health records of 1.6 million patients to develop Streams, an app meant to spot patients at risk of acute kidney injury. According to the UK Information Commissioner's Office, the Trust shared that information without people's knowledge or consent and without giving them the option to opt out, and in 2017 the ICO concluded the Trust should have been much more transparent about how it used patients' personal information (ICO).

The commercial logic of health data is straightforward and powerful: data that enables better risk stratification, more accurate diagnosis, more effective treatment selection, or more precise drug development is valuable to pharmaceutical companies, insurers, and healthcare providers. The individuals who generated that data, through the vulnerability of illness and through encounters with clinical care systems, capture little of that value. This asymmetry is not unique to health data, but it is more morally fraught in health than in most other domains, because the power differential between patients and healthcare institutions is acute, consent is compromised by the urgency of care needs, and the potential for data-derived decisions to harm the data's subjects, through insurance discrimination, employment decisions, or differential access to care, is concrete and proximate.

The re-identification of nominally anonymised health data is a persistent technical challenge that makes the concept of de-identification less protective than it sounds. Rocher, Hendrickx, and de Montjoye (2019) estimated that 99.98% of Americans would be correctly re-identified in any dataset using 15 demographic attributes, which challenges the adequacy of the de-identification release-and-forget model (Nature Communications). Genomic data presents a particularly acute challenge: it is inherently identifying (it encodes ancestry, family relationships, and individual distinctiveness), it is permanent (unlike a password, you cannot change your genome), and it is consequential (it predicts health risks, carries insurance implications, and provides identifying information about family members who have not consented to share their data).

The regulatory framework for health data in the United States remains fragmented. HIPAA provides protection for data held by covered entities (healthcare providers, health plans, healthcare clearinghouses) and their business associates, but excludes the vast majority of consumer health data generated outside formal clinical settings. State-level privacy laws, including California's CPRA and Colorado's CPA, provide some additional protections but vary significantly in their scope and enforcement. The EU's GDPR provides stronger baseline protections for health data than the US patchwork.

AI in Diagnostic Imaging

Diagnostic imaging covers radiology, pathology, ophthalmology, and dermatology. It is the clinical domain in which AI has achieved some of the strongest performance benchmarks. The reasons are structural: imaging produces standardised, high-dimensional data amenable to machine learning analysis; there are large, well-curated training datasets available; and the task of classification from images is one for which deep learning methods have demonstrated particular strength across many domains.

Rajpurkar and colleagues (2017) at Stanford reported that CheXNet, a convolutional neural network trained on ChestX-ray14, a public dataset of more than 100,000 frontal-view chest X-rays, exceeded the average performance of four practicing academic radiologists at detecting pneumonia, measured by F1 score. Some commentators took this as a sign that AI would soon make radiologists redundant. In the author's view that conclusion was overstated: the result applied to a single task on one curated test set, not to the full range of imaging presentations or the clinical context radiologists work in.

Esteva and colleagues (2017), also from Stanford, demonstrated that a convolutional neural network could classify skin cancer lesions at a level comparable to board-certified dermatologists, achieving equivalent sensitivity and specificity on a binary task of distinguishing malignant from benign lesions in a curated dataset. The paper appeared in Nature and attracted significant attention. As described above, Daneshjou and colleagues (2022) later found that dermatology AI models performed worse on dark skin tones.

FDA authorizations of AI-enabled devices have grown steeply. One review of the FDA's public list, last updated in September 2024, counted 1,247 devices, of which 956 (77%) were authorized by the radiology panel (FDA Law Blog, 2025). Wu and colleagues (2021) found that the pre-market evidence for devices cleared through 2020 was mostly retrospective, which leaves a gap between the controlled conditions of regulatory submission and the heterogeneous conditions of clinical deployment.

The deployment of AI diagnostic tools in clinical practice raised fundamental questions about the allocation of responsibility for diagnostic errors. When a radiologist misses a finding on a chest CT, the error can be attributed, investigated, and learned from within existing professional accountability frameworks. When an AI system misses the same finding, the accountability is distributed among the system's developers, the institution that deployed it, the radiologist who reviewed the AI output, and the regulatory body that approved the device. This diffusion of accountability is not merely a legal problem. It affects whether errors are identified, investigated, and corrected, which determines whether AI diagnostic systems improve over time in the way that human clinicians improve through feedback.

The clinical workflow integration challenge, separate from the question of algorithmic performance, has proven to be one of the most practically significant barriers to beneficial AI deployment in radiology. AI diagnostic tools are not free-standing clinical services; they must be integrated into the radiology workflow in ways that preserve, rather than undermine, the clinician's ability to exercise independent judgment. Research on "automation bias," the tendency of human operators to over-trust automated recommendations and under-weight their own contrary observations (Goddard, Roudsari, and Wyatt, 2012), suggests that poorly designed AI integration can degrade overall system performance even when the AI component performs well in isolation, by anchoring clinicians to AI outputs that are incorrect and dampening the critical evaluation those outputs should receive.

Automation and Clinical Displacement

The question of whether AI will displace clinical workers, and if so which categories, has generated extensive commentary and rather less empirical certainty. Early predictions that radiologists would be made redundant within a decade have not come true as of 2026. In the author's view they rested on a misunderstanding of what radiologists and GPs actually do.

There is a distinction between tasks that AI can automate within clinical roles and the wholesale replacement of clinical roles themselves. Radiology, for example, involves image interpretation, the task where AI has demonstrated strong benchmark performance, but also clinical consultation, procedure guidance, report communication, integration of imaging findings with clinical context, and the management of incidental findings that require clinical judgment about follow-up. Automating image interpretation, even perfectly, would automate one component of radiology practice rather than the practice itself.

A subtler risk is that AI and digital tools change the composition of clinical work, adding supervision of automated outputs and quality assurance tasks while possibly reducing time for direct patient interaction. Whether that happens, and whether clinicians experience it as an improvement, are empirical questions this essay does not settle.

At the level of the health workforce as a whole, the concern is not primarily about clinician displacement but about clinical task allocation. AI-enabled task automation creates the technical possibility of assigning clinical tasks to workers with lower formal qualifications, using AI as a substitute for specialist judgment rather than as an augmentation of it. The economic incentives for healthcare systems to exploit this possibility are real. Whether it results in equivalent or better patient outcomes, and for which patient populations, are empirical questions that require careful study rather than confident assertion in either direction.

Nursing and allied health professions face a different version of the automation question. Tasks that are physically performed by nurses, such as patient mobility assistance, medication administration, and vital sign monitoring, are candidates for robotic automation. Tasks that involve communication, emotional support, relationship-building, and the exercise of clinical judgment in complex interpersonal contexts are substantially more resistant to automation, though not entirely immune as conversational AI systems improve. The distribution of automation risk across nursing tasks is uneven in ways that have significant implications for nursing practice and for the therapeutic relationship that nurses maintain with patients.

Consent, Privacy, and Oversight Gaps

The informed consent framework that governs clinical research and, in principle, clinical care was developed in a context where the encounter between patient and healthcare system was bounded, identifiable, and finite. A patient consenting to a clinical procedure or a research study understood (in the idealised model) what was being proposed, what the alternatives were, what the risks were, and what the benefits might be. That information was sufficient to support a meaningful choice. AI-enabled medicine has introduced conditions that the consent framework struggles to accommodate.

When a patient's electronic health record data is used to train an AI clinical decision support system, the "consent" involved is typically buried in a terms-of-service document signed at registration, referencing data use policies that most patients have never read. The specific use, training an AI system that will make recommendations affecting the care of future patients, is neither disclosed nor explained in terms accessible to the average patient. The data may be used by the healthcare institution itself, by a technology partner, or by a third-party AI developer, with each step in the chain involving additional complexity that the original consent relationship was not designed to manage.

HIPAA's "minimum necessary" standard and de-identification pathway sit awkwardly with AI model training, in the author's view. Training a large AI model involves exposing it to the entirety of available data, not a theoretically justified subset. The concept of "minimum necessary" data for AI training is difficult to operationalise meaningfully.

Regulatory oversight of AI in clinical practice has been fragmented and inconsistent. The FDA's Software as a Medical Device framework provided a pathway for AI clinical tools to receive regulatory approval, but section 3060(a) of the 21st Century Cures Act excludes certain clinical decision support software from the device definition (FDA guidance), so such tools need no FDA pre-market review. In the author's view, that leaves a category of clinical tools with real potential to affect patient safety without a mandatory efficacy or safety evaluation.

Mittelstadt and colleagues (2016) identified inscrutable evidence, the difficulty of understanding why an algorithm produces a particular output, as one source of accountability gaps. In clinical contexts, this opacity has specific consequences. A physician who makes a clinical error can be asked to explain their reasoning, and that explanation can be evaluated by peers, disciplinary bodies, and courts. An AI system that produces a harmful recommendation cannot provide this explanation in the same sense. The "black box" character of deep learning systems, technically explicable to experts through attention maps and feature importance analyses but not interpretable in the way that clinical reasoning is interpretable, creates a genuine accountability deficit.

In December 2024 the FDA issued final guidance on predetermined change control plans for AI-enabled device software functions, which lets a manufacturer describe in advance the modifications it intends to make (FDA, 2024). Continuous learning systems still raise acute oversight questions: if a system's algorithm changes after deployment, how does the regulatory authorisation granted at the time of initial approval remain valid? How are downstream effects on patient care monitored? How are clinicians informed of model updates that may change the system's recommendations? In the author's view these questions remain without fully satisfactory answers.

How to Avoid Algorithmic Bias: What Equitable AI in Healthcare Would Require

AI in healthcare today is characterised by demonstrated technical capability in constrained domains, significant bias in deployment, inadequate oversight frameworks, and inequitable data practices. The path to a future in which AI genuinely reduces health disparities and improves outcomes equitably across populations is neither short nor simple. But the components of that path are sufficiently clear that the absence of progress along them can be recognised as a choice rather than an inevitability.

Representative training data is the most foundational requirement. AI systems trained on non-representative datasets will not perform equitably across the full diversity of patient populations, and no amount of post-hoc calibration fully compensates for training data that systematically underrepresents specific populations. This requires active investment in data collection from underrepresented communities, including communities that have historically had reason to distrust medical research institutions. The Tuskegee Syphilis Study and other historical research abuses created a legacy of justified mistrust in African American communities, which in the author's view affects research participation. Equitable AI in healthcare is inseparable from the broader project of repairing the relationship between medical institutions and underserved communities.

Prospective algorithmic auditing, meaning mandatory evaluation of AI system performance across demographic subgroups before deployment and continuously thereafter, is a technical requirement that has regulatory implications. The FDA's action plan for AI/ML-based software as a medical device, published in January 2021, committed to a patient-centered approach that includes transparency to users (FDA bulletin, 2021). A regulatory requirement for pre-deployment demographic subgroup performance disclosure, with meaningful thresholds for acceptable disparity, would create enforceable standards where voluntary commitments have proven insufficient.

Meaningful consent for health data use in AI development requires a different architecture than the current terms-of-service model. Several proposals offer paths toward more genuine consent, including dynamic consent platforms that allow patients to make granular, ongoing choices about data use, and fiduciary frameworks that require healthcare institutions to act in patients' interests in data governance decisions. None has achieved widespread adoption, partly because the current system is financially advantageous to the institutions that would have to change it.

Clinical AI governance at the institutional level, including multidisciplinary review of AI tools before deployment, ongoing performance monitoring, and clear accountability structures for AI-related adverse events, is achievable within existing institutional frameworks and does not require regulatory change. In November 2023 the American Medical Association issued principles for the development, deployment and use of augmented intelligence, including proactive identification and mitigation of bias (AMA, 2023). In the author's view, principles alone are too general to give operational direction. More detailed, mandatory governance frameworks, analogous to institutional review board requirements for human subjects research, would raise baseline standards across the sector.

Ultimately, equitable AI in healthcare requires confronting the fact that healthcare inequity is not primarily a data problem or an algorithm problem. It is a structural problem, rooted in the distribution of resources, power, and opportunity in the broader society. AI cannot correct inequities in access to care, in exposure to environmental health risks, in the economic determinants of health, or in the structural racism that produces differential health outcomes across racial and ethnic groups. What AI can do is avoid amplifying those inequities by learning from data that reflects them, and be designed and deployed in ways that explicitly prioritise equity rather than treating it as a secondary consideration. That requires technical choices, regulatory requirements, and institutional commitments. It also requires political will, which is ultimately a social question rather than a technical one.