Equity and AI

AI learns from the past, and the past isn't fair. Health data reflects who has been seen, measured, believed and funded. When technology is built on that data without care, it can reproduce inequalities, or make them worse, while appearing neutral.

This page puts emancipatory knowing (see Ways of knowing) into practice: noticing whose interests a technology serves, who it fails, and acting on it.

How bias enters

Unequal historical records, unrepresentative training images, and flawed proxies (such as past healthcare spend instead of clinical need).

Bedside rule

Trust the person over the device: if a score, oximeter reading, or AI transcript contradicts what you see and hear, escalate on clinical judgement.

Organisational duty

Under the Public Sector Equality Duty (Equality Act 2010, s.149), trusts and universities must assess and monitor AI impact across groups.

What the evidence shows

Interactive Emancipatory Knowing Evidence Explorer

Inspect Five Documented Ways Health AI & Devices Create Inequality

Select an evidence domain to see how bias enters the technology, what nurses do at the bedside, and what NHS/university governance must check.

Pulse Oximeters & Optical Skin-Tone Bias

DHSC Equity in Medical Devices Review (2024)

Key finding: Pulse oximeters can overestimate arterial oxygen saturation (SpO2) in people with darker skin tones, masking occult hypoxaemia.

How the Inequality EntersOptical calibration historically relied predominantly on lighter-skinned cohorts, leading to higher melanin absorption error and delayed escalation when staff trust the number alone.

Bedside Nurse Action (Emancipatory Knowing)Trust the person over the reading: assess respiratory rate, work of breathing, speech, agitation, and mucous membranes—and escalate on clinical concern regardless of SpO2.

Organisational & Reporting Duty (EqIA / Datix / LFPSE)Log suspected device disparity via Datix/LFPSE and MHRA Yellow Card, and check trust EqIA procurement standards (18 review recommendations accepted by UK Government).

Example	What was found	Source
Pulse oximeters	Can overestimate oxygen levels in people with darker skin tones more than in people with lighter skin, risking missed or delayed treatment. The UK government accepted all 18 recommendations of the independent review, including on AI.	Equity in Medical Devices: Independent Review, 2024
A population health algorithm	A widely used US algorithm used past healthcare spending as a stand-in for need. Because less had been spent on Black patients with the same needs, it judged them healthier, and fewer were flagged for extra care. Predicting need directly largely removed the bias.	Obermeyer, Z. et al. (2019) Science, 366(6464), pp. 447–453
Speech recognition	Five commercial speech recognition systems made almost twice as many errors for Black speakers as for white speakers (average word error rate 0.35 against 0.19).	Koenecke, A. et al. (2020) PNAS, 117(14), pp. 7684–7689
Chatbots and race-based medicine	Four large language models, asked medical questions, each repeated at least one debunked race-based claim, for example about kidney function.	Omiye, J.A. et al. (2023) npj Digital Medicine, 6, 195
Clinical images	Teaching materials have relied on images of white skin, so signs on Black and brown skin are missed. AI image tools trained on similar material can repeat the gap.	Mukwende, M., Tamony, P. and Turner, M. (2020) Mind the Gap, St George's, University of London

The same patterns can affect people by disability, age, sex and gender, language, neurodivergence, class and where they live. They are harder to see when the people affected aren't involved in designing, testing or reviewing the technology.

Why it matters for nurses

• You are often the person who notices. Nurses see when a tool works badly for a particular person: a reading that doesn't fit the person in front of you, a transcript that garbles someone's words, a leaflet that assumes a family that isn't there.
• The Code expects it. Treat people as individuals and uphold their dignity (standard 1), and treat people fairly and without discrimination (standard 20).
• Organisations have legal duties. Under the Public Sector Equality Duty (Equality Act 2010, section 149), public bodies must have due regard to eliminating discrimination and advancing equality of opportunity. That includes when they buy and use AI.

What nurses can do

1. At the Bedside

Trust the Person Over the Device

If a reading, risk score, or AI transcript contradicts what you see and hear, escalate on your clinical judgement and log speech or skin-tone performance failures via Datix/LFPSE.

2. In Your Team & Trust

Ask for the EqIA

Request the Equality Impact Assessment (EqIA) before any AI tool goes live, involve patient/carer and staff networks early, and monitor outcomes disaggregated by group.

3. In Nursing Education

Teach with Diverse Cases

Pair AI diagrams with resources like Mind the Gap, and use biased AI outputs in Spot the error exercises to build emancipatory knowing.

Questions to ask about any AI tool

Equality Impact Assessment (EqIA) ChecklistEquality Act 2010 (s.149)

5 Questions to Ask Before & After Deploying an AI Tool

1. Training & testing representation:
   Whose data was this tool trained and tested on? Does that include people with the skin tones, accents, languages, ages, and disabilities of the people we care for?

2. Disaggregated performance:
   How does the tool perform across different protected characteristics, and who has independently verified those error rates?

3. Failure impact:
   What happens clinically when the tool gets it wrong, and which patients or families are most likely to be harmed or delayed?

4. Co-design & voice:
   Which patients, carers, and frontline nursing staff (including inclusion networks) were involved in deciding to buy or deploy it?

5. Post-deployment monitoring:
   How will we audit outcomes by group after go-live to detect if the tool is widening inequalities?

Quick Clinical Self-Check

During a community visit, an approved ambient AI scribe produces a draft note that repeatedly misattributes and omits key symptom descriptions spoken by a patient with dysarthria and a strong regional accent. What should the nurse do?

Further reading

• Department of Health and Social Care (2024) Equity in Medical Devices: Independent Review, final report, and the government response.
• Koenecke, A. et al. (2020) Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14), pp. 7684–7689. doi:10.1073/pnas.1915768117.
• Obermeyer, Z. et al. (2019) Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), pp. 447–453. doi:10.1126/science.aax2342.
• Omiye, J.A. et al. (2023) Large language models propagate race-based medicine. npj Digital Medicine, 6, 195. doi:10.1038/s41746-023-00939-z.
• NHS Race and Health Observatory.