AI risks · Bias and discrimination

Can AI discriminate?

AI systems learn from data that reflects existing inequalities, and can repeat or amplify them when used to make decisions about people. This is documented, including in Sweden, and the EU now sets binding rules for high-risk uses.

Happening nowSocietal impact:

How bias arises

Bias here means systematic errors that favour some groups over others. The International AI Safety Report 2025 found that general-purpose AI systems frequently show biases related to race, gender, culture, age, disability and political opinion. One reason is the training data, which over-represents English-speaking and Western material. Design choices also matter. Recent research has found subtler forms too, such as different answers depending on the dialect a user writes in.

Technical fixes have improved things considerably but do not always work, and the line between a harmful stereotype and accurate knowledge about the world can be hard to draw.

When bias reaches decisions

The risk becomes concrete when AI is used to decide who gets checked, hired or approved. The EU Agency for Fundamental Rights (FRA) showed in 2022 how bias can grow over time in predictive policing: if police patrol where an algorithm points, they record more crime there, which feeds back into the next version of the algorithm. FRA also found strong bias in tools that detect offensive speech, related to ethnic origin, gender, religion and sexual orientation.

In Sweden, the Data Protection Authority (IMY) opened an inspection in June 2025 into how the Social Insurance Agency (Försäkringskassan) selected people for targeted checks of temporary parental benefit for caring for a sick child (VAB), and whether an AI system was involved. The agency replied that it had used a machine-learning-based risk profile but had stopped using it, with no plans to reintroduce it. IMY therefore closed the case in November 2025 without examining it further.

What is being done

The EU AI Act bans social scoring and AI that predicts an individual's risk of committing crimes; these bans have applied since February 2025. AI used in employment, education, access to essential public and private services, law enforcement, migration and the justice system is classed as high-risk. Such systems must meet requirements on data quality, logging, documentation and human oversight. After the 2026 amendments, these rules apply from December 2027.

FRA recommends regular bias testing both before and after a system is put into use, and stronger equality bodies with specialist staff. Discrimination law and data protection law already apply to automated decisions today.

What you can do

  1. If you receive a decision from an authority or company, you can ask how it was made and whether automated tools were used.
  2. Appeal decisions you believe are wrong; a human review is often possible.
  3. If you suspect discrimination, contact the equality body in your country (in Sweden, the Equality Ombudsman, DO).
  4. If your organisation uses AI for decisions about people, test the results for different groups before and after launch.

Sources

  1. International AI Safety Report 2025 – International AI Safety Report (chair: Yoshua Bengio), 2025
  2. Bias in algorithms – Artificial intelligence and discrimination – European Union Agency for Fundamental Rights (FRA), 2022
  3. Beslut efter tillsyn enligt dataskyddsförordningen – Försäkringskassan (IMY-2025-11828) – Integritetsskyddsmyndigheten (IMY), 2025
  4. AI Act – regulatory framework for AI – European Commission, 2026

Data updated: 2026-11-19

More risks

AI risks