The Dutch government risks exacerbating racial discrimination through the continued use of unregulated algorithms in the public sector, Amnesty International said in a damning new analysis of the country’s childcare benefit scandal.
The report Xenophobic Machines exposes how racial profiling was baked into the design of the algorithmic system used to determine whether claims for childcare benefit were flagged as incorrect and potentially fraudulent. Tens of thousands of parents and caregivers from mostly low-income families were falsely accused of fraud by the Dutch tax authorities as a result, with people from ethnic minorities disproportionately impacted. While the scandal brought down the Dutch government in January, sufficient lessons have not been learnt despite multiple investigations.
“Thousands of lives were ruined by a disgraceful process which included a xenophobic algorithm based on racial profiling. The Dutch authorities risk repeating these catastrophic mistakes as human rights protections are still lacking in the use of algorithmic systems,” said Merel Koning, Senior Advisor on Technology and Human Rights at Amnesty International.
“Alarmingly, the Dutch are not alone. Governments around the world are rushing to automate the delivery of public services, but it is the most marginalized in society that are paying the highest price.”
Amnesty International is calling on all governments to implement an immediate ban on the use of data on nationality and ethnicity when risk-scoring for law enforcement purposes in the search of potential crime or fraud suspects.
Discriminatory loop
From the start, racial and ethnic discrimination was central to the design of the algorithmic system introduced in 2013 by the Dutch tax authorities to detect incorrect applications for child benefits and potentially fraud. The tax authorities used information on whether an applicant had Dutch nationality as a risk factor and non-Dutch nationals received higher risk-scores.
Parents and caregivers who were selected by the system had their benefits suspended and were subjected to hostile investigations, characterized by harsh rules and policies, rigid interpretations of laws, and ruthless benefits recovery policies.This led to devastating financial problems for the families affected, ranging from debt and unemployment to forced evictions because people were unable to pay their rent or make payments on their mortgages. Others were left with mental health issues and stress on their personal relationships, leading to divorces and broken homes.
The design of the algorithm reinforced existing institutional bias of a link between race and ethnicity, and crime, as well as generalizing behaviour to an entire race or ethnic group.
These discriminatory design flaws were reproduced by a self-learning mechanism that meant the algorithm adapted over time based on experience, with no meaningful human oversight. The result was a discriminatory loop with non-Dutch nationals flagged as potentially committing fraud more frequently than those with Dutch nationality.
Absence of accountability
When an individual was flagged as a fraud risk, a civil servant was required to conduct a manual review but was given no information as to why the system had generated a higher-risk score. Such opaque ‘black box’ systems, in which the inputs and calculations of the system are not visible, led to an absence of accountability and oversight.
“The black box system resulted in a black hole of accountability, with the Dutch tax authorities trusting an algorithm to help in decision-making without proper oversight,” said Merel Koning.
A perverse incentive existed for the tax authorities to seize as many funds as possible regardless of the veracity of the fraud accusations, as they had to prove the efficiency of the algorithmic decision-making system. Parents and caregivers who were identified by the tax authorities as fraudsters were for years given no answers to questions about what they had done wrong.
The findings of Xenophobic Machines are to be presented at a United Nations General Assembly side event on algorithmic discrimination on 26 October. This year, Amnesty International is launching an Algorithmic Accountability Lab – a multidisciplinary team tasked with carrying out investigations and campaigning on the human rights risks of automated decision systems in the public sector. Amnesty International is calling on governments to:
- Prevent human rights violations in relation to the use of algorithmic decision-making systems, including by implementing a mandatory and binding human rights impact assessment before the use of such systems.
- Establish effective monitoring and oversight mechanisms for algorithmic systems in the public sector.
- Hold those responsible for violations to account and provide effective remedy to individuals and groups whose rights have been violated.
- Stop the use of black box systems and self-learning algorithms where the decision is likely to have a significant impact on the rights of individuals.
