Biases in data

Flawed data is characterized as non-representative, lacking information, historically biased or otherwise “bad” data.¹ It leads to algorithms that produce unfair outcomes and amplify any biases in the data. AI systems that use biased results as input data for decision-making create a feedback loop that can also reinforce bias over time. This cycle, where the algorithm continuously learns and perpetuates the same biased patterns, leads to increasingly skewed results. 

Bias can also arise during the training phase if data is incorrectly categorized or assessed. Sometimes, algorithms can “learn” from data correlation rather than causation, as they do not possess the abilities to understand the difference. When this happens, the output of the algorithm can be biased in that the model failed to consider other factors in the data that may be of more importance.

A commonly cited example of correlation bias is a hypothetical model that determines a causal relationship between increased shark attacks and higher ice cream sales. In reality, both situations tend to occur during summer and only possess a correlating relationship.

Biases in algorithmic design

Algorithms design can also introduce bias. Programming errors, such as an AI designer unfairly weighting factors in the decision-making process, can unknowingly transfer into the system. Weighting is often a technique to avoid bias, as it involves adjustments to data so that it better reflects the actual population. However, it might require assumptions from designers, which can lead to inaccuracies and introduce bias. Developers might also embed the algorithm with subjective rules based on their own conscious or unconscious biases.

Biases in proxy data

AI systems sometimes use proxies as a stand-in for protected attributes, like race or gender. However, proxies can be unintentionally biased as they might have a false or accidental correlation with the sensitive attributes they were meant to replace. For example, if an algorithm uses postal codes as a proxy for economic status, it might unfairly disadvantage certain groups where postal codes are associated with specific racial demographics.

Biases in evaluation

Biases in evaluation occur when algorithm results are interpreted based on the preconceptions of the individuals involved, rather than the objective findings. Even if the algorithm is neutral and data-driven, how an individual or business applies the algorithm’s output can lead to unfair outcomes depending on how they understand the outputs.

Discrimination and inequality

Biased algorithmic decisions reinforce existing societal disparities faced by marginalized groups and these human biases lead to unfair and potentially harmful outcomes from AI systems. While many of the most common AI applications might seem low-stakes (such as search engines, chatbots and social media sites) other applications of AI can influence life-altering decisions. The use of biased AI tools within areas like criminal justice, healthcare and hiring could yield devastating results.

For example, the marginalization of African American people in the past is reflected in historical arrest data from Oakland, California in the United States. If this data is used to train a current predictive policing algorithm (PPA), the decisions made by the PPA are likely to reflect and reinforce those past racial biases.

Legal and reputational damage

Organizations that use biased AI systems could face legal consequences and reputational damage, as biased recommendations can have what’s known as a disparate impact. This is a legal term referring to situations where seemingly neutral policies and practices can disproportionately affect individuals from protected classes, such as those susceptible to discrimination based on race, religion, gender and other characteristics.

Protected groups adversely affected by biased AI decisions might file lawsuits, potentially leading to significant financial liabilities, long-term reputational damage and condemnation from stakeholders. Organizations could also face financial penalties if they are found to be in violation of any applicable antidiscrimination laws.

Erosion of trust

Biased results from AI tools erode trust in AI in multiple ways. If an organization is found to have biased AI systems, they might lose the trust of stakeholders within the business who no longer have confidence in the algorithmic decision-making processes. These stakeholders might also no longer consider the optimization value of AI to outweigh its risk and lose confidence in the technology overall.

Algorithmic bias can also lose the trust of customers. It takes only one case of discrimination to ruin brand reputation, especially in the era of fast-spreading news. Trust in AI is especially important to retain with marginalized groups like people of color, who already experience bias and discrimination in the physical world.

Bias in criminal justice

United States courts use the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) tool to assess defendants’ recidivism risk. A study by ProPublica revealed that the tools algorithm might have classified white and Black defendants differently. For example, in the risk assessment, Black defendants were twice as likely as white defendants to be misclassified as being a higher risk of violent recidivism. The company that created the tool disputes this analysis; however, it does not disclose the methods used to arrive at risk scores.²

Bias in predictive policing

Researchers built their own predictive policing algorithm trained on victim report data from Bogotá, Colombia. When they compared the model’s predictions with actual crime data sets, they found major errors. For example, it predicted 20% more high-crime locations than reality in districts with a high volume of reports. But this reflected a social bias: Black people are more likely to be reported for a crime than white people.³

Bias in healthcare

In healthcare, underrepresentation of minority groups in data can skew predictive AI algorithms. For example, computer-aided diagnosis (CAD) systems have been found to return lower accuracy results for Black patients than white patients. 

Bias in recruitment
 

Amazon abandoned an AI recruiting tool after discovering it systematically discriminated against female job applicants. Developers trained the hiring algorithm using resumes from past hires—who were predominately male. As a result, the algorithm unfairly favored keywords and characteristics found in men’s resumes.⁴

Bias in financial services

Bias within financial services can have severe consequences for people’s livelihoods, as historical data can contain demographic biases affecting creditworthiness, loan approvals and more. For example, a study from the University of California, Berkeley showed that an AI system for mortgages routinely charged minority borrowers higher rates for the same loans when compared to white borrowers.⁵

Bias in image generation

Academic researchers found gender bias in the AI image generator Midjourney. During the analysis of over 100 generated images, they also found instances of racial, class and age bias in the results. For example, when asked to create images of people in specialized professions, it showed both younger and older people, but the older people were always men, reinforcing gender bias regarding women in the workplace.⁶

Bias in facial recognition systems

Research from MIT found that some general purpose commercial facial recognition systems—used for matching faces in photos, for example—were unable to recognize darker-skinned individuals. Recognition was even worse for darker-skinned women. Training data that misrepresented real demographics skewed the results.⁷

Bias in pricing

After a Chicago law forced ride-hailing companies to disclose their fares, researchers discovered that Uber and Lyft’s pricing algorithm charged more for drop-offs in neighborhoods with high non-white populations.⁸

Diverse and representative data

Machine learning is only as good as the data that trains it. For AI to better reflect the diverse communities it serves, a far wider variety of human beings’ data must be represented in models. Data fed into machine learning models and deep learning systems must be comprehensive and balanced, representative of all the groups of people and reflective of the actual demographics of society.

Bias detection and mitigation

No computer systems are ever fully “trained” or “finished.” Ongoing monitoring and testing (through initiatives like impact assessments, algorithmic auditing and causation tests) can help detect and correct potential biases before they create problems. Processes like the “human-in-the-loop” system require recommendations to be reviewed by humans before a decision is made to provide another layer of quality assurance.

Transparency and interpretability

AI systems can be “black boxes,” which makes it difficult to understand their outcomes. Transparent AI systems clearly document and explain the underlying algorithm’s methodology and who trained it. The more people understand how AI systems are trained and tuned and how they make decisions, the more individual stakeholders and society at large can trust AI’s accuracy and fairness.

Inclusive design and development  
 

Inclusive AI starts with a diverse and interdisciplinary team of AI programmers, developers, data scientists, ML engineers and more who are varied racially, economically, by educational level, by gender, by job description and other demographic metrics. Diversity within design and development will bring different perspectives to help identify and mitigate biases that might otherwise go unnoticed.