Algorithmic Bias

Algorithmic bias occurs when a computer system's outputs create or reinforce unfair outcomes, such as discriminating against a particular group. This often happens because the data used to train the system reflects existing human biases. For example, if a company's past hiring data shows they primarily hired men, an AI trained on that data might learn to unfairly favor male candidates, even if gender isn't explicitly mentioned. The system simply learns patterns from the past—including the unfair ones.

Ask this: What historical biases might be hidden in the data used to make a decision about me?

Algorithmic bias isn't usually a case of programmers intentionally writing discriminatory code. Instead, bias typically creeps in through the data fed into the system. If historical data reflects societal inequalities, an algorithm will learn to replicate and even amplify those same inequalities, mistaking correlation for causation.

This can have serious real-world consequences. Consider a loan approval algorithm trained on historical data from a bank that practiced redlining. The algorithm may learn to associate certain zip codes with higher risk, unfairly denying qualified applicants simply based on where they live, thus perpetuating a cycle of economic disadvantage.

Similarly, in healthcare, an algorithm designed to predict patient risk might use past healthcare spending as a proxy for sickness. This approach was found to systematically disadvantage Black patients, who due to systemic factors have historically spent less on healthcare for a given level of need. The algorithm could then incorrectly flag them as healthier than they are, denying them access to necessary care programs [7].

The key takeaway is that “data-driven” does not automatically mean “objective.” We must critically examine the data sources, model assumptions, and real-world impacts of any automated system to ensure it promotes fairness, not just historical efficiency.

Research on algorithmic bias, often called algorithmic fairness, explores its sources, impacts, and mitigation strategies. It distinguishes between different types of bias (e.g., historical, measurement, representation) and proposes mathematical definitions of 'fairness,' which can sometimes be in conflict, creating difficult tradeoffs.

• Barocas & Selbst (2016) argue that algorithmic bias can lead to discriminatory outcomes legally recognized as disparate impact, even without any discriminatory intent on the part of the creators. [1]
• Kleinberg et al. (2016) and Chouldechova (2017) show it is generally impossible to satisfy both calibration (predictive parity) and equal error rates across groups when the underlying prevalence of the outcome (the base rate) differs between groups. [2, 6] [2]
• Buolamwini & Gebru (2018) exposed significant accuracy disparities in commercial facial recognition systems, which performed far worse on darker-skinned women than on lighter-skinned men, highlighting the problem of biased representation in training datasets. [3]
• Obermeyer et al. (2019) found that a widely used commercial algorithm for predicting health risk systematically disadvantaged Black patients. It used past healthcare costs as a proxy for need, which underestimated the illness severity of Black patients compared to equally sick white patients. [7]

Defining and measuring 'fairness' is a major challenge, as there are over twenty different mathematical definitions, and they often conflict. What is fair to one group may be unfair to another by a different metric. Furthermore, 'de-biasing' an algorithm is not a purely technical fix; it requires deep domain knowledge and addressing the root societal inequalities that the data reflects. Simply removing protected attributes like race or gender from data is often ineffective, as other variables (like zip code) can act as strong proxies.