Algorithms and Bias

Background

While there is widespread agreement that algorithmic bias exists, defining it precisely is tricky, as there are different ways to measure bias. Using the example of an algorithm that assesses risk for criminal behavior, one might measure bias in several different ways (Huq, 2019). One could look at aggregate risk scores for various groups and see if they differ. Or, one could determine if the same initial risk score resulted in the same final risk score for people with different demographic characteristics. One could determine whether the rate of false positives and/or false negatives varied for each demographic group. Efforts to improve the fairness of an algorithm on one measure may actually lead to worse performance on another measure. For example, Speicher, et al (2018), borrowed the concept of inequality indices, which are used by economists, and applied them to biased algorithms. This provided a way to quantify bias. But they also found that efforts to minimize between-group bias may actually increase within-group bias.

For purposes of this article, a biased algorithm will be defined as one that that unfairly and/or inaccurately discriminates against a certain person or group of people, especially on the basis of protected categories such as race and/or gender. In some cases (as will be discussed below), the bias is not in the algorithm per se but rather in the data which it uses.

Unfortunately, examples of algorithmic bias are not difficult to find. A study found that names associated with people of color were far more likely to return search results that were negative (in this case, related to arrest records) than neutral or positive. A black-identified name was 25% more likely to return an ad for an arrest record (Sweeney, 2013). In an experience that went viral in 2015, Jacky Alcine and his friend, both African American, were tagged as “gorillas” by Google Photos (Garcia, 2016). Google responded promptly by removing the auto-tags for terms that might be offensive, but they didn’t fix the underlying problem, which was the algorithm’s inability to properly identify people with darker skin (Monea, 2019).