This review of empirical studies on algorithmic bias revealed significant advancement on various fronts, but a significant amount of work is still remaining to enhance our understanding of the phenomenon. As represented in Figure 1, empirical studies in algorithmic bias can be reasonably distinguished in three themes:
First, based on Internal algorithmic characteristics, which include properties of the algorithm, for instance, data used, and AI algorithm (Cowgill & Tucker, 2019). Therefore, it consists of literature discussing data biases, and bias in the algorithmic process, such as, how variables are weighed, and decisions made by institutions. Second, the evaluation of algorithmic bias from the perspective of how an algorithm is interpreted by users, for instance, perceived algorithmic fairness (Shin, 2020). Third, evaluation of Algorithmic bias literature from the behavioural science aspect (Behavioural Algorithmic bias)-, for instance, few people blindly trust algorithmic output, whereas sceptics interrogate algorithms to unveil some form of biases in algorithms. We argue that these issues are insufficiently explored and demand further empirical research. First, the key finding of past literature will be elaborated and then we focus on an area that deserves empirical validation.
4.1 Results from Previous Empirical Studies –
(i) Internal Algorithmic Characteristics aspect of algorithmic bias literature-
Empirical studies on internal algorithmic characteristics of algorithmic bias consist of : (i) defining algorithmic fairness, (ii) demonstrating algorithmic bias, and (iii) mitigating algorithmic bias. These studies described the technical and firm-level issues which lead to algorithmic bias.
(i) Defining algorithmic fairness - First, empirical studies defining algorithmic fairness will be discussed in this section. Fairness is being defined for three different levels - individuals, groups, and subgroups (Mehrabi et al. 2021). A review by Mehrabi et al. (2021) provided a widely used definition of fairness, these definitions were driven through empirical studies. Fairness definition at various levels are as follows: at the group level: Demographic parity (Dwork et al. 2012; Kusner et al., 2017), Conditional statistical parity (Corbett-Davies et al., 2017), Equalized odds (Hardt et al., 2016), Equal opportunity (Hardt et al., 2016), Treatment equality (Berk et al., 2021) Test fairness (Chouldechova, 2017); at subgroup level: Subgroup fairness (Kearns et al., 2018; 2019); and at individual level: Fairness through unawareness (Grgic-Hlaca et al., 2016., Kusner et al., 2017) Fairness through awareness (Dwork et al., 2012) Counterfactual fairness (Kusner et al., 2017) (Mehrabi et al. 2021).
(ii) Demonstrating algorithmic bias - Second, empirical studies focused on algorithmic bias focused on demonstrating the presence of bias in an algorithmic system is being discussed. Amini et al. (2018), developed a Deep Neural Network (DNN) to address training data imbalance and potential bias in autonomous driving systems. Kay et al. (2015) demonstrated the presence of gender bias in image searches related to various occupations. Various studies have exhibited the presence of gender bias and racial discrimination in ads shown to users in online systems (Datta et al., 2014; Sweeney, 2013). Study by Angwin, Larson, Mattu, & Kirchner (2022) found that algorithms for predicting repeat offenders in the criminal justice system tend to discriminate on the basis of race. Torralba & Efros (2011) conducted a comparison study on popular image datasets and evaluated them on various criteria, such as relative data bias, cross-dataset generalization, and so on. Schmidt (2015), identified the presence of bias in word embedding. Caliskan, Bryson, & Narayanan, (2017) demonstrated that text corpora are the exact imprint of our historical bias, be it being morally neutral for flowers, and insects, problematic for gender/race, similar distribution of gender with careers and first name. Also, some studies demonstrated the presence of algorithmic bias in digital platforms, for instance, Lambrecht, & Tucker (2019), explored gender bias in ads provided by algorithms on Facebook, related to jobs in science, technology, engineering, and math fields; Dash et al. (2021), found that on Amazon platform sponsored recommendation were biased toward Amazon private label products; Xie, Yang & Yu (2021), elaborated the algorithmic bias in news recommendation in China’s digital media; Park, Yu & Macy (2023) claimed that on Airbnb selection of same race endorsement by the consumer is due to top searches provided by recommender system and not due to content of the recommendation. Papakyriakopoulos & Mboya (2023), demonstrated racial bias in Google searches and claimed that discriminatory algorithmic outcomes resulted because of the training data set, and attitude of firm owners, and the algorithm designer. Lin et al. (2023), demonstrated algorithmic bias in leading search engine autocomplete, where discrimination was based on race, gender, and sexual orientation.
Several algorithmic bias assessment tools are also developed, which can assess fairness in a system. For instance, Saleiro et al. (2018) presented Aequitas, which is an open-source fairness and bias audit toolkit, that lets users test models for different biases and fairness matrices for individual/group/subgroups. It provides reports that help machine learning researchers, data scientists, and policymakers make informed and fair decisions for deployed algorithmic systems. Another toolkit is AI Fairness 360 (AIF360), which is developed with the aim of facilitating fairness in algorithms to be used in industrial settings and developing a framework for researchers to enable evaluation of algorithms. Its package consists of various fairness metrics and their explanations, and algorithms to mitigate biases in models and datasets. It also has an interactive web experience, to facilitate practitioners, and data scientists to implement the most appropriate tool while searching for solutions or in their work product (Bellamy et al., 2019).
(iii) Mitigating algorithmic bias - Third, the focus is made on empirical studies on algorithmic bias which discuss the mitigation of bias in an algorithmic system. Amini et al. (2019) targeted the biased data set having under-representation of a segment of society and developed an algorithm which mitigates potentially unknown and hidden biases in training data. The study by Bellamy et al. (2019), discusses about toolkit AIF360 can mitigate bias present in models and datasets. Various studies developed a method to mitigate bias borrowed from Saleiro et al. (2018) are : Hardt et al. (2016); Kamishima et al. (2011); Feldman et al. (2015); Kleinberg et al. (2016); Corbett-Davies et al. (2017); Zafar et al. (2017); Kearns et al. (2019); Noriega-Campero et al. (2019). A review study by Mehrabi et al. (2021), mentions several empirical studies in different domains (such as regression, clustering, natural language processing, and so on) to combat unfairness and bias in AI in order to attain fairness. Basically, methods to mitigate algorithmic bias lie in three categories- pre-processing, in-processing, and post-processing (Mehrabi et al., 2021; Schwartz, et al., 2022). Understanding fairness presence or bias in algorithmic systems will help people understand how and where an algorithmic bias can affect users and systems, and help researchers to identify potential points where biased or discriminating outcomes by algorithm will have negative consequences.
(iv) Attitude of firm owners, and the algorithm designer - Groves et al. (2024), discussed the failure of the algorithmic auditing system implemented in New York City, in July 2023, used for auditing automated employment decision-making tools (AEDTs). It occurs due to - narrow AEDT definition, flawed transparency-driven theory by law, industry lobbying, and challenges faced by auditors while data accessing. The study also provided recommendations for policymakers to develop a better algorithm auditing system.
(ii) Evaluation of algorithmic bias from consumer perception approach.
As AI is becoming widespread, it is important to address questions, such as how an algorithm is interpreted by users, and how users understand algorithm-based systems (Shin, 2020). Starke, C., Baleis, J., Keller, B., & Marcinkowski, F. (2022), provided a systematic literature review of empirical studies on the perception of algorithmic fairness, the study included 58 empirical papers consisting of various disciplines and domains. Since, algorithmic decision-making, has some drawbacks: unfair algorithmic systems can systematically strengthen societal biases, causing marginalization of minorities, and without any restriction could harm certain society members (Veale and Binns, 2017; Žliobaitė, 2017). It is important because algorithms play an important role in various domains, such as interaction with business, education, government, and entertainment. Therefore, people are viewing the outside world from the algorithm’s lens.
(i) FATE ( Fairness, Accuracy Transparency, and explanation) – Transparency - Previous research studies were highly focused on the techno-centric approach, but now the focus has shifted to the user-centric design of algorithms. Public concern about bias in algorithmic systems discriminating based on race, gender, or other characteristics, also led to a call for transparency in problematic algorithmic systems (Goodman & Flaxman, 2017; O’neil, 2016; Weld & Bansal, 2019). For instance, a study by Springer & Whittaker (2020), utilized an empirical method for understanding users’ reactions toward transparent systems. Results tell that, initially user anticipated a transparent system to be better but changed their beliefs after experiencing a transparent system. Chen, Mislove & Wilson (2016), developed a methodology to detect algorithmic pricing, and empirically test its prevalence in the Amazon marketplace. The study explored the characteristics of algorithmic sellers and their strategies. The study aimed to increase transparency in such practices. Diakopoulos & Koliska (2017), conducted empirical research on algorithmic transparency in news media. Findings suggested the challenge of the human role in the adoption of algorithmic transparency, such as lack of incentive to organization and abundance of information to users. According to Lee et al. (2019), transparency and control over outcomes positively influence the fairness perception of the algorithm, as it includes humans in final decision-making. According to Datta et al. (2016), reporting transparency could have the potential for privacy breaches, and hence study explored the transparency-privacy trade-off. The result proved that transparency reports with the addition of little noise could be made private. Bujold, Parent-Rocheleau & Gaudet (2022), suggested that transparency of algorithmic surveillance positively affects procedural justice; and algorithmic performance management positively affects distributive justice; and procedural and distributive justice negatively influence the turnover rate.
FAT - Although algorithms have exhibited the potential to provide improved services to consumers, issues such as fairness, transparency, and accuracy (FAT) are intertwined with the algorithm’s operation (Shin & Park, 2019). In this domain, several issues remain unsolved and problematic, for instance, whether the algorithm is fair or biased, the issue of assigning accountability in case of harmful outcome from the algorithm, and the issue of justifying goals, actions, and operations by the algorithm (Castelvecchi, 2018; Shin, 2019). Abdu et al. (2023), conducted content analysis of published articles on FAT, to identify in algorithmic fairness literature how race is being formalized and conceptualized. Results showed in algorithmic fairness literature racial categories are applied inconsistently and little explanation is provided for it. And asked the algorithmic fairness community to re-examine the racial classification to align the field’s intervention with its values. According to Solyst et al. (2023), youths have a better capability to identify and articulate algorithmic bias. Young people, who have less awareness of technology and societal structure, can work with adults having better knowledge can lead to the development of fair and responsible AI. These studies will help in designing fair, transparent, and accountable AI, multidisciplinary teams will be required for implementing such a system (Turchi et al., 2024).
Explainable AI- Shin (2021) explored the explainable AI approach, in this study, ‘causability’ was conceptualized as an antecedent of explainability. The result suggested that including explanatory cues and cuasability, will increase users’ trust in AI, as it brings transparency and accountability to AI. The study by John-Mathews (2021) talks about concerns about post-hoc explanations of black-box AI, a trend in explainable AI. Findings reported that post-hoc explanations have the tendency to provide biased information by algorithms’ mechanisms or manipulate users to divert their attention.
(ii) Algorithmic experience (AX): Due to the importance of algorithmic experience (AX), various empirical studies have been conducted to identify the process through which users create the perception of the algorithm. These studies provided guidelines for developers, to develop fair and responsible AI systems (Turchi et al., 2024).
(iii) Trust: The study by Shin et al. (2020), proposed an Algorithmic acceptance model as an analytic framework for human-algorithm interaction. As per result algorithmic experience is influenced by the user’s understanding of fairness, transparency, and accuracy (FAT) and other experiences, which in turn are related to trust in the algorithm. According to the OECD (2019) and the European Commission (2019), trustworthy AI consists of four principles, among which fairness is one. However, it requires technical solutions to understand the societal implication of unfair algorithmic decisions (Barabas et al., 2020; Sloane and Moss, 2019), however various studies addressed when and why citizens perceive algorithmic decisions to be unfair.
Consumer fairness perception is necessary for implementing human-centric AI, as it informs developers to include ethical concerns for algorithmic systems to be implemented in the societal context (Kieslich et al., 2022). This could help to achieve a society-in-loop approach while designing an Algorithmic system (Rehwan, 2018). There is a need for more research work from theoretical as well as methodological stand from the context of non-Western countries, to develop a harmonized view on conceptualization and measurement of the algorithm’s fairness perception.
(iii) Evaluation of Algorithmic bias literature from the behavioural science aspect (Behavioural Algorithmic bias)-
The third domain of empirical studies in algorithmic bias is which explores algorithmic bias from a behavioural science aspect, because of the human-in-loop approach in AI implementation (Schwartz et al., 2022). Therefore, AI decision-making can be affected by human biases, these biases are related to how a user perceives AI output while making the decision. AI is susceptible to these biases across the AI lifecycle once the AI application is deployed. There are various human biases, as they are a fundamental part of humans, the field around these biases is behavioural economics (Slovic & Tversky,1982; Schwartz et al., 2022).
(i) Automation Bias – Automation bias occurs when users perceive the output of algorithms as being objective or factual. The user posits that the computer offers statistical computing that is devoid of bias or subjective influence. For instance, few people blindly trust algorithmic output (automation bias), whereas skeptics interrogate algorithms to unveil some form of biases in algorithms. There is inconsistency in the literature on automation bias. For instance, a study by Fritsch et al. (2022), using experimental method, showed that radiologists reading mammograms, supported by an AI-based system have a susceptibility to exhibit automation bias. They also mentioned the need to consider the effect of other such human-machine interactions. Whereas study by Alon-Barkat & Busuioc (2023), suggested that in the public sector bureaucrats are not prone to automation bias, but if the algorithmic output is as per group stereotypes (selective adherence bias), it may lead them to accept the decision. The finding by Wright, Chen, Barnes & Hancock (2016), suggests that providing the reason for outcome improves performance and reduces automation bias; however, providing information that creates ambiguity increases complacency, which reduces performance and increases automation bias. According to Horowitz & Kahn (2023) the knowledge, familiarity, and experience of AI, if lower likely causes algorithmic aversion and at an average level likely causes automation bias. Only after getting highly exposed to AI individuals become more balanced on whether to rely on AI decision aid or not. Jones-Jang, S. M., & Park, Y. J. (2023), explained two important psychological processes of how the failure of AI is evaluated by users. One mechanism when individuals have high expectations from AI’s consistent performance causes ‘automation bias’; and then they get frustrated by poor performance leading to ‘algorithmic aversion’. Also, the study by Kim et al. (2023) mentioned that users' social identity can influence their perception of a biased algorithm, i.e., different social groups’ lived experiences of discrimination influence their inference of biased algorithms. Schemmer, Kühl, Benz & Satzger (2022), stated that the impact of explanation on automation bias is dependent on the domain and kind of explanation. Explainable AI may increase dependency on AI, but if there is a confusing explanation, it either increases automation bias or leads to algorithmic aversion. Vered, Livni, Howe, Miller & Sonenberg (2023) conducted a study to design an explanation for an automated system. The study suggested that when automation bias is low no explanation is required to be offered, whereas when high performance is required and the willingness to accept automation bias, then there will be great use of explanation. According to Kupfer, Prassl, Fleiß, Malin, Thalmann & Kubicek (2023), individuals who received information of system error tend to have low automation bias, and individuals informed of their responsibility tend to have high automation bias. Also, there are other empirical behavioural studies in the algorithmic bias domain.
(ii) Interpretation Bias – Interpretation bias can be regarded as a mechanism via which users might introduce bias into an ostensibly neutral outcome. Bias occurs when individuals interpret an unclear output based on their personal internalized biases. For instance, a study by Lopez & Garza (2023) revealed that when users receive negative evaluation, they report high evaluation fairness for the human evaluator v/s AI evaluator, on the other hand, if users receive positive evaluation, they report a statistically insignificant difference in evaluation fairness level of user v/s AI. There is a lack of studies in behavioural science about algorithmic systems.
(iii) Feedback loop Bias - A notable characteristic of computation-based systems is their ability to generate additional data through all user activities. The algorithm is acquiring knowledge through the analysis of user behaviours. The study by Agan et al. (2023), focused on social media algorithms, suggests that the algorithm also fails because many a time, the choices made by users, deviate from their actual preferences, as the choices made by users are based on some specific context (known as automatic behaviour). These biases creep into the system, which can lead to biased behaviour by the algorithm which the user did not intend to, therefore automatic behaviour by the user leads to algorithmic bias. Therefore, there is a need to explore the behavioural aspect of AI from the user perspective since AI is going to be an integral part of the working environment and daily life of individuals. This understanding will help organizations provide enhanced services and facilities to users and will help employees make better use of AI services.
The Table of literature of empirical studies is presented in Appendix A.
4.2 Need for more empirical studies in the future –
Now, this section examines the various facets and highlights the requirements of empirical studies:
Numerous definitions and approaches to fairness have been proposed and explored in the existing literature. However, it is important to note that the study in this field still is not complete. Algorithmic bias and fairness still have numerous research opportunities (Mehrabi et al., 2021). This section describes challenges in research of fairness and opportunities for under-studied research to conduct the study. Challenges: There exist a few unresolved challenges in fairness literature which is required to be addressed. These are: (1) Synthesizing a fairness definition. From the Machine Learning perspective, various definitions of fairness are being proposed in the literature. The definitions cover a broad spectrum of cases and hence exhibit some degree of disparity in fairness conceptualization. Therefore, it is very difficult to understand whether one fairness definition is applicable under different fairness conditions. Combining all fairness definitions into a single definition is still an open research problem, it will make algorithmic system evaluation more comparable and standardized. Having a standardized definition of fairness will help to deal with the issue of incompatibility with some of the current fairness definitions. (2) Need to move to equity from equality. Most of the fairness definitions present in the literature focus on equality, i.e., to ensure equal quantity allocation of resources among all groups or individuals. Little focus is being placed on equity, which means the allocation of resources to groups or individuals for them to succeed (Gooden, 2015). Therefore, operationalization of the equity definition and studying whether it enhances or contradicts the existing fairness definition can be an attractive research direction. (3) Identifying unfairness. Provided fairness definition, it should be possible to detect the presence of biases or unfairness in the dataset. Efforts have been made to address this issue of data bias, by identifying instances of Simpson's Paradox in arbitrary datasets (Alipourfard et al., 2018), still, more attention is required to be put on issues of unfairness because of the presence of multiple definitions and the absence of a process to detect them (Mehrabi et al., 2021).
Apart from the definitional inconsistency of fairness in literature, there is a requirement to conduct an empirical study on users’ perceptions of the algorithmic fairness domain. For instance, developing a framework of algorithmic fairness by including four primary fairness dimensions: distributive, procedural, interactional, and informational fairness, enhances understanding of human-computer interaction (Starke et al., 2022). Also, from the perspective of AI-employee integration in organizations also opens the avenue of empirical studies, such as, (i) Cognitive issues - How do decision-makers trust output provided by AI systems? What controls are required when AI provides abnormal results and needs human intervention? (ii) Relational issues - How can employees build trust with an AI system/robot? And (iii) Structural Issues - How can we reskill workers to work successfully with AI systems? What type of technological and relational training is needed for nontechnical employees working with AI systems? (Makarius et al., 2020). From the perspective of automation bias, there is a need for a deep understanding of the impact of explanation on automation bias. For instance, conducting mediation analysis and structural equation modeling to investigate the mediation effect. Such understanding will help to develop better human-AI collaboration (Schemmer et al., 2022).
Various literature reviews and conceptual studies have mentioned multiple research gap in the field of algorithmic bias. The study by Kordzadeh & Ghasemaghaei (2022) made recommendations for future information systems research they were – first, to examine the mechanism which influences user’s behaviour towards algorithmic bias when users experience algorithmic bias. Second, there is a need to understand how and within which circumstances algorithmic bias influences user behaviour towards outcomes provided by the algorithm, for instance, individual characteristics, task characteristics, technology characteristics, organizational characteristics, and environmental characteristics. These propositions majorly focus on the user’s behaviour and can help determine the fairness perception of the system developer. Whereas Puntoni, et al., (2021) acknowledged that AI technology is being embedded in various products which could impact consumers’ experience, hence the following question arises- Does difference in users’ awareness of biased algorithms make users feel misunderstood by AI? How does wrong social classification by AI output affect the choice and behaviour of consumers? How does the user decide which variable is used by AI for providing personalized recommendations? Which kind of classification leads consumers to feel misunderstood? Does the nature of the task impact the likeliness of feeling misunderstood? Also, Lopez & Garza (2023) mentioned that researchers can work to identify whether a negative judgment for “complex” tasks will influence users’ perception of fairness for human’s V/s AI.
Starke et al. (2022) mentioned the lack of studies from non-Western contexts, so researchers must focus on the impact of algorithmic bias among users from various countries. Also, there is a need for more theoretical and methodological groundwork to develop synchronized concepts and measurements of algorithmic fairness. Also, researchers should focus on interdisciplinary research to provide empirical evidence to - developers for the proper designing of algorithms; and to decision-makers for implementing algorithms following a society-in-loop framework. There is a requirement to integrate these three aspects of the study, i.e., internal algorithmic characteristics, which consists of technical solutions and goals defined by organizations for AI development; user AI fairness perception, which discusses the four aspects of AI design fairness, accountability, transparency and explainability (FATE); and human behavioural science which describe the human biases while interacting with smart systems. Combining these three aspects will provide the framework for designing an algorithm, as shown in Figure 2.