Insights into suggested Responsible AI (RAI) practices in real-world settings: A systematic literature review

doi:10.21203/rs.3.rs-4707426/v1

AI-enabled systems have significant societal benefits, but only if they are developed, deployed, and used responsibly. We systematically review 45 empirical studies in real-world settings to identify suggested Responsible AI (RAI)practices to ensure that AI-enabled systems uphold stakeholders' legitimate interests and fundamental rights.

Our findings highlight eleven areas of suggested RAI practices: harm prevention, accountability, fairness and equity, explainability, AI literacy, privacy and security, human-AI calibration, interdisciplinary stakeholder involvement, value creation, RAI governance, and AI deployment effects. Our findings also show that there are more discussions about how RAI is supposed to be practiced than existing RAI practices. Ad hoc implementation of RAI practices in real-world settings is concerning because almost 80% of the AI-enabled systems reported in the 45 included articles are applied in use cases that can be categorised as high-risk settings, and over half are reported in the deployment phase. Our findings also highlight the crucial role of stakeholders in ensuring RAI. Identifying stakeholders into user, non-user, and primary stakeholders can thus help understand the dynamics of the settings where AI-enabled systems are (to be) deployed and guide the implementation of RAI practices.

In conclusion, although there is a consensus that RAI practices are a necessity, their implementation in real-world is still in its early day. The involvement of all relevant stakeholders is irreplaceable in driving and shaping RAI practices. There is a need for more comprehensive and inclusive RAI research to advance RAI practices in real-world settings.

Artificial Intelligence and Machine Learning

Psychology

Human-computer interaction

ethical AI

AI governance

socio-technical

trustworthy AI

“When we think about [AI], we need to put human dignity, human well-being—human jobs—in the centre of consideration.” – Dr. Fei Fei Li [1]

Software systems enabled by artificial intelligence (AI-enabled systems)[1] are increasingly used for various purposes in real-world settings [2-4]. For example, AI-enabled systems are deciding who gets hired [5], fired, or promoted [6], who gets a loan or life insurance [7] and who receives which medical treatment [8]. AI-enabled systems are also driving cars [9], delivering care to elderly patients [10, 11], diagnosing health symptoms [12], cleaning airports [13], and managing a city’s waste [14].

Yet, AI-enabled systems are still “emerging technologies” that, despite their potential for society, also come with the significant risk of negative consequences [4]. One particularly severe example is the childcare benefits scandal (“toeslagenaffaire”) in the Netherlands, where a self-learning algorithm used by Dutch tax authorities wrongly accused thousands of families of fraud, disproportionately targeting ethnic minorities, and resulted in devastating financial problems for those families [15]. It is because of cases like this that establishing normative principles for “responsible AI” (RAI) development, deployment, and governance – with the objective to prevent or minimise the risk of negative consequences – is seen as a global priority [16]. Attesting to this, Shneiderman [17] reported that there are more than 400 policy documents on RAI. Numerous initiatives have also been aimed at defining the principles that comprise RAI [18, 19].

Yet, RAI principles, while important to define, do not guarantee RAI in practice (Mittelstadt, 2019). One reason for the principle-practice gap is that several competing frameworks of RAI principles have been developed which are not all in complete alignment [20]. Moreover, even if there is convergence around the key principles of RAI, there could still be considerable differences in how practitioners interpret them and implement them in practice at both organisational and individual levels [20].

Our review thus focuses on investigating RAI practices in real-world settings which present unique challenges and implications that may not be apparent in theoretical discussions. Here, RAI practices are defined as a set of practices to ensure that AI-enabled systems are developed, designed, deployed, used, and managed in a way that upholds the legitimate interests and fundamental rights of the stakeholders in the setting where the AI-enabled systems are (to be) deployed [21]. To our knowledge, there is still little research investigating what RAI practices are suggested within real-world settings. The overall objective of this paper is thus to review the empirical studies of real-world AI-enabled system applications to identify the suggested RAI practices expressed, attempted, or implemented by stakeholders in these settings. We will identify and discuss gaps and areas for further improvement. Accordingly, our main research question is:

RQ: What RAI practices are suggested in real-world settings?

Specifically, we will investigate:

What RAI practices do stakeholders of AI-enabled systems in real-world applications suggest?
What examples exist in the literature that illustrate attempts to implement RAI practices in real-world settings and what can be learned from them?
What are the gaps and areas for improvement for RAI research and practices?

Our findings can inform researchers, policymakers, industry leaders, and practitioners how to set the right priorities in navigating the implementation of RAI practices in real-world settings. Importantly, findings can be used to build the groundwork for RAI best practices for real-world settings.

[1]Any system that contains or relies on one or more AI components. AI components are a distinct unit of software that performs a specific function or task within an AI-enabled system and consists of a set of AI models, data, and algorithms, which, through implementation, creates an AI component

2.1 Literature search and selection process

We used a systematic literature review (SLR) to search and select articles relevant to addressing our research question. This methodology minimises bias in article selection and produces reliable results [22]. The SLR method focuses on delivering a comprehensive overview of available state of the art evidence based on existing literature [23]. Our SLR followed the Preferred Reporting Item for Systematic Reviews and Meta-Analysis (PRISMA) standards [24].

2.1.1 Search strategy and inclusion criteria

The search terms/strings utilised in the literature search and justification for their inclusion can be found in Table 1 (see Appendix 1 for search syntaxes). We identified relevant articles from ACM digital library [25], Scopus [26], and PubMed [27] databases. We included PubMed database because our pilot research showed that much of the work in RAI was originated, applied, or linked to healthcare. The inclusion criteria used to select relevant articles and their justification can be found in Table 2. We only included articles that met all inclusion criteria.

2.1.2 The selection process of the included articles

The PRISMA flowchart shown in Figure 1 outlines the process for selecting the articles we included in the review [24]. Three reviewers (JKK, IBJ, TAB) independently and manually screened the titles and abstracts to reduce biases. Four reviewers (JKK, IBJ, TAB, AB) conducted manual full-text screening on 382 articles against the inclusion criteria in Table 2 prior to one reviewer (IBJ) and two authors (TAB, AB) performing the final extraction that led to the 45 included articles. Throughout the selection process, regular meetings were held between the reviewers and authors to discuss articles. Group consensus was used to resolve any disagreements.

2.2 Data extraction

2.2.1 Categorisation of the suggested RAI practices

We utilised content analysis with inductive coding when examining the suggested RAI practices described in the included articles. In contrast to deductive coding, which starts with a theoretical framework and codes against this framework, inductive coding starts by observing and exploring a topic in question, in this case suggested RAI practices, to look for similarities and patterns. The content analysis involved generating initial codes for each article and then collating codes with similar content into themes. This process was iterated until all similar content was grouped together. The themes assigned were inspired by themes used in the included articles themselves and discussions among reviewers and authors.

Importantly, when reviewing, we discovered that the included articles did not consistently or clearly distinguish between actual, perceived, planned, or suggested RAI practices. To address this, we chose to treat all identified themes under a term, "suggested RAI practices." This approach allows us to present a comprehensive view of the current discussions on RAI without imposing distinctions that were not clear in the primary sources, while also aligning with our research question.

2.2.2 Extracted information

When reviewing, we learned that both users and non-users could be stakeholders of suggested RAI practices. Users directly interacted with the AI-enabled systems. Non-users did not directly interact with the AI-enabled system but were directly involved in the development, deployment, use, or management of the AI-enabled system. In addition, primary stakeholders, could be users or non-users, were the direct recipients of the benefits and risks of AI-enabled systems. We believed that this learning was valuable and a critical part of RAI practices, thus, as a result, we identified and categorised the stakeholders involved in each included article into these roles: user, non-user, and primary stakeholders.

In summary, we extracted the following information from the 45 included articles systematically: title and citation, author(s), publication year, geographical location of data collection or study, study goal, setting (e.g., an industry, domain, or field), where in the AI lifecycle phase, methodology, subjective or objective measures used, the roles of AI-enabled systems, documented suggested RAI practices, stakeholder roles (primary, users and non-users), and the level of criticality associated with the use of proof of concepts (POCs) or AI-enabled systems. Content analysis was used for data extraction. Common themes were clustered together for each category. Group consensus was used to resolve any disagreements.

3.1 Summary of the 45 included articles

The majority of the 45 included articles (88.89%) were published after the year 2020 (Fig. 2). When disclosed, the articles collected data most commonly in the USA (24.44%) and multiple countries (22.22%) (Fig. 3). Healthcare was the most common real-world setting where AI-enabled systems were used (46.67%) (Fig. 4). Qualitative methods (e.g. interviews, focus groups, observation) using subjective measures [28], were the most common study methodology and measures (75.56%) (Table 3). Finally, almost 80% of the included articles aimed to analyse stakeholder perspectives on AI-enabled systems and RAI practices in specific real-world use cases (Table 4).

Of the 45 included articles, 17 involved both user and non-user stakeholders in their studies, 10 involved only user stakeholders, and 14 involved only non-user stakeholders (Appendix 2). We identified 17 articles that involved users who were also primary stakeholders and one that involved a non-user who was also the primary stakeholder (Table 5 and Appendix 2).

We categorised the AI-enabled systems or POCs used or described in the 45 included articles into phases of AI lifecycle, roles, and the level of risk associated with use. Deployment of AI-enabled systems in a setting was the most common phase of AI lifecycle (51.11%) (Fig. 5). Almost half of the included articles (42.22%) used AI-enabled systems as AI-assisted decision-making systems (Table 6 and Appendix 3). Following the categorisation of levels of risk by the EU AI ACT, we categorised the use cases where the AI-enabled systems or POCs identified in the 45 included articles were (to be) applied into unacceptable risk, high risk, limited risk, or minimal risk [29, 30]. Our findings show that most of the included articles (80%) applied AI-enabled systems or POCs for use cases in high-risk settings (Fig 6). High-risk settings were those AI-enabled systems being used in settings that could endanger citizens' lives and health, determine access to education and career opportunities, ensure product safety, affect employment and self-employment, determine access to essential services, interfere with fundamental rights, manage migration and border control, and influence justice and democratic processes. See Appendix 3 for the description of the real-world applications where AI-enabled systems or POCs were used in the included articles.

3.2 The Suggested RAI practices

We categorised the suggested RAI practices into 11 themes: harm prevention, accountability, fairness and equity, explainability, AI literacy, privacy and security, human-AI calibration, interdisciplinary stakeholder involvement, value creation, RAI governance, and AI deployment effects. See Table 7 for theme descriptions and Appendix 4 for included articles and associated themes.

3.2.1 Harm Prevention

Conducting an initial evaluation. Harm prevention practices were suggested to begin by evaluating whether the use of AI-enabled systems for a specific use case or setting was useful, appropriate, and unlikely to lead to any obvious potential harm [31]. If the evaluation went poorly, then the use of AI-enabled systems was recommended against.

For example, in one included article, the value in integrating an emotion AI-enabled system into hiring services was analysed. It was found that the system unfairly obstructed job applicants from effectively using their emotional capital in the recruitment process, and exploited them in a manner that benefited the hiring organisations but disadvantaged the job applicants [32]. The AI-enabled system was determined to be biased against candidates’ facial expressions along racial, gender, and disability lines because its facial emotion recognition capabilities performed poorly on such candidates. Such evaluation led to the conclusion to discourage use of the AI-enabled system in that setting as it would lead to potential harm to certain stakeholders.

On the other hand, if the initial evaluation of the AI-enabled system went well, then ensuring the accuracy and safety of the system was suggested as the next step. The articles included in our review described several practices for accomplishing this, as described in the next section.

Ensuring accuracy and safety of AI-enabled systems. Engaging potential real-world stakeholders in the development and testing phases was mentioned to help ensure that the AI-enabled systems performed as intended in actual settings [11, 33, 34]. For example, one article found that an AI-enabled floor cleaning robot at an airport caused slips and falls and nearly hit a visitor [14], an event that was not predicted during testing prior to deployment. Specifically, involving primary stakeholders, such as patients, in evaluating AI outputs were mentioned in multiple articles to enhance the system's accuracy and reliability [34, 35].

In addition, comparing and testing similar AI-enabled systems or models in different settings was suggested as another important part of assessing their accuracy and safety. This process was suggested to reveal the AI’s strengths and weaknesses and helped in the selection of the most suitable AI-enabled systems for specific use cases [34]. This process was also suggested to be complementary to efforts to find independent evidence of AI-enabled system claims for ensuring their accuracy and safety, instead of relying solely on information from AI vendors and suppliers [34].

Continuous monitoring and adjustment were identified as a measure to ensure accuracy and safety in the rapidly changing and unpredictable settings where AI-enabled systems were used [34, 36]. Various methods were highlighted, including:

o Tracking trends and detecting anomalies [7].

o Conducting retroactive pilots before deploying new versions [7].

o Continuously calibrating and adjusting the AI-enabled systems to adapt to changes in the physical environment [13].

o Monitoring and evaluating user feedback to prevent skewed outputs resulting from inadequate user feedback [5].

Collecting real-world data. Ensuring high-quality and high-integrity data for training AI-enabled systems and models was identified as crucial for preventing harm and fostering stakeholder trust [37, 38]. Additionally, balancing the collection and processing of high-quality data with privacy preservation was reported to often mean that raw data was rarely provided. Therefore, recognising that the data used to train AI-enabled systems might be heavily processed and anonymised was highlighted as key to maintaining its quality and relevance [39]. The included articles also concluded that effort to collect real-world data to focus on relevance and targeted data only to ensure high-quality data processing, leaving out irrelevant data that was reported to often be collected as well [39]. In cases where only synthetic data was available or ethical, aligning and validating this data with real-world data was mentioned as necessary to ensure its quality and integrity [36].

3.2.2 Fairness and equity

Incorporating user feedback to improve fairness. Our review identified that RAI practices incorporated from an early phase of development to ensure the fairness of AI models was crucial. For example, one included article suggested that using a tool called “model cards” looking at both ethical and unethical reflections and actions was proven to be useful for encouraging developers of AI-enabled systems to take a more critical approach and incorporating fairness elements into the development [40].

For AI-enabled systems already deployed, it was suggested for fairness practices to involve incorporating user feedback, for example, using disparate impact as a fairness metric [41, 42]. This approach was done by allowing users to change predicted labels and train new AI models based on the adjustment or change causal relationships between AI model attributes. However, it was suggested that incorporating user feedback required mechanisms to quality assure the feedback, and ensure that feedback that could make the model even less fair was not incorporated [41, 43]. Several mechanisms to quality assure user feedback were proposed, including:

· Quantifying the discrepancy in the quality of user feedback [43].

· Focusing on fairness criteria that users found important including affordability, equality of opportunity, and individual fairness [41].

· Including feedback from non-WEIRD (Western, Educated, Industrialised, Rich, Democratic) users [41].

· Given the variability of fairness perceptions across geographic locations and local cultures, it was also highlighted to approach the expansion or restriction of user populations thoughtfully [41].

The continuous evaluation of fairness was thus recommended to adapt to changing stakeholder expectations and dynamic real-world conditions [40, 44].

Testing for bias and discrimination. Other articles referred to testing for biases [41, 45, 46] and to determine when an AI model could be considered as fair [45]. Evaluating AI-enabled systems for biases, discrimination, and inconsistencies against certain groups or sub-cohorts was indicated as a necessity not only to take appropriate mitigation actions, but also because the likelihood of biases and inconsistencies was apparent [43, 45]. For example, an included article found that the reliability of explanations produced by an AI-enabled system was inconsistent across hospitals and sub cohorts, and this was likely to produce explanation biases against certain surgical groups [43].

Importantly, an analysis of AI-enabled systems regarding their biases and discrimination was suggested to be executed at an individual level, not on an aggregated level or on overall performance, by comparing individuals to a baseline in the same cohort and socio-demographic group [7]. Such action was suggested to increase transparency for stakeholders as well. This was highlighted because variability in stakeholder priorities, datasets, and use cases was found to influence actions to mitigate biases and discrimination [44].

Ensuring the equity of benefits of use of AI-enabled systems. It was reported that in some cases, benefits of use of AI-enabled systems were not received equitably across target groups due to lack of infrastructure such as poor access internet and limited socio-demographic data of target stakeholders [45, 47]. The included articles concluded that it was crucial to consider the real-world setting and local conditions where AI-enabled systems were to be used and build necessary infrastructure prior to deployment to ensure equity [45].

3.2.3 Accountability

There was a consensus in the included articles that RAI practices needed to be accompanied by accountable actions. The general perception was that only humans or organisations could be made accountable for the responsible management of AI-enabled systems and its outcomes [9]. Assigning accountability was highlighted to rely on scalable and systemic organisational structures and processes, rather than to rely on individuals to avoid finger pointing discussions [48]. Our review identified that accountability was suggested to be assigned to all stakeholders but for different areas. For example, AI vendors were accountable for ensuring accurate AI models, and users to ensure proper intended use [49]. In cases where the users were not the ones affected by AI output but the ones making decisions, users would be accountable for using the AI output responsibly and for communicating to the primary stakeholders about the AI output and how the AI output affected their decision making [35].

3.2.4 Explainability

[8]Disclosing relevant information. In general, the included articles suggested to include the following information to facilitate explainable AI and foster trust in AI-enabled systems:

· Data sources, prediction accuracy, how the data sources (input) were used to train AI models in which context or use case [35].

· Calculations or processes behind AI output, risk factors, and AI limitations [7, 12].

· Demonstration of how responsible the process of developing AI-enabled systems such as considerations behind design choices and risks [40].

· Sharing the perspective of what the AI-enabled systems were “seeing” [50].

Making explanations adaptable. The included articles suggested that making AI explanations adaptable to individual user qualifications and profiles made them more user centric. Several approaches on what made AI explanations adaptable emerged:

· Adapting how much explanation was provided and in which styles and formats to suit different user groups [51].

· Selecting which explanation methods (e.g., decision tree approximation, local rules or trees) based on what was most appropriate for which user groups [52].

· Making AI explanations more interactive or conversational by allowing users to ask questions and request more information, rather than providing a one-off information [52].

On the other hand, aligning AI explanations to users’ explanations was discouraged in another article [43]. This was because doing so could result in missing opportunities of learning new explanation methods or perspectives from what users already knew. Instead, this included article suggested to consider other indicators of reliability of explanations, such as alignment with contextual factors in the task domain and leveraging AI-enabled systems to automatically decode relevant information, without relying solely on human input [43].

3.2.5 AI literacy

Ensuring awareness. Our review identified practices to ensure stakeholder awareness about the use and potential impact of AI-enabled systems [52]. Practices aimed at ensuring awareness were mentioned as particularly significant when stakeholders were consenting to use AI-enabled systems owned by private sectors, as it helped them to understand their rights and how their data was processed [47]. Involving stakeholders was essential to the effectiveness of these efforts, particularly for users or other stakeholders who might not be able to give consent, such as those with dementia, developmental delays, cognitive impairments, or limited understanding [36, 52].

Educating and training. Our review highlighted the importance of providing AI education and training to stakeholders, such that they would know how (and how not) to use AI-enabled systems and better understand and be able to interpret AI output and recommendations [50]. AI education and training was also associated with correcting improper use of AI-enable systems and preventing undesired outcomes. For example, when a self-diagnosis health chatbot was used by users for nontherapeutic purposes, deviating from the chatbots intended purpose, this usage generated a large amount of noisy training data and affecting the chatbots’ performance negatively [12]. Several considerations related to AI education and training were identified in our review:

· First, it was important to provide education and training to all relevant stakeholders, not only users. Tailored education and training for each stakeholder group was suggested as more effective compared to that of general education and training. Moreover, it was found that educating stakeholders through practical training and direct user-AI interaction was more impactful than theoretical training [48].

· With regards to training content, it was suggested to focus the education and training for these stakeholders on both understanding and identifying potential errors in AI outputs or recommendations [8, 35, 40, 52]. Understanding AI output and recommendations was reported to be particularly important in a setting such as healthcare where clinicians needed to explain AI outputs and recommendations to themselves, their patients, peers, and communities [35].

· [48]Furthermore, RAI-specific training was identified as useful for enabling stakeholders to advocate for RAI practices within their settings [48].

Critically evaluating companies’ claims. The included articles emphasised the importance of enabling stakeholders to understand and critically evaluate company claims about AI-enabled systems [32, 53]. Effective AI education and training were found to help stakeholders in critically assessing these claims and finding supporting evidence. Several focus areas emerged from our review, including:

· Evaluating how companies’ profit-driven nature influenced their fairness promise [47].

· Evaluating how a company’s transparency promise was balanced with the need to protect IP and identifiable information [47, 54].

· Carefully scrutinising the claims that the use of AI-enabled systems eliminated human biases [32].

· Carefully scrutinising the claim that AI-enabled systems were accurate and objective when performing health-related diagnoses to humans [53].

· Finding information on and understanding who made decisions about the development and design of AI-enabled systems. For example, one article found that clients involved in development often lacked AI-related knowledge but they were treated as key decision-makers who affected the end result of the AI-enabled systems and their claims [46].

3.2.6 Privacy and security

Several practices from the included articles were aimed at ensuring that AI-enabled systems respected individuals' privacy rights and securely processed sensitive data. These included:

· Using granular non-identifying data to train AI models; data that did not contain sufficient detail to re-identify individuals when cross-validation with other sources of data [39].

· Establishing an agreement among users on what data could be shared among which user groups [36], as was where to store sensitive and personal output and recommendations produced by AI-enabled systems [8].

· Establishing practices to ensure compliance with GDPR [55], and enforcing informed consent to process and share user data and explicitly inform users that users could ask for their data to be deleted at any time [36, 47].

· [39]Moreover, when video data was used, one included article demonstrated that blurring faces could be done without reducing data utility [56].

3.2.7 Human-AI calibration

Aligning expectations. Included articles identified that a mismatch between stakeholder expectations and AI-enabled system capabilities happened when users projected over-optimism (i.e., overreliance risks) towards AI-enabled systems, and when AI-enabled systems were perceived as merely a tool to achieve a particular task rather than as a complex transformative technology [35]. Several efforts were reported to be effective to align expectations:

· At an organisational level, these included shaping organisational culture and norms [13, 57] and organisational mechanisms ensuring AI-enabled systems’ accuracy and safety to minimise consequences of overreliance [8].

· At an individual level, this included allowing users to have a direct first-hand experience with AI-enabled systems in real-world settings [9], and allowing users decide on how much they would like to use AI-enabled systems to prevent overreliance [47].

It was also reported that when there was an agreement of conclusions between AI-enabled systems and domain experts, AI output and recommendations were used to validate domain experts’ conclusions, and vice versa. However, sometimes conclusions made by AI-enabled systems and domain experts were not aligned. Several ways to deal with such cases were brought up, including:

· Prioritising human conclusions over AI-enabled system outputs was not encouraged due to human expectations and biases [54].

· Conducting a validity assessment on the AI output and recommendations [42].

· Allowing users to directly interact with AI-enabled systems to construct comprehensive mental models about the entire AI pipeline to help them understand the output and recommendations [35].

· Supporting users by calibrating their confidence in a hypothesis based on available evidence [58].

· Comparing AI output and recommendations with not only the domain experts’ and users’ knowledge and experience, but also other similar AI-enabled systems, and peers [37].

Requiring human oversight. In our review, we found that human oversight of AI-enabled systems was required by stakeholders [35, 40]. Ways to implement human oversight included:

· Incorporating user feedback into AI training data [42].

· Increasing transparency and explainability on how the output and recommendations were produced [7].

· Standardising how to conduct a human oversight to reduce variability of human oversight practices [9].

· [50]Empowering stakeholders with greater rights to monitor and intervene in use of AI-enabled systems if necessary [47], and by giving more explanations and controls to users [59].

Considering stakeholder backgrounds and characteristics. Several articles indicated that human-AI calibration needed to consider stakeholder background and characteristics. Particularly influential were factors such as personality traits, work experience, age, personal preferences, technological concerns, personal experiences, individual differences, and baseline trust levels to technology [45, 52, 54, 60].

Trust, mistrust (absence of trust), and distrust (the belief not to trust) in AI-enabled systems. Furthermore, fostering trust in AI-enabled systems was suggested to be coupled up with efforts to address and identify stakeholder mistrust and distrust. For example, stakeholder concerns about the accuracy of AI output and the implications of incorrect output were to be addressed first as part of trust building [8].

User interface (UI) and user experience (UX). UI was found to determine the UX and was reported to be important for enabling effective human-AI calibration. Here, we summarise key UI features emerged from the included articles:

· Giving stakeholders full or greater control over UI features such as which information to be put on display (e.g., the level of actionability, amount of information, language) [11, 12, 61].

· Considering user profiles for the design of size and shape of the UI [61], usability of user login [36], and interfaces concerning position of buttons, colours, alarm tones [36, 62].

· Accommodating minority user groups by incorporating settings of user preferences and personalised AI features into UI (e.g., additional features for elderly users to include bigger font size, and a speech interface) [55, 62].

· Non-dependency on stable wireless network connection to use AI-enabled systems [62].

· Conducting all tests of AI-enabled systems prior to deploying them to ensure user personalisation [12, 36].

3.2.8 Interdisciplinary stakeholder involvement

Involving stakeholders. The included articles consistently highlighted the importance of involving potential target users and other stakeholders in all phases and iterations of the AI-enabled systems’ lifecycle for successful and responsible AI-enabled system deployment. This included involving users and other stakeholders in:

· Ensuring fit-for-purpose AI-enabled systems [63].

· Specifying their concerns and needs [61]such that they could be incorporated into the system design requirements [55].

· Validating AI models by evaluating the accuracy of outputs [7].

· Identifying RAI issues during testing and ensuring their feedback on RAI factors were incorporated to correct AI models [11, 32, 36].

· Improve the reliability and fairness of AI explanations by a way of incorporating their feedback as training data [43].

· Ensuring ethical AI practices [14].

· Ensuring that AI-enabled systems' behaviours upheld human dignity and values during operation [11].

Identifying the correct users and stakeholders to include was reported as an important first step to improve the accuracy of these practices [45, 59]. The user identification process was also reported to identify reasons for empowering users to adopt AI-enabled systems [14, 57], and help stakeholders recognise their responsibilities in using the AI-enabled systems [40]. Several considerations for identifying user groups emerged in our review, including:

· Developing a strategy to ensure the identified user groups are appropriate [52].

· Developing a strategy to accurately identify user groups while maintaining confidentiality and privacy [49].

· Distinguishing between actual users and other key stakeholders, such as clients, in the development of AI-enabled systems [46].

· Developing a strategy to engage particularly marginalised groups with limited access to technology [45].

Encouraging interdisciplinary teamwork. Our review highlighted the importance of encouraging and inviting all stakeholders to shape AI-enabled systems [51]. The value of interdisciplinary teamwork and collaboration emerged in our review included:

Specific for the development phase

Interdisciplinary teamwork was suggested to bridge the gap between the design of AI-enabled systems and stakeholder understanding of ethical implications [36, 54, 59]. The interdisciplinary nature of such co-designing approach was proven to make the process of developing an AI-enabled system fair since AI experts, users, and other stakeholders with limited technical knowledge were all involved and well-informed in making each decision [36]. Interdisciplinary teamwork was found to ensure privacy of data used and legal compliance through open and transparent discussions between AI developers, product managers, clients, and legal team [54].

Specific for the deployment phase

Interdisciplinary teamwork between users, domain experts, and AI vendors was suggested to be valuable for fine-tuning AI-enabled systems to fit local contexts [42]. It was reported that continuous interdisciplinary teamwork and user training were key to ensure AI-enabled systems kept creating value for their stakeholders [14]. This was documented to be achieved through continuous user feedback and enabled stakeholder discussions and interactions [60].

From development to deployment phases

Involving interdisciplinary experts from technical, non-technical, industry, domain, user, design, and legal fields was consistently reported as a necessity for cross-learning in developing, deploying, and using AI-enabled systems in a responsible manner [9, 33]. Interdisciplinary collaboration was also reported as a necessity in addressing complex ethical, legal, and social challenges in the development and deployment of AI-enabled systems [37, 42, 51, 52, 60]. Interdisciplinary teamwork was reported to be fostered through requesting and incorporating stakeholder feedback to refine an AI-enabled system, as well as creating channels or mechanisms to enable interactions and teamwork between and among stakeholder groups vertically and horisontally [14, 48, 54, 58].

3.2.9 Value creation

Our review identified that initially deployed AI-enabled systems often aimed to create value by enhancing productivity or improving decision-making capabilities [7]. However, it was reported that these goals would likely evolve as new input and training data were integrated, reflecting real-world changes [45]. Thus, approaches were needed to ensure that AI-enabled systems continuously created value for their stakeholders. These included:

· Continuously measuring the impact of created value by assessing benefits against risks, managing risks effectively, and identifying key success factors in AI deployment and adoption [35, 47].

· Consistently incorporating RAI practices following a holistic approach throughout the AI lifecycle rather than on separate phases [40, 44, 46, 64].

· Continuously investing adequate resources in the deployment and adoption phase to prevent AI-enabled systems from being abandoned after development and to ensure the promised value was realised [13, 54].

· Considering and managing the expectations, aversions, perceptions, and trust levels of users and stakeholders that were identified as predictors of the perceived value of AI-enabled systems [42].

· Ensuring that the trade-off between the value created and undesired effects to certain stakeholder groups was minimised [42].

· Communicating all potential benefits and risks of the use of AI-enabled systems to all stakeholders [54].

3.2.10 RAI governance

Our review identified several approaches for establishing processes, structures, and policies to oversee and ensure that AI-enabled systems were developed, deployed, and used responsibly.

At an organisational level

· Seeking leadership commitment to set up or improve organisational support for RAI practices and practitioners or ethicists [64, 65].

· Enforcing organisational structure and channels, formal and process-driven measures, standards, and best practices favourable for fostering RAI practices [48].

· Fostering a favourable organisational culture for RAI practices [48].

· Implementing proactive RAI practices by encouraging individuals to be RAI champions [48], while supporting and establishing a clear responsibility and function for RAI or AI ethicists [65].

· Introducing and implementing RAI organisational policies such as enforcing the use of RAI checklist, providing incentives for exercising RAI practices and consequences for not exercising them [48, 49, 64, 65].

· Developing a strategy to measure the impact of RAI governance [47, 48, 50, 59].

· Developing a strategy for managing and addressing the adaptive nature of AI-enabled systems [50].

· Conducting internal or external RAI audits by [41, 64, 66, 67]:

o auditing AI model’s fairness and equity by evaluating output for different socio-demographic backgrounds and underrepresented sub-groups,

o investing in checking data validity,

o assessing data representativeness, and appropriateness of training data for intended purpose,

o assessing bias in input data,

o performing intersection/cross analyses, and

o identifying concrete fair and unfair instances.

At a national/governmental level

· Protecting stakeholders from AI-enabled system companies’ claims that were misleading, incorrect, and fraudulent [32].

· Implementing and guiding stakeholders through consumer product protection such as from defective AI-enabled systems [67].

· Operationalising RAI through legal standards [64].

· Developing a strategy or regulations for addressing post-market device surveillance through an algorithm change protocol to deal with adaptive algorithms and AI opacity [50].

· Supporting SMEs and start-ups in RAI governance through providing necessary resources for post-market surveillance of AI-enabled systems [50].

· Standardising and investing in RAI audit processes to minimise variability and increase audit accessibility [49].

· Supporting RAI audits through enabling interdisciplinary collaboration between auditors and stakeholders for successful audits [66].

3.2.11 AI Deployment effects

Our review found that discrepancies between the envisioned AI-enabled system and its real-world application were frequently observed. These discrepancies, often referred to as unanticipated or unforeseen consequences, were unintended effects caused by overlooked issues during the development phase [13]. Several methods for anticipating undesired AI deployment effects were reported in the articles reviewed. These included:

· Conducting a mapping analysis of all stakeholders and potential effects of deployment of AI-enabled systems to these stakeholders and communicating the analysis results to them [50].

· Anticipating stakeholders’ potential to re-skill and up-skill themselves to contend with the deployed AI-enabled systems [13, 14].

· Incorporating local contexts and norms of the settings where AI-enabled systems were to be deployed, starting in the development phase [51, 59].

· Tailoring AI-enabled systems to target stakeholder groups’ backgrounds, needs, and characteristics [13, 31].

· Co-designing and directly involving stakeholders in the development and deployment phases [36, 61].

· Implementing resources and organisational changes necessary to ensure adoption, such as making resources available to act upon AI output or recommendations [8]. [8].

· Addressing all stakeholder concerns, including potential legal liability [13, 14, 42].

· Securing investments, especially for deployment and post-deployment phases, rather than development phase alone [13].

· Ensuring that the value created by deployed AI-enabled systems outweighs the consequences for the stakeholders (e.g. added workload) [8].

“….we cannot use AI to sidestep the hard work of organizing society so that where you are born, the resources of your community, and the labels placed upon you are not the primary determinants of your destiny.” ― Joy Buolamwini

We have reviewed 45 relevant articles systematically and our findings illuminate eleven themes related to how RAI is suggested to be practiced in real-world settings (Table 7). There are several key findings from our review. First, our findings show that 60% of the included articles focused on analysing stakeholder perspectives on AI-enabled systems and only 17.78% focused on stakeholder perspectives on RAI practices (Table 4). In addition, when reviewing, we identified more discussions in the literature about how RAI is supposed or expected to be practiced rather than observations and discussions of actual, existing RAI practices. These findings show that implementation of RAI practices in real-world settings is likely still in its infancy and relatively inconsistent [68]. Accordingly, there is a need for more documentation of and learning from RAI practices in real-world. These findings confirm the motivation of our review, as well as call for more research focused on fostering and documenting RAI practices in real-world settings.

Second, the ad hoc nature of RAI practice implementation in real-world settings, as found in our review, is especially concerning because our findings also show that almost 80% of the AI-enabled systems or POCs reported in the 45 included articles are (to be) applied in use cases that can be categorised as high-risk settings (Fig. 6 and Appendix 3), according to the description provided in the EU AI ACT [16]. In addition, our finding shows that more than half of the AI-enabled systems or POCs reported in the 45 included articles are in the deployment phase (Fig. 5 and Appendix 3), involving not only technical considerations but also integration of AI-enabled systems into real-world workflows. These findings pinpoint that AI-enabled systems have the transformative potential in critical settings such as healthcare where the implications of use of these systems can have profound impacts, both positive and negative, on individual stakeholders and society. Importantly, the prevalence of AI-enabled systems in high-risk settings and in the deployment phase found in this review highlights further the critical need to implement RAI practices to ensure stakeholder fundamental rights and legitimate interests are upheld.

Third, our findings show that the themes interdisciplinary stakeholder involvement and human-AI calibration are central, being discussed in 28 and 24 of the included articles, respectively (Table 7). In addition to the finding that shows that almost 80% of included articles focused on analysing stakeholder perspectives on AI-enabled systems and RAI practices, almost all our included articles involved user, non-user stakeholders, or both in their studies (Table 5 and Appendix 2). These findings highlight significant research interest in understanding stakeholder perspectives on the suggested RAI practices and how RAI practices should be conducted in real-world settings, as well as highlighting the critical role stakeholders play in ensuring that AI-enabled systems are developed, deployed, and used in a responsible way. These findings are in line with previous studies that also pinpoint that stakeholder role is central to RAI and the fact that their needs and interests drive RAI practices [19, 48].

Fourth, when reviewing, we discovered that that stakeholders of suggested RAI practices can be users, non-users, and primary stakeholders, as mentioned in the method section as well. Our review demonstrates that it is possible to identify stakeholder roles across different real-world settings into user, non-user, and primary roles (Appendix 2). This simple role identification can reveal how AI-enabled systems may impact each role and how each role may, in turn, influence the development, deployment, and use of these systems. This understanding can help to illustrate the dynamics of the setting where AI-enabled systems are (to be) deployed and guide the implementation of RAI practices. For example, ICU clinicians using AI-enabled systems to forecast a risk score in critical instability in patients are user stakeholders who directly interact with the AI-enabled system [61]. The patients, however, are the primary, non-user stakeholders of the risk score because they are affected by, but don’t interact with, the AI-enabled system. In addition, the developers and designers of this AI-enabled system are non-user stakeholders who are responsible for its safety and accuracy but do not directly interact with the AI-enabled system themselves. Due to the nature of these roles, the perceptions of RAI practices are likely to be different among primary, user, and non-user stakeholders within settings where an AI-enabled system is (to be) deployed [69], and thus their different perceptions should be captured to provide a more complete picture of RAI. Future research, accordingly, should focus on how each stakeholder role contributes and is affected by the development, deployment, and use of AI-enabled systems, as well as assigning accountability to each role.

Fifth, our findings show that stakeholder background, characteristics, and preferences are acknowledged to influence suggested RAI practices, and lead to the need for the design and use of AI-enabled systems to be adaptable and adjustable. This finding suggests that tailoring RAI practices based on stakeholder profiles is needed to ensure that AI-enabled systems are inclusive and create value for their stakeholders. Future research should focus on exploring which suggested RAI practices are more relevant and effective for different stakeholder groups and roles across phases of the AI lifecycle to ensure a more comprehensive understanding.

Sixth, our findings show that the most used method and measure by the included articles is qualitative and subjective which depend on human responses to questions or human observations and interviews [28] (Table 3). These findings are likely to be related to the fact that almost 80% of included articles aimed at analysing stakeholder perspectives (Table 4). Future research should thus focus on exploring more objective measures and quantitative methods, or mixed methods approaches, to provide a more comprehensive understanding of RAI perspectives and balance subjective and objective data.

There are several limitations to our review. First, our review does not include grey literature, which may contain other relevant or more recent work on RAI. Similarly, there may be other RAI initiatives or practices that are not documented or published as articles and thus are not found in our search. Second, there may be geographical bias, as there are more studies on certain regions (e.g., the USA and European countries) and fewer on others, potentially overlooking other suggested RAI practices influenced by cultural elements. Third, we can only use the information provided in the included articles for conducting the analyses, which means misinterpretation can occur, for example, in interpreting the suggested RAI practices as well as the lifecycle phases, levels of risk, and roles of the reported AI-enabled systems of POCs. Fourth, our review does not document evidence of the impact of the suggested RAI practices. This is because none of the included articles focused on measuring the impact, there is very little documented or reported evidence of the suggested RAI practices, and most of the data is in the form of descriptions of real-world stakeholder or expert suggestions of how RAI is supposed or expected to be practiced, perceptions of the development or deployment process of AI-enabled systems, or evaluation of AI-enabled systems. Future work should thus focus on exploring methods to measure and find evidence of the impact of RAI practices in real-world settings.

Because of the transformative impact of AI-enabled systems in real-world settings, RAI has become an important topic in ensuring that AI-enabled systems are developed, deployed, and used in a responsible manner, upholding the fundamental rights and legitimate interests of their stakeholders. At the moment, although there is a consensus that RAI practices are a necessity wherever AI-enabled systems are (to be) used, implementation of such practices in real-world is still in its early day. This is a concern especially when AI-enabled systems are already being deployed in high-risk settings.

In pursuing RAI practices, identifying and involving relevant stakeholders, both users and non-users, in the development through to especially the deployment and post-deployment phases of AI-enabled systems is irreplaceable. The fact that user and non-user stakeholders are driving and shaping RAI practices pinpoints the importance of initiatives and efforts to foster more involvement and coordination across groups. Identifying stakeholder roles into user, non-user, and primary stakeholders can also help understand the dynamics of the settings where AI-enabled systems are (to be) deployed and guide the implementation of RAI practices. Accordingly, there is a need for more comprehensive and inclusive RAI research to advance RAI practices in real-world settings.

McKinsey and Company. Author Talks: Dr. Fei-Fei Li sees ‘worlds’ of possibilities in a multidisciplinary approach to AI. 2023 [cited 2024; Available from: https://www.mckinsey.com/featured-insights/mckinsey-on-books/author-talks-dr-fei-fei-li-sees-worlds-of-possibilities-in-a-multidisciplinary-approach-to-ai.
Chui, M., et al., The state of AI in 2022-and a half decade in review. https://www. mckinsey. com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2022-and-a-half-decade-in-review#/, 2022.
Jan, Z., et al., Artificial intelligence for industry 4.0: Systematic review of applications, challenges, and opportunities. Expert Systems with Applications, 2023. 216: p. 119456.
Maslej, N., et al., The AI Index 2024 Annual Report. April 2024, AI Index Steering Committee, Institute for Human-Centered AI, Stanford University,: Stanford, CA.
Li, L., et al. Algorithmic hiring in practice: Recruiter and HR Professional's perspectives on AI use in hiring. in Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 2021.
Houser, K.A., Can AI solve the diversity problem in the tech industry: Mitigating noise and bias in employment decision-making. Stan. Tech. L. Rev., 2019. 22: p. 290.
Maier, M., et al., Improving the accuracy and transparency of underwriting with AI to transform the life insurance industry. AI Magazine, 2020. 41(3): p. 78-93.
Petersson, L., et al., Ethical considerations in implementing AI for mortality prediction in the emergency department: Linking theory and practice. Digit Health, 2023. 9: p. 20552076231206588.
Chu, M., et al. Work with AI and Work for AI: Autonomous Vehicle Safety Drivers’ Lived Experiences. in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 2023.
Sinclair, A.J., A.J. Girling, and A.J. Bayer, Cognitive dysfunction in older subjects with diabetes mellitus: impact on diabetes self-management and use of care services. All Wales Research into Elderly (AWARE) Study. Diabetes Research and clinical practice, 2000. 50(3): p. 203.
Kim, J.-W., et al., A care robot with ethical sensing system for older adults at home. Sensors, 2022. 22(19): p. 7515.
Fan, X., et al., Utilization of Self-Diagnosis Health Chatbots in Real-World Settings: Case Study. J Med Internet Res, 2021. 23(1): p. e19928.
Fox, S.E., et al., Patchwork: the hidden, human labor of AI integration within essential work. Proceedings of the ACM on Human-Computer Interaction, 2023. 7(CSCW1): p. 1-20.
Kang, E.Y. and S.E. Fox. Stories from the Frontline: Recuperating Essential Worker Accounts of AI Integration. in Proceedings of the 2022 ACM Designing Interactive Systems Conference. 2022.
Amnesty International, Xenophobic machines: Discrimination through unregulated use of algorithms in the Dutch childcare benefits scandal. 2021.
European Commission. Shaping Europe’s digital future: AI Act. [cited 2024; Available from: https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai#:~:text=High%20risk,life%20(e.g.%20scoring%20of%20exams.
Shneiderman, B., Responsible AI: Bridging from ethics to practice. Communications of the ACM, 2021. 64(8): p. 32-35.
Mikalef, P., et al., Thinking responsibly about responsible AI and ‘the dark side’of AI. 2022, Taylor & Francis. p. 257-268.
Deshpande, A. and H. Sharp. Responsible ai systems: who are the stakeholders? in Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society. 2022.
Jobin, A., M. Ienca, and E. Vayena, The global landscape of AI ethics guidelines. Nature machine intelligence, 2019. 1(9): p. 389-399.
DNV. DNV-RP-0671 Assurance of AI-enabled systems. 2023 [cited 2024; Available from: https://standards.dnv.com/hearing/5BDF814775DB435C8AF32242F73FCB71/01.
About Cochrane Reviews. [cited 2023; Available from: https://www.cochranelibrary.com/about/about-cochrane-reviews.
Peričić;, T.P. and S. Tanveer. Why systematic reviews matter. 2019 [cited 2024; Available from: https://beta.elsevier.com/connect/why-systematic-reviews-matter?trial=true.
Moher, D., et al., Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. International journal of surgery, 2010. 8(5): p. 336-341.
ACM Digital Library. [cited 2024; Available from: https://dl.acm.org/.
Scopus. [cited 2024; Available from: https://www.scopus.com/home.uri.
Pubmed. Available from: https://pubmed.ncbi.nlm.nih.gov/.
Muckler, F.A. and S.A. Seven, Selecting performance measures:" Objective" versus" subjective" measurement. Human factors, 1992. 34(4): p. 441-455.
Tzavaras, A., Ethical and trustworthy Artificial Intelligence: BSI’s introduction to the European Artificial Intelligence Draft Act (AIA). 2023, The British Standards Institution.
European Commission. Shaping Europe’s digital future: AI Act. Available from: https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai#:~:text=High%20risk,life%20(e.g.%20scoring%20of%20exams.
Burema, D., et al. A sector-based approach to AI ethics: Understanding ethical issues of AI-related incidents within their sectoral context. in Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. 2023.
Roemmich, K., et al., Values in Emotion Artificial Intelligence Hiring Services: Technosolutions to Organizational Problems.(2023). 2023.
Percy, C., et al., The need for knowledge extraction: Understanding harmful gambling behavior with neural networks, in ECAI 2016. 2016, IOS Press. p. 974-981.
Wilhelm, T.I., J. Roos, and R. Kaczmarczyk, Large language models for therapy recommendations across 3 clinical specialties: comparative study. Journal of Medical Internet Research, 2023. 25: p. e49324.
Verma, H., et al. Rethinking the role of AI with physicians in oncology: revealing perspectives from clinical and research workflows. in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 2023.
Cenci, A., et al., The participatory value-sensitive design (VSD) of a mHealth app targeting citizens with dementia in a Danish municipality. AI Ethics, 2023: p. 1-27.
You, Y., et al. The medical authority of AI: A study of AI-enabled consumer-facing health technology. in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 2021.
Zając, H.D., et al. Ground Truth Or Dare: Factors Affecting The Creation Of Medical Datasets For Training AI. in Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. 2023.
Cortés, U., et al., The ethical use of high-performance computing and artificial intelligence: fighting COVID-19 at Barcelona Supercomputing Center. AI and Ethics, 2022. 2(2): p. 325-340.
Nunes, J.L., et al. Using model cards for ethical reflection: a qualitative exploration. in Proceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems. 2022.
Nakao, Y., et al., Toward involving end-users in interactive human-in-the-loop AI fairness. ACM Transactions on Interactive Intelligent Systems (TiiS), 2022. 12(3): p. 1-30.
Li, L., et al., Algorithmic Hiring in Practice: Recruiter and HR Professional's Perspectives on AI Use in Hiring, in Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 2021, Association for Computing Machinery: Virtual Event, USA. p. 166–176.
Kiyasseh, D., et al., A multi-institutional study using artificial intelligence to provide reliable and fair feedback to surgeons. Commun Med (Lond), 2023. 3(1): p. 42.
Percy, C., et al. Lessons learned from problem gambling classification: Indirect discrimination and algorithmic fairness. in Proc. AAAI Fall Symposium for Social Good. 2020.
Ismail, A., et al., Public Health Calls for/with AI: An Ethnographic Perspective. Proceedings of the ACM on Human-Computer Interaction, 2023. 7(CSCW2): p. 1-26.
Hartikainen, M., et al. Human-centered AI design in reality: a study of developer companies’ practices: a study of developer companies’ practices. in Nordic human-computer interaction conference. 2022.
Liu, R., S. Gupta, and P. Patel, The application of the principles of responsible AI on social media marketing for digital health. Information Systems Frontiers, 2023. 25(6): p. 2275-2299.
Rakova, B., et al., Where responsible AI meets reality: Practitioner perspectives on enablers for shifting organizational practices. Proceedings of the ACM on Human-Computer Interaction, 2021. 5(CSCW1): p. 1-23.
Costanza-Chock, S., I.D. Raji, and J. Buolamwini. Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem. in Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 2022.
Brass, I. and A. Mkwashi. Risk Assessment and Classification of Medical Device Software for the Internet of Medical Things: Challenges arising from connected, intelligent medical devices. in Proceedings of the 12th International Conference on the Internet of Things. 2022.
Inuwa-Dutse, I. FATE in AI: Towards Algorithmic Inclusivity and Accessibility. in Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization. 2023.
Liao, Q.V., D. Gruen, and S. Miller. Questioning the AI: informing design practices for explainable AI user experiences. in Proceedings of the 2020 CHI conference on human factors in computing systems. 2020.
Ma, A., E. Patitsas, and J. Sterne. You Sound Depressed: A Case Study on Sonde Health’s Diagnostic Use of Voice Analysis AI. in Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. 2023.
Dhanorkar, S., et al. Who needs to know what, when?: Broadening the Explainable AI (XAI) Design Space by Looking at Explanations Across the AI Lifecycle. in Proceedings of the 2021 ACM Designing Interactive Systems Conference. 2021.
Chatterjee, A., et al., ProHealth eCoach: user-centered design and development of an eCoach app to promote healthy lifestyle with personalized activity recommendations. BMC Health Serv Res, 2022. 22(1): p. 1120.
Marschik, P.B., et al., Open video data sharing in developmental science and clinical practice. iScience, 2023. 26(4): p. 106348.
Lee, M.K., et al. Participatory algorithmic management: Elicitation methods for worker well-being models. in Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 2021.
Rastogi, C., et al. Supporting human-ai collaboration in auditing llms with llms. in Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. 2023.
Yildirim, N., et al. Investigating how practitioners use human-ai guidelines: A case study on the people+ ai guidebook. in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 2023.
Svendsen, M.J., et al., One size does not fit all: Participants’ experiences of the selfBACK app to support self-management of low back pain—a qualitative interview study. Chiropractic & manual therapies, 2022. 30(1): p. 41.
Helman, S., et al., Engaging Multidisciplinary Clinical Users in the Design of an Artificial Intelligence-Powered Graphical User Interface for Intensive Care Unit Instability Decision Support. Appl Clin Inform, 2023. 14(4): p. 789-802.
Klemme, I., et al., Integration of the vision of people with diabetes Into the development process to improve self-management via diabetes apps: Qualitative interview study. JMIR diabetes, 2023. 8(1): p. e38474.
Malins, S., et al., Developing an Automated Assessment of In-session Patient Activation for Psychological Therapy: Codevelopment Approach. JMIR Med Inform, 2022. 10(11): p. e38168.
Ali, S.J., et al. Walking the walk of AI ethics: Organizational challenges and the individualization of risk among ethics entrepreneurs. in Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. 2023.
Rismani, S. and A. Moon. What does it mean to be a responsible AI practitioner: An ontology of roles and skills. in Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. 2023.
Lu, J., et al., Considerations in the reliability and fairness audits of predictive models for advance care planning. Front Digit Health, 2022. 4: p. 943768.
Raji, I.D., et al. The fallacy of AI functionality. in Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 2022.
Wang, Y., M. Xiong, and H. Olya. Toward an understanding of responsible artificial intelligence practices. in Proceedings of the 53rd hawaii international conference on system sciences. 2020. Hawaii International Conference on System Sciences (HICSS).
Glomsrud, J.A. and T.A. Bach, The Ecosystem of Trust (EoT): Enabling effective deployment of autonomous systems through collaborative and trusted ecosystems. arXiv preprint arXiv:2312.00629, 2023.
Chuchu, N., et al., Smartphone applications for triaging adults with skin lesions that are suspicious for melanoma. Cochrane Database of Systematic Reviews, 1996. 2018(12).

Table 1 The search terms

	Search strings	Justification
1	“Artificial intelligence” OR “AI”	To ensure the inclusion of an AI-enabled system
2	AND “responsible” OR “ethic” OR “trustworth”	To filter articles discussing RAI, ethical, or trustworthy AI: the main elements of RAI practices today
3	AND “transparen” OR “fair” OR “accountab” OR “bias” OR “accurate” OR “accuracy” OR “robust” OR “explainab” OR “interpretab” OR “secure” OR “security” OR “privacy” OR “benefic” OR “explicab” OR “justice” OR “non-maleficence” OR “controllab*” OR “diversity” OR “non-discriminative”	We used inductive reasoning to ensure the inclusion of all related keywords within our knowledge that might extract or were related to any RAI practices
4	AND “manage” OR “practical” OR “in practice” OR “step” OR “organize” OR “organise” OR “activities” OR “frontline” OR “do” OR “how to” OR “document” OR “reproduceab” OR “monitor” OR “practice”	To filter articles that demonstrated practices of RAI, rather than RAI principles, concepts, or framework
5	AND “industry”^a	To filter RAI applications in an industry setting and exclude those without a clear application domain
^aOnly used for Scopus and ACM digital database because PubMed database was already focused on an industry (i.e., healthcare) *Used as a wildcard character to represent one or more characters in a string

Table 2 Inclusion criteria

Inclusion criteria	Description	Justification
Time of publication	January 1, 2012, through to November 9, 2023	To include only recent studies
Language	English	For accessibility and availability purposes
Type	Peer-reviewed articles or conference proceedings	To ensure a relatively similar quality assurance process
Study methodology	Empirical (e.g. qualitative, quantitative, SLR)	To build evidence
Real-world applications	Applied in a real-world setting	To identify suggested RAI practices for real-world applications
AI-enabled systems	Inclusion of a concrete AI-enabled system or a proof of concept (POC)	To avoid discussions related to AI-enabled systems in a general sense
Intended purpose	Applied in a specific use case or a context	Due to RAI being contextual and to avoid discussions related to use of AI-enabled systems in a general sense
RAI focus	Focused on socio-technical factors of RAI, not technical factors	To explore deeper societal, ethical, and other non-technical impacts of AI-enabled systems for their stakeholders and society

Table 3 Study methodology and measures (N=45)

	Subjective measures	Objective measures	Subjective and objective measures	N
Qualitative and quantitative methods	4	0	1	5
Qualitative methods	34	0	0	34
Quantitative methods	2	2	1	5
Systematic Literature Review	1	0	0	1
N	41	2	2	45

Table 4 Overview of study goals of included articles and their description (See appendix 2 for more details)

Study goal	N (%)	Description
Analysis of stakeholder perspectives on AI-enabled systems	27 (60%)	Analyses on stakeholder perspectives on development, deployment, or use of AI-enabled systems in real-world settings.
Analysis of stakeholder perspectives on RAI practices	8 (17.78%)	Analyses of stakeholder perspectives on how RAI should be practiced in real-world settings.
Analysis of AI-enabled systems	10 (22.22%)	RAI analyses of AI-enabled systems by authors of included articles
Total	45 (100%)

Table 5 Stakeholder involvement and roles (See Appendix 2 for detailed information)

	User stakeholders involved?		Non-user stakeholders involved?
	All	Users who were also primary stakeholders^a	All	Non-users who were also primary stakeholders^a
Yes	31	17	27	1
No	9	7	16	0
N/A	5	2	2	1
Total	45	26	45	2
^aIf the included articles did not mention primary stakeholders, we made assumptions about who the primary stakeholders would be in their use cases.

Table 6 Descriptions of the Roles of AI-Enabled Systems (See Appendix 3 for detailed information)

The role of the AI-enabled system or POC	N (%)	Description
AI assisted decision-making	19 (42.22%)	AI-enabled systems were used to assist (expert) users in making decisions by, for example, providing analyses, recommendations, or predictions
Human controlled-AI	10 (22.22%)	AI-enabled systems were used merely as a tool to help humans achieve a goal that required close human oversight, supervision, and intervention to function.
Collaborative AI	3 (6.67%)	AI-enabled systems were used to collaborate with users to achieve a goal
Emotional AI	2 (4.44%)	AI-enabled systems that were designed to respond to user emotions
Explainable AI	1 (2.22%)	AI-enabled systems designed to provide understandable insights into their processes, decisions, and outputs, allowing users to comprehend how and why specific outcomes were reached
Mix roles	10 (22.22%)	Multiple AI-enabled systems or POCs being developed, deployed, or used, and when there was no information or description of the individual AI-enabled systems or POCs because the focus of these included articles was capturing the stakeholder perspectives in specific use cases
TOTAL	45 (100%)

Table 7 Overview of the eleven themes of the suggested RAI practices (See Appendix 4 for included articles and associated themes)

The suggested RAI practices	N=number of included articles	Description
Harm prevention	N=17	Ensuring that AI-enabled systems were used with caution, refraining from causing harm to others
Accountability	N=7	Holding individuals or organisations accountable for their actions, decisions, and/or impacts when AI-enabled systems were in use
Fairness and equity	N=10	Ensuring fairness, equity, and the avoidance of bias or discrimination in AI development and deployment
Explainability	N=15	The ability to explain or make understandable the behaviours and decisions of an AI-enabled systems to a human audience
AI literacy	N=13	The knowledge and understanding necessary to comprehend, effectively use, and critically evaluate AI-enabled systems and their implications.
Privacy and security	N=7	Ensuring that AI-enabled systems respect and secure individuals' privacy rights and sensitive data
Human-AI calibration	N=24	Aligning the capabilities, expectations, and interactions between stakeholders and AI-enabled systems to achieve optimal performance and outcomes
Interdisciplinary stakeholder involvement	N=28	Involving stakeholders directly in the decision-making processes of AI-enabled systems by providing input, feedback, and validation at all phases of the AI lifecycle
Value creation	N=13	Ensuring that the use of AI-enabled systems to create value for their stakeholders
RAI governance	N=13	Establishing processes, structures, and policies to oversee and ensure that AI-enabled systems were developed, deployed, and used responsibly
AI Deployment effects	N=11	Evaluating the socio-ethical and individual impacts resulting from the deployment and use of AI-enabled systems

The authors declare no competing interests.

6Appendices.docx

Insights into suggested Responsible AI (RAI) practices in real-world settings: A systematic literature review

Status:

Version 1

Abstract

Figures

1 INTRODUCTION

2 METHODS