We adapted the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) eight-step guidelines for conducting and reporting an MCDA (9). Each step is presented in Fig. 1 and detailed further below. The initial four steps describe setting up the problem and the available choice options along with the criteria upon which those options can be judged. In the current project, the first four steps were considered pre-protocol work that shaped our cross-sectional survey. The fifth step involves conducting a cross-sectional survey of national stakeholder preferences for changes within and between criteria. The methods for the first to fifth steps are detailed in the methods section, followed by the sixth and seventh steps in the analysis and the results sections. The final step is to report the findings to the funder and stakeholders.
Step 1 - Defining The Problem
During early consultations with hospital teams, we would present up to 20 nudge intervention options to optimise medication use previously identified in a systematic review (17). The hospital teams appeared overwhelmed and often did not choose any option, plausibly due to “choice overload” (19). The research team determined that a multi-criteria decision analysis (MCDA) could provide a robust and transparent framework to facilitate these decisions. To further simplify alternative intervention options, the research team categorise the 20 intervention options into a smaller number of nudge intervention types. Our categorisations were cross-checked and approved by representatives at Public Health England.
Step 2 - Selecting Criteria
To select the criteria upon which the intervention types would be judged, a core group of 10 quality improvers was formed including four doctors, two pharmacists, and four managers. The core group was asked to review the 41 constructs from the Consolidated Framework for Implementation Research (CFIR) (18). Within this framework, constructs are organised across five domains: characteristics of the innovation in question (e.g., two constructs include adaptability and trialability), individuals involved (e.g., knowledge and self-efficacy), inner setting (e.g., compatibility and available resources), outer setting (e.g., incentives and patient needs), and the process encouraging uptake (e.g., planning and opinion leaders). Each improver was asked to rate each construct based on their perceived importance, from 1 (least important) to 5 (most important). Then, each construct was ranked based on the sum of those scores. The top 10 constructs were selected as the criteria considered in the present MCDA.
Step 3- Measuring Performance
To measure levels of quality for each criterion, three-point Likert scales were created to indicate high, medium, and low performance levels.
Step 4 - Scoring Alternatives
Two researchers (SK and UT) used consensus discussions to assess the expected performance of each type of intervention against each criterion. Next, two additional researchers (KAS and IV), each of whom had at least 10 years of experience implementing nudge interventions in healthcare settings, cross-checked these decisions. Next, a performance matrix was assembled with the agreed rankings. A RAG system was used to indicate the quality criteria rankings, such that low-quality rankings would be designated in red, medium in amber, and high in green.
Step 5 – Weighting Criteria
A cross-sectional survey was conducted between October 2021 and December 2021. The survey was designed using 1000minds Decision-making software.
Participants. To capture more diverse views across a national context, our participants included staff involved in making suggestions or decisions about the implementation of the quality improvement projects for NHS hospital organisations in England such as doctors, nurses, pharmacists, and quality improvement managers (20). As we planned to recruit our participants via email, we anticipated a low rate of those emails being opened, 20–25%, and that only 15–30% of those opened would be completed (21, 22). We worked with the Health Foundation (described below) to identify at least 4,000 potential participants and to ensure at least 100 completed the survey.
Identification and Recruitment of participants. The opportunity to take part in the survey was initially advertised via email to 4,439 members of the Health Foundation’s Q-Community. The Health Foundation is a charitable foundation charged with improving healthcare quality. The Q-Community encompasses individuals who self-identify as healthcare quality improvers in England. At the end of the survey, participants were asked to identify other potential participants, i.e., snowball recruitment. The survey was also advertised by researchers on Twitter and LinkedIn. Participants were asked to complete the survey within two weeks. One reminder was sent to encourage completion.
Survey. The survey was designed according to the Potentially All Pairwise Ranking of All Possible Alternatives (PAPRIKA) method (23). PAPRIKA is a method for scoring additive multi-attribute values using pairwise rankings of alternatives (24). Each question asked participants to choose between two alternative intervention types differing according to two criteria with differing performance levels. An example question appears in Fig. 2. Future questions were adaptively selected based on participant responses to previous questions.
Step 6 – Calculating Aggregate Scores
Our research team calculated the aggregated scores for each intervention type when the criteria were unweighted and weighted. Unweighted scores were calculated by adding the intervention scores against each criterion based on the researcher team’s assigned performance levels. Weighted scores were calculated using an additive model that combines scores and weights in a way that is consistent with stakeholders’ expressed preferences. The form of an additive function was given according to the formula
$${V}_{j} = \sum _{i=1}^{n}{S}_{ij}. {W}_{i},$$
where Vj is the overall value of intervention j, Sij is the score for intervention j on criterion i, and Wi is the weight attached to criterion i. Then, each intervention type was rank ordered according to its total score.
Step 7 – Dealing With Uncertainty
To assess the consistency of participant responses we developed a decision-analytic model comprising criteria, weights, and scores in TreeAge® Pro R1. A probabilistic sensitivity analysis was carried out using a Monte Carlo simulation with 1000 iterations for the following three scenarios: (a) incorporating uncertainty in scoring the alternatives, (b) incorporating uncertainty in the weighting preference, and (c) incorporating uncertainty for both the scoring alternatives and weighing preference. A gamma distribution was used for scores. Because preference weights are positive proportions that totalled one, we selected a Dirichlet distribution to preference weights in the simulation (25). The means of observed data were used as parameters for this distribution.
Step 8 – Reporting And Examining Findings
The initial results were shared with the Health Foundation, and the published report will be provided to the Q-Community members.