The ability to perform policy analyses in a systematic and reproduceable manner is a competency found across a number of disciplines, such as public health [2, 14, 15], economics [16], public administration [17, 18], and law and legal studies [19]. In this article, we provided a detailed description of how policy analysis can be performed systematically from a qualitative research viewpoint by amending existing best practices in evidence synthesis methods.
The process of systematically mapping policies in one or more policy regions is identical regardless of policy regions under study: (1) develop a comprehensive search strategy for policy databases; (2) review and extract relevant policies and legislation directly from original sources; (3) develop a multi-layered search strategy for scientific databases; (4) merge policy and academic publications according to the eligibility criteria; (5) acquire further information through searching reference lists of included policies and articles; and (6) merge the three searches into one single data repository for further analysis. However, there are some specific factors that can become relevant depending on the policy region under study. For instance, some countries host multi-level or federated policy environments (e.g., Belgium, Germany, and the United States have policies at the national level and at the state level), meaning that – in order to properly assess the total policy environment at the national level – a number of sub-national regions also need to be included to ensure a complete overview of the active policies [23, 38]. Some policy repositories also may be missing data as a result of, for example, geopolitical developments or technical capabilities [25, 26], in which case a greater reliance is put on academic and grey literature.
It must be noted that the contents of this methodology are inherently limited due to the fact that it solely investigates written policy, meaning they cannot be used to gauge how these policies are put in practice. Nevertheless, they can provide information on one of the “building blocks” of the studied system and how this system is designed at the highest level. That said, it is possible to use the contents of this policy mapping methodology as a starting point to explore options for future policy developments. To do this, it is vital that the findings of this policy mapping methodology are contextualized using other forms of empirical data, such as impact assessments or simulations, or public and expert consultations [39, 40].
One type of follow-up analysis that is particularly well-suited to the data collected through a policy mapping methodology is a qualitative comparative analysis (QCA). The results obtained through policy mapping methodologies can be considered individual case studies, with each studied policy region forming its own case study. Historically, a common challenge of comparing qualitative case studies studies was to capture the complexity of the terminology, show different possible pathways through which an outcome can be achieved, and discuss their respective consequences for practice [41]. The data output of policy mapping frameworks – if done consistently across projects – typically results in a medium number of cases (i.e., 12 or more), which enables the use of systematic comparison and intensified case-based methodologies, such as a QCA to further identify patterns in data that can potentially point towards factors that positively or negatively influence the studied outcome measure [33]. QCA involves a mixed-method approach and is used to explore and analyse quantitative and qualitative data by using set theory and Boolean minimization to identify patterns in data [42]. QCA typically uses case studies in which cause and effect are determined and explained. The conditions are classified as either necessary or sufficient. Necessary conditions are essential for consistently producing an outcome, though require other (sufficient) conditions to be present. Sufficient conditions, on the other hand, can individually produce an outcome, though inconsistently [43]. By triangulating the findings of a policy mapping method with the outputs of a QCA, several key points can be identified in which policy has harmonised or diverged across policy regions over time. This can be used to inform possible future policy developments. To do this, however, it is important to consider an array of future policies and their subsequent outcomes. Scenario planning involves the definition of numerous possible future policy directions, some of which may be desirable while others can be detrimental, as to better prepare professionals to be successful in a constantly shifting environment [44], and has been used by the European Commission to set the agenda on the future of the European Union [45].
In conclusion, we argue that there is a clear need to improve current policy collection and qualitative analysis methods. Our methodological toolkit gives a point-by-point description of how to design, perform, and report a systematic and reproduceable policy mapping article, as well as how that policy data can be analysed further to lead to prospective policy action points.