Based on the framework of emergency capability assessment of ISO 22325 [16], the process is divided into three steps. First, a well-established system (similar to a handbook) should be built, which includes all levels and all potential indices. Second, the process control is introduced to discuss the implementation/execution of the system, i.e., how to use the well-established system according to the actual situation. The third step is the measurement of the system through test and exercise. The comparative study and analytic hierarchy process (AHP) are used to build the basic framework of our index system. Then a comprehensive index system is established based on literature research and expert interviews. At last, a practical case will be used to illustrate the implementation and measurement of our system.
3.1 Comparative Study of Related Work
To make our evaluation tool more scientific and useful, the comparative study with several widely used tools or lessons is conducted to report what they have done and form our evaluation framework, without making any judgement on the effectiveness of the approaches or recommending any specific tool. It’s a question to select the compared tools (i.e., the bases of our study), and some principles are listed as below:
- Widely used or referenced in the evaluation of COVID-19 response measures,
- Demonstrated by experts,
- Similar in the purpose and scope,
- Different in internal relation,
- Comparable in indicators,
- Unified in concepts to ensure comparability.
Four analogical evaluation tools or lessons were finally selected, including Oxford COVID-19 Government Response Tracker (OxCGRT, University of Oxford), Four Levels of Response Measures (FLRM, Tsinghua University), Universities’ Preparedness and Response Towards Multi-Hazards (UPTRMH, International Institute of Disaster Science (IRIDeS) at Tohoku University, Keio University, and Tsinghua University), and the Manchester Briefings COVID-19 (ManBC, The University of Manchester). They either track in a systematically way by evaluating the time and stringency or hold periodically seminar sharing their experience, cases, and lessons in response to COVID-19. Above tools have similarities and commonalities in the aim of evaluating and prompting the response for COVID-19. However, they have unique characteristics to serve for their own evaluation purposes, and many aspects are different, such as concept, aim, framework, and indicators. The detailed information is described as shown in Fig2.
OxCGRT developed by Oxford, is a relatively perfect framework to track and compare policy responses around the world. It has evolved several versions with the development of COVID-19 and policies, starting with 9 indicators, to 20 indicators, then to 23 indicators. Empirical researches have been conducted based on the data from more than 180 countries to record the number and strictness of government policies. The tool pays more attention on the public health system and some closure/containment measures at the city level and during the response phase. It mainly focuses on policy making, rather than the implementation of policy. For example, indicators affecting policy execution (i.e., the public acceptance and the reaction of policies) are not taken into consideration in this tool [24-27].
FLRM conducts a correlation analysis between the strictness of government response measures and epidemic trends of COVID-19. It consists of seven categories with 51 indicators to assess the measures from 13 selected countries. It is a whole-of-society approach, including country, city, community, and individual levels. The individual involvement is taken into account to obtain the public support during the fight to COVID-19. However, the system is not built, and the framework and indicators are relatively incomplete.
UPTRMH keeps a record of what happened and success/failures to learn from the experience and prepare for the next hazardous events. 26 case studies from 13 countries and regions are analyzed in this project. It focuses on the response of a typical community – campus, to strengthen their current strategy and plan to reduce multiple disaster risks and respond efficiently [28], which can increase the understanding of the response at local community level.
The Manchester Briefings on COVID-19 (ManBC) are series of international lessons and examples which may prompt thinking on the recovery from COVID-19, as well as other information from a range of sources[4]. It can provide experience for government emergency planners and resilience officers through reporting what others have done. The effect of community participation and support on the COVID-19 response is discussed, such as public leadership, social protection, volunteer policy, and partnerships [29].
Our index system is intended to become an assessment tool for countries or regions. Several requirements should be satisfied. 1) Indicators should involve multi-levels, including country-, city-, local community- and individual-level. 2) Both policy level and implementation level are taken into account. 3) The assessment is treated as a continuous process, including response phase and preparedness phase. 4) Community support and individual involvement will be considered in our system. 5) Empirical researches will be conducted in our future research when relevant data/database is supplemented.
3.2 Analytic Hierarchy Process (AHP) for Setting Up the Index
AHP is a decision-making method that decomposes the elements into goals, criteria, programs, and other aspects. Both qualitative and quantitative analysis can be carried out. AHP is used to set up the framework under the concept of a multi-level strategy of public health management. It is a multi-level hierarchy which includes Level, Category, and Indicators. The Level consists of country, city, local community, and individual from top to bottom. There are several categories for each level based on the function of each level. For example, the Country level has the power to express commands and deploy national resources (e.g., sending medical teams to key areas). Similar to the OxCGRT, containment and closure, response in economic system, and response in public health system constitute the main body of the framework. Containment and closure are replaced by the category of command, control and coordination in the Country level. The participation and support of local community have big significance through current researches and practical experience. Individuals are only responsible for implementing measures, including wearing masks, staying at home, keeping social distancing, etc. The Individual level is not classified into categories further. Each Category includes several indicators through literature research and expert interviews, which will be specified in the next section. The hierarchical model is defined as the hierarchical index system assessing local resilience capability in COVID-19, as shown in Fig 3.