We developed a pediatric virtual care evaluation framework (Figure 1) based on the four domains proposed by STEM9 and other established evaluation frameworks and tools. 17,30,32 Working group members agreed the framework should support comprehensive evaluation of virtual care programs and appropriate integration of virtual care based on patient factors (e.g., age, diagnosis, socioeconomic status), timing in the care pathway, and mode of delivery. We also agreed to use the term "virtual care"3 rather than telehealth or telemedicine to highlight the applicability of this framework to all types of virtual care regardless of technology or synchronicity. We believe this framework can be applied in a variety of contexts ranging from community-based practice to multi-site health centres, and oversight or funding agencies (e.g., government health ministry).
Framework structure
Our framework has four measurement domains that parallel STEM: health outcomes, health delivery, individual experience, and program implementation. Under these four domains, we have specified 19 sub-domains and provided a structure that can readily support development of organization-wide or program-specific evaluation questions that illustrate scope, and associated measures, data sources, and bases of comparison for each. Using this tool, teams can apply the framework to a variety of patient care settings (see Tables 1-4 for examples and Additional File 2 for the full framework); articulate their own evaluation questions under each subdomain, and identify measures, data sources and bases of comparison that can be used to answer each question.
Framework domains and sub-domains
The first domain, Health Outcomes, includes clinical measures of individual and population health within three sub-domains: physical, mental and behavioural, and quality of life (QoL). This domain aligns with the ultimate goal of helping children and youth achieve their best life by going beyond direct measures of health to include the impact of health status on the quality of life of patients and caregivers. We added “behavioural” to the mental health sub-domain to ensure that our framework is relevant for programs delivering developmental and behavioural health services (e.g., applied behaviour analysis, complex care) and provides a holistic view of the impact of virtual care on pediatric health outcomes. Data sources are primarily patient charts, assuming patient and caregiver self-reports and other measures of QoL are included therein. Comparators include internal or external databases or clinical practice guidelines (depending on the evaluation questions). Table 1 provides example evaluation questions for the physical health outcomes sub-domain along with associated measures, data sources and bases of comparison.
The second domain, Health Delivery, includes five sub-domains: access to care, privacy, safety, efficiency, and effectiveness of care delivery. We have defined the scope of these sub-domains as follows. Access to care considers both timeliness (e.g., time to first visit, wait times) and equity (e.g., to what extent the program provides equitable access to care or addresses access or equity gaps present in other programs). Privacy examines the patient environment and the extent to which the tools and processes used to deliver virtual care adhere to privacy and confidentiality standards. Safety considers the potential for virtual care to introduce or address safety risks (e.g., adverse event reporting or risk assessments). Efficiency considers both time and cost (e.g., the financial and time-cost-difference for patients to travel to the hospital compared to attending a virtual visit). Finally, effectiveness considers how well the virtual care program delivers care as intended in terms of quality and quantity, and can be linked back to program objectives and compared to in-person care as appropriate. While organized slightly differently, four of these five sub-domains are consistent across AAP STEM,9 Ontario Health Quality17 and CHIBE31 frameworks. We added privacy because it is a theme that appears regularly in telehealth experience surveys, especially when evaluating eMental Health programs33–35 and can be impacted by virtual models of care delivery. Examples of evaluation questions, measures, and data sources for the privacy sub-domain are presented in Table 2.
The third domain, Individual Experience, includes seven sub-domains: usefulness, ease of use, interaction quality, technology reliability, satisfaction and future use, patient-centeredness, and workload burden. To ensure a holistic evaluation of any virtual care program (implemented or envisioned), patient, family/caregiver, provider, and support staff experiences are all considered essential components of high-quality care.6,23 To support consistent and comprehensive evaluation of the individual experience, the defining components of usability and themes that appear in validated survey tools (e.g., Telehealth Usability Questionnaire36) and CHIBE31 are represented within the sub-domains. Usefulness is the degree to which a system or tool supports a desired function or goal (e.g., perception of how well the virtual care program supports access to care, clinical outcomes or reduces cost),37 while ease of use speaks to how easy it is to use, regardless of utility. Interaction quality is the quality of the interpersonal interaction between patient/caregiver and provider that is facilitated by the virtual care system/tool, for example how well participants felt they were able to see, hear, and express themselves.36 In alignment with the quadruple aim,22,23 sub-domains also include patient centeredness (e.g., whether patients should receive care in-person or virtually based on need or personal choice) and individual workload burden (e.g., patient and/or caregiver, provider workload or wellness). Table 3 presents example evaluation questions for the ease-of-use sub-domain along with associated measures, data sources and bases of comparison.
The fourth domain, Program Implementation, highlights key factors impacting system change and sustainability, and includes four sub-domains: leadership engagement and structure (institutional buy-in, provision of policy or guidance), resources (human and financial), training and support (availability and appropriateness), and infrastructure and technology (functionality, performance, and security). The last two sub-domains reflect the CHIBE framework.31 Here, infrastructure and technology considerations relate to information systems and device capabilities rather than a technology usability evaluation, which falls under the individual experience domain. These four sub-domains are considered important to facilitate planning for implementation and to support decision-making regarding additions or changes to the reach and depth of virtual care programs over time. This domain does not emphasize the identification of KPIs and value definitions as in STEM,9 because our framework is designed to identify measures and bases of comparison for each sub-domain. Once measures have been identified, these can be used as a repository from which to identify program-specific KPIs. Example evaluation questions, measures, data sources and bases of comparison for the infrastructure and technology sub-domain are presented in Table 5.
How to use this framework
The first step in using this evaluation framework is to define the objectives of the virtual care program. Excellent resources are available outlining how to develop program objectives. For example, program leaders could consider developing SMART aims following quality improvement methodology.38,39 The next step is to develop evaluation questions for each sub-domain that speak directly to the objectives of the virtual care program. For example, when selecting health outcomes evaluation questions, consider whether health outcome targets and impacts should be defined at the individual or population level, or both. For each evaluation question, identify measures that can be used to answer the question (e.g., hemoglobin A1C value as an outcome measure for children and youth with diabetes). Measures may come from encounter records (e.g., observations documented by healthcare providers) or patient-reported outcomes. The operational definitions of each measure should be described so data retrieved by different operators, processes and organizations are consistent and reproducible. Identify existing data sources to determine what data are already available versus need to be collected (e.g., electronic health record data or patient/provider survey results). For each source, consider an appropriate data collection strategy (e.g., qualitative, quantitative, or mixed methods) and the bases of comparison that can be used to determine success. Bases of comparison include an in-person care program, a comparable virtual care program (internal or external), clinical norms, or clinical guidelines as appropriate.
One of the key attributes of evaluation quality is feasibility. This framework provides an organized approach to deciding on outcomes of interest for different program stages (see WHO guidance18), priorities, and durations of evaluation (e.g., a program duration may be too short to assess health outcomes, but sufficient to assess program implementation). To demonstrate framework feasibility and how evaluation questions and measures tie back to specific program objectives, we applied the framework to an existing pediatric type 1 diabetes virtual care program (see Additional File 3). This allowed us to explore how the evaluation framework supported development of a program-specific evaluation plan. We felt that the breadth of the 19 subdomains covered all potential areas of interest for evaluation. This exercise also highlighted the need to consider which sub-domains are most relevant in unique contexts. While the pediatric type 1 diabetes virtual care program example illustrates a comprehensive evaluation plan for one clinical area, and the framework’s utility in developing program-specific evaluation questions and measures that tie back to program objectives, it is not practical for every evaluation to address all sub-domains of the framework. We believe it is important to consider all domains when scoping out an evaluation, but discrete elements of this framework can be used as required, depending on evaluation objectives, audience, and resources.