Multi-criteria Decision Analysis for Health Technology Assessment of Intensive Care Ventilators for Pediatric Patients.

doi:10.21203/rs.3.rs-786470/v1

Download PDF

Research

Multi-criteria Decision Analysis for Health Technology Assessment of Intensive Care Ventilators for Pediatric Patients.

https://doi.org/10.21203/rs.3.rs-786470/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

The technological complexity and heterogeneity of intensive care ventilator models currently available on the market together with the heterogeneity in pediatric patients (0 to 18 years old), make the choice of the best machine for pediatric healthcare setting crucial.

This paper is aimed at addressing all the critical aspects linked to the implementation of intensive care ventilators in a pediatric setting, highlighting the most relevant technical features and describing the methodology to conduct health technology assessment (HTA) for supporting the decision-making process.

Four ventilators models were included in the assessment process. A decision-making support tool (DoHTA method) based on Analytic Hierarchy Process, was applied. 28 Key Performance Indicators (KPIs) were identified, defining the safety, clinical effectiveness, organizational, technical, and economic aspects. The Performance scores of each ventilator have been measured with respect to KPIs integrated with the total cost of ownership (TCO) analysis, leading to a final rank of the four possible technological solutions.

Results

The final technologies’ performance scores reflected a deliver valued, contextualized, and shared outputs, detecting the most performant technological solution for the specific hospital context. HTA results had informed and supported the pediatric hospital decision-making process.

Conclusions

This study, identifying and discussing the pros and cons of innovative features of ventilators and all the evaluation criteria and aspects to be taken into account during the evaluation process, can be considered as a valuable proof of evidence as well as a reliable and transferable method for conducting a decision making process in a hospital context.

Health Economics & Outcomes Research

Intensive Care Ventilator

Intensive Care Unit

Pediatric

HTA

Hospital Decision Making

AHP

Multi Criteria Decision Analysis

Mechanical ventilation has been considerable progressing over the last 20 years, thanks to the development of more efficient and sophisticated technical features.

It has gradually led to the development of different ventilation modes that had significantly improved the clinical effectiveness of ventilation techniques and patient–ventilator synchrony(1), leading in the meantime to reduction of time spent on ventilation, staff workload and intensive care unit (ICU) costs (2) (3).

Even if the variety of currently available ventilation modes and technical features are a great chance to solve respiratory failures, health professionals need to be aware about the technology and the risks that they could bring to the system (4).

For this reason, a comprehensive evaluation of the technical characteristics of ICU ventilators results pivotal to understanding the strengths and weaknesses of each ventilator, particularly because the ventilators’ efficacy provided by manufacturers may be significantly different from the clinical effectiveness in a real clinical setting.

The complexity of pediatric ventilation is firstly linked to the variable size of patient ranging from premature patient below 1 kilogram to adolescents weighing 70 kilos. For this reason, the heterogeneity in patients’ age and size should be carefully taken into account during an evaluation process. For instance, as non-invasive ventilation (NIV) and high flow nasal cannula oxygen (HFNC), are the established therapies within the pediatric ward (5), in order to be able to treat all possible kinds of patients, each ICU bed should be provided with a ventilator that includes NIV and HFNC functions as well.

In addition to the technological complexity and the heterogeneity of patients, the heterogeneity of ventilator machines is another aspect to be carefully analyzed.

Moreover, the presence of several ICUs in the same hospital, with high care complexity and with different medical specialties, makes the theme of technologies’ heterogeneity crucial. Major issues are associated to the learning curve of healthcare professionals in transferring the theory into clinical practice. The learning curve theory is usually based on homogeneous products. The higher the heterogeneity of intensive care ventilator models within the same clinical setting is, the higher the difficulties in covering the clinical needs with enough skilled health professionals able to manage each of these complex technologies.

Another critical aspect associated with intensive care ventilation are costs. Recent years have been characterized by a significant increase of healthcare costs (6, 7). Expenses of Intensive Care unit (ICU) usually cover up to 20% of overall hospital’s costs (8) especially when considering those patients who require prolonged mechanical ventilation (9).

All the critical aspects mentioned above make the selection of the intensive care ventilator model that best fits with the specific healthcare setting needs difficult. It exposed the problems in taking a well-informed decision about a suitable technology throughout an effective and reliable tool for prioritization.

Considering the high complexity of decision-making process for ICU ventilators selection and taking advantage of the renewal plan for the intensive care ventilators, the hospital management of Bambino Gesù Children’s Hospital requested a detailed Health Technology Assessment (HTA) study aiming at the replacement of the intensive care ventilators in three clinical departments.

The primary aim of this study, indeed, was to provide an exhaustive overview of the assessment criteria that should be considered when comparing the performances of different ventilators models and to show an effective and reliable evaluation method. The secondary aim was to give clinicians a comprehensive and relevant report on these performances. The paper describes in detail the evaluation process that has been conducted within our hospital context offering a clear examination of the ventilators’ impact on the possible risk and clinical benefits for patients, the entire hospital organization, and the hospital budget.

Thanks to HTA process that represents a multidisciplinary process and a method of evidence synthesis, it was possible to determine the ICU ventilators’ main technical characteristics, to measure their performances, to analyze the pros and the cons of different ventilators’ models, and to sum up all the information gathered and elaborated in an evidence-based final recommendation, actively supporting the complex decision-making process. Different aspects such as clinical effectiveness, safety, costs, technical, organizational and social, ethical, and legal aspects of the ventilators were analyzed. According to the literature, complex decision-making processes have been usually carried out through Multi-Criteria Decision Analysis (MCDA) method in several fields, even though very few applications to guide resource allocation decisions in health care have been recorded, so far. As MCDA results an effective and reliable tool able to identify the criteria and their weights for priority setting, resulting in a rank ordering of interventions, we decided to adopt MCDA as an integral part of the HTA evaluation (10), as an analytical quantitative instrument focused on supporting the decision-making process between alternative products.

As the HTA is a multidisciplinary and multidimensionality process, a working group, composed by professionals with different professional skills, was established to identify all the pertinent aspects to be analyzed in the assessment of different ventilator systems. The working group involved eighteen professionals: twelve medical doctors; five Biomedical Engineers; one Health Economist. Table 1 illustrates the different responders' professional profile and hospital's Department, which they are addressed to.

Table 1

Profile of professionals involved in the study.
Code	Professionals' Profile	Hospital Department
1	Medical Doctor	Emergency department
2	Medical Doctor	Intensive Care Unit
3	Medical Doctor	Intensive Care Unit
4	Medical Doctor	Intensive Care Unit
5	Medical Doctor	Intensive Care Unit
6	Medical Doctor	Intensive Care Unit
7	Medical Doctor	Intensive Care Unit
8	Medical Doctor	Cardiac Intensive Care Unit
9	Medical Doctor	Cardiac Intensive Care Unit
10	Medical Doctor	Cardiac Intensive Care Unit
11	Medical Doctor	Cardiac Intensive Care Unit
12	Medical Doctor	Hospital Health Direction
13	Biomedical Engineers	Clinical Engineering Department
14	Biomedical Engineers	Clinical Engineering Department
15	Biomedical Engineers	Health Technology Assessment Unit
16	Biomedical Engineers	Health Technology Assessment Unit
17	Biomedical Engineers	Health Technology Assessment Unit
18	Health Economist	Health Technology Assessment Unit

The HTA process was carried out integrating the evaluation conducted following the EUnetHTA CoreModel©, as a guideline for HTA processes [10] (11), and the Analytic Hierarchy Process(AHP), as outlined in Decision-oriented Health Technology Assessment (Do-HTA) method (10) and graphically schematized in Fig. 1.

Figure 1: DoHTA method (10)

This method proposes to organize the collected evidence of the selected technology and the consequences of its use in different Key Performance Indicators (KPIs), which are arranged in a hierarchical decision structure (cf AHP). The integration of the analytical approach (AHP) within the qualitative assessment tool (EunetHTA), allows quantifying opinions and transforming them into a coherent decision model assessing all factors and their interactions in a decision domain, leading to define a final rank of several health interventions. The process was described in the flowchart presented in Additional File 1 (Figure S1). The idea is that a decision problem can be decomposed into a hierarchy of more sub-problems each of which can be measured and analyzed independently. AHP mathematical process using pairwise comparisons, eigenvector method for deriving weights, and a method to verify the “consistency” of judgments, is able to integrate all KPIs evaluation in a final numerical result, which represents the alternatives' relative ability to achieve the decision goal. The whole process, previously published by the authors under the name of DoHTA, it is characterized by seven steps described in Fig. 1.

The first phase of the HTA process is aimed at defining and detailing the decision problem. The aim of the HTA process was to select the ventilator model that best fits with the hospital needs. Within the variety of models and manufactures currently available on the market, the working group drew up the minimum tenders’ requirements that the ventilators should have to be considered eligible for the assessment process. More specifically all the ventilators’ models must have a touch screen < 21’’ and must be approved for neonatal and pediatric patients. Finally, four ventilators models that meet the inclusion criteria were selected. The HTA process was carried out involving the three hospital departments that took part to the renewal plan: The Intensive Care Unit (ICU), the Emergency Department and the Cardiac ICU. Health professionals tested for a month each model involved in the study.

The aim of the study is to show the method used to carry out the decision making process highlighting the KPIs that were used to assess the ventilators models. The KPIs identified and the method presented here will be extremely helpful for other hospitals (pediatric or not) that have to select the intensive care ventilators that best feet with their needs, especially during COVID-19 pandemic.

The entire HTA process will be shown as a case study calling the different ventilators’ models as Model 1, Model 2, Model 3 and Model 4.

After defining the decision problem, a preliminary study was conducted to acquire a general overview of the subject and to define the main criteria representing discriminant assessment factors between the new technology under evaluation and the state of art.

The main evaluation criteria (Domains) were identified, by the working group, among those singled out by EUnetHTA Core Model© (11), which represents the European guideline for each HTA process and intends to coordinate the collected evidence of the selected technology in different “assessment elements”. In this way, the technological innovation, could be analyzed according to safety, clinical effectiveness, costs, social, ethical, legal, and organizational aspects.

Each evaluation domain was better defined throughout several KPIs, in order to detail the decision problem and describe which aspect of each evaluation area, the working group intends to take into account. To outline a clear, detailed, and proper definition of the indicators, a literature search was carried out. It aimed at identifying the detailed indicators through which the safety, efficacy, costs, organizational and technical aspects of intensive care ventilators could be evaluated. More specifically, studies that contain relevant information on safety ventilators related aspects, on clinical effectiveness and economic evaluations were considered for the inclusion. According to AHP, the KPIs identified were arranged in a hierarchy decision tree. More specifically, the “goal of the decision” was split into the main evaluation criteria (i.e. the domains), each one is divided in turn, into KPIs, covering all the pertinent aspects to be analyzed in the assessment of the different ventilators’ systems.

All indicators and domains were judged in terms of performances’ values (Assessment phase) and relative weights (Appraisal phase) through pairwise comparisons. Taking into account the evidence gathered from the literature, together with the professionals’ expertise, after the ventilators’ test period, each member of the working group provided his judgement on the ventilators’ performances (in terms of KPIs) as well as on the relative weights of each assessment element of the decision tree (12). Results of AHP method reveals the global weight of each KPIs, indicating the relevance of each assessment element within the overall evaluation as well as their final performance values, providing a priority ranking of all alternatives technologies in terms of their overall preferences.

Economic evaluation

The economic evaluation was carried out considering all business proposals and comparing all purchase costs. The analysis of Total Cost of Ownership (TCO) has also been carried out. It is a financial estimation focused on helping managers to determine direct and indirect costs of a product or system. This analysis informed the economic and evaluation domain results.

Sensitivity analysis

Finally, a sensitivity analysis was performed to test the stability and the robustness of the alternatives’ ranking. The analysis was conducted comparing the models with the highest performance value, to identify the elements representing source of uncertainty and to determine the impact of this variability on the stability of the assessment results. It was applied to both weights and performance system. Sensitivity analysis was carried out by calculating the minimum changes on values of each criterion needed to reverse the current ranking of alternative technologies (13).

Evidence gathering

Throughout the critical review of the literature, integrated with expert opinion and information from devices’ manufacture’s companies and tenders’ specifications, the working group selected 28 KPIs which represent the main factors for discriminating the technologies under evaluation.

Hierarchy construction

The working group decided to include in the analysis five domains: safety, clinical aspects, costs and economic evaluation, technical characteristics, organizational aspects, whereas Legal, social and ethical domains (which, usually, are included in the Core Model©) were not included because they do not represent discriminant factors between the technologies under evaluation. Each domain was then described by a subgroup of the 28 KPIs previously identified. A decision tree was designed and showed in Fig. 2.

Figure 2: Hierarchical decision tree (Hierarchy construction): domains, Level 1-and Level 2-key performance indicators.

More specifically, as the aim of this HTA process was to compare the performances of four ventilators models, each KPI was described by a number of tenders’ specifications, each of which is measurable and intrinsically provides an effect within the related domain. To give an example, it is believed that better complications rate (patient safety), might be reached if the ventilator’s model guarantees the minute-volume ventilation mode or if it provides the compliance and resistance monitoring system. Detailed information and description of Lev 1 and 2 KPIs are shown in Table 2.

Table 2

provides detailed information and description of Lev 1 and 2 KPIs are shown in Table 2. Each KPI was described by a number of tenders’ specifications, each of which is measurable and intrinsically provides an effect within the related domain.
SAFETY
Patient safety
Complications during ventilation
	Adjustable screen (ergonomics)
	Ventilator display configurability Flow pattern/waveform adjustment
	Number waveforms simultaneously displayed
	Solid line waveforms displayed
	Online help
	Programmable sigh breath function in frequency and amplitude
	Ventilation with minute volume guaranteed
	Resistance and static dynamic compliance monitored
	Lung recruitment tools (PV loops)
	Automated broncho aspiration, pre oxygenation, post oxygenation procedures
	Inspiratory Trigger mechanism
	Expiratory Trigger mechanism
	Apnoea alarm
Necessity of endotracheal intubation
	NIV leak compensation
	Automatic leak compensation, trigger flow mechanism
	Endotracheal and tracheostomy tube compensation
	NIV leak calculations
Complications post ventilation
	Endotracheal cuff pressure control
Technological risks
Ventilation parameters alarms
	Programmable alarms
Technical alarms
Safety mechanisms
	System control for involuntary ventilator settings change
	Confirmation ventilator settings
	Calibration flow and oxygen sensors
	RFID technology
Homogeneity of technologies
Other technological risks for patients
	Adverse events
CLINICAL EFFECTIVENESS
Customization of ventilation
	Frequency and duration of recorded trends
	tracheal P-V loop visualization (with ad without CO2 monitoring)
	Programmable sigh breath function in frequency and amplitude
	APRV independent of the ventilator cycle.
	Proportional assist ventilation (PAV)
	Pressure support mode (PSV)
	Resistance and static dynamic compliance monitored
	Bilevel positive airway pressure (BIPAP)
	NIV leak compensation
	Duration of the inspiratory phase
	Duration of the expiratory phase
	Lung recruitment tools (PV loops)
	Manual and Automated P0. 1 measurement
	Advanced ventilation modes
	inspiratory Trigger mechanism
	Expiratory Trigger mechanism
	Measurement of maximal inspiratory pressure (or NIF)
	NIV leak calculations
Patient comfort
	NIV
Reduction of ventilation time
	Ventilator display configurability Flow pattern/waveform adjustment
	Variable Pressure support mode (PSV)
	Automated weaning systems
COSTS AND ECONOMIC EVALUATION
	Adjustable screen (ergonomics)
	Automated weaning systems
	High flow oxygen therapy
	Nebulization systems integrated into ventilators
	Purchase cost
	Consumables
	Replacement parts
	Technical assistance
TECHNICAL CARACTERISTICS
Features of technology
Parameters visualization
	Adjustable screen (ergonomics)
Technical performances
	Nebulization systems integrated into ventilators
	Automated broncho aspiration, pre-oxygenation, post oxygenation procedures
	Data export to CCE
	Accuracy of ventilator parameters measurements (tidal volume, respiratory rate fraction of inspired oxygen, etc.)
Management
Post-sales services
	Preventive maintenance frequency
	Clinical assistance
	Technical assistance
Training course for operators
	Support service and training maintenance course for operators
ORGANIZATIONAL ASPECTS
Operating principles of technology
Ease of use and ergonomics of technology
	Adjustable screen (ergonomics)
	Online help
	Measurement of percentage levels of oxygen
	Calibration flow and oxygen sensors
	NIV leak compensation
	Automated broncho aspiration, pre-oxygenation, post oxygenation procedures
	High flow oxygen therapy
	Data export to CCE
	Ease of use for main features controls
	Screenshot download
	Nebulization systems integrated into ventilators
	Preventive maintenance frequency
	Support and training maintenance service
	Clinical assistance
	Technical assistance
	Learning curve
	Sterilization management
	First level maintenance (set up, sterilization)
	Tidal Volume min and max
	Availability of transport ventilator with compatible accessories
	Availability of Neonatal ICU ventilators and/or compatible accessories
Impact on existing healthcare system
Availability of beds
	Automated weaning systems

AHP indicators' weights and priorities

AHP results are showed in Table 3 and graphically represented in Figs. 3 and 4. More specifically, the ring plot (Fig. 3), represents the unified weights’ system considering the results of all the involved professionals' judgements, pertaining to the “domains” layer of the decision tree, as gathered from pairwise comparisons and mathematical calculations in accordance with the AHP method. Results showed that the most important aspects to be considered when comparing different ventilators’ models are the safety and the clinical effectiveness reaching respectively 45,95% and 31,47% of the total weight, followed by the organizational aspects (9,51%), technical characteristics (7,80%) and costs & economic evaluation (5,27%). Detailed information about each indicator weight are listed in the Table 3 (2nd column).

Table 3

List of Indicators gathered from literature review, with the relative global weights and performances percentages’ scores.
	Weights	Performance Model 1	Performance Model 2	Performance Model 3	Performance Model 4
		76,74%	89,77%	87,49%	64,04%
SAFETY	45,95%	36,52%	42,36%	41,93%	28,66%
Patient safety / Clinical Risks	32,40%	26,15%	30,04%	29,85%	20,53%
Complications during the ventilation	18,02%	13,07%	17,98%	15,87%	10,73%
Necessity of endo-tracheal intubation	4,68%	3,38%	4,68%	4,32%	2,83%
Complications post-ventilation	9,70%	9,70%	7,37%	9,66%	6,97%
Technological Risks	13,55%	10,37%	12,32%	12,09%	8,13%
Alerts for ventilators’ parameters	4,39%	3,12%	4,39%	3,94%	2,90%
Technical alerts	1,78%	1,31%	1,03%	1,78%	0,56%
Safety working	3,55%	2,58%	3,52%	3,02%	1,99%
Homogeneity of technology	2,71%	2,69%	2,71%	2,23%	2,23%
Other technological risks for the patient	1,12%	0,67%	0,67%	1,12%	0,45%
CLINICAL EFFECTIVENESS	31,47%	23,93%	30,29%	29,06%	20,00%
Customization of ventilation	8,61%	6,47%	8,44%	7,75%	5,51%
Patients comfort	6,21%	4,15%	6,21%	5,01%	3,77%
Reduction of ventilation time	8,12%	5,67%	7,94%	7,77%	4,32%
Reduction of weaning time	8,53%	7,64%	7,70%	8,53%	6,40%
COSTS	5,27%	2,79%	1,03%	1,24%	5,27%
Total Cost of Ownership	5,27%	2,79%	1,03%	1,24%	5,27%
TECHNICAL CHARACTERISTICS	7,80%	6,42%	7,44%	6,79%	4,45%
Features of technology	2,61%	1,85%	2,61%	2,23%	1,59%
Parameters visualization	1,42%	0,94%	1,42%	1,18%	0,80%
Technical performances	1,19%	0,91%	1,19%	1,04%	0,80%
Management of the technology	5,19%	4,57%	4,83%	4,57%	2,86%
Post-sales service	2,60%	1,97%	2,23%	1,97%	1,30%
Training course for operators	2,60%	2,60%	2,60%	2,60%	1,56%
ORGANIZATIONAL ASPECTS	9,51%	7,08%	8,65%	8,47%	5,66%
Operating principles of technology	6,10%	4,02%	5,57%	5,06%	3,10%
Ease of use and ergonomics of technology	1,81%	1,32%	1,76%	1,55%	1,04%
Maintenance aspects	0,63%	0,43%	0,63%	0,55%	0,41%
Versatility to use the technology with pediatric patients	2,85%	1,79%	2,85%	2,46%	1,50%
Homogeneity of devices	0,81%	0,47%	0,33%	0,50%	0,16%
Impact on existing healthcare system	3,41%	3,06%	3,08%	3,41%	2,56%
Availability of beds	3,41%	3,06%	3,08%	3,41%	2,56%

Figure 3: Ring plot illustrating the unified weights’ system (which represents the percentage level of importance of the various domains with respect to the overall evaluation) pertaining to the “domains” layer of the decision tree, as gathered from pairwise comparisons and mathematical calculations in accordance with the AHP method).

Figure 4: Histogram chart specifying the computed performance (global and per domain) of Intensive care ventilators. The ventilators were compared with their alternatives with respect to every lowest indicator according with the AHP method.

More specifically, within the safety domain the “patients’ safety” weights twice the “technological risks”. The “patient safety” KPI, indeed, covers the 70% of the weight of the domain. It is most represented by the “complication during ventilation” that doubled the weight of “Complication post ventilation”.

The “alert for ventilators’ parameters” represents the most critical KPI regarding the “technological risks”. The others KPIs (safety mechanisms, risk related to the heterogeneity of ventilators models and the adverse events/recalls) assume more or less the same relevance within the evaluation.

The clinical effectiveness domain, which represents the second domain in order of importance, is described by a number of technical specifications which more or less equally affect the clinical outcomes such as the possibility to customize the ventilation, patients comfort related to the availability of non-invasive ventilation mode, the reduction of ventilation time and the weaning time.

The technical characteristics domain was mainly described by different aspects of the technology management and by a number of specifications emerged from technical documentation of different devices’ producers.

Regarding the organizational aspects, the impact that different operating principles of technologies have on the organization doubled the “interface with other healthcare systems or technologies”. More specifically, the former includes the analysis of the “ease of use and ergonomics of technology”, the “maintenance”, the “versatility to use the technology with pediatric patients” and the availability of homogeneous technologies, compared to the latter that has been measured in terms of availability of beds where the technologies could be installed.

Finally, results on economic evaluation, aimed to compare all business proposals considering all purchase costs and hypothesizing 10-year life-cycle cost estimate per each ventilator. Table 3, as an example showed the comparison within the four models.

The histogram chart (Fig. 4), instead, specifies the computed performance (global and per domain) of the different ventilator models (Models 1, 2, 3 & 4). According to AHP (12, 14)[12, 14][12, 14] performances were expressed as percentages and derived from the elaboration of the pairwise comparisons made by all professionals involved.

As shown in the Fig. 4, the Model 2 seems to be the best option comparing with the other Models. The final performance value of the Model 2 has led 2.3% over Model 3 (89,76% vs. 87,49%) and 13% over Model 1 (89,76% vs. 76,74%). Table 3 (3rd, 4th, 5th and 6th columns) gives detailed information about each indicator performance value for the four Models presented.

Regarding the safety aspects, described by patient’s safety and the technological related risks, nevertheless the Model 2, seems to be safer than the others, reaching 42.36% performances score, it does not significantly differ from Model 3 overall safety performances score (41.93%), whereas the Model 1 and 4 performed worst.

It resulted that Model 2 provides several technical functions that, limiting the complication during ventilation, improve the patient safety, including trigger mechanisms, apnea alarm, resistance and static dynamic compliance monitoring, lung recruitment tools (PV loops), higher number waveforms simultaneously displayed and adjustable screen for ergonomics. It emerged that the necessity for endotracheal intubation is reduced if the ventilator model provides compensation for large leaks during NIV, trigger flow mechanism and tube compensations. All these aspects are better represented by Model 2 that is able to provide the best performances with respect of these technical characteristics. Model 1 provides control of optimal cuff pressure during the entire ventilation period, helping to prevent the complications post-ventilation such as ventilator associated pneumonia (VAP) and tracheal injuries.

To reduce the technological risks, Model 2 provides more programmable alarms than the other models as well as a system control for potentially dangerous involuntary changes to ventilation settings.

In relation to the clinical effectiveness, the Fig. 4 shows that the Model 2 outperforms the other Models even though performances score is quite like that reached by Model 3. Model 2 offers a better customization of ventilation providing a number of advanced ventilation modes and innovative technical characteristics aiming at reducing asynchronies that are a frequent issue in ventilated patients. As the better performant, regarding the customization of the ventilation, Model 2 provides the continuous and reliable measurement of static compliance of the lung and thorax, the automatic trigger mechanism, lung recruitment tools to easily and safely perform lung recruitment maneuvers, the possibility to program frequency and amplitude of sigh breath function, etc... Model 2, throughout advances in variable Pressure support ventilation mode (PSV), improves synchrony and reduces the ventilation time. Model 3, instead, with automated weaning modes provide the possibility to reduce weaning time.

The technical characteristics domain was described by a number of specifications emerged from technical documentation of different devices’ manufacturers and by different aspects of the technology management. Even regarding this domain, Model 2 resulted the best performant (Fig. 4). In detail, as mentioned before, Model 2 provides more advanced technology features to make easier and more intuitive the visualization of information during ventilation such as adjustable screen and the possibility to simultaneously visualize more than one curve or loop. Model 2 also ensures high level accuracy and the availability of a wide measure range of monitored parameters (e.g., tidal volume, respiratory rate fraction of inspired oxygen, etc..). Moreover, regarding the post-sales service, while the manufacturer of Model 2 offers higher preventive maintenance frequency and continuous clinical and technical assistance than the others, the manufacturers of all four ventilators models offer the same conditions in terms of support services and training for biomedical technicians.

Organizational aspects analyze different aspects of the management of the technology and the impact of its introduction on the existing workflows.

Model 2 also performs better considering the ease of use and the ergonomics of the technology, providing some technical characteristics (i.e. the adjustable screen and the possibility to simultaneously visualize more than one curve or loop technology) that make the ventilator more usable and enhance the human-machine interface aspects. Moreover Model 2 is also the more versatile model for the pediatric use, because of the wider range of tidal volume.

Finally, results on economic evaluation, as showed in Table 4, aimed to compare all business proposals considering all purchase costs and hypothesizing 10-year life-cycle cost estimate per each ventilator. From this analysis it resulted the Model 4 as the best option with a total cost of ownership of about € 30.742,66. The highest performance score, indeed, is attributed to Model 4 (the most cheaper) with 5.27% in comparison to Model 1, 3 and 2 (2.79%, 1.24% 1.03%, respectively) (Fig. 4).

Table 4

Economic evaluation: Total Cost of ownership.
	Model 1	Model 2	Model 3	Model 4
Purchase cost	€ 23.395,08	€ 25.479,24	€ 27.414,91	€ 13.783,34
Replacement parts	€ 5.686,90	€ 1.750,00	€ 5.052,00	€ 5.198,00
Consumable items	-	-	€ 5.395,50	-
Maintenance Costs (Full Risk)	€ 14.972,85	€ 10.701,28	€ 9.650,05	€ 11.761,32
Total cost of ownership	€ 44.054,83	€ 37.930,53	€ 47.512,46	€ 30.742,66

Sensitivity analysis

Sensitivity analysis was carried out comparing the models with the highest final performances value to test the stability and reliability of the solution. The Model 2 seems to be the best option, even if its total performances score is very close to that of Model 3 (89.76% vs. 87.49%, respectively). The more similar the different models’ final scores are, the higher the probability to reverse the final solution. As the final performances value of the Model 2 has led only 2.3% over Model 3, the sensitivity analysis was carried out between these two models.

Sensitivity analysis results on performances showed that to reverse the current ranking of the two different models, at least one of the following conditions have to be verified:

A 3% reduction of Model 2 safety performance and a simultaneous 3% increase of Model 3 safety performance.
A 5% reduction of Model 2 clinical effectiveness performance and a simultaneous 5% increase of Model 3 clinical effectiveness performance.

Results showed the instability of this solution. A minimum variation in the evaluation parameters final scores (3% in safety and 5% in clinical effectiveness), might reverse the final decision. It can result in a substantial equivalence between the Model 2 and Model 3.

Results on weights, instead, showed that to reverse the current ranking of different models a 3.5-fold increase of cost and economic evaluation weight and a simultaneous 14% reduction of the other dimensions’ weight should be occurred. It revealed, instead, a greater stability of the solution from the weight system point of view because such level of importance to costs, instead of organizational or clinical effectiveness aspects, has been considered inappropriate by those responsible for the decision making process in our hospital.

This project was commissioned by the hospital management for the renewal of all hospital’s ventilators.

For these reasons, an HTA process was carried out in order to identify the best suitable model of critical care ventilator available on the market by comparing four different models.

The Do-HTA method (10) was used to yield a structured outcome able to support the decision-making process concerning the final decision on choosing the best technology to be adopted. Therefore, evidence gathered from the literature review and professionals’ expertise and judgments per each unit, with their experience and knowledge, were integrated as results of Do-HTA method.

An important point of this study was the identification of KPIs useful to compare the ventilator models and the construction of the decisional hierarchy tree. The main informative indicators we found were about safety, clinical effectiveness, technical characteristics, cost & economic evaluation, and organizational aspects. The safety and the clinical effectiveness have been defined as the most important domains within the assessment. They covered more than 75% weight of the total evaluation.

As regards the safety domain, the most important indicator into the patient safety (Lev-1 KPI) is the “complication during the ventilation”, followed by the “complication post-ventilation” and “necessity of endo-tracheal intubation”. Data from literature review showed that an inappropriate use of ventilators may cause complication during ventilation called ventilator-induced lung injury (15). To avoid that, several innovative technical functionalities have been improved, including trigger mechanisms, apnea alarm, resistance and static dynamic compliance monitoring, lung recruitment tools (PV loops), higher number waveforms simultaneously displayed and adjustable screen for ergonomics. Regarding the complication post-ventilation, a recent prospective cohort study showed that the hospital mortality occurrence is related to the incidence of postoperative lung complications (16), even the ventilator-associated pneumonia (VAP), which is common for pediatrics’ patients in ICU, significantly affects the morbidity and mortality of pediatric patients (17) (18) (19). The evaluation of this KPI (complication post-ventilation) has been conducted considering the management of endotracheal tube cuff pressure, representing an important element for the prevention of ventilator-associated pneumonia in patients receiving mechanical ventilation.

Some studies showed that endotracheal intubation-related complications are likely to occur in intubated pediatric patients. (20). This issue might be avoided if the ventilator model is able to optimize the NIV (non-invasive ventilation) operation, which is increasingly being used in pediatric units (21). Some studies showed that the use of NIV in pediatric patients, provides a rapid improvement of the respiratory acute injury, avoiding the requirement for endotracheal intubation reducing the related complications (22) (23).

In relation to the technological risks indicator, based on our analysis, the “alerts for ventilators parameters”, resulted to be the most important one. It could be observed, according to the literature and even after the evaluation process, that the patient safety could be, even fatally, compromised, when a situation requiring an alarm activation, is not recognized by the ventilator (21). The “technical alarms” KPI is referred to the potential issues arising from ventilator’s malfunctions (24) that might be overcome by the availability of programmable alarms. Ventilators also may have safety mechanisms including control system for involuntary ventilator settings changes, calibration of flow and oxygen sensors and RFID technology (25).

The homogeneity of technologies resulted a pivotal aspect in the evaluation. Ensuring that as many ventilators as possible belong to an identical make and model, reduces the learning curve of operators as well as the risk of errors. Finally, while choosing a ventilator model it is important to be aware of the related adverse events and/or recalls. These aspects are evaluated into the “other technological risks for patients”.

Comparing the performances of the four ventilators models with respect of these safety indicators, results showed that the Model 2 seems to be the safest one between the four technologies compared.

Evidence helped us to find out more about the clinical effectiveness domain and its indicators. Its Lev-2 KPIs are: “customization of ventilation”, “patients’ comfort”, “reduction of ventilation time”, “reduction of weaning time”. According to the literature and based on the assessment, customized patient-ventilator interface may be useful in long-term ventilation, improving the patient-machine interaction. A number of advanced ventilation modes and innovative technical specifications (resistance and static dynamic compliance monitoring, trigger mechanism, lung recruitment tools -PV loops-, programmability of frequency and amplitude of sigh breath function) guarantee the customization of ventilation. Literature review also showed that the improvement of patient/device interaction is pivotal for a better patient comfort as well as for the reduction of ventilation time. Optimizing the comfort for the patient represents also an important goal (26). The duration of mechanical ventilation has to be as short as possible and to this end, in case of neonatal patient, the ventilator system, throughout the variable support pressure ventilation mode, should provide a synchronous interaction with the infant (27). The optimization of the duration of ventilation time might reduce the incidence of post-operative complications and can improve patients’ clinical outcomes (28). Finally, the reduction of weaning time is another important aspect to be considered in the choice of the most appropriate ventilator model. Ventilators with automated weaning modes can overcome the problem that clinicians might under-recognize the patient's ability to breathe without assistance, prolonging ventilation time and increasing the complications incidence (29).

The technical characteristics domain has been characterized by several technical specifications gotten from the different devices’ manufactures and from some websites as ECRI and similar others. Technology features refer to the ventilation parameters visualization (adjustable screen, the possibility to simultaneously visualize more than one curve or loop, ease to visualize information during ventilation) and to other technical aspects as the availability of ventilators accessories and consumables such as nebulizer, the possibility to export data to EHR as well as accuracy and the availability of a wide measure range of monitored parameters. Moreover, also the management of the technology plays a crucial role, considering the importance attributed to the post-sales service, maintenance, and the training course for the users.

In relation to the cost and economic evaluation, all business proposals have been analyzed. In this way it was possible to make an evaluation of the economic trend relative to the purchase of all consumables for the use of the device. The analysis of total cost of ownership has been carried out (30). A detailed analysis of all service intervals and preventive maintenance requirements per each ventilator model was carried out. It was also looked for pricing information for consumables, repair parts, preventive maintenance kits. Based on the information collected and on discussions with companies, health professionals, economists, and biomedical experts, it was estimated the annual cost of ownership value per each ventilator. At this stage, a 10-year life-cycle cost estimate per each ventilator was calculated. Results showed that Model 2 resulted more expensive than the others, because of its peculiarities.

Our analysis ended with the organizational aspects domain. Such analysis helped us to make a prevision of the impact that the deployment of new ventilators might have on existing healthcare systems (1). In this domain it was possible to enhance two aspects strictly related to the selection of the ventilator model that best meets the hospital needs: the ventilator operating principles that affects the organizational aspects and the impact of the technology on the existing workflows. As for the former, KPIs included the “ease of use and ergonomics of the technology” (technical characteristics that make the ventilator more usable and enhance the human-machine interface aspects, affecting the mental workload), the “maintenance aspects”, the “versatility to use the technology with pediatric patients” due to the possibility to arbitrarily set the minimum and maximum Tidal Volume values and the “homogeneity of devices” related to the availability of portable and or neonatal ventilators of the same manufacturer or the availability of compatible accessories, to be used in the same department or in the whole hospital. The latter aspect referred to the availability of ICU beds and to the possibility to decrease the length of ICU stay if a reduction of the mechanical ventilation time occurred (31). This aspect was carefully taken into account and it was associated to the ventilator model’s availability of advanced weaning ventilation modes. However, the literature review results showed no significant evidence associating with the reduction of weaning ventilation time and/or ICU stay when these advanced ventilation modes are used (32, 33).

To sum it all up, from our analysis it can be summarized that the safety (in particular that one relative to the patient safety) and the clinical effectiveness indicators resulted as the most important aspects.

The evaluation of the performance value relative to all parameters, as above described, has led to define a rank between the four models considered in the analysis. Model 2 reached the best performance value, followed by the other models. However, from the economic analysis and from a performances values point of view, even though Model 2 resulted more expensive than the others, because of its peculiarities, it may be considered as a great added value for the Hospital.

As presented in the results section, a substantial equivalence can be observed between Model 2 and 3 performance values: Model 2 has led, indeed, only 2.3% over Model 3. For this reason, the sensitivity analysis was carried out and it emerges that the final ranking is not robust, because with a minimum parameter variation (3% in safety performances and 5% in clinical effectiveness performances), the final decision can be inverted, confirming a substantial equivalence of the two products. From the weight system point of view, instead, results revealed a substantial stability of the final ranking. It means that the weight system elaborated is robust and coherent with the decision problem even if some changes occurred in the professionals’ judgements on the relative importance of the evaluation criteria.

Do-HTA method, therefore, has permitted to evaluate the performances of the selected ventilators, providing reliable evidence about the four ventilators models and therefore supporting the hospital decision makers.

This paper provided a general overview of the most relevant features to consider in the assessment of intensive care ventilators for pediatric patients. It discussed the main characteristics of these technologies as well as the potential implications of introducing their different innovations (proposed by the manufacturers involved in this study) into the pediatrics ICU/Emergency units. The Hospital Decision Makers, after the HTA process which has entailed evaluating the weights’ system, averaging departments’ preferences, and performing sensitivity analysis, rationally decided to conscientiously and objectively assign the rank of best performer to the Model 2.

Our evidence-based approach, identifying and discussing the pros and cons of innovative features of ventilators and all the evaluation criteria and aspects that be considered when a new ventilator model has to be acquired, can be deemed as a valuable proof of evidence that conducting an HTA process before the acquisition of a technology is essential to make the best, most rational and reliable decision. This paper also provides detailed information on the methodology used to carry out the assessment. Thanks to sensitivity analysis, results appear reliable and, if accurately adapted, they could be transfer to similar hospital contexts.

HTA: Health Technology Assessment

Do-HTA: Decision Oriented Health Technology Assessment

KPIs: Key Performance Indicators; ICU: Intensive Care Unit

NIV: Non-invasive ventilation

HFNC: High flow nasal cannula oxygen

MCDA: Multiple Criteria Decision Analysis

AHP: Analytic Hierarchy Process

VAP: Ventilator-associated pneumonia.

TCO: Total Cost of Ownership

Acknowledgments

We thank all professionals for their contribution during the different phases of the HTA study. In particular, we would like to thank all of them, who, with their assistance, helped us to conduct this study answering to face-to-face interviews and filling out all needed questionnaires for the project.

Funding

Not Available.

Consent for Publication

Not applicable.

Availability of data and materials

The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Authors’ contributions

GT, DP, and RB gave, as doctors, their clinical contribution to the HTA Project and provided a critical revision to the manuscript. MR coordinated the entire HTA project and reviewed the manuscript. FF supported MR in coordination of HTA process and acquired data needed for the analysis. MA performed MCDA and statistical analyses and drafted the manuscript. FN and CC analyzed the technical features. LP reviewed the manuscript. PD and NP were the technical and clinical referring, respectively. All authors have read and approved the final version of the manuscript.

Ethics approval and consent to participate

Not Available.

Competing interests

The authors declare that they have no conflicts of interest.

1. Lellouche F, Brochard L. Advanced closed loops during mechanical ventilation (PAV, NAVA, ASV, SmartCare). Best Pract Res Clin Anaesthesiol. 2009 Mar;23(1):81-93.

2. Morato JB, Sakuma MT, Ferreira JC, Caruso P. Comparison of 3 modes of automated weaning from mechanical ventilation: a bench study. J Crit Care. 2012 Dec;27(6):741 e1-8.

3. Petter AH, Chiolero RL, Cassina T, Chassot PG, Muller XM, Revelly JP. Automatic "respirator/weaning" with adaptive support ventilation: the effect on duration of endotracheal intubation and patient management. Anesth Analg. 2003 Dec;97(6):1743-50.

4. Iodice F, Testa G, Amodeo A, Cogo P. Inappropriate use of neurally adjusted ventilator assist. BMJ Case Rep. 2012 Sep 3;2012.

5. Morley SL. Non-invasive ventilation in paediatric critical care. Paediatr Respir Rev. 2016 Sep;20:24-31.

6. Dasta JF, McLaughlin TP, Mody SH, Piech CT. Daily cost of an intensive care unit day: the contribution of mechanical ventilation. Critical care medicine. 2005;33(6):1266-71.

7. Zilberberg MD, Shorr AF. Prolonged acute mechanical ventilation and hospital bed utilization in 2020 in the United States: implications for budgets, plant and personnel planning. BMC health services research. 2008;8(1):1-10.

8. Moerer O, Plock E, Mgbor U, Schmid A, Schneider H, Wischnewsky MB, et al. A German national prevalence study on the cost of intensive care: an evaluation from 51 intensive care units. Critical Care. 2007;11(3):1-10.

9. Cox CE, Carson SS, Govert JA, Chelluri L, Sanders GD. An economic evaluation of prolonged mechanical ventilation. Critical care medicine. 2007;35(8):1918.

10. Ritrovato M, Faggiano FC, Tedesco G, Derrico P. Decision-Oriented Health Technology Assessment: One Step Forward in Supporting the Decision-Making Process in Hospitals. Value Health. 2015 Jun;18(4):505-11.

11. HTA core model for medical and surgical interventions. European Network for Health Technology Assessment (EUnetHTA).

12. Saaty TL. The Analytic Hierarchy Process, Planning, Priority Setting, Resource Allocation. 1980.

13. Triantaphyllou E. A Sensitivity Analysis Approach for Some Deterministic Multi-Criteria Decision-Making Methods. 1997.

14. Saaty TL. Decision making with the analytic hierarchy process. Int J Services Sciences 2008;1(1):83-98.

15. Sutherasan Y, Vargas M, Pelosi P. Protective mechanical ventilation in the non-injured lung: review and meta-analysis. Crit Care. 2014 Mar 18;18(2):211.

16. Slutsky AS. History of Mechanical Ventilation. From Vesalius to Ventilator-induced Lung Injury. Am J Respir Crit Care Med. 2015 May 15;191(10):1106-15.

17. Wright ML, Romano MJ. Ventilator-associated pneumonia in children. Semin Pediatr Infect Dis. 2006 Apr;17(2):58-64.

18. Iosifidis E, Stabouli S, Tsolaki A, Sigounas V, Panagiotidou EB, Sdougka M, et al. Diagnosing ventilator-associated pneumonia in pediatric intensive care. Am J Infect Control. 2015 Apr 1;43(4):390-3.

19. Aelami. Ventilator-associated pneumonia in neonates, infants and children. Antimicrobial Resistance and Infection Control. 2014.

20. Lizy C, Swinnen W, Labeau S, Poelaert J, Vogelaers D, Vandewoude K, et al. Cuff pressure of endotracheal tubes after changes in body position in critically ill patients treated with mechanical ventilation. Am J Crit Care. 2014 Jan;23(1):e1-8.

21. Teague WG. Noninvasive ventilation in the pediatric intensive care unit for children with acute respiratory failure. Pediatr Pulmonol. 2003 Jun;35(6):418-26.

22. Chen CW, Wu CP, Dai YL, Perng WC, Chian CF, Su WL, et al. Effects of implementing adaptive support ventilation in a medical intensive care unit. Respir Care. 2011 Jul;56(7):976-83.

23. Mas A, Masip J. Noninvasive ventilation in acute respiratory failure. Int J Chron Obstruct Pulmon Dis. 2014;9:837-52.

24. Top 10 Health Technology Hazards for 2016 - A Report from Health Devices November 2015. 2016.

25. Teague WG. Noninvasive ventilation in the pediatric intensive care unit for children with acute respiratory failure. Pediatric pulmonology. 2003;35(6):418-26.

26. Pavone M, Verrillo E, Caldarelli V, Ullmann N, Cutrera R. Non-invasive positive pressure ventilation in children. Early Hum Dev. 2013 Oct;89 Suppl 3:S25-31.

27. Cheifetz IM. Invasive and noninvasive pediatric mechanical ventilation. Respir Care. 2003 Apr;48(4):442-53; discussion 53-8.

28. Ball L, Dameri M, Pelosi P. Modes of mechanical ventilation for the operating room. Best Pract Res Clin Anaesthesiol. 2015 Sep;29(3):285-99.

29. Morato JB, Sakuma MT, Ferreira JC, Caruso P. Comparison of 3 modes of automated weaning from mechanical ventilation: a bench study. Journal of critical care. 2012;27(6):741. e1-. e8.

30. Estimating Cost of Ownership for Intensive Care Ventilators. 2014.

31. Lellouche F, Brochard L. Advanced closed loops during mechanical ventilation (PAV, NAVA, ASV, SmartCare). Best practice & research Clinical anaesthesiology. 2009;23(1):81-93.

32. Rose L, Schultz MJ, Cardwell CR, Jouvet P, McAuley DF, Blackwood B. Automated versus non-automated weaning for reducing the duration of mechanical ventilation for critically ill adults and children: a cochrane systematic review and meta-analysis. Critical Care. 2015;19(1):1-12.

33. Sutherasan Y, Vargas M, Pelosi P. Protective mechanical ventilation in the non-injured lung: review and meta-analysis. Annual Update in Intensive Care and Emergency Medicine 2014. 2014:173-92.

Additionalfile.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Multi-criteria Decision Analysis for Health Technology Assessment of Intensive Care Ventilators for Pediatric Patients.

Status:

Version 1

Abstract

Figures

Background

Methods

Economic evaluation

Sensitivity analysis

Results

Evidence gathering

Hierarchy construction

AHP indicators' weights and priorities

Sensitivity analysis

Discussion

Conclusion

Abbreviations

Declarations

References

Supplementary Files

Status:

Version 1