The Statistical Approach in Trial-based Economic Evaluations Matters: Get your Statistics Together!

doi:10.21203/rs.3.rs-134947/v1

Download PDF

Research Article

The Statistical Approach in Trial-based Economic Evaluations Matters: Get your Statistics Together!

https://doi.org/10.21203/rs.3.rs-134947/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background: Baseline imbalances, skewed costs, the correlation between costs and effects, and missing data are statistical challenges that are often not adequately accounted for in the analysis of cost-effectiveness data. This study aims to illustrate the impact of accounting for these statistical challenges in trial-based economic evaluations.

Methods: Data from two trial-based economic evaluations, the REALISE and HypoAware studies, were used. In total, 14 full cost-effectiveness analyses were performed per study, in which the four statistical challenges in trial-based economic evaluations were taken into account step-by-step. Statistical approaches were compared in terms of the resulting cost and effect differences, ICERs, and probabilities of cost-effectiveness.

Results: In the REALISE study and HypoAware study, the ICER ranged from 636,744€/QALY and 90,989€/QALY when ignoring all statistical challenges to -7,502€/QALY and 46,592€/QALY when accounting for all statistical challenges, respectively. The probabilities of the intervention being cost-effective at 0€/ QALY gained were 0.67 and 0.59 when ignoring all statistical challenges, and 0.54 and 0.27 when all of the statistical challenges were taken into account for the REALISE study and HypoAware study, respectively.

Conclusions: Not accounting for baseline imbalances, skewed costs, correlated costs and effects, and missing data in trial-based economic evaluations may notably impact results. Therefore, when conducting trial-based economic evaluations, it is important to align the statistical approach with the identified statistical challenges in cost-effectiveness data. To facilitate researchers in handling statistical challenges in trial-based economic evaluations, software code is provided.

Health Economics & Outcomes Research

Health Policy

cost-benefit analysis

methodology

clinical trial

multiple imputation

baseline imbalances

skewed costs

Previous research shows that the statistical quality of many trial-based economic evaluations is poor.
This study evaluates the impact of simultaneously accounting for various statistical challenges on the results of trial-based economic evaluations.
Ignoring these statistical challenges in trial-based economic evaluations may notably impact the results.
It is important to avoid misalignment between data characteristics and the statistical approach.
To facilitate researchers in handling statistical challenges in trial-based economic evaluations, software code is provided. Stata syntax is provided as supplementary file 1.

Economic evaluations aim to inform resource allocation decisions in healthcare by evaluating whether the additional health benefits of an intervention justify its additional costs[1]. In many countries, economic evaluations are increasingly being accepted as a formal decision criterion for the reimbursement of pharmaceuticals and other healthcare technologies[2, 3]. Consequently, there is growing interest in economic evaluations of healthcare interventions.

Although great improvements in the conduct and reporting of economic evaluations along clinical trials have been made in previous years[4], literature shows that the quality of the applied statistical methods is typically far from optimal[4–6]. Often, baseline imbalances, the skewed nature of cost data, and the correlation between costs and effects are not adequately accounted for[7, 8]. Additionally, missing data are frequently handled using “naïve” imputation methods, such as mean imputation and last observation carried forward[7–11]. Failure to appropriately account for these statistical issues when analyzing trial-based economic evaluations is of great concern, because use of inadequate statistical methods may lead to biased results, and consequently invalid decisions resulting in a potential waste of scarce resources[12].

In recent years, various studies have been performed on how to deal with baseline imbalances, skewed costs, correlated costs and effects, and missing data in trial-based economic evaluations[8–10, 13–18]. A more detailed description of these statistical issues and how to deal with them can be found in Box 1[19]. Although previous studies have investigated how to deal with these statistical challenges in trial-based economic evaluations separately, the impact of simultaneously accounting for these statistical challenges on the results of trial-based economic evaluations has not yet been explored. The current study aims to address this gap in knowledge, by analyzing data from two previous trial-based economic evaluations, the REALISE and HypoAware studies, whilst step-by-step taking into account the aforementioned statistical challenges in the analysis of the data.

Box 1 Statistical challenges in trial-based economic evaluations
1) Baseline imbalances
It is commonly assumed that the random allocation of participants in trial-based economic evaluations ensures that observed and non-observed characteristics are well-balanced across study groups. Nevertheless, some between-group differences in baseline values and/or important prognostic factors regularly occur[20]. Failure to account for such baseline imbalances will likely lead to biased results[20, 21]. Various methods have been suggested to account for baseline imbalances in trial-based economic evaluations, including mean difference adjustment, regression-based adjustment, and matching methods[14, 16, 20–23].
2) Skewed costs
Costs are generally right-skewed as there are relatively few participants with (very) high costs and it is impossible to incur negative costs. Consequently, the assumption of normality of standard parametric tests, such as t-tests and linear regression analyses, is violated[4, 12]. Although the normality assumption is violated by the skewed nature of costs, if the sample size is large enough the central limit theorem ensures that sample means will be normally distributed and standard parametric statistical methods may be used[24]. Log-transformations and standard non-parametric tests (e.g., Mann-Whitney U) are unsuitable for trial-based economic evaluations, since both methods fail to provide an estimate of the mean difference in costs, whereas this is required by decision-makers to allow for estimations of the total budget impact of a new intervention[1, 8, 25, 26]. Suggested methods that are suitable to account for the skewness of costs, while simultaneously comparing mean costs are non-parametric bootstrapping and generalized linear models assuming distributions that fit the data best (e.g. Gamma, Log-Normal and Inverse-Gaussian)[1, 4, 8, 12, 15, 25–29].
3) Correlated costs and effects
Costs and effects are typically correlated and hence their correlation should be accounted for[30–32]. Proposed methods include non-parametric bootstrapping, which can account for the correlation between costs and effects by resampling them in pairs[10], and seemingly unrelated regression (SUR), in which two separate regression models are specified simultaneously; i.e. one for costs and one for effects. In SUR, the correlation between costs and effects is accounted for through correlated error terms[33, 34].
4) Missing data
In clinical trials, missing data are common[1, 25, 26, 35]. This is of great concern for trial-based economic evaluations, because total costs are calculated as the sum of several cost components measured at different time points. If one resource use item or one time point is missing, total costs will also be missing[1, 25, 26]. According to Rubin[35], missing data can be classified in three mechanisms. First, if missing values are not dependent on any observed or unobserved variable, data are said to be missing completely at random (MCAR). Second, when missing values are related to one or more observed variables, but not the missing value itself, data are said to be missing at random (MAR). Third, when missing data depends on the missing values itself, data are said to be missing not at random (MNAR). In case of MAR and MNAR, bias may be introduced when analyses are restricted to complete cases or when naïve imputation methods, such as mean imputation or last observation carried forward, are used[1, 25, 26, 36]. Only, when missing data can validly assumed to be MCAR or the proportion of missing data is low (i.e. <5%), naïve imputation methods may be used[37, 38]. For all other situations, “naïve” imputation methods are discouraged, because - amongst others - they do not account for the uncertainty related to imputing missing values[9, 11, 39–41]. More advanced methods to account for missing data assuming MAR in trial-based economic evaluations include multiple imputation and statistical models with maximum likelihood estimation[9, 11, 39, 40].

2.1 Data

To evaluate the impact of whether or not accounting for baseline imbalances, skewed costs, correlated costs and effects, and missing data in trial-based economic evaluations, empirical data from two previously published trial-based economic evaluations were used, the REALISE and HypoAware study.

REALISE study

In the Rehabilitation After Lumbar disc Surgery (REALISE) study, early rehabilitation after lumbar disc surgery was compared to no referral after lumbar disc surgery among 169 participants (intervention group: n = 92; control group: n = 77). Resource use was measured from a societal perspective at 6, 12 and 26 weeks follow-up using cost questionnaires[42]. Resource use was valued using Dutch standard costs[43]. Utility values were based on the EuroQol (EQ-5D-3L), which was administered at baseline and 3, 6, 9, 12 and 26 weeks follow-up[42]. Utility values were estimated using the Dutch tariff for the EQ-5D-3L[44]. Quality-adjusted life years (QALYs) were calculated using linear interpolation between measurement points.

HypoAware study

In the HypoAware study, the HypoAware intervention (a blended, group and online psycho-educational intervention based on the evidence-based Blood Glucose Awareness Training) was compared to usual care among 137 participants (intervention group: n = 71; control group: n = 66)[45]. Resource use was measured from a societal perspective at 2, 4, and 6 months follow-up using cost questionnaires. Utility values were based on the EuroQol (EQ-5D-5L), which was administered at baseline, 2, 4, and 6 months follow-up. Utility values were estimated using the Dutch tariff for the EQ-5D-5L[46]. Quality-Adjusted Life-Years (QALYs) were calculated using linear interpolation between measurement points.

Tables describing baseline characteristics of the REALISE and HypoAware study populations are included in the Appendix (Supplementary Tables 1 and 2). For a detailed description of both studies, the reader is referred elsewhere[42, 45-47].

2.2 Statistical analysis

In total, 14 full economic evaluations were performed for both the REALISE and HypoAware study. In the first analysis, a statistical approach was used, in which baseline imbalances, the skewed nature of cost data, the correlation between costs and effects and missing data were ignored. Thus, this approach simply compared the difference in costs and effects between both groups using t-tests, including only participants with complete cost and effect data, while assuming that both costs and effects were normally distributed and that costs and effects were not correlated. Although this statistical approach ignores all of the challenges in trial-based economic evaluations, it is still being used in practice[7, 8, 48]. Step-by-step, the analyses accounted for the different statistical challenges, until in the final approach all of the statistical challenges were accounted for using the following methods:

- Baseline imbalances:

Regression-based adjustment was used[16, 49, 50]. Costs and effects were corrected for their baseline value, if available, and for relevant confounding variables. Variables were considered to be a confounder if the estimated regression coefficients for the cost or effect differences changed by 10% or more when the possible confounding factor was added to the model[50, 51]. For the REALISE study, confounders of costs were participants’ baseline mental health status, physical health status, risk of future work disability, fear-avoidance beliefs about work, treatment credibility and treatment expectations. Confounders of effects included the participants’ baseline utility value, mental health status, back pain, and risk of future work disability. For the HypoAware study, confounders of costs were the participants’ baseline costs, number of severe hypoglycemia episodes during the previous 6 months, and wearing a real-time sensor. Confounders of effects comprised the participants’ baseline utility value and marital status.

- Skewed costs:

Non-parametric bootstrapping with 5,000 replications was used[33, 52-54]. The non-parametric bootstrap is a data-based resampling method to estimate statistical uncertainty, without making any distributional assumptions[52]. Bootstrapped confidence intervals were estimated using the bias-corrected and accelerated bootstrap method. The advantage of using bias-corrected and accelerated bootstrapping over percentile bootstrapping, is that it adjusts better for skewness and bias of the sampling distribution, resulting in more accurate confidence intervals[52, 55]. In the REALISE study, the skewness of costs was 1.70 and the kurtosis was 5.75 (excess kurtosis 2.75). In the HypoAware study, the skewness of costs was 1.39 and the kurtosis was 3.90 (excess kurtosis 0.90). The positive skewness indicates that the distribution is skewed to the right and the excess kurtosis indicates a long right tail (i.e. relatively many outliers).

- Correlation between costs and effects:

Seemingly unrelated regression (SUR) analysis was used in which two separate regression models were specified simultaneously (i.e. one for costs/one for effects)[33, 34]. In the REALISE study, the correlation between costs and effects was ρ = -0.42. In the HypoAware study, the correlation between costs and effects was ρ = -0.44. A negative correlation indicates that individuals with worse outcomes induce higher costs.

- Missing data:

Missing data were assumed to be MAR[35]. Multiple Imputation by Chained Equations (MICE) with predictive mean matching (PMM) was used to predict and impute the missing values based on observed data[26, 56]. PMM was used to deal with the skewed distribution of costs[18]. The advantage of PMM is that it is more robust against non-normal data than linear regression estimation methods, as it uses the observed distribution of the data and non-existing values cannot be imputed[57]. The number of imputed datasets was increased until the loss of efficiency was less than 5%, resulting in 10 imputed datasets for the REALISE study and 20 imputed datasets for the HypoAware study[58]. The imputed datasets were analysed separately to obtain a set of estimates, which were then pooled using Rubin’s rules[35] to obtain overall estimates, variances, and confidence intervals[35, 58, 59]. In the REALISE study, 33 (24%) participants had missing cost data and 21 (15%) had missing effect data. In the HypoAware study, 28 (17%) participants had missing cost data and 20 (12%) had missing effect data.

An overview of the 14 analytical approaches used in this study as well as the statistical challenges they account for can be found in Table 1. For all approaches, incremental costs and QALYs, 95% confidence intervals around incremental costs and QALYs, incremental cost-effectiveness ratios (ICERs) and cost-effectiveness accessibility curves (CEACs) were estimated and compared. ICERs were calculated by dividing incremental mean costs by incremental mean QALYs. CEACs were estimated using the Incremental Net Monetary Benefit (INMB) approach[60]. CEACs represent the probability of an intervention being cost-effective (y-axis) for a range of different ceiling ratios (x-axis) and provide a summary measure of the joint uncertainty surrounding costs and effects[61, 62]. All analyses were performed in StataSE 16® (StataCorp LP, CollegeStation, TX, US).

2.3 Comparison of the statistical approaches

Statistical approaches were compared in terms of how sensitive the point estimates are to changes in the statistical approaches (i.e. value sensitivity) and how sensitive the conclusion of an economic evaluation is to changes in statistical approaches (i.e. decision sensitivity)[63]. Value sensitivity was assessed by comparing incremental costs and QALYs, the corresponding confidence intervals, and ICERs across the 14 statistical approaches. Decision sensitivity was assessed by comparing the CEACs of the 14 statistical approaches. For comparing and interpreting the CEACs, thresholds of 0 €/QALY gained, 10,000 €/QALY gained and 23,300 €/QALY gained (i.e. about 20,000 £/QALY gained) were used, which refer to a situation in which decision-makers are not willing to pay anything per QALY gained, the Dutch willingness-to-pay (WTP) thresholds (i.e. between 20,000€/QALY gained and 80,000 €/QALY gained depending on disease severity) and the British National Institute for Health and Care Excellence (NICE) threshold, respectively.

3.1 Value sensitivity

In the REALISE study, cost and effect differences ranged from -€782 and -0.001 when ignoring all statistical challenges (analysis 1) to -€82 and 0.011 when accounting for all of them (analysis 14), respectively. The associated ICERs ranged from 636,744 €/QALY gained in analysis 1 to -7,502 €/QALY gained in analysis 14. In analyses 1 to 4, the intervention was less costly and less effective than the control, whereas in analyses 5 to 14 it was less costly and more effective than the control, indicating dominance of the intervention over the control condition. However, in all analyses statistical uncertainty, in terms of confidence interval width, was considerable (see Table 2).

In the HypoAware study, cost and effect differences ranged from -€142 and -0.002 when ignoring all statistical challenges (analysis 1) to -€462 and 0.010 when accounting for all of them (analysis 14), respectively. The associated ICERs ranged from 90,989 €/QALY gained in analysis 1 to 46,592 €/QALY gained in analysis 14. In analyses 1 to 8, the intervention was less costly and less effective than control, in analyses 5 to 8 the intervention was less costly and more effective than the control, indicating dominance of the intervention over the control condition, whereas in analyses 9 to 14 it was more costly and more effective than the control. However, in all analyses statistical uncertainty, in terms of confidence interval width, was considerable (see Table 3).

3.2 Decision sensitivity

In the REALISE study, at a willingness-to-pay of 0 €/QALY, 10,000 €/QALY, and 23,300 €/QALY, the probabilities of cost-effectiveness of the intervention as compared to control were 0.67, 0.57, and 0.55, respectively, when ignoring all statistical challenges (analysis 1) and 0.54, 0.57 and 0.59, respectively, when accounting for all of them (analysis 14). In the HypoAware study, at a willingness-to-pay of 0 €/QALY, 10,000 €/QALY, and 23,300€/QALY, the probabilities of cost-effectiveness of the intervention as compared to control were 0.59, 0.53, and 0.51, respectively, when ignoring all statistical challenges (analysis 1) and 0.27, 0.33, and 0.40, respectively, when accounting for all of them in analysis 14 (Table 4; Figure 1; Figure 2).

The current findings indicate that failure to adequately account for baseline imbalances, skewed costs, correlated costs and effects, and missing data in trial-based economic evaluations may have a substantial impact on cost-effectiveness results.

Correction for baseline imbalances showed to have a large impact on the point estimates for both costs and effects, with the impact being most pronounced for costs. When accounting for skewed costs using bootstrapping, in some cases the estimated statistical uncertainty (i.e. confidence intervals) around cost differences increased. However, for the majority of the statistical approaches, the confidence interval width was relatively similar between bootstrapped and non-bootstrapped statistical approaches. Taking into account the correlation between costs and effects had no large effects on the point estimates, nor on the statistical uncertainty surrounding both outcomes. When using different methods to account for missing data, point estimates as well as the amount of statistical uncertainty differed considerably between analyses. These methods consequently had the largest impact on the probabilities of the interventions being cost-effective compared with the control.

Overall, these results indicate that point estimates as well as statistical uncertainty reflected in the probabilities of cost-effectiveness are notably affected when adjusting for baseline imbalances, skewed costs, and/or missing data in trial-based economic evaluations. In the case studies presented, all four statistical challenges were present, indicating that a statistical approach that takes into account all of these challenges simultaneously was the most appropriate approach and is expected to lead to the most valid results and conclusions.

4.1 Strengths and limitations

The current study is the first to systematically evaluate the impact of simultaneously adjusting for baseline imbalances, skewed costs, correlated costs and effects, and missing data in trial-based economic evaluations on cost differences, effect differences, ICERs, and statistical uncertainty. Another strength is that, step-by-step, all statistical challenges were accounted for until all statistical challenges were dealt with, thus also showing the separate impact of accounting for each of these challenges. Finally, all of the applied statistical methods have previously been found to be valid (see references in Box 1). The main limitation of this study is generalizability, as the findings likely depend on the characteristics of the datasets that were analyzed. In addition, the applied statistical methods were employed from a frequentist approach. Bayesian methods are generally more flexible and the interpretation of their results is more intuitive than those of frequentist methods[64-68]. However, Bayesian methods are generally more complex to implement and are less commonly known to most healthcare researchers. Therefore, in line with Gomes et al. [69] we think that frequentist approaches are more likely to improve current practice. Another limitation is that the true outcomes of the REALISE and HypoAware study are not known. Therefore, the performance of the combination of statistical methods used in this study could not be assessed. Finally, although we found a considerable impact of using either one of the statistical approaches on the probabilities of cost-effectiveness in both studies, the conclusions of the studies did not change. Nonetheless, it can be easily imagined that a difference of 0.20 in the probability of cost-effectiveness between analyses 1 and 14 at a ceiling ratio of 10,000 €/QALY (HypoAware study) can lead to a different conclusion in other situations.

4.2 Comparison to other studies and implications for further research and practice

One can argue that it is already known that when using different statistical approaches, different results will be obtained. The results of this study reinforce this message and show that it is of utmost importance to align the statistical approach with the statistical challenges identified in a specific dataset. Nonetheless, current practice still shows discrepancies between the statistical approach used and the statistical challenges present in a dataset. For instance, missing data and skewed costs are still inappropriately handled in many trial-based economic evaluations[7, 8, 48], but can have a large impact on the results as illustrated by this study. Amongst others, failure to appropriately handle these issues may be due to a lack of consensus about what the most optimal methods for dealing with baseline imbalances, skewed costs, correlated costs and effects, and missing data in trial-based economic evaluation are.

Based on the literature, the statistical methods used in this study are currently considered amongst the most appropriate methods. However, statistical methods are in continuous development. For example, multiple imputation is nowadays generally recommended to deal with missing data[19, 35, 70-73]. However, Twisk et al. [74] showed that multiple imputation was not necessary when using longitudinal mixed model analyses to estimate clinical effects, although it is unclear whether this also holds for cost-effectiveness data. Additionally, missing data was assumed to be MAR, however, this assumption might not always hold and data could be MNAR. Recently, an increasing number of guidelines and studies emphasize the importance of checking for possible departure from the MAR assumption[28, 36, 75-77]. It is recommended to perform sensitivity analyses, using other methods such as selection and/or pattern-mixture models[76]. Furthermore, the handling of clustered data or longitudinal data was not investigated in this study, whereas failure to account for clustering will underestimate statistical uncertainty, can lead to inaccurate point estimates, and may in turn lead to incorrect inferences[10, 78, 79]. It is also important to note that the statistical challenges identified in this study are only a selection of possible statistical issues that might arise when analyzing trial-based economic evaluations[19]. Therefore, in order to improve the statistical quality of trial-based economic evaluations, it is helpful to expand the health economic literature by laying out the current state of play regarding statistical methods for trial-based economic evaluations, and to develop guidance and frameworks in which specific statistical methods are recommended to be used in trial-based economic evaluations.

4.3 Conclusion

The current study emphasizes the importance of adequately accounting for baseline imbalances, skewed costs, correlated costs and effects, and missing data in trial-based economic evaluations by demonstrating that ignoring them may lead to different cost-effectiveness results. Therefore, when conducting trial-based economic evaluations, it is of utmost importance to first check the data to identify statistical challenges that need to be accounted for in the analysis, and then adequately deal with them. Furthermore, it is worthwhile to develop consensus among researchers about frameworks and guidelines on how to best analyze trial-based economic evaluations. To facilitate researchers in appropriately dealing with baseline imbalances, skewed costs, correlated costs and effects, and missing data in trial-based economic evaluations, a software code (Stata) is provided in Supplementary File 1 for the most advanced statistical approach.

Ethics approval and consent to participate

The HypoAware study protocol was approved by the Ethics Committee of the VU University Medical Centre certified by the Central Committee on Research involving Human Subjects in the Netherlands (NL47354.029.13, registration number 2014.007). The REALISE study protocol was approved by the Medical Ethics Review Board of the VU University Medical Centre in September 2011 (registration number NL35897.029.11). All participants gave written informed consent. All methods were performed in accordance with the relevant guidelines and regulations.

Consent for publication

Not applicable

Availability of data and material

The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.

Competing interest

Raymond W.J.G. Ostelo reports grants from The Netherlands Organization for Scientific Research , grants from The Netherlands Organization for Health Research and Development, grants from SIA-RAAK PRO, grants from European Centre for Chiropractic Research Excellence (ECCRE), grants from EUROSPINE, grants from FRIESLAND Zorgverzekeraar, grants from Scientific College Physiotherapy (WCF) of the Royal Dutch Association for Physiotherapy (KNGF), grants from CZ Health Care Insurance and grants from The European Chiropractic Union (ECU), outside the submitted work; Frank Snoek reports grants from ZonMW - Dutch Health Services Research, during the conduct of the HypoAware study; Maurits W. van Tulder reports non-financial support from various professional organizations, others from Swedish and Canadian governmental grant agencies and grants from various research grant agencies, outside the submitted work; Elizabeth N. Mutubuki, Mohamed El Alili, Judith E. Bosmans, Teddy Oosterhuis and Johanna M. van Dongen declare that they have no conflict of interest.

Funding

Not applicable.

Author contributions

ENM and ME are shared first authors ENM: Writing the introduction, results and discussion, running the analyses, interpretation and reflection, revising the manuscript. ME: Writing the methods, results and discussion, writing the syntax, interpretation and reflection, revising the manuscript. JEB: Interpretation and reflection, reviewing the manuscript. FS: Reviewing the manuscript, data collection in HypoAware study. TO: Reviewing the manuscript, data collection in REALISE study. RO: Interpretation and reflection, reviewing the manuscript. MVT: Reviewing the manuscript. JMvD: Study rationale and design, interpretation and reflection, reviewing the manuscript, checking the syntax.

Acknowledgements

Both the REALISE and HypoAware studies were funded by the Netherlands Organization for Health Research and Development (REALISE-171102010 and HypoAware - 837001406).

Authors' information

¹Department of Health Sciences, Faculty of Science, VU Amsterdam, Amsterdam Movement Sciences Research Institute, the Netherlands

²Department of Health Sciences, Faculty of Science, VU Amsterdam, Amsterdam Public Health Research Institute, the Netherlands

³Department of Medical Psychology, Amsterdam UMC, VU Amsterdam, Amsterdam, the Netherlands

⁴Department of Epidemiology and Biostatistics, Amsterdam UMC, Location VUmc, Amsterdam Movement Sciences Research Institute, the Netherlands

⁵Department of Physiotherapy & Occupational Therapy, Aarhus University Hospital, Aarhus, Denmark

Drummond MF, Sculpher, M. J., Claxton, K., Stoddart, G. L., & Torrance, G. W. Methods for the economic evaluation of health care programmes: Oxford university press; 2015.
Barbieri M, Drummond, M., Rutten, F., Cook, J., Glick, H. A., Lis, J., ... & ISPOR Good Research Practices Economic Data Transferability Task Force. What do international pharmacoeconomic guidelines say about economic data transferability? Value in Health. 2010;13(8):1028-37.
van Lier LI, Bosmans, J. E., van Hout, H. P., Mokkink, L. B., van Den Hout, W. B., de Wit, G. A., ... & van der Roest, H. G. Consensus-based cross-European recommendations for the identification, measurement and valuation of costs in health economic evaluations: a European Delphi study. The European Journal of Health Economics. 2018:1-16.
Ramsey SD, Willke, R. J., Glick, H., Reed, S. D., Augustovski, F., Jonsson, B., ... & Sullivan, S. D. Cost-effectiveness analysis alongside clinical trials II—an ISPOR Good Research Practices Task Force report. Value in Health. 2015;18(2):161-72.
Adetugbo K, & Williams, H. How well are randomized controlled trials reported in the dermatology literature? Archives of dermatology. 2000;136(3):381-5.
Montané E, Vallano, A., Vidal, X., Aguilera, C., & Laporte, J. R. Reporting randomised clinical trials of analgesics after traumatic or orthopaedic surgery is inadequate: a systematic review. BMC clinical pharmacology. 2010;10(1):2.
El Alili M, van Dongen, J. M., Huirne, J. A., van Tulder, M. W., & Bosmans, J. E. Reporting and analysis of trial-based cost-effectiveness evaluations in obstetrics and gynaecology. Pharmacoeconomics. 2017;35(10):1007-33.
Doshi JA, Glick, H. A., & Polsky, D. Analyses of cost data in economic evaluations conducted alongside randomized controlled trials. Value in Health. 2006;9(5):334-40.
Gabrio A, Mason, A. J., & Baio, G. Handling missing data in within-trial cost-effectiveness analysis: a review with future recommendations. PharmacoEconomics-open. 2017;1(12):79-97.
Gomes M, Grieve, R., Nixon, R., & Edmunds, W. J. Statistical methods for cost-effectiveness analyses that use data from cluster randomized trials: a systematic review and checklist for critical appraisal. Medical Decision Making. 2012;32(1):209-20.
Díaz-Ordaz K, Kenward, M. G., Cohen, A., Coleman, C. L., & Eldridge, S. Are missing data adequately handled in cluster randomised trials? A systematic review and guidelines. Clinical Trials. 2014;11(5):590-600.
Ramsey S, Willke, R., Briggs, A., Brown, R., Buxton, M., Chawla, A., ... & Reed, S. Good research practices for cost‐effectiveness analysis alongside clinical trials: the ISPOR RCT‐CEA task force report. Value in health. 2005;8(5):521-33.
Faria R, Gomes, M., Epstein, D., & White, I. R. A guide to handling missing data in cost-effectiveness analysis conducted within randomised controlled trials. Pharmacoeconomics,. 2014;32(12):1157-70.
Hoch JS, Briggs, A. H., & Willan, A. R. Something old, something new, something borrowed, something blue: a framework for the marriage of health econometrics and cost‐effectiveness analysis. Health economics. 2002;11(5):415-30.
Thompson SG, & Nixon, R. M. How sensitive are cost-effectiveness analyses to choice of parametric distributions? Medical Decision Making,. 2005;25(4):416-23.
van Asselt AD, van Mastrigt, G. A., Dirksen, C. D., Arntz, A., Severens, J. L., & Kessels, A. G. How to deal with cost differences at baseline. Pharmacoeconomics. 2009;27(6):519-28.
Vroomen JM, Eekhout, I., Dijkgraaf, M. G., van Hout, H., de Rooij, S. E., Heymans, M. W., & Bosmans, J. E. Multiple imputation strategies for zero-inflated cost data in economic evaluations: which method works best? The European Journal of Health Economics. 2016;17(8):939-50.
White IR, Royston, P., & Wood, A. M. Multiple imputation using chained equations: issues and guidance for practice. Statistics in medicine. 2011;30(4):377-99.
van Dongen JM, El Alili M, Varga AN, Guevara Morel AE, Jornada Ben A, Khorrami M, et al. What do national pharmacoeconomic guidelines recommend regarding the statistical analysis of trial-based economic evaluations? Expert Review of Pharmacoeconomics & Outcomes Research. 2020;20(1):27-37.
Manca A, Hawkins, N., & Sculpher, M. J. Estimating mean QALYs in trial‐based cost‐effectiveness analysis: the importance of controlling for baseline utility. Health economics. 2005;14(5): 487-96.
Nixon RM, & Thompson, S. G. Methods for incorporating covariate adjustment, subgroup analysis and between‐centre differences into cost‐effectiveness evaluations. Health economics. 2005;14(12):1217-29.
Sekhon JS, & Grieve, R. D. A matching method for improving covariate balance in cost‐effectiveness analyses. Health economics. 2012;21(6):695-714.
Manca A, Hawkins N, Sculpher MJ. Estimating mean QALYs in trial‐based cost‐effectiveness analysis: the importance of controlling for baseline utility. Health economics. 2005;14(5):487-96.
Nixon RM, Wonderling D, Grieve RD. Non-parametric methods for cost-effectiveness analysis: the central limit theorem and the bootstrap compared. Health Econ. 2010;19(3):316-33.
van Dongen JM, van Wier, M. F., Tompa, E., Bongers, P. M., van der Beek, A. J., van Tulder, M. W., & Bosmans, J. E. Trial-based economic evaluations in occupational health: principles, methods, and recommendations. Journal of occupational and environmental medicine. 2014;56(6):563.
Petrou S, & Gray, A. Economic evaluation alongside randomised controlled trials: design, conduct, analysis, and reporting. Bmj. 2011;342(1548).
Baio G. Bayesian Models for Cost-Effectiveness Analysis in the Presence of Structural Zero Costs. Value Health. 2014;17(7):A584.
Gabrio A, Mason AJ, Baio G. A full Bayesian model to handle structural ones and missingness in economic evaluations from individual-level data. Stat Med. 2019;38(8):1399-420.
Ng ES-W, Grieve R, Carpenter JR. Two-stage nonparametric bootstrap sampling with shrinkage correction for clustered data. The Stata Journal. 2013;13(1):141-64.
Polsky D, Glick, H. A., Willke, R., & Schulman, K. Confidence intervals for cost–effectiveness ratios: a comparison of four methods. Health economics. 1997;6(3): 243-52.
Willan AR, Briggs, A. H., & Hoch, J. S. Regression methods for covariate adjustment and subgroup analysis for non‐censored cost‐effectiveness data. Health economics. 2004;13(5):461-75.
Glick H, & Doshi, J. Designing economic evaluations in clinical trials. Economic Evaluation in Clinical Trials. Oxford, UK: Oxford University Press; 2010.
Willan AR. Statistical analysis of cost-effectiveness data from randomized clinical trials. Expert Rev Pharmacoecon Outcomes Res. 2006;6(3):337-46.
Willan AR, Briggs AH, Hoch JS. Regression methods for covariate adjustment and subgroup analysis for non-censored cost-effectiveness data. Health Econ. 2004;13(5):461-75.
Rubin DB. Multiple imputation for nonresponse in surveys. New York ;: Wiley; 1987. xxix, 258 p. p.
Leurent B, Gomes M, Carpenter JR. Missing data in trial-based cost-effectiveness analysis: An incomplete journey. Health Econ. 2018;27(6):1024-40.
Eekhout I, de Vet HC, Twisk JW, Brand JP, de Boer MR, Heymans MW. Missing data in a multi-item instrument were best handled by multiple imputation at the item score level. J Clin Epidemiol. 2014;67(3):335-42.
White IR, Carlin JB. Bias and efficiency of multiple imputation compared with complete-case analysis for missing covariate values. Stat Med. 2010;29(28):2920-31.
Lipsitz SR, Fitzmaurice, G. M., Ibrahim, J. G., Sinha, D., Parzen, M., & Lipshultz, S. Joint generalized estimating equations for multivariate longitudinal binary outcomes with missing data: an application to acquired immune deficiency syndrome data. Journal of the Royal Statistical Society: Series A (Statistics in Society). 2009;172(1): 3-20.
Noble SM HW, Tilling K. Missing data in trial‐based cost‐effectiveness analysis: the current state of play. Health economics. 2012;21:187-200.

Zhang Z. Missing data imputation: focusing on single imputation. Annals of translational medicine. 2016;4(9).
Oosterhuis T, van Tulder, M., Peul, W., Bosmans, J., Vleggeert-Lankamp, C., Smakman, L., ... & Ostelo, R. Effectiveness and cost-effectiveness of rehabilitation after lumbar disc surgery (REALISE): design of a randomised controlled trial. BMC musculoskeletal disorders. 2013;14(1):124.
Hakkaart-van Roijen L, Tan S, Bouwmans C. Handleiding voor kostenonderzoek, methoden en standaard kostprijzen voor economische evaluaties in de gezondheidszorg. Geactualiseerde versie. 2010;2010.
Lamers L, Stalmeier P, McDonnell J, Krabbe P. Measuring the quality of life in economic evaluations: the Dutch EQ-5D tariff. Nederlands tijdschrift voor geneeskunde. 2005;149(28):1574-8.
de Wit M, Rondags, S. M. P. A., van Tulder, M. W., Snoek, F. J., & Bosmans, J. E. Cost‐effectiveness of the psycho‐educational blended (group and online) intervention HypoAware compared with usual care for people with Type 1 and insulin‐treated Type 2 diabetes with problematic hypoglycaemia: analyses of a cluster‐randomized controlled trial. Diabetic Medicine. 2018;35(2):214-22.
Rondags SM, de Wit, M., van Tulder, M. W., Diamant, M., & Snoek, F. J. HypoAware-a brief and partly web-based psycho-educational group intervention for adults with type 1 and insulin-treated type 2 diabetes and problematic hypoglycaemia: design of a cost-effectiveness randomised controlled trial. BMC endocrine disorders. 2015;15(43).
Rondags SM, de Wit, M., Twisk, J. W., & Snoek, F. J. Effectiveness of HypoAware, a brief partly web-based psychoeducational intervention for adults with type 1 and insulin-treated type 2 diabetes and problematic hypoglycemia: a cluster randomized controlled trial. Diabetes Care. 2016;39(12):2190-6.
Gomes M, Grieve R, Nixon R, Edmunds WJ. Statistical methods for cost-effectiveness analyses that use data from cluster randomized trials: a systematic review and checklist for critical appraisal. Med Decis Making. 2012;32(1):209-20.
Hoch JS, Briggs AH, Willan AR. Something old, something new, something borrowed, something blue: a framework for the marriage of health econometrics and cost-effectiveness analysis. Health Econ. 2002;11(5):415-30.
McNeish DM. Analyzing clustered data with OLS regression: The effect of a hierarchical data structure. Multiple Linear Regression Viewpoints. 2014;40(1):11-6.
Twisk JW. Inleiding in de toegepaste biostatistiek. Houten: Bohn Stafleu van Loghum ; 2017.

Barber JA, Thompson SG. Analysis of cost data in randomized trials: an application of the non-parametric bootstrap. Stat Med. 2000;19(23):3219-36.
Doshi JA, Glick HA, Polsky D. Analyses of Cost Data in Economic Evaluations Conducted Alongside Randomized Controlled Trials. Value in Health. 2006;9(5):334-40.
Ramsey SD, Willke RJ, Glick H, Reed SD, Augustovski F, Jonsson B, et al. Cost-effectiveness analysis alongside clinical trials II-An ISPOR Good Research Practices Task Force report. Value Health. 2015;18(2):161-72.
Briggs AH, Wonderling DE, Mooney CZ. Pulling cost-effectiveness analysis up by its bootstraps: a non-parametric approach to confidence interval estimation. Health Econ. 1997;6(4):327-40.
Glick H, Doshi JA, Sonnad SS, Polsky D. Economic evaluation in clinical trials. Second edition. ed. Oxford: Oxford University Press; 2015. x, 252 pages p.
Vink G, Frank, L. E., Pannekoek, J., & Van Buuren, S. Predictive mean matching imputation of semicontinuous variables. Statistica Neerlandica. 2014;68(1):61-90.
Sterne JA, White IR, Carlin JB, Spratt M, Royston P, Kenward MG, et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. Bmj. 2009;338:b2393.
White IR, Royston P, Wood AM. Multiple imputation using chained equations: Issues and guidance for practice. Stat Med. 2011;30(4):377-99.
Hoch JS DC. Advantages of the net benefit regression framework for economic evaluations of interventions in the workplace: a case study of the cost-effectiveness of a collaborative mental health care program for people receiving short-term disability benefits for psychiatric disorders. J Occup Environ Med. 2014;56(4):441-5.
Fenwick E, O'Brien BJ, Briggs A. Cost-effectiveness acceptability curves--facts, fallacies and frequently asked questions. Health Econ. 2004;13(5):405-15.
van Hout BA, Al MJ, Gordon GS, Rutten FF. Costs, effects and C/E-ratios alongside a clinical trial. Health Econ. 1994;3(5):309-19.
Griffin SC. Dealing with uncertainty in the economic evaluation of health care technologies. York, United Kingdom: University of York; 2010.

Al MJ, Van Hout BA. A Bayesian approach to economic analyses of clinical trials: the case of stenting versus balloon angioplasty. Health Econ. 2000;9(7):599-609.
Gabrio A, Baio G, Manca A. Bayesian Statistical Economic Evaluation Methods for Health Technology Assessment. Oxford Research Encyclopedia of Economics and Finance2019.
Gabrio A, Hunter R, Mason AJ, Baio G. Pitfalls of adjusting for mean baseline utilities/costs in trial-based cost-effectiveness analysis with missing data. arXiv preprint arXiv:180507149. 2018.
Grieve R, Nixon R, Thompson SG. Bayesian hierarchical models for cost-effectiveness analyses that use data from cluster randomized trials. Medical Decision Making. 2010;30(2):163-75.
Mason AJ, Gomes M, Grieve R, Carpenter JR. A Bayesian framework for health economic evaluation in studies with missing data. Health economics. 2018;27(11):1670-83.
Gomes M, Ng ES, Grieve R, Nixon R, Carpenter J, Thompson SG. Developing appropriate methods for cost-effectiveness analysis of cluster randomized trials. Med Decis Making. 2012;32(2):350-61.
Burton A, Billingham LJ, Bryan S. Cost-effectiveness in clinical trials: using multiple imputation to deal with incomplete cost data. Clin Trials. 2007;4(2):154-61.
El Alili M, van Dongen JM, Huirne JAF, van Tulder MW, Bosmans JE. Reporting and Analysis of Trial-Based Cost-Effectiveness Evaluations in Obstetrics and Gynaecology. PharmacoEconomics. 2017;35(10):1007-33.
Faria R, Gomes M, Epstein D, White IR. A Guide to Handling Missing Data in Cost-Effectiveness Analysis Conducted Within Randomised Controlled Trials. PharmacoEconomics. 2014;32(12):1157-70.
White IR, Royston P, Wood AM. Multiple imputation using chained equations: Issues and guidance for practice. Statistics in Medicine. 2011;30(4):377-99.
Twisk J, de Boer M, de Vente W, Heymans M. Multiple imputation of missing values was not necessary before performing a longitudinal mixed-model analysis. J Clin Epidemiol. 2013;66(9):1022-8.
Faria R, Gomes M, Epstein D, White IR. A guide to handling missing data in cost-effectiveness analysis conducted within randomised controlled trials. Pharmacoeconomics. 2014;32(12):1157-70.
Leurent B, Gomes M, Cro S, Wiles N, Carpenter JR. Reference-based multiple imputation for missing data sensitivity analyses in trial-based cost-effectiveness analysis. Health Econ. 2020;29(2):171-84.
Leurent B, Gomes M, Faria R, Morris S, Grieve R, Carpenter JR. Sensitivity Analysis for Not-at-Random Missing Data in Trial-Based Cost-Effectiveness Analysis: A Tutorial. Pharmacoeconomics. 2018;36(8):889-901.
Grieve R, Nixon, R., & Thompson, S. G. Bayesian hierarchical models for cost-effectiveness analyses that use data from cluster randomized trials. Medical Decision Making. 2010;30(2):163-75.
El Alili M, van Dongen JM, Goldfeld KS, Heymans MW, van Tulder MW, Bosmans JE. Taking the Analysis of Trial-Based Economic Evaluations to the Next Level: The Importance of Accounting for Clustering. PharmacoEconomics. 2020;38(11):1247-61.

Due to technical limitations, tables are only available as a download in the Supplemental Files section.

Download PDF

Editorial decision: Major revision
10 Mar, 2021
Reviews received at journal
07 Feb, 2021
Reviewers agreed at journal
06 Feb, 2021
Reviewers agreed at journal
18 Jan, 2021
Reviewers invited by journal
17 Jan, 2021
Editor assigned by journal
13 Jan, 2021
Editor invited by journal
13 Jan, 2021
Submission checks completed at journal
13 Jan, 2021
First submitted to journal
23 Dec, 2020

You are reading this latest preprint version

The Statistical Approach in Trial-based Economic Evaluations Matters: Get your Statistics Together!

Status:

Version 1

Abstract

Figures

Highlights

1. Introduction

2. Methods

2.1 Data

REALISE study

HypoAware study

2.2 Statistical analysis

- Baseline imbalances:

- Skewed costs:

- Correlation between costs and effects:

- Missing data:

2.3 Comparison of the statistical approaches

3. Results

3.1 Value sensitivity

3.2 Decision sensitivity

4. Expert Opinion

4.1 Strengths and limitations

4.2 Comparison to other studies and implications for further research and practice

4.3 Conclusion

Declarations

Ethics approval and consent to participate

Consent for publication

Availability of data and material

Competing interest

Funding

Author contributions

Acknowledgements

Authors' information

References

Tables

Supplementary Files

Status:

Version 1