Of the 1459 unique articles retrieved from the database search, 132 studies met the inclusion criteria after full-text screening and were thus included in the review. The characteristics of these studies are presented in Table 1. Our review identified studies from 37 different countries. Three quarters of the studies were from Sub-Saharan African countries (74%), followed by South Asia (11%). The vast majority of the studies were published in the last decade, and more than half were published after 2014 (55%), suggesting an increase in the use of RHIS data for research purposes over time. Most of the studies included an analysis of RHIS data (97%), and a few used RHIS data to inform the study but did not describe analysis of RHIS data. One study, for example, used information from RHIS to justify for the selection of the indicators to be used at the individual-level in their study. Among the studies that analyzed RHIS data, most utilized an ecological study design (79%). Of those, more than half included statistical inferences (61%), while the remaining studies only used RHIS data for descriptive purposes (39%). Nearly a fifth of the studies were mixed methods or case studies (18%), a third of which included statistical analyses of RHIS data (33%). A quarter of articles included a description of how they managed missing data (25%), while only a small number of studies described how they detected and dealt with extreme values (14%).
[Table 1 to be placed here.]
Types of disease and research purpose
Figure 2 shows the different research purposes for which RHIS data were used, along with the health topics investigated. The most common purpose of the studies was program evaluation (51%). RHIS data have been used to evaluate a wide range of interventions, ranging from programs that targeted specific diseases to interventions or policies that affected multiple types of diseases or health services. These included: the effect of malaria control strategies 30–36, user fee exemption policies 37–40, health financing schemes 41–44, interventions on health governance 45–53, the administration of new vaccines and vaccination campaigns 54–56, as well as community-level interventions such as approaches to enhance community participation and improve referrals from traditional birth attendants in increasing the demand for maternal and child care 57–59.
Additionally, RHIS data were used to monitor or assess service provision (23%) and to describe disease epidemiology (17%). Similar to the program evaluation studies, these studies also investigated a diverse set of health services and the allocation of healthcare resources. Some of these studies found large discrepancies between RHIS data and an estimated disease burden in populations or highlighted the lack of service provision. A few studies also used RHIS data to describe specific programs 60–64, conduct impact evaluations (non-programmatic) 65–68, and estimate costs 69,70. Most of the studies investigated a communicable disease (95%), of which malaria was most studied health condition (24%). A few studies focused on mental health (2%), diabetes mellitus (1%), and permanent tooth extraction (1%). Only two studies used RHIS data to research the health workforce or the equity of funding allocations 71,72.
Analytic methods using RHIS data
Among articles that conducted statistical analyses using RHIS data (n=68), time series analyses to test or account for trends were most commonly performed (25%), followed by geostatistical analyses (16%), pre-post comparisons (15%), interrupted time series (ITS) (10%), and difference-in-difference analyses (7%). Other longitudinal analyses (13%), other cross-sectional analyses (12%), and scenario analysis on cost effectiveness (2%) were also conducted. Table 2 presents the range of methodologies identified across studies using RHIS data, as well as the corresponding articles.
[Table 2 to be placed here.]
Time series analysis
Time series analysis using RHIS data was most often applied to evaluate programs and identify disease epidemiology, with one study assessing the impact of an infectious disease outbreak on primary health service utilization 73. Studies analyzed indicators using large quantities of monthly or yearly data to estimate change (range of time units: 5 – 168). For instance, two-thirds of the studies analyzed three or more years of monthly data. Many of the studies utilized the highly disaggregated nature of the data by using either facility or district level data, with the exception of two studies which modelled national trends 33,74. Studies commonly applied strategies to account for temporal autocorrelation and the correlation between geographical units, including generalized linear models 58, multi-level analysis 75,76, and ordinary least-squares regression with adjustment for seasonality and lag 34,37,77. Among studies that modelled multiple facilities or administrative regions, random effects were commonly applied to account for heterogeneity.
In addition to RHIS data, a number of included studies incorporated data from external sources in their models based on geographical location such as district or region. Studies of malaria, for example, commonly included climate data from satellites in their models to control for important temporal factors, for example precipitation, humidity, and temperature 77,78. Other studies incorporated information from other national community surveys, health facility surveys, and program data as covariates 34,75. While most studies controlled for potential confounders by including covariates in analytic models, one study on maternal health service applied propensity score matching to further remove biases from differences in covariate distribution 37.
Geostatistical analysis
Geostatistical analyses using RHIS data were predominantly conducted for epidemiological purposes and the monitoring and assessment of service provision by exploiting geospatial information included in the RHIS at the facility or district level. Three of the studies that applied geostatistical analysis were cross-sectional, while the remainder were spatial-temporal. About half of the studies focused on malaria, of which three compared and illustrated various kriging methods to provide a reliable estimate of malaria burden amid missing reporting 79–81, and one study applied geostatistical modeling to select the most relevant health facility indicators for severe malaria outcomes 82. Studies on other topics investigated the spatial or spatial-temporal dynamics of malaria in pregnancy 83, childhood diarrhea 84, clustering of malaria and HIV 85, and meningitis 86. About half of the studies did not include data from external sources, and others triangulated data sourced from satellite data, Demographic and Health Surveys, national Malaria Indicator Surveys, and Service Delivery Indicator Surveys in their analyses. Studies that included covariates in the geostatistical analysis applied Bayesian hierarchical Poisson model or Bayesian geostatistical negative binomial models 82,87,88.
Pre-post comparison analysis
Pre-post comparison was commonly applied among studies that used RHIS data for program evaluation, and several studies used simple descriptive statistics to compare the periods before and after interventions. As pre-post comparison is subject to the limitation of temporal confounders and secular trends, two of the studies included contextual factors in regression modelling 35,89.
Interrupted time series analysis
Most of the studies that conducted ITS analysis used it to evaluate interventions, and one assessed the impact of an infectious disease outbreak on maternal and child health service use 68. The studies used large quantities of monthly data to model trend and level change (range of time unit: 44 – 132). RHIS data were minimally aggregated in these studies, which mostly analyzed facility or district level data, and similar to studies using time series analysis, accounted for autocorrelation through incorporating autoregressive structures or clustered standard errors in their modelling.
As ITS analyses are generally unaffected by confounding variables that do not change over time by design 90, baseline characteristics were typically not included in these models. Nonetheless, ITS analyses can be affected by time-varying confounders that rapidly change and some models included contextual factors from other data sources, such as climate and program data. To strengthen the quasi-experimental design, two studies also included a contrast group of time series to control for contextual changes that occurred at the same time as the interventions 38,45.
Difference-in-difference analysis
Five studies applied difference-in-difference techniques using a wide range of time periods (range of time units: 4 – 48) and level of geographical units (facility, district, provincial). Only one study included contextual characteristics from other data sources in its analysis. Analytic methods varied from descriptive comparison between and within intervention and control groups 41,59,91,92, to ordinary least square regression with propensity score matching 42.
Impact of research using RHIS data
Most of the studies that conducted statistical analyses using RHIS data were published in journals with impact factors (88%, figure 3), two-thirds of which were two or higher, and more than a fifth of which were greater than three. Among those studies published in journals with the highest impact factors, most of them focused on program evaluation (53%), followed by monitoring and assessment of service provision (20%), epidemiology (20%) and impact evaluation (7%). These studies encompassed a range of health topics commonly studied using RHIS data.
Strategies to circumvent RHIS data quality issues
Data quality is commonly cited as a barrier to using RHIS data in research, and slightly more than a quarter of the included studies described the strategies that they used to handle missing data and/or identify extreme values (table 3). These strategies consisted of exclusion, imputation, interpolation, verification, and accounting for missing data in modeling. Exclusion of missing data was the most common practice, and among studies that used this technique, they excluded facilities from the analytic samples 38,41,45,52,65,91,93–98, restricted the study period based on explicit criteria 54,99, or applied sensitivity analysis to compare various exclusion criteria 41,100,101. Imputation methods varied from assigning specific values to the missing observation 42,86,91,102–104, to various modeling strategies such as conditional autoregressive model 87, generalized linear regression 103, and iterative singular value decomposition 103. A sensitivity analysis was also conducted to select a specific imputation strategy 103. Interpolation involves predicting values at unsampled locations. Methods described included the use of space-time kriging 79–81, and the adjustment of results by calibrating with other relevant information 52,53,55. Some studies assumed data were missing at random, which was accounted for in specific modeling methods such as mixed-effect models 65,103. When the source of data could be reached, some studies also described verifying the missing information using registries where the original data were recorded 39,78,99,105,106.
[Table 3 to be placed here.]
Slightly fewer articles described methods to identify and handle extreme values in the RHIS data, of which three types of strategies emerged: setting specific thresholds, visual inspection, and analytic assessment. Thresholds were set based on the distribution of the data, such as proportions or standard deviations from univariate regression. Several studies used visual inspection of outliers 38,81, while the use of jackknifing analysis and the identification of influential points through Cook’s distance statistics were also applied 107,108. Upon identification of extreme values, several strategies were utilized: exclusion, replacement with the average value, replacement with the missing value, verification with a data source, or discounting the observation in statistical estimation. However, studies that replaced the extreme value with an explicit value potentially introduced bias into their estimates. A few studies also described the strategies applied to assess the reliability of the RHIS data, some of which were routine processes administered in the health systems 39,106.