Study Design
The study team carried out a mixed methods study to assess performance of Sierra Leone’s IDSR and CBS systems, combining seven quantitative and qualitative research components: i) desk review of MoHS surveillance documents and international literature on CBS for outbreak prone diseases, ii) spatial and time series analysis of CBS alerts, iii) household survey to estimate incidence of symptoms and health seeking behavior, iv) health facility survey on IDSR and CBS experiences, v) household case verification study, vi) costing analysis and vii) qualitative study focusing on health system factors (Fig. 1).
The study team convened a stakeholders’ workshop with the MoHS, international and country partners to agree on assessment objectives and methods. To operationalize the study objectives we identified research indicators for each study component, based on international literature on surveillance system evaluation and stakeholder consultation(11).The assessment was performed on both CBS and IDSR, but the methods and results presented in this manuscript address only the CBS portion of the assessment therefore results of the case verification survey are not presented. As CBS is intended to supplement the IDSR system, the interface between CBS and IDSR is addressed where relevant throughout this manuscript (9).
Data management and analysis
Data collection for all studies was done in April 2019. Eight chiefdoms within four districts (two per district) were purposefully selected with stakeholder input based on three criteria: 1) availability of functioning IDSR and CBS for specific lengths of time prior to data collection, 2) representative rural and urban areas, and 3) representative geography. Chiefdoms were selected in Port Loko, Koinadugu and Pujehun districts, and Tonkolili was selected as a control, where CBS reporting was not taking place (Fig. 2).
We developed data collection tools for all study components, and programmed household, health facility, and case verification survey tools in OpenDataKit (ODK). Forms were stored on a central server and uploaded onto the ODK Collect app on Android devices. Questions were formulated to mirror Demographic and Health Survey language for comparability with Sierra Leone’s and other DHS surveys. Quantitative and qualitative data collection teams were trained on tools, survey methodology and ethical issues. Tools were piloted in a health facility before beginning field work.
Household survey
While the primary aim of the household incidence survey was to serve the assessment of the IDSR system, we also collected information on health seeking behavior and awareness of CBS. Household survey locations were selected as part of a multi-stage cluster sampling design. Within each chiefdom, 13 enumeration areas (EAs) were randomly selected by Statistics Sierra Leone, and per EA 20 households were randomly selected following a systematic skip pattern for a total sample size of 2,060 households. The household survey was conducted primarily in Krio with the primary caretaker of under 5 year old children. Household survey field teams collected and uploaded 2,079 digital forms to the server, averaging 260 households per chiefdom. Seven households with incomplete data were dropped, leaving 2,072 households in the final dataset. 9,793 case-based records were created to enumerate all household individuals including those who moved away or died in the prior year. Age data were collected from 9,452 individuals, of which 26.2% (n = 2,475) were under age five, and 73.8% (n = 6,977) who were 5 or above.
Health facility survey
Four health facilities were surveyed in each of two chiefdoms of Port Loko, Pujehun and Koinadugu for a total of 24 health facilities. The purpose of the health facility survey was to describe CBS recording and reporting practices from facility staff perspectives, including their interactions with CHWs, PSs, and the District Health Management Team (DHMT). Facilities were selected purposefully to ensure a sufficient number of CBS alerts was available for the case verification survey (to verify notified cases, not described here). Data collectors also extracted patient information from CBS case report forms (CRFs) showing suspected cases of acute watery diarrhea (AWD), measles, or maternal deaths for the 1st through 17th epidemiological week of 2019. Data collection teams uploaded 24 health facility forms to the server.
Quantitative analysis of the household and health facility surveys was conducted in Stata version 15 SE software including descriptive statistics, and for the costing analysis descriptive statistics were calculated in Microsoft Excel.
Qualitative study (IDIs and FGDs)
For the qualitative study we applied various methods including the desk review, and IDIs and FGDs with stakeholders at all levels in Sierra Leone’s health system on IDSR and CBS. At the district level, 15 IDIs were conducted with the DHMT including members of the RRT, hospital and district lab staff. At chiefdom level 15 IDIs were conducted with primary health unit (PHU) staff. At community level 15 IDIs were carried out with CHWs, PSs, and village leaders (VL), and FGDs were conducted with community members. In total we performed 45 IDIs and 6 FGDs.
Following transcription of qualitative interviews, two researchers identified main themes from a sample of 10 transcripts to develop a coding framework. Coding was performed in Microsoft Excel by three researchers. When new themes were identified in the remaining transcripts, they were discussed and added to the framework. After coding, descriptive summaries were written for each code and linked the assessment framework’s research questions.
DHIS2 data extraction (spatial and time series analysis)
We extracted measles and diarrhea disease notifications from the MOHS’s IDSR DHIS2 module for all chiefdoms across Sierra Leone from 2017 through March 2019. We performed time series analysis of CBS suspected cases of diarrhea and measles and IDSR disease notification rates per 100,000 population from October 2016 – March 2019. We also performed Local Indicators of Spatial Autocorrelation (LISA) analysis in GeoDa using the Bayesian shrinkage technique to identify whether CBS alerts of 2018 per 100,000 population showed spatial clustering or outliers at the chiefdom level. CBS LISA maps were compared with IDSR LISA maps to identify similarities and dissimilarities in observed spatial patterns of comparable CBS alerts and IDSR notifications.
Costing analysis
We designed a retrospective micro-costing study from a health system perspective based on WHO guidelines and relevant literature (12–16). CBS surveillance activity costs were collected and aggregated for a one year period from 1 January to 31 December 2018. Cost prices were obtained from governmental disease surveillance budgets, program records and expert opinion. We used paper based questionnaires to interview surveillance program staff about resources used. Four interviews were conducted with representatives of MoHS Disease Surveillance Office, CDC, eHealth Africa and WHO. We visited regional laboratories in Port Loko, Makeni and Bo and conducted one interview per lab. We conducted 13 district level interviews in Koinadugu, Pujehun and Tonkolili, each. Per district, eight interviews were conducted in the urban chiefdom with a representative from the DHMT, district hospital, private hospital, Community Health Center (CHC), Community Health Post (CHP), Mother and Child Health Post (MCHP), one CHW peer supervisor (PS) and one CHW. In the rural chiefdom five interviews were conducted with representatives from CHC, CHP, MCHP, one CHW PS and one CHW.
Cost categories included building, equipment, human resources, training, internet, travel, office supplies, booklets and lab supplies. District level population size estimates and number of health system entities (e.g. number of PHUs and CHW per district) were collected from the 2015 census and MoHS program data respectively (17–19).
Cost data were recorded in Sierra Leonean Leones (SLL) and US dollars (US$) using the mean annual 2018 exchange rate (20). Data were entered into Microsoft Excel and resource cost components were multiplied by prices. Costs were aggregated to calculate the annual 2018 unit cost per CBS activity and health system entity. The budget impact (total annual costs in 2018) was estimated based on projections. The unit costs were multiplied by the number of health system entities for Koinadugu, Tonkolili, Pujehun and for Sierra Leone nationwide. Based on the budget impact we calculated cost per capita and the distribution across health system levels. To analyze cost drivers we divided the costs by the budget impact for all cost categories. Cost effectiveness estimates were calculated by dividing the budget impact by the number of CBS suspected measles, diarrhea and maternal death cases alerted in 2018 for Koinadugu, Tonkolili and Pujehun.
Ethical Considerations
Ethical approval
was obtained from the Sierra Leone Ethics and Scientific Review Committee (SLESRC) prior to the commencement of this study. Collection of personal identifiers were required as part of the assessment’s primary data analysis component and were deleted after creation of the final study dataset.