Study settings
This study compared two samples to validate the SDQ. These are: (1) a community sample; and (2) a clinical sample. The community sample consisted of children recruited from the community; one district in the capital city. The clinical sample consisted of children who visited a psychiatry outpatient service in Mongolia. There was no gold standard questionnaire to compare with the SDQ in Mongolia. In addition, there were no known groups which consist only of children with mental disorders and only of children without mental disorder. Thus, in the present study, we aimed to validate the SDQ through examining the discriminating ability of the SDQ between a community sample and clinical sample although there was a possibility that the community sample might include children with mental health problems and the clinical sample might include children without mental health problem. This method for SDQ validity has been applied in previous studies.(21, 22)
Community sample
The community sample consisted of participants in a study that evaluated the effectiveness of physical activity on academic achievement and cognitive function among children in elementary schools. The details of this study are described elsewhere.(23) Participants were children in their 4th year at 10 public primary schools in Sukhbaatar district, which is one of nine districts in the capital city, Ulaanbaatar. Inclusion criteria were: (1) attendance at a public school in the Sukhbaatar district; (2) written consent from parents or guardians; and (3) age-appropriate literacy in Mongolian. Exclusion criteria were: (1) comorbidities or contraindications prohibiting participation in an exercise program; and (2) enrollment in a special needs program. The population of the district is roughly 10% and 5% of the population of Ulaanbaatar and Mongolia, respectively. The district stretches from urban city center to non-urban area where the infrastructure is not enough developed so that the socioeconomical background of the participants are diverse. There is no apparent difference in demographic characteristics in this district compared with other districts in Ulaanbaatar. Thus, population representativeness of the community sample was considered high.
Clinical sample
The clinical sample was recruited at the child and adolescent mental health outpatient service at the National Mental Health Center. The National Mental Health Center is a tertiary-level hospital and it is the only one specialized hospital for mental health in Mongolia. Almost all children with severe mental disorders or intellectual disability in Ulaanbaatar are considered to visit this National Mental Health Center.
The inclusion criteria were: (1) younger than 20 years old; (2) visiting Child and Adolescent Mental Health Outpatient Service at the National Mental Health Center between 1st December 2018 to 31st March 2019; (3) written consent from parents or guardians; and (4) parents’ or guardians’ literacy in Mongolian. There were no exclusion criteria. In this analysis, children aged between 4 to 17 years old were included in the analysis due to the target age range of the SDQ.
Measures
Socio-demographic characteristics and the Strengths and Difficulties Questionnaire were obtained from a parent or guardian in both community and clinical samples. Clinical diagnosis was obtained in the clinical sample.
The Strengths and Difficulties Questionnaire
The Strengths and Difficulties Questionnaire (SDQ) is a 25-item questionnaire for child and adolescent mental health problems. It is for 4-17-year-old children and adolescents. It is used for clinical assessment, epidemiological study and screening of psychiatric disorders. The 25-items are answered using a 3-point scale, “certainly true”, “somewhat true” and “not true” and scored from 0-2 points. The items yield 5 subscale scores that range from 0 to 10 including: (1) emotional symptoms; (2) conduct symptoms; (3) hyperactivity/inattention; (4) peer relationship problems; and (5) prosocial behavior. Summing emotional, conduct, hyperactivity/inattention and peer relationship subscale scores yields a total difficulties score that ranges from 0 to 40. The SDQ uses cut-off scores that are defined using normative data banding in three categories: normal (80th percentile and less), borderline (80th -90th percentile) or abnormal (90th percentile and more). The Mongolian version was obtained from the official website.(13) For the screening of psychiatric disorders, the SDQ has been shown to have moderate sensitivity and high specificity in other languages.(14, 22, 24-26)
In the clinical sample, clinical diagnosis was assessed by recording consultation or reviewing medical records. The 10th revision of International Statistical Classification of Diseases and Related Health Problems is used conventionally in Mongolia and was used in the present study.
Statistical analysis
Receiver Operating Characteristic analysis
To validate the SDQ, Receiver Operating Characteristic (ROC) analysis was performed. ROC curves are curves drawn by plotting sensitivity and specificity for all possible thresholds. The discriminating ability of the total difficulties score between the community and clinical samples was assessed by evaluating the area under the curve (AUC). The AUC of total difficulties scores was calculated among the entire sample, subdividing by gender. AUC values of 1.0 means perfect discriminating ability and AUC values of 0.5 means no discriminating ability at all. Conventionally, AUC values of 0.5-0.7 are considered low accuracy, 0.7-0.9 are considered moderate accuracy and 0.9-1.0 are high accuracy.(27) The analysis was done with pROC library on R version 3.4.4.(28)
This analysis had an assumption that the prevalence of psychiatric disorders among the clinical sample was substantially higher than that of the community sample. This analysis did not have an assumption that either none of the participants in the community sample had a psychiatric disorder or all the participants in the clinical sample had a psychiatric disorder.
To assess the discriminating ability of subscale scores, AUC of each subscale score was calculated.
Although the clinical sample consisted of patients at a child and adolescent psychiatric outpatient service, some participants in the clinical sample might not have a psychiatric disorder. If many in the clinical sample did not have a psychiatric disorder, it might be difficult to examine the discriminating ability of the SDQ. To solve this problem, a sensitivity analysis was conducted using the community sample and a subsample of the clinical sample participants which only included those with definite psychiatric diagnoses.
Cut-off score by normative banding
Normative data for the SDQ total difficulties score of the entire community sample were described. As the etiology of child and adolescent mental health problems has gender differences, normative data by gender were also described.(29) Normative data of the 5 subscale scores were described (Supplementary material).
To determine the original UK version SDQ cut-off scores, banding of the normative data of the SDQ total difficulties scores was done to divide percentiles into abnormal and borderline categories.(22) In the present study, the same banding method was applied to the normative data to determine the cut-off scores of the Mongolian version.
Comparison with the cut-off score candidates by ROC analysis
The cut-off score by normative banding was compared with the cut-off score candidates using ROC analysis which has a balance between sensitivity and specificity. For ROC analysis, the best cut-off score was analyzed by two methods: (1) determining the point closest to the top-left point of the plot which means perfect discriminating ability (100% sensitivity and 100% specificity); and (2) Youden’s J statistics which uses the point that maximizes the distance to the line of no discriminating ability (connecting the point of 100% sensitivity and 0% specificity and the point of 0% sensitivity and 100% specificity).(27, 30, 31) The candidates from the ROC determined cut-off score were compared with the cut-off score by normative banding.
Sensitivity and specificity
Though we did not have an assumption that either the community sample did not include any participants with psychiatric disorders or that the clinical sample did not include any participants without psychiatric disorders, sensitivity, specificity, positive likelihood ration, and negative likelihood ratio to discriminate participant’s group (clinical sample or community sample) were calculated for each cut-off score. Sensitivity meant the proportion of above threshold participants in the clinical sample. Specificity meant the proportion of below threshold children in the community sample The proportion of children above each cut-off score was calculated.