Study Design
This psychometric validation study included patients with schizophrenia who were participating in a Phase II, randomized, double-blinded, placebo-controlled, parallel group study to evaluate the efficacy (cognition and everyday living skills), safety, and tolerability of four orally administered doses of BI 409306 during a 12-week treatment period (clinicaltrials.gov: NCT02281773) [18]. Healthy individuals (control group) were recruited separately from subgroups at the same study sites, but using a separate, parallel-study protocol (Clinicaltrial.gov: NCT01505894). The study was conducted in accordance with the ethical principles of the Declaration of Helsinki [19], International Council for Good Clinical Practice [20], and applicable country-specific regulatory requirements. It was reviewed by the New England Research Institute’s institutional review board (IRB) and each study site received individual IRB approval via Alpha IRB.
Patients
All patients/individuals were 18–55 years of age. The patient group had a diagnosis of schizophrenia (as per the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition) and had received stable antipsychotic treatment for ≥8 weeks prior to randomization. Patients had no more than a ‘‘moderate’’ severity rating on hallucinations and delusions (Positive and Negative Syndrome Scale [PANSS]) and no more than “moderate” depression scores (PANSS general psychopathology syndrome depression item). The control group were free of major psychiatric illness, neuropsychological impairment and history of antipsychotic drug use. All patients, or their legally accepted representatives, provided written informed consent.
Assessments
The patient group completed the PRECIS instrument at baseline (randomization or Visit 2), Visit 4 at 6 weeks, Visit 5 at 9 weeks, and end of treatment (EOT) visit at 12 weeks. The PRECIS instrument was initially comprised of 35 items answered via a five-category Likert scale (ie, 1=not at all/not at all hard, 2=a little bit/a little bit hard, 3=somewhat/somewhat hard, 4=quite a bit/quite hard, 5=very much/very hard) [15].
Identification of an appropriate algorithm to convert the responses into numerical values was carried out once the final structure of the PRECIS instrument was determined, as described below. In addition to the PRECIS instrument, the CANTAB and the MATRICS Consensus Cognitive Battery (MCCB) were used to objectively assess cognitive function. The SCoRS was conducted to subjectively assess cognitive impairment. Patients completed the CANTAB and MCCB during Screening, Visits 2 and 4, and EOT. The SCoRS was completed at Visit 2 and EOT. The control group completed the original 35-item PRECIS instrument, the MCCB, CANTAB, and SCoRS at a single visit.
Statistical Analyses
Analyses were conducted using SAS version 9.4 software (SAS Institute, Cary, CA, USA). Descriptive statistics were calculated for demographic variables (age, gender, race, and ethnicity). For individual items in the PRECIS instrument, the mean, standard deviation (SD), range, and ceiling and floor effects were calculated. Effects of gender (female vs male), age (≤45 years vs >45 years) and race-ethnicity (white vs black vs other) on PRECIS item scores (35-item instrument) were examined using standard statistical tests.
Responses to the PRECIS instrument were converted into a total score (unweighted), calculated using the sum of the 35-item scores divided by 35 (average item score). Individual domain scores (scores for each category of items) were computed as the average score across all items within each domain. Any data points missing ≥10% of data were excluded. If more than one item was missing from a domain, the domain score for the patient was excluded. Individual items with less than adequate reliability or validity were identified and eliminated or modified. Items that met the following pre-defined elimination criteria were considered poorly performing and were considered for elimination from the 35-item instrument:
-
floor or ceiling effects of >50% among patients
-
≥10% missing data
-
relatively low factor loadings across the domains (<0.025)
-
Cronbach alpha inflation of >10% above the original value upon item removal
-
lower internal or test–retest reliability (<6)
-
failure to distinguish statistically between patients and controls in discriminatory validity testing (statistical cut-off: p=0.05).
Analyses were repeated on the revised instrument once poorly performing items were removed. Item finalization was performed over two rounds of analysis and alternative scoring procedures were considered.
Factor Analysis
An exploratory factor analysis (EFA) was conducted to determine the dimensionality of the latent variables (ie, domains) being assessed by the 35-item PRECIS instrument and to identify common factors (domains) among the 35 items. All pre-treatment scores in the patient group, and for patients and controls combined, were used for these analyses. The control group (N=88) had insufficient sample size for a separate factor analysis. The number of derived factors was based on eigenvalues (threshold of >1.0), a scree test, and the interpretation of simple structure. Both Promax and Varimax rotations were employed to identify correlated factors and uncorrelated factors, respectively, to evaluate the strength of domains. A preliminary confirmatory factor analysis (CFA) was carried out after the poorly performing items had been eliminated and a 24-item version of the PRECIS instrument was available for testing.
Reliability
Reliability was assessed using assessments of internal reliability and test–retest replicability. Internal reliability, or inter-item consistency, of the original 35-item PRECIS instrument was evaluated by means of Cronbach’s alpha [21, 22] based on data obtained at baseline from patients with schizophrenia. Cronbach’s alpha ranges from 0 to 1, with higher values indicating increased reliability. Inter-item correlations were analyzed using Spearman’s (non-parametric) rank order correlation coefficients. Unfortunately, repeated administrations of the PRECIS items during the pre-treatment or run-in phase of the study, as would ideally be used for test–retest purposes, were not included during the main clinical trial (NCT02281773). These analyses were therefore performed during the last phase of the clinical trial (Weeks 9–12). Intra-class correlation coefficients (ICCs) were used to analyze test–retest reliability of the responses to the 35 individual PRECIS items between Visit 5 (Week 9) and EOT (Week 12). To control for potential treatment-related changes associated with the active drug condition, we included only subjects in the placebo condition, who also had stable composite CANTAB scores during this final study period (defined as >1-point change in either direction) between the last two study visits.
As patients had been randomized to receive treatment that may have influenced their cognitive impairment, this potentially reduced the reliability of the test–retest analysis. To minimize this variability, test–retest analyses were conducted in a subset of placebo-treated patients with stable CANTAB scores (defined as <1-point change in either direction). ICCs, percent agreement scores and paired t-test comparisons of scores were calculated to compare PRECIS instrument responses at Visit 5 and EOT. Correlations of ≥0.70 were defined as good reliability [23]. This analysis was repeated for the revised PRECIS total and factor scores, once poorly performing items had been eliminated.
Validity
For both the original 35-item and the subsequent 24-item versions of the PRECIS instrument, discriminant validity assessed whether it was possible to differentiate between the patient and control groups. PRECIS responses from both groups recorded at randomization were compared using an exact Wilcoxon two sample (non-parametric) test. Convergent and divergent validity were assessed by correlating pre-randomization PRECIS scores with related and unrelated domains from other validated instruments (CANTAB, MCCB, and SCORS), using Spearman’s (non-parametric) rank order correlation coefficients for patients with schizophrenia. Strong and poor convergent validity was defined as correlations of ≥0.70 or ≤0.40, respectively. These analyses were repeated on the revised, 24-item PRECIS instrument to ensure its validity.