Response Processes Evidence. Eleven cognitive interviews were conducted over a two-month period. Participants were almost evenly split between the undergraduate (45%) and graduate (55%) levels of study, were mostly female (63.6%), and studied in a variety of different areas. Problem codes were used to identify issues with the instrument. Table 2 shows the number of problem codes marked in each category for each participant. [TABLE 2 HERE]
Examples of interview responses are included in Table 3. No problems with recall or bias were identified by any of the participants. The majority of the problems identified were regarding clarity, with participants asking for clarification on items, and suggesting rewordings to make items more explicit. For example, one participant explained that the item “completing assignments” should be clarified to emphasize having to manage multiple assignments simultaneously. As a result, this item was changed to “having multiple assignments due around the same time.” Both redundancies were identified among stressors within the learning environment theme. Students suggested that “asking my professor a question,” “asking my professor to remark something,” and “asking my professor for clarification on a grade” were largely addressing the same stressor: interaction with faculty. Therefore, these items were collapsed into a single item. Three items were added as per participants’ recommendations to ensure comprehensiveness of the index. Students questioned the relevance of some items within the personal stressors theme to one’s experience as a student (e.g., “worrying about my personal appearance”), though as these problems were rarely identified, the items were ultimately left in for the subsequent phase of testing. Finally, several students struggled with the response options used to evaluate frequency. While these options were initially more specific (e.g., a few times per year, per month, per week), they were altered to be more general (e.g., rarely, sometimes, regularly) in response to participants’ feedback. Additionally, the highest frequency response option was changed from “always” to “almost always” to dissuade participants from shying away from the extreme option. [TABLE 3 HERE]
Content Evidence. Using a snowball sampling approach, we were successful in recruiting a sample of 65 post-secondary students to serve as our panel of participants (Table 4). Our goal at this stage was to recruit a demographically varied sample of students across provinces and areas of study in order to gain a broader perspective on student stress. The majority of participants were female (76.6%) university students (95.7%) from Ontario (79.5%), at the undergraduate (30.4%) or master’s degree level (34.8%), with an average age of 24 years (SD= 3.5). [TABLE 4 HERE]
A total of 38 items demonstrated I-CVIs < 0.7, the recommended cut off for retention (15,17). All items met the cutoff for clarity. While some items’ relevancy ratings did not meet the I-CVI cutoff, we carefully considered these, choosing to retain those that we considered were important to the overall comprehensiveness of the instrument. For example, some items were retained due to being prevalent in focus groups held with students during the item pool development phase of this program of research (e.g., worrying about getting into a new program after graduation, feeling pressured to socialize). Others were retained if we thought the item CVI might have been higher had the sample we used contained more students to whom the item applied (e.g., having to take student loans, and working on one’s thesis). Items addressing sexual harassment and instances of discrimination on campus were also retained despite falling below the threshold, as we considered these were important potential student experiences of campus culture that every institution should seek to monitor. Table 5 displays the retained items along with their relevance and clarity content validity indices, organized by domain of stress. A complete list of the stressors originally tested is available in the supplementary materials to this article. [TABLE 5 HERE]
Relations to Other Variables. A total of 535 participants completed the initial pilot test survey, representing a response rate of 11%. Most participants were female (74.0%), single (64.9%), lived off campus with roommates (62.1%), self-reported their GPA to be between 80-89% (41.7%), and studied full-time (92.1%), and at the undergraduate level (65.5%). The majority of participants were between the ages of 19 and 21 years (63.7%), with an overall average age of 24.5 years (SD = 7.0). International students made up about 9% of the sample. Of the 535 students, a total of 350 completed the second survey (response rate 65%) with a similar demographic breakdown (Table 6). [TABLE 6 HERE]
To calculate “scores” on the PSSI, we dichotomized responses to reflect whether participants experienced some level of stress response to stressor (coded as 1) or did not experience or did not find the experience stressful (coded as 0). For each respondent, we then summed the number of stressors experienced to derive an absolute count, which was treated as a “score”. We used the non-parametric Spearman’s rho to calculate correlations between the PSSI scores and those on other instruments, as the data were not normally distributed. Table 7 and Figure 1 show the correlations between each of the instruments tested. As hypothesized, the PSSI demonstrated a positive, moderate correlation with both the PSS-10 and K-10. As expected, the PSSI also demonstrated a negative correlation with the CD-RISC, indicating that as the number of stressors experienced increased, participants’ resilience scores decreased (20). While the correlation coefficient was relatively modest, this is not surprising given the subjectivity associated with stressful experiences, as well as the individual nature of psychological resilience (24,25). [FIGURE 1 HERE]
Internal Structure Evidence. Results of an exploratory factor analysis (data not shown) supported our hypothesis that the PSSI would take the internal structure of an index, rather than a scale. That is to say that no clear groups of items emerged as viable subscales, making it appropriate to treat each item as an individual causal indicator of our underlying construct of post-secondary student stress.
With respect to test-retest reliability, a total of 365 students completed the second survey, and 350 responses were successfully matched using unique identifiers. Respondents completed the first iteration of the survey over the course of a four-week period. Invitations to complete the second survey were staggered in order to ensure that at least two weeks had passed between responses. As described above, we summed the absolute count of stressors experienced at each time point and examined the correlation between the average PSSI “scores” for the total sample. We sought the recommended minimum reliability coefficient of 0.7 (9). The PSSI demonstrated strong test-retest reliability (rs = 0.78; 95% CI 0.74, 0.82) comparable to that of the Perceived Stress Scale (PSS-10, rs = 0.79; 95% CI 0.74, 0.83), which has been demonstrated to have consistent test-retest reliability averaging around 0.7 for a two-week interval (26).
While we hypothesized that students’ stress levels were likely to remain fairly static over a two-week period, we added a variable to the second survey in order to account for the possibility of a significantly stressful event producing a change in PSSI scores. Respondents were asked whether an event had occurred that caused them extreme stress since they submitted their last survey. Removing all respondents who indicated ‘yes’ (21.4%) from the dataset, we repeated our correlation analysis (n=273), which revealed the test-retest reliability of the PSSI to be largely unchanged (rs = 0.78; 95% CI 0.73, 0.82). Finally, we repeated the correlation analysis among only those who experienced an extremely stressful events (n=71). Here, we saw a slight decrease in the correlation for the PSSI scores (rs = 0.76; 95% CI 0.64, 0.84). Table 7 depicts all tests conducted for test-retest reliability. [TABLE 7 HERE]