This longitudinal study evaluated the impact of an educational package developed and implemented using the KTA framework, and which aimed to increase PROM knowledge and utilisation in a small cohort of chiropractors in Australia. The results are presented and discussed according to three of the four levels of the NWKM – reaction, learning, and behavior [32]. Overall, the reaction to the educational package was very positive, with most participants noting improved knowledge and a high level of satisfaction with the material. Although there was a significant improvement in knowledge after receiving the educational package, which was retained at 12 weeks, these changes were small and did not translate to improvements in confidence, attitude, or frequency of PROM usage. The findings suggest that changes in knowledge alone may be insufficient to influence the frequency of PROM utilisation by chiropractors.
Level 1 - Reaction
It is important to evaluate the participant’s reactions to the course or training when evaluating the success of the educational intervention [33]. Overall, the findings confirmed a positive reaction to the educational package. Most respondents reported that the intervention was favourable, engaging, and relevant to their needs. These findings could therefore be important to create clinical behaviour change as Oreg, Vakola [34] found that organizational and behaviour change are not well implemented unless there is a positive reaction.
Survey 1 included items which allowed respondents to communicate directly with the content creators and inform design and content of the educational package [28, 35]. Importantly, 98% of those surveyed felt that the educational package they engaged with was highly relevant to their needs and matched their learning style; the latter of which has been associated with improved learning outcomes in educational [36]. Furthermore, 81% of the survey respondents were either somewhat or strongly satisfied with the overall educational package and 88% stated that the educational package was effective at increasing their knowledge and they would recommend it to their colleagues. Interestingly, only 70% of the respondents noted that they would recommend changes to their practice procedures after viewing the educational package. While research suggests that participant reaction to learning is important such measures may not correlate to how much participants learnt, or whether their behaviour changed [37-40]. For example, a recent study of acupuncturists noted that while the reaction to the training is important, levels 2 (learning) and 3 (behaviour) of the NWKM are the most relevant to determine the success of the program [41].
Level 2 - Learning
Level two of the NWKM is concerned with determining the degree of learning, knowledge, confidence, and skill acquisition because of training [26, 32, 42, 43]. To assess knowledge, evaluation of the participant’s recall, understanding and application of the learning should be measured [44]. Recall and understanding were assessed using three questions included in each survey and covered a range of topics including the definition of PROMs, categories of PROMs, and the identification of PROMs examples using multiple-choice questions. The results show a significant improvement in knowledge scores between survey 1 and survey 2, which were retained at survey 3, suggesting the intended learning outcomes were achieved and the educational package was successful [45, 46]. Nevertheless, the improvements in knowledge were modest (~10%), and while statistically significant, may be insufficient to affect a change in participant behavior and attitude towards PROMs. Despite a recent survey of chiropractors in Australia suggesting that improving clinician understanding of PROMs and why/when to use PROMs would improve utilisation rates, our study, together with others, suggest that knowledge, in and of itself, may not be sufficient to change behavior [47, 48]. Similarly, confidence of the learner, which has been suggested to be a gap between learning and behaviour [49], was also not changed as a result of the educational package. We found no difference in confidence scores between survey 1, and surveys 2 and 3. Research suggests that motivation and confidence are key determinants to create behavioral change [50]. Boyce, Robertson [51] further suggest that building confidence is important if intentions are to be translated into behaviour change. Nevertheless, while confidence may bridge the gap between knowledge and behavior, improving knowledge alone seems insufficient to change confidence. Overall, this study suggests that the provision of an online educational package aimed to improve knowledge of PROMs in chiropractors had no effect on their confidence in understanding what PROMs were available or when and/or how to use them.
Level 3 - Behaviour
Level 3 of the NWKM evaluates to what extent the newly acquired knowledge or skills have been practically applied [42, 52]. This level also assesses the degree to which the participants apply what they learned and modified their behaviour based on the intervention [53]. In the current study we evaluated behavior change using questions related to two subthemes: 1) attitude toward PROMs; and 2) frequency of PROM utilisation. Participant’s attitude toward PROMs were evaluated using 16 questions, which covered a variety of attitudinal aspects relating to the importance and influence of PROMs in clinical practice. Of the 16 questions, only one question was significantly different after the provision of the educational package. Participants more strongly agreed with the statement: ‘Health professionals should use patient reported outcome measures to monitor treatment outcomes using valid and reliable tools’ in survey 2, compared to survey 1. The educational package may have highlighted the arrival of the “era of accountability” with more pressure placed on health care providers to provide treatment evidence [54]. The last decade has seen an increasing focus and interest not only in a patients symptoms, but also in documenting the patient experience and their interactions with healthcare providers [55]. Nevertheless, the overall results do not support a change in clinician attitudes toward PROMs after receiving the educational package. Whilst difficult to determine, it is likely that respondents already had a very favorable view towards PROMs and so there may have been a ceiling effect limiting the impact of the educational package on participant attitudes. Previous studies do show that most chiropractors have a favorable view of PROMs [5, 15], however, these views do not necessarily correlate to higher levels of usage [15].
Although 70% of the respondents reported that they would recommend changes to their practice procedures after viewing the educational package, the data did not support that those changes occurred. Furthermore, despite most respondents acknowledging the importance of PROMs and the need for clinical change to include PROMs, the clinical implementation did not increase after receiving the educational package. Except for health-related PROMs, the reported frequency of PROM utilisation did not change from survey 1 to surveys 2 and 3. We found a small but significant increase in the reported use of health-related PROMs from survey 1 to survey 3. Although there was a similar trend toward increased use of functional PROMs, these differences were not significant. These improvements may be explained, at least in part, by two factors. First, the current study, as well as previous studies [8], show that functional and health-related PROMs are less frequently used by chiropractors in clinical practice, compared to pain-related PROMs, and so their frequency of use may be more likely changed with an educational package that promotes awareness of these PROMs. Secondly, and further to this point, the educational package was specifically designed to address these practice gaps by providing a comprehensive summary of numerous health- and functional-related PROMs relevant to each body region. It is possible that much of this content may have been perceived as ‘new and novel’ and therefore may be more likely to elicit a measurable change in behaviour, compared to the more commonly used pain-related PROMs.
Overall, the educational package did not affect a measurable change in participant’s confidence, attitude, or frequency of PROM utilisation. Although we used the KTA framework to design the content of the educational package, we did not include any specific strategies to promote translation, but rather relied on knowledge to drive behavior change. Clearly, knowledge alone is not sufficient to affect behavioral change. Although knowledge has been consistently identified as a barrier/facilitator to PROM utilisation in healthcare, multiple other barriers exist which were not addressed in the current study and may be more influential in changing behavior. Time was the most reported barrier to PROM usage by chiropractors [5,8]. Although the educational package provided links to source material there were no specific implementation strategies included in the package that may have eased the time burden of using PROMs in clinical practice. The provision of all relevant PROMs to chiropractic care, including background information, via a single electronic application may be a future strategy to better align PROM knowledge with PROM access, and reduce the time the burden of accessing PROMs in a clinical setting.
It is possible that if this study had continued over a longer period of time, this may have increased the likelihood of measuring behavioral change. Axtell, Maitlis [56] found that the amount of learning transferred into practice one month post intervention was a strong predictor for learning transferred one year post intervention. According to Kirkpatrick D [57] the most accurate time to evaluate behaviour change is at least three months after the training was applied, although literature suggests that behaviour change may not occur until six months, to enable learners sufficient time to put their new skills into practice [58]. Therefore, a longer-term follow-up (up to 12 months) may have been needed to identify whether the short-term changes in knowledge changed behaviour.
Limitations
In addition to the short follow-up period, there are several other limitations to this study. First, the low response rate to the cohort survey may limit the strength and generalisability of the findings. Although online surveys are a common research method, their success, particularly amongst health care professionals has been questioned [59]. Cunningham, Quan [59] imply that low survey response rates are common within the health care profession. Another limitation is bias, which is an inherent issue in the design of surveys [60]. Although the authors aimed to reduce selection bias and population bias by inviting all members of Australia’s main chiropractic associations to participate, an unintentional responder bias may have occurred. Due to the low response rate, the survey respondents may not be representative of the entire chiropractic profession in Australia. A further limitation of this study was that there was no control group. Additionally, given the substantial drop-out between survey 1 and survey 2, it is possible the participants who completed all three surveys may have already had favorable views of PROMs, and relatively high utilisation rates, and so there may have been a potential ceiling effect associated with the educational package that affected changes in confidence, attitude, and frequency of PROM usage.