The analysis of the interviews revealed multiple themes regarding emergency physicians’ perceptions of the difficulties and strengths of critical appraisal, as well as strategies of how to motivate residents to learn and maintain critical appraisal skills based on their own experiences.
At the start of the interview, each physician was asked to define critical appraisal in their own terms. Interviewees described critical appraisal as “A structural approach to assessing, analyzing, and making your own conclusion with respect to an article in a journal, book that you read based on the preset set of criteria.”
Major themes identified during the interview analysis included: 1. Barriers to learning critical appraisal including lack of interest, time limitations, and perceived difficulty of critical appraisal 2. Importance of critical appraisal 3. Motivation for critical appraisal 4. Facilitating engagement with critical appraisal 5. Teaching critical appraisal 6. Level of competency with critical appraisal.
1. Barriers to learning critical appraisal
Interviewees described three main problematic aspects of learning critical appraisal skills: lack of interest or relevance, time limitations, and perceived difficulty to learn these skills. In combination, these barriers make it difficult for emergency physicians, especially trainees, to learn critical appraisal skills. “It’s a skill that has nothing to do or very little to do with clinical medicine. It’s a very significant amount of knowledge to understand and requires – on some level, it’s like learning a new language, and it just requires a significant investment of time that most physicians are not willing to undertake...”
a) Lack of interest or relevance
One main difficulty that many interviewees described was the boring nature of critical appraisal in itself. One physician stated that critical appraisal was “a dry topic and doesn’t engage the learners, especially someone who is young…” Similar to the idea of critical appraisal being uninteresting was the concept that these skills feel somewhat out of the scope of what most emergency residents want to learn during their training.
b) Time limitations
Residents noted the difficulty in trying to learn critical appraisal skills on top of the core EM knowledge they are required to learn, stating, “I think it’s hard especially as a resident to just be focusing on even just learning the literature.” Attending physicians involved in education also felt that there was limited time to include critical appraisal skills training within conference.
c) Perceived difficulty
Critical appraisal skills were seen as difficult to understand compared with learning the more practical knowledge of how to diagnose and treat medical emergencies. Residents noted that “most people don’t like doing things they’re not good at, right” and also emphasized that “It sometimes feels like you won’t ever reach that expert level so you just kind of give up.”
Attending physicians noted that it could be hard to learn critical appraisal skills because of the challenges of understanding research methodology as well as the sheer amount of medical literature available to review.
2. Importance of critical appraisal
Despite these barriers, all interviewees described the value of learning critical appraisal skills. Attendings noted the increasing importance of evidence-based medicine to guide physicians’ practice as a reason for all physicians to develop and maintain their critical appraisal skills.
“We’re moving further and further away from eminence-based medicine which is basically experience driven and more towards evidence-based medicine which is obviously based on research studies.” Many interviewees felt that critical appraisal skills were of even greater value to emergency physicians because of the large amount of medical knowledge that they were responsible to understand for clinical decision-making.
Although many attending physicians noted that they improved their critical appraisal skills after graduating residency, they expressed concern that if these skills were not encouraged, residents may not practice their critical appraisal skills as attendings. “I certainly spent more time learning critical appraisal as an attending than I did during residency. But if you didn’t learn it during residency and you end up in sort of a community practice, the chances of you ever getting a chance to meaningfully do that again are small.”
Residents felt that learning critical appraisal skills was valuable for their training by helping them to provide better patient care, and also felt that these skills would benefit them in the future when they would be responsible for their own continued medical education.
3. Motivation for critical appraisal
Most physicians noted both extrinsic and intrinsic motivating factors for learning critical appraisal, with intrinsic motivating factors perceived as having a bigger and more lasting influence overall.
Extrinsic Motivation
Many attending physicians noted that they were required to do a research project during residency. This assigned work forced them to become better at critical appraisal in order to complete the requirement. One physician held very little interest in critical appraisal until she had to do it as part of her research project.
But I still didn’t care, you know, all that much and – where I really learned it was midway through residency I had to do a research project, and my project was, you know an original study that I designed that I got the IRB for…ultimately my article got published in a journal, so I had to actually do it for real, and it made me appreciate it in such a…more real way where I actually had to do it on my own...Because I’m very invested in something that’s my own project, so that’s when I actually forced myself to do it.
Although many physicians did not express enthusiasm for performing critical appraisal as a resident, the initial assigned research project allowed them to understand critical appraisal on a more meaningful level.
Intrinsic Motivation
Intrinsic motivating factors were described by all attending physicians as contributing to a greater interest in critical appraisal. Most physicians described a desire to learn critical appraisal skills in order to provide better patient care and to keep up to date on current medical literature. Physicians also felt greater intrinsic motivation to understand critical appraisal in order to justify their clinical decisions once they became attendings who needed to teach residents.
4. Facilitating engagement with critical appraisal
Attending physicians and residents emphasized the importance of finding ways to make critical appraisal skills more relevant to residents. Distinct from practical teaching strategies, emergency physicians noted three major ways to support residents’ engagement with critical appraisal: normalization of critical appraisal skills within the ED culture, connection of critical appraisal skills to patient care, and connection of critical appraisal skills to resident research.
Almost all physicians emphasized the importance of promoting critical appraisal within the culture of the ED. Attending physicians noted that the ED where they trained and their attending mentors held a major role in shaping both their interest in critical appraisal and how likely they were to continue using it.
But you have to normalize it as an expectation, meaning that the senior residents and the attending staff role model critical appraisal in their teaching and on shift and in other places…You have to make it feel not so much like a task but just like a normal part of the culture.
Emergency physicians also noted the importance of engaging residents in critical appraisal by relating it to patient care. A few physicians also felt that residents could become more engaged in critical appraisal by providing them with greater opportunities for meaningful research.
5. Teaching critical appraisal
In terms of the practical aspects of teaching critical appraisal, attending physicians noted the importance of thinking about critical appraisal as a tool or skillset rather than a topic.
You have to sit down and learn critical appraisal separate from conjunctivitis or myocardial infarction. It should be a tool. It shouldn’t be a topic itself.
One physician felt that critical appraisal could be seen as similar to a language. Most emergency physicians should learn the basics in order to understand how to interpret medical literature, but very few physicians needed to become truly ‘fluent’ in critical appraisal. “You should have, like, a relatively constrained, unambitious set of skills that – sort of a curriculum. I think it should be a relatively small curriculum that focuses on the basics so that there’s at least some fluency with the language and people can understand, you know, what people are – what methodologists are even talking about when they say things like cohort study or a p-value, but often our critical appraisal curriculum is overambitious.”
Many physicians noted the importance of incorporating medical literature into residency conference with an emphasis on performing critical appraisal frequently. Many physicians felt that the best way to have residents learn critical appraisal skills was to make the resident teach critical appraisal or be in charge of their own projects. Residents also felt that being in charge of projects involving critical appraisal or being asked to teach critical appraisal themselves would enhance their learning.
6. Level of competency with critical appraisal
Physicians in general felt that residents should have a basic foundation in critical appraisal skills which they could continue to develop over their careers.
"I think we should all have a sort of a foundation and that we should have a specialized subset of emergency physicians and other methodologists who can tell the rest of us which studies are good and which studies are bad and what we should take from various studies…"
Nearly everyone interviewed, however, from residents to the ED chair, felt that they could be better at critical appraisal.