A total of 14 ED physicians participated across the three workshops and all were aware of the existence of the CSRS tool prior to participation. Participant characteristics are summarized in Table 1.
Workshop 1
Participant feedback
Participants validated the barriers identified in previous qualitative work (discomfort using the CSRS, lack of confidence, lack of knowledge and skills, and uncertainty around interpretation) and highlighted three additional barriers. These included struggling with how to apply the CSRS recommendations, inapplicability of CSRS for some patient clinical presentations, and a lack of buy-in from the broader medical team (i.e., cardiologists and internists). Physicians described that system-level barriers may prevent the ability to apply recommendations, including lack of access to outpatient cardiac monitors, difficulty keeping patients for six hours of ED observation, and the ‘black hole’ following patients’ discharge due to unknown continuity of care.
Having a system so that when using the score, you know what’s happening to the patients and what you are recommending is going to follow-through would help. The mild-risk patients I think everyone is fine with, a lot of high-risk patients we’ll end up admitting, but then the high-risk that you sent home or the intermediate-risks that go home, we send to 4-5 different cardiology services and I might get a letter back from 1 out of the 10 patients that I sent. You feel like they’re being sent into this black hole where you don’t really necessarily have a good sense of what happens to them even though that’s what you recommend. The thrombosis clinic is so helpful as when you don’t have a CT scan at night, they get seen the next day and you can use rule to have a definitive management. But I know that it may be far beyond the scope of the rule. (Workshop#1, Participant 5, Women)
The lack of collective buy-in is an important barrier to CSRS use, as ED physicians described the importance of all team members being aware of and having confidence in the resulting course of treatment for a given patient.
For emerg doctors you are worried to accept the responsibility of discharging people at home. I think there needs to be something coming from cardiologists to say “yes, that is acceptable to discharge someone home”; you need to know there is a timely follow-up; it is hard to adopt rule if we don’t feel it is accepted widely by specialists as well. So, we feel supported to safely use it. (W#1, P1, W)
Co-designing strategies
Participants described that effectively addressing the above barriers requires multiple strategies deployed using various dissemination channels. The need to target a broad audience (i.e., ED physicians, cardiologists, internists) with consistent exposure over time (i.e., repeat messaging) was emphasized.
Grand rounds, presentations during physicians’ meetings, posters, study sheets all over the place, research assistants remind you to use it; seeing publications, seeing it on social media, pretty embedded in our group; importance of multiple strategies. (W#1, P4, M)
Participants also highlighted the need to leverage existing structures, including integration of the CSRS into the electronic medical record (EMR), discussing the CSRS at standing educational meetings, and displaying a poster in the workplace environment. Participants felt that holding combined grand rounds with ED physicians and specialists (i.e., cardiologists and internists) would be a coordinated strategy to addressing multiple barriers simultaneously. How these educational events are promoted is important to stimulate interest and excitement, including highlighting the credibility of the speaker. Having a journal club with ED physicians and specialists to review evidence around CSRS would be useful. Podcasts can be another interesting channel to disseminate knowledge around CSRS. Participants shared the example of emergency medicine reviews and perspectives (EM: RAP), a perceived trusted online resource, which is a monthly emergency medicine audio series encompassing continuing medical education. Displaying a poster in the ED was suggested as a helpful visual cue, with participants describing the usefulness of the CT head rule poster (37, 38) as an example.
Flow diagram of what you would do with each category; I think a lot of people in emergency medicine like “if this, then that,” to know which way to flow. That can help to take some of the thought process out if it as long as it is standardized across colleagues/specialists. We want to be practicing along with our colleagues and specialists, so having consensus with colleagues to follow the diagram with appropriate clinical practice and applying the rule appropriately, I think would help too. (W#1, P4, M)
Resulting action
The research team created two prototype posters for discussion in workshop 2 (Additional file 5). The two posters encompassed similar content with different displays. In poster #1, information around “For whom the CSRS must be applied” and “When to use CSRS” was highlighted. Poster #2 focused on the proposed course of treatment prior to the application of the CSRS. In both prototype posters, the CSRS was illustrated along with the three risk levels and their proposed practice-based recommendations. In response to the need for education about the CSRS, we distributed eight previously developed educational videos to participants of workshop 2 (Additional file 6). We asked participants to focus on the four videos that provided 1) an overview of CSRS, as well as the risk of serious adverse events and the recommendations for 2) low, 3) medium, and 4) high-risk patients.
Workshop 2
Participant feedback on barriers
Participants described the inapplicability of CSRS for some clinical presentations, a lack of buy-in from the broader medical team, and discomfort using CSRS. One participant explained the source of discomfort in using the tool as a feeling of hesitancy:
We reviewed this in journal club at [our hospital] and the biggest thing that came up and common to clinical decision rule is the clinical Gestalt at the end (…) Physicians felt hesitant using tool arguing it is telling me if it is vasovagal or cardiac syncope, seems counterintuitive to make decision in order to use tool to tell you something you already know. (W#2, P2, W)
Participants also highlighted the lack of evidence about how the CSRS practice-based recommendations impact patients’ outcomes as a barrier to uptake.
Co-designing strategies
Educational meetings (grand rounds) and materials (educational videos) were discussed extensively during the second workshop as relevant implementation strategies. Participants emphasized the need to expand the target of the implementation efforts and to onboard the head of department and nursing staff, in addition to physicians. There was agreement that educational meetings could be utilized to promote general awareness of the CSRS and to encourage a nuanced discussion about its application. However, a range of opinions were expressed on the best format for those educational meetings (e.g., combined grand rounds, case-based discussion in small group). Participants also highlighted the need for a local champion to model the application and use of the CSRS and further influence uptake among colleagues.
I think case-based rounds is great. I think to get hospital buy-in I’m thinking smaller community hospitals. I think having combined rounds with cardiology, medicine, and emerg to go over the score and how to apply it, and what are monitoring implications - I think that’s helpful as a group. That way the discussion happens with all the key players and the barriers to implementing this. As opposed to presenting it in silos really when a patient comes all these people are important, so combining a strategy could be helpful. (W#2, P2, W)
Excerpt of the educational videos were played during the workshop to solicit individual reactions and group discussion. All the components and features (e.g., written summary, questions, graphics, videos, and links to scientific papers) were perceived as useful and the content was perceived as clear, concise, relevant, and credible. The duration of videos was reasonable if viewed out of the workplace but were too long to be viewed during a shift. It was suggested that a five-minute video that includes the main information would be an ideal length and would facilitate wider dissemination of the CSRS.
When reviewing the two poster prototypes, participants suggested the need to simplify the posters, separating the explicative notes (i.e., additional information) from the care pathway and move those notes as footnotes using a different font (e.g., smaller fonts for footnotes) to make the content easier to read, and adding QR codes to facilitate immediate access to the CSRS online calculator.
Resulting action
We prepared a brief written summary of evidence (three short paragraphs) to support each practice recommendation for each patient group (low, medium and high-risk) (Additional file 7) and re-designed the posters in response to feedback and added three QR codes – 1) How to use CSRS, 2) Recommendation evidence, and 3) Online calculator. We also pulled e-mail communication (Additional file 9) from a previous pilot study [Remote Cardiac Monitoring of Higher-Risk Emergency Department Syncope Patients after Discharge (REMOSYNC) study; yet to be published], as an example to communicate positive feedback on patient impact (i.e., an example where home monitoring detected a patient arrythmia) as well as messaging to remind physicians to use the CSRS.
Workshop 3
Co-designing strategies
Participants discussed the perceived usability, usefulness, and operationalization of the following strategies: the online calculator, the summary of evidence, the poster, the local champion, and the e-mail communication.
All participants tested the CSRS online calculator prior to the workshop (Additional file 8). Participants suggested ways of improving the usability of the online calculator: 1) reviewing the wording of some criteria to avoid misleading interpretations, 2) adding a “not drawn” response option to this question “elevated troponin level”, 3) adding an access to evidence, and 4) adding access to practice-based recommendations for low, medium, and high-risk patients (i.e., what to do with the risk score). All participants had an intention to use the online calculator but for different purposes: utilization in practice and as an education tool with medical students. Some would use it only if it is integrated into EMR and will not use it if it is part of a mobile application.
Participants would find it useful to discuss in length the summary of evidence in grand rounds or in another type of educational meeting, as an initial evidence uptake. Getting easy access to evidence was considered important: tying evidence to online calculator and to EMR would be one way to improve its access.
Thinking back to other scores, or decision rules that are on calculator… It does bug me sometimes when I'm not able to access like a summary of why that's the recommendation or why that's the rule but again having an optional because if you already know it you don't need to come up every time if you forget or you want to know about the medium risk what exactly are the details having the option to go easily access from the rule would be nice. (W#3, P5, W)
Participants shared their reactions towards the second prototype poster and found it usable (i.e., easy to follow, simple, and appealing (nice colors)). They would use it as a reminder and as a prompt to apply CSRS. They would also refer their colleagues to this poster, which is seen as a way of giving credibility for their ED syncope management course of treatment.
I can refer to that poster, maybe give me a little bit of credibility if I'm advocating for an admission where I'm getting pushed back. (W#3, P2, M)
Participants highlighted that such a type of poster would be helpful, especially at the early stages of the CSRS implementation process in ED. However, participants identified some barriers of using such a poster: risk of poster fatigue and no space to display it in their clinical settings. They would probably not use the QR codes for these reasons: they assumed that if they used CSRS, they would already be familiar with the underlying evidence (so no need to scan QR codes for additional information); and perceived lack of skills of using QR codes. If one QR code had to be kept within the poster, it would be the online calculator.
Participants perceived that the local champion could play multiple roles: speaker in educational meetings, connection between the clinical and research team, monitors the implementation process over time, and provides in-person and/or written feedback:
After the six months check-up can be within the department if you have that champion, is the one that can do that link up. The first six months I think will give you enough information, does that local champion can be the one and that links back with the research team and see what is it at that point, having someone locally I think is significantly better to get like off the cuff comments and things like that and how they wish it was changed, applied or supports, I think it has better chance of getting quality feedback and regular feedback. (W#3, P3, M)
Finally, e-mail communication with feedback would be useful for ED physicians to convince them to use CSRS. Participants had different opinions on how and by whom feedback could be delivered, such as through educational outreach, one-to-one discussion with local champion, and e-mail communication. They would like to be provided feedback by the research team (especially the CSRS developer) and by a cardiologist within their hospital. This quote speaks to quality indicators that would be of interest:
I think for me anecdotal feedback is really helpful. So with the implementation of like the electronic records and EPIC (EMR), actually getting responses from the referrals that I make and similarly like for this type of thing, getting even anecdotal or like n of 5 feedback from cardiology on the results of the Holter monitor well, over time, I think, build up to convince me to use the rule. So I think that there should be someone at each site who's trying to collect that information, like, based on what was the risk level patient has, did they have a Holter or not? And are the numbers that we're seeing, matching up what the what the actual CSRS showed. (W#3, P5, W)
Resulting action
We synthesized the findings across the three workshops and selected the strategies to be developed and deployed as part of a future pilot study. The parameters of these strategies are outlined in Table 2.
Table 2
Summary of the strategies and their parameters over the three workshops
Strategy
|
Required content
|
Mode of delivery
|
Delivery source
|
Target audience
|
Target outcome
|
Educational meetings (e.g., grand rounds, case-based rounds, case discussion)
|
Nuances, barriers, and pitfalls when using CSRS; evidence underlying CSRS and recommendations; cost/resources (e.g., cardiac monitors)
|
Online
In-person
|
CSRS experts, cardiology, general medicine
|
All locations where CSRS will be applied
Diverse stakeholdersa
|
Improve knowledge of and comfort in using CSRS
Improve skills on how to use CSRS
|
Educational videos
|
How to deal with ultra-low-risk criteria and troponin; what to do with risk score
|
Online
|
CSRS experts
|
Journal clubs
|
Scientific papers - development and validation of CSRS
|
Online
|
CSRS experts
|
Medical staffb
|
Improve knowledge about evidence
|
Online calculator
|
How to deal with troponin criterion; what to do with the risk score (recommendations)
|
Webc, mobile application, EMR
|
Not applicable
|
CSRS usersd
|
Improve CSRS integration into workflow
|
CSRS integration into EMR
|
Interpretation of the risk score; what to do with the risk score
|
EMR
|
Local champion
|
Roles: Speaker, monitor the implementation process, support the teams; adapt and tailor implementation strategies; provide feedback.
Preferred attributes: Strong leadership skills, know how to apply CSRS and recommendations, positive influence on colleagues.
|
In-person
|
Local emergency medicine physicians and cardiologists (each site)
|
CSRS usersd
|
Improve collective buy-in
|
Poster
|
Care pathway, how to deal with troponin criterion, recommendations
|
Paper
QR codes
|
Not applicable
|
Diverse stakeholdersa
|
Improve collective buy-in
|
Dissemination of evidence summary
|
Impact of CSRS practice-based recommendations on patients’ outcomes
|
Online calculator, QR codes,
In-person
|
Electronic content
|
Diverse stakeholdersa
|
Improve knowledge
|
Feedback
|
CSRS impacts on providers’ practice; numbers of cardiac monitor referrals and of arrythmias detected
|
In-person and written
|
Champions
|
CSRS usersd
|
Improve skills and adoption of behaviour (CSRS uptake)
|
Prompts
|
Invitation to use CSRS, image with arrythmia detected (feedback)
|
Email
communication
|
CSRS experts
Champions
|
CSRS usersd
|
Remind physicians to use CSRS and its positive consequences
|
a - Emergency medicine physicians, family physicians working at ED, any consultants who are asked for high-risk patients, cardiologists, internists, nurses (including nurse practitioner), support from head of department |
b – Internal medicine, cardiology and emergency medicine physicians |
c – MDCalc (53) |
d – All emergency medicine physicians and residents |