Interactive Participatory Audit and Feedback (IPAF): Theory-based development and multi-site implementation outcomes with specialty clinic staff

doi:10.21203/rs.3.rs-26043/v1

Download PDF

Methodology

Interactive Participatory Audit and Feedback (IPAF): Theory-based development and multi-site implementation outcomes with specialty clinic staff

https://doi.org/10.21203/rs.3.rs-26043/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 31 May, 2021

Read the published version in Implementation Science Communications →

Version 1

posted

You are reading this older preprint version

Read the latest preprint version →

Background: Theory-based implementation strategies, such as audit and feedback (A&F), can improve adoption of evidence-based practices. However, few strategies have been developed and tested to meet the needs of specialty clinics. In particular, frontline staff can execute CVD risk-reduction protocols, but A&F strategies to support them have not been well examined. The objective of this paper is to develop a novel A&F strategy, Interactive Participatory A&F (IPAF), and evaluate as an implementation strategy to support specialty clinic staff.

Methods: We developed IPAF by combining theories to inform staff goals (Self-Regulation Theory) and address their psychological needs for relatedness, autonomy, and confidence (Self-Determination Theory). We facilitated IPAF fidelity by developing a theory-based facilitation tool: a semi-structured worksheet to guide flexible discussion of target behaviors, perceived barriers, goals, and action plans. We evaluated IPAF in the context of eight quasi-experimental implementations in specialty clinics across two health systems. Following a Hybrid Type 2 effectiveness-implementation design, we reported intervention outcomes for CVD risk-reduction elsewhere. This paper reports implementation outcomes associated with IPAF, focusing on feasibility, acceptability, fidelity, and adoption. We evaluated implementation outcomes using mixed-methods data including Electronic Health Records (EHR) data, team records, IPAF worksheets, and staff questionnaire responses.

Results: Eighteen staff participated in 99 monthly, individual, synchronous (face-to-face or by phone) IPAF sessions during the first six months of implementation. Subsequently, we provided over 375 monthly feedback emails. Feasibility data revealed high staff attendance (90-93%) and engagement in IPAF sessions. Staff rated questionnaire items about acceptability of IPAF highly. IPAF records and staff responses demonstrated fidelity of delivery and receipt of IPAF. Adoption of target behaviors increased significantly (all P-values < 0.05) and was maintained for over 24 months.

Conclusions: We developed and evaluated an interactive participatory A&F strategy to improve implementation of evidence-based interventions. The IPAF toolkit combines two complementary theories: Self-Regulation Theory and Self-Determination Theory. Findings support feasibility, acceptability, and fidelity of IPAF, and staff adoption and maintenance of target behaviors. By evaluating multi-site implementation outcomes in specialty clinics with frontline staff, we have extended prior research on clinic protocols and A&F beyond primary care settings and providers.

Health Policy

audit and feedback

theory-based strategies

implementation strategies

implementation outcomes

effectiveness-implementation hybrid

frontline staff

specialty clinics

preventive care

cardiovascular disease

medical assistants

Audit and feedback (A&F) is an established implementation strategy[1] to improve uptake of evidence-based practices, but its effectiveness remains variable across studies, with effect sizes stagnating since 2003.[2, 3] Implementation science experts argue that applying theory to design, measure, and report implementation strategies can strengthen implementations.[2, 4–7] Yet, in the most recent Cochrane review of 140 A&F trials to improve professional practice and patient outcomes, only 14% of studies (n = 20) used theory to inform A&F and only 9% (n = 13) reported how theory was applied.[4] In addition, key characteristics of how A&F strategies were designed and used are often missing or inconsistently addressed.[4, 6] Important characteristics include: the format and frequency of feedback, the role of the person giving feedback, the type of interpersonal interaction, and whether instruction is provided about specific future goals and action plans.[4, 6] Finally, when people share feedback verbally and in a face-to-face manner, their discussions with individuals are likely to differ. Such differences in interpersonal interactions can alter the consistency of delivery or receipt of A&F. Thus, researchers who study A&F need to address the fidelity, that is, the consistency of delivery and receipt of A&F over time and across participants.[8, 9]

A&F can promote guideline-concordant care by improving how health care professionals follow clinical practice protocols, but has primarily been studied with physicians, nurses, and pharmacists.[4] Evidence-based protocols can also be executed by frontline staff such as medical assistants or nurses. This is well illustrated in the area of cardiovascular disease (CVD) prevention, where clinic protocols executed by medical assistants have improved control of blood pressure (BP) from 50% to > 80% of patients in U.S. primary care settings.[10] In specialty clinics, however, clinicians discuss CVD risk factors such as BP or tobacco in only 10% of relevant specialty visits, despite routinely assessing them.[11, 12] Non-vascular specialty clinics have not implemented protocols to address high BP and tobacco use, which are the most prevalent risk factors for CVD in US adults. In particular, although many rheumatology populations face increased inflammatory CVD risks,[13] rheumatologists typically consider addressing CVD risk factors to be outside their scope of practice. Because frontline staff routinely assess risk factors such as BP and tobacco, integrating evidence-based protocols could promote routine action to address immediate care and follow-up needs. If specialty clinics employed CVD risk-reduction protocols, then they could reach many more of the two million specialty patients who have CVD events annually, recognizing that specialty visits are nearly equal to primary care visits in the US.[14]

The objective of this paper is to address the need for detailed reports on the use of theory to guide A&F, through development and evaluation of a theory-based strategy, Interactive Participatory A&F (IPAF). Specifically, we aimed to:

Develop IPAF along with a semi-structured tool to support application of IPAF with fidelity, guided by behavioral and motivational theories.
Evaluate IPAF in the context of eight quasi-experimental implementations of evidence-based interventions to reduce CVD risk.

Our evaluation of IPAF focused on the implementation outcomes of feasibility, acceptability, fidelity, and adoption.[9] By reporting on the application of IPAF in eight implementations as an exemplar, this paper describes methodology to enable others to provide theory-based, interactive participatory feedback to promote target behaviors.

Theory-Based Development of IPAF

Consistent with calls to apply theory to the development of interventions and implementation strategies,[4, 5, 15] we applied complementary, behavioral and motivational theories to develop a novel A&F strategy and select evaluation measures. The two theories we used were Self-Regulation Theory (SRT) and Self-Determination Theory (SDT). The following two sections describe how the theories guided our A&F development in a complementary way: SRT informed what goals to achieve and SDT informed how to interact to achieve goals.

How Self-Regulation Theory Guided IPAF Development

We applied SRT to the development of IPAF as an implementation strategy, consistent with recommendations by experts in SRT[16, 17] and A&F.[4, 6] As an illustration, assume a clinic goal is to increase rates of BP re-measurements to 80% among patients whose initial BPs were high. This clinic goal provides a standard against which we could compare staff’s rates of repeat BPs in an audit. Individual staff can iteratively compare their current behaviors against the clinic goal, whereby discrepancy can motivate behavior change.

The concept of discrepancy in SRT is critical to A&F at the individual level, as shown in Fig. 1. When feedback to individuals indicates that their behaviors are inconsistent with reference points, then people likely experience dissatisfaction.[5, 16, 17] If staff receive feedback that their behaviors are inconsistent with intervention goals that they agreed to support, then they likely would experience discrepancy. As a result, they would be motivated to improve their target behaviors, have future behaviors align with the intervention goals, and experience less discrepancy in the future.

Consistent with SRT, specific behavioral strategies are helpful in adopting new behaviors. These strategies include barriers identification, goal setting, and action planning,[18] as shown in Table 3. These are highly relevant to staff’s processes of adopting target behaviors, as demonstrated in Fig. 2.

Table 1

Intervention and Implementation Components: Context of IPAF application
Intervention Components
1) Check risk factor 2) Advise patient on risk factor relation to specialty condition 3) Connect patient to risk-reduction resources
Implementation Components
1) Engage staff in planning implementation
2) Educate staff about protocols and rationale
3) Remind staff of protocol steps using electronic health record alerts
4) Feedback based on audits of staff’s target behaviors, with IPAF approach: interactive, participatory discussions about barriers, solutions, goals, and action steps

Table 2

Characteristics of Participants
Table 2. Characteristics of Participants
Audit and Feedback (n = 18 staff)*	%
Registered Nurse	50
Medical Assistant	50
Female	94
Questionnaires (n = 30 staff)	%
Prior experience with quality improvement	40
Years of clinical experience
<1	14
1–4	36
5–10	14
>10	36
*n = 18 unique individuals (14 positions) over four 6-mo. periods

Table 3

Conceptual Rationale for the IPAF Components and Tool
IPAF Components of Discussion		Applications of Concepts from Theory	Theory
I. Introduction
A	Showing respect for people and offering choice.	Relatedness, interacting in a humanistic manner, and respecting autonomy meet human needs.	SDT
B	Clarifying what to expect about events can influence levels of anxiety and attention.	Beliefs influence emotions and behavior.	SRT
II. Clarifying purpose of meeting
A	Inviting exchange of information and clarifying purpose of feedback conveys respect. Avoiding judgment can create positive environment for dialogue, learning. Offering meaningful choices.	Relatedness and autonomy, volition or choice, can support corresponding behavior	SDT SRT
B	Clarifying purpose of project. If leaders build relationships, recognize how contextual factors can influence target behaviors, staff likely to believe leaders understand their situation.	Belief in purpose, relatedness, and autonomy can support corresponding behavior.	SRT SDT
C	Clarifying what to expect. Inviting exchange and offering choices. Recognizing, respecting staff’s abilities to contribute to project. Inviting mutual dialogue.	Relatedness, autonomy and perceived competence can support corresponding behavior	SDT
III. Offering choice
A	Offering meaningful choices can elicit staff participation.	Autonomy supports corresponding behavior	SDT
IV. Obtaining feedback from staff
A	Focus staff’s attention on their recent behaviors vs. project objectives; discuss capability and desirable behavior.	Comparison of current behavior vs. reference point, feedback, congruence can support behavior; competence and relatedness	SRT SDT
B	If leaders recognize how many factors can influence target behavior, staff may believe that leaders understand their situation, be non-judgmental, and share honestly.	Comparison, feedback, congruence; relatedness	SRT SDT
C	Acknowledging situations with less than ideal behavior and focusing on capabilities may invite honest feedback.	Congruence; perceived competence, relatedness	SRT SDT
V. Sharing performance data with staff
A	Focusing staff’s attention on their recent behaviors and project objectives can stimulate staff comparison of same.	Comparison, feedback, congruence	SRT
B	Offering choice of goal; Directing attention to short-term goals and project objectives.	Autonomy, congruence	SRT SDT
VII. Planning action steps
A	Offering choice of steps, planning collaboratively an action plan, setting up situation for building perceived competence to meet personal goals and project objectives.	Autonomy, perceived competence	SDT SRT
* Concept descriptions and assumptions are clarified on first mention. For brevity, subsequent mentions state only the concept.
SRT = Self-Regulation Theory[16, 19, 21, 35]; SDT = Self-Determination Theory[19–21]

We also more broadly applied SRT to the interventions and overall implementation approach, as shown in Fig. 1. Using SRT, researchers can describe, explain, and predict how people manage (i.e., regulate) themselves to reach their goals over time. At the organizational level, SRT applies to our choosing reference points for the patient population, that is, high BP or readiness to quit tobacco. At the staff and patient level, SRT applies to individuals; people need clear reference points against which to evaluate risk factors. In summary, SRT informed what goals to achieve.

How Self-Determination Theory Guided IPAF Development

We applied SDT, a theory regarding motivation and behavior,[19–21] to guide the development of IPAF, as shown in Table 3. According to SDT, all people have three, inherent psychological needs relevant to behavior: relatedness, autonomy (i.e., choice), and perceived confidence (i.e., self-efficacy).[19, 21] When these needs are met, people are more motivated to engage in relevant behaviors.[19, 21] Randomized, controlled studies, based on SDT, have demonstrated improvements in work behaviors.[19, 21–23] If staff’s psychological needs were met, then staff would be more motivated to adopt target behaviors than if their psychological needs were ignored or thwarted. Accordingly, we designed IPAF to meet staff’s psychological needs during individual feedback sessions. In summary, SDT informed how to interact to achieve goals.

Application of IPAF as an Implementation Strategy

Design

We evaluated implementation outcomes associated with IPAF in the context of a broader pre-, post-, quasi-experimental evaluation of CVD risk-reduction interventions.[24] That evaluation followed a Hybrid Type 2 effectiveness-implementation design,[25] attending to both intervention and implementation outcomes, We reported elsewhere the intervention outcomes: a) timely patient follow-up in primary care after high BPs and b) referrals to the tobacco quit line after assessing readiness to quit.[24, 26, 27]

In this paper, we evaluated the implementation outcomes of feasibility, acceptability, fidelity, and adoption,[9] focusing on measures relevant to IPAF. We delivered IPAF sessions to individuals synchronously (in-person or by phone) for six months after the beginning of implementation, and later asynchronously by email for over 24 months. Table 1 shows the components of the CVD-risk reduction interventions and the implementation package within which IPAF was used to provide feedback.

Setting and Sample

We evaluated IPAF in a total of eight implementations, representing two interventions (BP Connect and Quit Connect)[24, 27–30] in four separate rheumatology clinics in two US health systems. Clinics A, B, and C were in a large, suburban, academic, multi-specialty practice; clinic D was a community clinic. Rheumatology clinics offer an ideal setting and specialty population to evaluate A&F with frontline staff as a strategy to implement CVD risk-reduction.[10, 31, 32] Our IPAF participants were all medical assistants and nurses who performed pre-visit rooming (i.e., vital signs, patient history, etc.) at the clinics. We collected mixed-methods data including their responses to questionnaires, (EHR) data, and team records such as IPAF worksheets.

Context

The components of our interventions and implementation package are shown in Table 1. This paper focuses on the development and evaluation of IPAF, the implementation component for providing feedback to staff. IPAF was used to improve staff’s target behaviors with two CVD risk-reduction interventions: BP Connect for high BP[24, 28] and Quit Connect for tobacco use.[27, 29] With a Check-Advise-Connect structure for both interventions, the target behaviors were to Check for addressable risk factors, confirming high BPs or readiness to quit tobacco; Advise patients on CVD risk, and Connect patients to relevant resources. Connecting consisted of offering follow-up arrangements for BP appointments with primary care or quit-line phone calls for tobacco cessation counseling.

Pre-IPAF Staff Education

We held one-hour educational sessions with staff in small groups at the beginning of each implementation. We explained intervention rationale, principles, and components. We shared relevant evidence to address BP and tobacco, encouraged interactive discussion, and provided scenarios for staff role-plays regarding the Check-Advise-Connect behaviors. These staff had not previously had responsibilities for confirming or addressing CVD risk factors with patients (i.e., BP level or readiness to quit tobacco) or for referring patients to resources (i.e., primary care or quit line). The interactive educational sessions concluded with staff demonstrating mastery of role-play dialogue and navigation of the EHR, and receiving information about the monthly, individual IPAF feedback they would receive.

Interactive Participatory Audit & Feedback

We describe IPAF as interactive and participatory because our theory-based IPAF tool is a semi-structured worksheet that guides IPAF facilitators to collaboratively address staff’s psychological needs, systematically inviting them to interactively discuss barriers and action planning, to improve their target behaviors. IPAF sessions consisted of three theory-based components:

Providing feedback to individual staff about their actual rates of target behaviors and directing their attention to the ideal target behaviors for the intervention, based on SRT.[16, 17]
Interactive, one-on-one discussions of their experiences, including barriers and goals for target behaviors, while simultaneously supporting staff’s psychological needs, based on SDT.[19-21]
Eliciting action plans with staff about how they could improve rates of target behaviors, based on SRT and evidence for behavior change strategies.[18]

Consistent with best practices from the most recent Cochrane review of A&F,[4] we delivered feedback monthly, individually, face-to-face when possible, by a respected colleague (not a supervisor), to improve staff’s target behaviors. The source of feedback (i.e., A&F facilitator) in clinics A, B, and C was a physician known to staff, a leader in the settings, not a direct supervisor of staff, and the project’s principal investigator (CB). In clinic D, the facilitator was a nurse researcher from another organization, with expertise in supporting nurses, known to staff only from engagement activities (AGB). The context for feedback with individual staff was synchronous for the first six months of implementation, in-person for clinics A, B, and C and by phone for clinic D. The IPAF facilitator met with individuals for up to 10 minutes, privately in a clinic room or by phone, at a mutually agreed time. After the first six months of each implementation, we shared feedback asynchronously by email, along with questions for staff to share their barriers, goals, and action steps regarding target behaviors. Staff sent their responses and goals to the facilitator by email. The frequency of feedback was monthly amounting to at least four synchronous sessions per individual between months one and six of each implementation, and over 375 monthly emails for up to four years thereafter (2016–2019).

Facilitators and participants collaboratively followed the IPAF tool as a guide for IPAF during the synchronous IPAF sessions, shown in Fig. 2. The purpose of the tool was to support fidelity of delivery and receipt.[8, 9] Rather than being a rigid script, the tool consisted of a worksheet that was semi-structured to guide flexible discussions. It included what topics to address with staff, based on SRT, and how to support staff’s psychological needs, based on SDT. Table 3 presents the concepts, theories, and rationale for the IPAF components included in the tool.

Feedback sessions addressed the SRT concepts reflected in Fig. 1. Guided by the IPAF tool (Fig. 2), sessions addressed: a) individuals’ rates of target behaviors, b) range of peers’ rates of target behaviors, grouped by clinic; and c) the desired personal and organizational goals to minimize staff’s perceived discrepancy between their rates and intervention goals. The IPAF facilitator explained the altruistic, long-term goal of controlling CVD in the rheumatology population, to highlight potential discrepancy between actual and desired behaviors and stimulate staff motivation.

Feedback sessions also addressed SDT concepts.[20] Guided by the IPAF tool (Fig. 2), the IPAF facilitator explained that the intent was to be collaborative, not judgmental. To respect individuals’ autonomy, the facilitator offered choices about the order of discussion topics, starting with either feedback on individual-level data or discussing how they thought the intervention was going. The facilitator elicited from individuals their barriers to engaging in target behaviors, possible solutions, goals for target behaviors, and action steps for the upcoming month.

Evaluation of Implementation Outcomes Associated with IPAF

Data Collection

Feasibility. We evaluated feasibility by whether we were able to collect rates of individual staff’s target behaviors. Staff were to document these behaviors in new EHR data fields to enable monthly audits. Information technology staff were to report rates of these behaviors monthly to the implementation team. Additionally, we evaluated the degree of staff participation in IPAF sessions, based on team records including notes from facilitators and debriefing sessions facilitators had with other team members after each IPAF session. We assessed both staff’s attendance at feedback sessions and their engagement in planned steps for behavior change during these sessions.

Acceptability. We evaluated staff’s opinions with selected items from a 22-item written questionnaire. Staff answered anonymously at month six. Consistent with SDT concepts, participants reported the degree to which the project team listened to them, responded to their ideas, flexibly tailored protocols, addressed barriers, and collaborated in planning. Participants responded on a 5-point scale from 1, “not at all” to 5, “extremely”. We also asked staff for comments at the end of the questionnaires. Finally, team records, including facilitator notes and post-session debriefing, documented the degree of participants’ engagement during the one-to-one feedback sessions.

Fidelity. We evaluated fidelity based on monthly IPAF worksheets and staff questionnaire responses. Completion of all sections of the IPAF worksheet for each session indicated fidelity of delivery. We evaluated fidelity of receipt with relevant questionnaire items, based on SDT and SRT. These included the extent to which participants felt respected by staff, found problem-solving helpful, and experienced goal setting as motivational during the feedback sessions.

Adoption. We measured staff’s rates of target behaviors, consistent with the Check-Advise-Connect structure of the two interventions. We defined adoption of the Check behavior as the rate of confirming the CVD risk factor: re-measuring high BPs or asking about readiness to quit tobacco. We defined adoption of the Connect behavior as the rate of offering follow-up: appointments with primary care for high BP, or electronic referrals to be called by the tobacco quit line. Moreover, in addition to actual rates, our theories suggest that intentions and perceived confidence can predict behavior. On the monthly IPAF worksheets, we evaluated staff’s goals for future rates of target behaviors to reflect intentions at the first and last interactive session. In questionnaires, we asked staff to rate their perceived confidence in addressing CVD risk factors pre- and post-intervention, on a 5-point scale, retrospectively for clinics A, B, and C and prospectively for clinic D.

Analyses

We generated descriptive statistics, including frequencies and standard deviations, for responses to questionnaire items and target behaviors. Paired t-tests were used to compare staff’s pre- and post-questionnaire responses.

Protection of human participants

We received approval for our project through the University of Wisconsin-Madison’s Health Sciences Institutional Review Board and the Gundersen Health System’s Institutional Review Board. According to policies covering research activities at both institutions, our project met exemption criteria for operational improvement activities, with permission to publish. We summarized participants’ data in aggregate and did not share individual audits with the supervisors of participants.

Across our eight implementations, 18 different staff participated in IPAF sessions, and we received 30 responses from them on the month six questionnaires. We had 100% response rates among participants employed at the time the questionnaires were administered. All but one staff (94%) were female and most had no prior research or quality improvement experience. Table 2 shows characteristics of IPAF participants and questionnaire respondents.

Feasibility

Both the audit and feedback components of IPAF demonstrated feasibility. Information technology staff were able to retrieve all monthly rates of target behaviors from the EHR, including new EHR fields documented by staff. We were able to provide feedback in 99 total monthly in-person or phone sessions with 18 different staff and over 375 monthly emails thereafter. Staff attendance to the feedback sessions was high: 90% at 72 of 80 planned sessions in clinics A, B, and C, and 93% at 25 of 27 planned sessions in clinic D. When participants missed a session, they were absent from work for illness or personal reasons. The sessions lasted seven minutes on average (range 3–10 minutes).

According to team records, including facilitator notes and IPAF worksheets, all participants engaged in identifying barriers to target behaviors, setting goals, and making action plans. In notes and debriefings, IPAF facilitators observed that staff were willing and able to set goals when prompted. Review of the IPAF worksheets showed that when staff were unable to set a percentage goal for infrequent behaviors, the worksheet accommodated setting a numerical goal (e.g. n = 2 quit line referrals). Across team records, a common approach was to mutually determine suggested goals based on prior rates, and achievable expectations, and the facilitator’s sense of an individual’s motivation in the project overall.

Acceptability

Responses to questionnaire items relevant to IPAF revealed high acceptability. In clinics A, B, and C, respondents (n = 20) reported that the project team listened (mean = 4.47 +/- 0.76 standard deviation); were responsive, (4.46 +/-0.73); flexible to ideas, (4.22 +/-0.97); and collaborative (4.44+/-0.53). Table 4 shows further details from all implementations. In open-ended questionnaire responses, staff reported appreciating seeing their data and practicing what they would say with patients in both training and individual IPAF sessions. Several staff reported eagerness to disseminate the protocols and implementation strategies, including IPAF, to other specialty clinics.

Table 4

Data on Implementation Outcomes: Acceptability, Fidelity, and Adoption
	BP Connect at Clinics A, B, C	Quit Connect at Clinics A, B, C	BP Connect at Clinic D	Quit Connect at Clinic D
	Mean (SD) unless otherwise indicated
Acceptability (questionnaire)	n = 10	n = 10	n = 4	n = 6
Listening	4.33 (1.00)	4.60 (0.52)	3.75 (0.96)	3.50 (1.38)
Responsiveness	4.22 (0.97)	4.70 (0.48)	3.75 (0.96)	4.00 (0.63)
Flexibility	4.22 (0.97)	NA	3.50 (1.29)	NA
Collaboration	4.44 (0.53)	NA	3.50 (1.00)	NA
Fidelity (questionnaire)	n = 10	n = 10	n = 4	n = 6
Respectfulness	4.90 (0.32)	4.89 (0.33)	4.00 (0)	4.67 (0.58)
Problem solving	4.90 (0.42)	4.78 (0.44)	3.33 (0.58)	4.67 (0.58)
Motivation from goals	3.70 (0.82)	4.56 (0.73)	2.50 (1.29)	NA
Confidence (pre)	2.60 (1.07)	2.10 (1.10)	3.33 (0.81)	2.5 (0.55)
Confidence (post)	4.00 (0.47)	4.30 (0.67)	3.75 (0.50)	3.67 (0.52)
Adoption (IPAF worksheets/ EHR data)
	n = 40	n = 32	n = 12	n = 15
Check goals at first session	50–90%	75%-100%	NA	100%
Check goals at last session	75–95%	75%-100%	NA	70%-100%
Check actual at baseline	2%	3%	0%	90%
Check actual 6 months post	98%	100%	80% peak	98%
Connect goals at first session	50–100%	50%-95%	NA	33%-80%
Connect goals at last session	50–100%	80–100%	NA	80%-85%
Connect actual at baseline	0%	< 1%	0%	< 1%
Connect actual 6 months post	73%	76%	68%	53% peak
* NA: data not available; BP Connect was implemented after Quit Connect in Clinic D; Clinic D A&F sessions were by phone; Connect step: offer primary care BP follow-up or tobacco quit line phone call; Pre-post confidence was measured retrospectively for clinics A, B, C and prospectively for D.

Fidelity

Data from the IPAF worksheets documented the completion of > 80% of all goal-setting topics and other discussion sections, supporting fidelity of delivery. Detailed findings are in Table 4. To summarize, in clinics A, B, and C, respondents (n = 20) reported: the implementation team was respectful (4.90 +/-0.33); problem-solving was helpful, (4.79 +/-0.43), and goal setting was motivational (4.13 +/-0.78). When asked about perceived confidence addressing BP and tobacco, only 10–20% said they had been “very” or “extremely confident” at baseline while 90% reported they were “very” or “extremely confident” after six months (P = 0.001), as shown in Fig. 3.

Table 4 shows further details from all eight implementations including our replication site, clinic D. Overall, responses from implementations in clinic D showed lower scores. That health system was understaffed and had 66% clinic staff turnover during implementation, thus new staff received abridged engagement and education.

Addressing Barriers. Using the IPAF tool to guide fidelity, facilitators asked participants to identify barriers to adoption of target behaviors when prompted. These barriers included: having high patient volume; being under-staffed; working with float staff; and working with some providers who rushed processes for rooming patients. Participants also generated solutions for barriers during IPAF sessions. For example, participants needed to address a barrier of time pressure to measure and re-measure BPs before visits with providers. Participants offered a solution using red laminated reminder cards on desks in patients’ exam rooms to alert providers and staff that repeat BPs were needed after provider visits. Using their solution resulted in a subsequent increase in rates of BP re-measurement.

Building Rapport. In addition to the structured discussion, the monthly IPAF sessions guided by the semi-structured IPAF tool, also afforded opportunities for reciprocal flexible communication, consistent with SDT. It was not uncommon for participants to share unsolicited details on changes in the work system or perceived staff views on the intervention or implementation process. Based on facilitator debriefings and participant comments, by having a consistent point of contact, facilitators also formed relationships with participants, so additional communication was able to occur. This fostered rapport and made going through structured feedback items more comfortable for both facilitators and participants.

Adoption

In the first IPAF sessions, staff’s goals for the Check behavior varied from 50–100%. In contrast, in the last IPAF sessions, staff’s goals for this behavior increased to 75–100%. Based on EHR data, staff’s actual rates of the Check behavior improved from 2–3% to 98–100% pre- to post-implementation in clinics A, B, and C. Clinic D had a peak of 80% for BP Connect, though ranges varied while experiencing understaffing. Staff’s rates of the Connect behavior rose from 0–1% to 68–76% pre- to post-implementation, with the exception of Quit Connect at clinic D. Table 4 shows further details from all eight implementations.

Advancing Research on Theory-based A&F as an Implementation Strategy

We have developed and evaluated a theory-based A&F strategy by applying complementary behavioral and motivational theories. Responding to the need for detailed reports of how theory guides A&F, [4] we delineated the theoretical concepts informing our IPAF components and tool in Table 3 and Fig. 2, respectively. Furthermore, we have posted these resources online as a toolkit available to download at https://www.hipxchange.org/AuditFeedback.[33]

Our application of complementary theories aligns with calls from implementation science experts to apply theories in general, and SDT/SRT in particular, to guide A&F. [4, 6] Our combination of theories and evidence about behavior change to guide IPAF is innovative. We applied concepts and evidence from SRT to inform different levels of our project, supported by behavior change research,[16, 17] which we incorporated into IPAF components such as barriers identification and problem solving.[16–18] We applied SDT concepts to interpersonal processes during IPAF sessions to support staff’s psychological needs and motivation for adoption of target behaviors.[21]

Advancing Research on Implementing Clinic Protocols with A&F

We have extended prior research on clinic protocols in primary care[10] and on A&F with physicians, pharmacists, and nurses by primarily targeting front line medical assistants.[4, 5, 10] We found that IPAF is a feasible and acceptable A&F strategy to use with frontline staff, such as medical assistants and nurses conducting pre-visit rooming (e.g. vital signs, patient history, etc.) in specialty clinics.

Feasibility was evidenced by: high rates of staff attendance; participation in the IPAF sessions; staff’s willingness to document Check and Connect target behaviors in new EHR fields; and our ability to retrieve these data for our audits. Our findings indicate that frontline staff in specialty clinics can co-participate in improvement processes and that our interactive participatory A&F strategy can support such improvements.

Acceptability was evidenced by positive staff reports about implementation. For example, in IPAF sessions, staff reported that problem solving was helpful and that goal setting was motivating. It was important to document support for acceptability because staff can resist change, especially since we had asked staff to adopt new professional roles for CVD risk reduction.

Fidelity of delivery was evidenced by the facilitators’ documentation on the IPAF worksheet. Fidelity of receipt was evidenced by staff’s responses to questionnaire items that reflected SRT and SDT concepts (e.g., IPAF discussions felt respectful) and by the fact that staff’s perceived confidence was higher after the implementations than prior to them.

Adoption was evidenced by improved rates of target behaviors at six months, and maintenance over 24 months, including checking risk factors (i.e., BP and tobacco) and connecting patients to CVD risk-reduction follow-up.[24, 26, 27, 34] Such positive findings are consistent with the theories guiding our project. When SDT concepts have been addressed in behavior change interventions, participants have not only initiated new behaviors but also maintained these new behaviors.[26] One explanation for the maintenance of target behaviors is that staff’s psychological needs were met by our application of humanistic concepts from SDT including interpersonal interactions with IPAF. Our findings indicate that nurses and medical assistants in specialty clinics can be partners in the implementation of clinic protocols and the collaborative use of implementation strategies such as IPAF to reduce CVD risks.

Limitations

Although this paper establishes strong multi-site implementation outcomes of a novel theory-based A&F strategy, we acknowledge limitations in our quasi-experimental design and the evaluation of IPAF in the context of other implementation components. Moreover, our reported sample size of staff receiving IPAF (n = 18) was modest. This limitation is arguably offset by our multi-site findings representing eight implementations across two health systems, including 99 feedback sessions and more than 375 monthly feedback emails offset this. Another limitation is that audit data for float staff were aggregated within staff data overall, yet float staff did not receive IPAF. Also, as a result of turnover, new staff received abridged engagement and education prior to IPAF, which may have undermined some SDT and SRT elements. Additionally, findings based on staff self-reports may be attributed to social desirability influences, given that the principal investigator was the IPAF facilitator in three of the four clinics. However, similar gains were confirmed in a different health system, where another facilitator successfully used the IPAF process and semi-structured tool.[34] Finally, our pre- post- measurement of staff’s perceived confidence was initially retrospective, which we addressed with prospective questionnaires in the fourth clinic.

Future research

Researchers and clinical improvement teams can apply and extend our innovative, theory-based IPAF approach to evaluate improvements in staff adoption of other target behaviors in other settings. Teams can download our IPAF toolkit, including its worksheet and theoretical details, at https://www.hipxchange.org/AuditFeedback. [33] We plan a future clinic randomized controlled study, with active A&F comparisons of individual versus group feedback, and larger sample sizes, to further test these methods.

We developed and evaluated IPAF, an interactive participatory A&F strategy to improve implementation of evidence-based interventions, building on two complementary theories, SRT and SDT. Findings support feasibility, acceptability, and fidelity of IPAF, as well as staff adoption and maintenance of target behaviors. By evaluating implementation outcomes in multi-site specialty clinics with medical assistants and nurses, we have extended prior research on clinic protocols and A&F beyond primary care settings and providers. Researchers and clinical improvement teams can use IPAF to help improve clinical outcomes such as reducing CVD risk factors.

Audit and Feedback (A&F)

Cardiovascular Disease (CVD)

Blood Pressure (BP)

Interactive Participatory Audit and Feedback (IPAF)

Electronic Health Records (EHR)

Self-Regulation Theory (SRT)

Self-Determination Theory (SDT)

Ethics approval and consent to participate

We received approval for our project through the University of Wisconsin-Madison’s Health Sciences Institutional Review Board (IRB) and the Gundersen Health System’s IRB. According to policies covering research activities at both institutions, our project met exemption criteria for operational improvement activities, with permission to publish. We summarized participants’ data in aggregate and did not share individual audits with the supervisors of participants.

Consent for publication

Not applicable.

Availability of data and material

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request per IRB approval.

Competing interests

All authors declare that they have no competing interests.

Funding

This work was supported by a peer-reviewed, independent investigator grant (PI-Bartels) titled: “Systems-Based CVD Prevention Protocols for Rheumatology Teams: A low cost multidisciplinary approach” from Independent Grants for Learning and Change (Pfizer) and Stepping Up in Specialty Clinics to Reduce Blood Pressure (NIH-NCATS UL1TR000427, through a Clinical & Community Outcomes Research Pilot (PI-Bartels) Clinical and Translational Science Award (CTSA) program, through the NIH National Center for Advancing Translational Sciences (NCATS), grant as well as the UW School of Medicine and Public Health. Bartels also received support from National Institutes of Health (NIH) National Institute of Arthritis, Musculoskeletal and Skin Diseases (NIAMS) (K23 #AR062381) for baseline data. The authors have no other direct financial, consultant, or institutional conflict of interest pertaining to this article. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. Funders had no role in the design, collection, analysis or interpretation of data, writing the manuscript, or deciding to submit for publication.

Authors’ contributions

ER and DRL are joint first authors. ER provided conception of the effectiveness-implementation hybrid design, implementation outcomes, all implementation components, and two interventions; acquisition, analysis, and interpretation of data; drafting and substantive revisions of the article. DRL guided conception of the theory-based audit and feedback strategy; analysis and interpretation of data; drafting and substantive revisions of the article. AGB provided acquisition, analysis, and interpretation of data; substantive revision of the article; CB obtained funding, provided conception of the two interventions, all four implementation components, and effectiveness-implementation hybrid design; acquisition, analysis, and interpretation of data; drafting and substantive revisions of the article. All authors read and approved the final manuscript.

Acknowledgements

The authors would like to thank Courtney Maxcy, Sarah Loring, Amanda Perez, Monica Messina, and the Health Innovation Program Staff for help with data collection and manuscript preparation.

Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, Proctor EK. and J.E. Kirchner. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21. PMCID: PMC4328074.
Colquhoun HL, Carroll K, Eva KW, Grimshaw JM, Ivers N, Michie S, Sales A. and J.C. Brehaut. Advancing the literature on designing audit and feedback interventions: identifying theory-informed hypotheses. Implement Sci. 2017;12(1):117. PMCID: PMC5622490.
Ivers NM, Grimshaw JM, Jamtvedt G, Flottorp S, O’Brien MA, French SD, Young J. and J. Odgaard-Jensen. Growing Literature, Stagnant Science? Systematic Review, Meta-Regression and Cumulative Analysis of Audit and Feedback Interventions in Health Care. J Gen Intern Med. 2014;29(11):1534–41.
Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O'Brien MA, Johansen M, Grimshaw J, Oxman AD. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Db Syst Rev. 2012(6):CD000259.
Ivers NM, Sales A, Colquhoun H, Michie S, Foy R, Francis JJ, and J.M. Grimshaw. No more 'business as usual' with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci. 2014;9:14. PMCID: 3896824.
Colquhoun HL, Brehaut JC, Sales A, Ivers N, Grimshaw J, Michie S, Carroll K, Chalifoux M, and K.W. Eva. A systematic review of the use of theory in randomized controlled trials of audit and feedback. Implement Sci. 2013;8:66. PMCID: 3702512.
Sales A, Smith J, Curran G, Kochevar L. Models, strategies, and tools - Theory in implementing evidence-based findings into health care practice. J Gen Intern Med. 2006;21:S43-S49. PMCID: 2557135.
Bellg AJ, Borrelli B, Resnick B, Hecht J, Minicucci DS, Ory M, Ogedegbe G, Orwig D, Ernst D, Czajkowski S. Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH behavior change consortium. Health Psychology. 2004;23(5):443–51.
Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Griffey R. and M. Hensley. Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda. Adm Policy Ment Health. 2011;38(2):65–76.
Frieden TR, King SM, Wright JS. Protocol-based treatment of hypertension: A critical step on the pathway to progress. JAMA. 2014;311(1):21–2.
Vreede AP, Johnson HM, Piper M, Panyard DJ, Wong JC, Bartels CM. Rheumatologists modestly more likely to counsel smokers in visits without rheumatoid arthritis control: an observational study. J Clin Rheumatol. 2017;23(5):273–7. PMCID: PMC5590221.
Bartels CM, Johnson H, Alcaraz Voelker K, Ogdie A, McBride P, Jacobs EA, Zhao YQ, Smith M. Frequency and Predictors of Communication About High Blood Pressure in Rheumatoid Arthritis Visits. J Clin Rheumatol. 2018;24(4):210–7. PMCID: PMC5962363.
Avina-Zubieta JA, Thomas J, Sadatsafavi M, Lehman AJ, Lacaille D. Risk of incident cardiovascular events in patients with rheumatoid arthritis: a meta-analysis of observational studies. Ann Rheum Dis. 2012;71(9):1524–9.
Aitken M, Kleinrock M, Lyle J, Caskey L. Medicine Use and Shifting Costs of Healthcare: A review of the use of medicines in the United States in 2013, I.I.f.H. Informatics, Editor. 2014, IMS Institute for Healthcare Informatics: Parisppany, NJ. p. 1–59.
Grol RP, Bosch MC, Hulscher ME, Eccles MP, Wensing M. Planning and studying improvement in patient care: the use of theoretical perspectives. Milbank Q. 2007;85(1):93–138. PMCID: 2690312.
Scheier M, Carver C, Armstrong G. Behavioral self-regulation, health, and illness, Handbook of health psychology, A. Baum, T. Revenson, and J. Singer, Editors. 2012, Psychology Press: New York. pp. 79–98.
Carver CS, Scheier MF. On the self-regulation of behavior. New York: Cambridge University Press; 1998.
Michie S, Abraham C, Whittington C, McAteer J, Gupta S. Effective techniques in healthy eating and physical activity interventions: a meta-regression. Health Psychol. 2009;28(6):690–701.
Deci EL, Olafsen AH, Ryan RM. Self-determination theory in work organizations: the state of a science. Annu Rev Organ Psych. 2017;4(1):19–43.
Ng JY, Ntoumanis N, Thogersen-Ntoumani C, Deci EL, Ryan RM, Duda JL, Williams GC. Self-determination theory applied to health contexts: a meta-analysis. Perspect Psychol Sci. 2012;7(4):325–40.
Ryan RM, Patrick H, Deci EL, Williams GC. Facilitating health behaviour change and its maintenance: Interventions based on self-determination theory. European Health Psychologist. 2008;10(1):2–5.
Fernet C, Austin S, Vallerand RJ. The effects of work motivation on employee exhaustion and commitment: An extension of the JD-R model. Work Stress. 2012;26(3):213–29.
Moreau E, Mageau G. The importance of perceived autonomy support for the psychological health and work satisfaction of health professionals: Not only supervisors count, colleagues too! Motiv Emot. 2012;36(3):268–86.
Bartels CM, Ramly E, Johnson HM, Lauver DR, Panyard DJ, Li Z, Sampene E, Lewicki K, McBride PE. Connecting Rheumatology Patients to Primary Care for High Blood Pressure: Specialty Clinic Protocol Improves Follow-up and Population Blood Pressures. Arthritis Care Res (Hoboken). 2019;71(4):461–70. PMCID: PMC6274604.
Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217–26. PMCID: PMC3731143.
Ramly E, Panyard D, Lauver D, Sampene E, Li Z, Johnson H, McBride P, Lewicki KS, Bartels CM. Sustained improvement in follow-up of hypertension in rheumatology patients: Results of an intervention sustainability assessment [abstract]. in American College of Rheumatology. 2016. Washington, DC: Arthritis Rheumatol.
Bartels CM, Ramly E, Panyard D, Lauver D, Johnson H, Li Z, Sampene E, Piper M. and M. McBride. Rheumatology Clinic Smoking Cessation Protocol Markedly Increases Quit Line Referrals [abstract]. Arthritis Rheumatol. 2017;69 (suppl 10).
Bartels CM, Ramly E, Panyard D, Lauver DR, Johnson HM, Lewicki K, McBride P. BP Connect Health Toolkit. 2017 [cited 2018 February 5]; Available from: https://www.hipxchange.org/BPConnectHealth.
Bartels CM, Ramly E, Panyard D, Lauver DR, Johnson HM, Lewicki K, McBride P. Quit Connect Health Toolkit. 2017 [cited 2018 February 5]; Available from: https://www.hipxchange.org/QuitConnectHealth.
Bartels CM, Panyard D, Lauver D, Sampene E, Li Z, Adsit R, McBride P, Johnson H, Lewicki K, Ramly E. Feasibility of a rheumatology staff protocol for tobacco cessation counselling and Quit Line electronic referral [abstract]. Arthritis Rheumatol. 2016;68(suppl 10).
Bartels CM, Johnson H, Voelker K, Thorpe C, McBride P, Jacobs EA, Pandhi N, Smith M. Impact of rheumatoid arthritis on receiving a diagnosis of hypertension among patients with regular primary care. Arthritis Care Res (Hoboken). 2014;66(9):1281–8. PMCID: PMC4269481.
Primdahl J, Clausen J. and K. Horslev-Petersen. Results from systematic screening for cardiovascular risk in outpatients with rheumatoid arthritis in accordance with the EULAR recommendations. Ann Rheum Dis. 2013;72(11):1771–6.
Bartels CM, Ramly E, Lauver D. Guide for Staff Interactive Participatory Audit - Feedback. 2018 [cited 2018 November 5]; Available from: http://www.hipxchange.org/AuditFeedback.
Chodara AM, Ramly E, White D, Johnson H, Gilmore-Bykovskyi A. and C.M. Bartels. Implementing a Staff Tobacco Cessation Protocol Increases Quit Line Referrals in a Community Rheumatology Practice [abstract]. Arthritis Rheumatol. 2018;70 (suppl 10).
Michie S, Fixsen D, Grimshaw JM, Eccles MP. Specifying and reporting complex behaviour change interventions: the need for a scientific method. Implement Sci. 2009;4:40. PMCID: 2717906.

Download PDF

Journal Publication

published 31 May, 2021

Read the published version in Implementation Science Communications →

Version 1

posted

You are reading this older preprint version

Read the latest preprint version →

Interactive Participatory Audit and Feedback (IPAF): Theory-based development and multi-site implementation outcomes with specialty clinic staff

Status:

Journal Publication

Version 1

Abstract

Figures

Background

Methods

Theory-Based Development of IPAF

How Self-Regulation Theory Guided IPAF Development

How Self-Determination Theory Guided IPAF Development

Application of IPAF as an Implementation Strategy

Design

Setting and Sample

Context

Pre-IPAF Staff Education

Interactive Participatory Audit & Feedback

Evaluation of Implementation Outcomes Associated with IPAF

Data Collection

Analyses

Protection of human participants

Results

Feasibility

Acceptability

Fidelity

Adoption

Discussion

Advancing Research on Theory-based A&F as an Implementation Strategy

Advancing Research on Implementing Clinic Protocols with A&F

Limitations

Future research

Conclusions

List of Abbreviations

Declarations

References

Supplementary Files

Status:

Journal Publication

Version 1