Phase One - Development
Part A – Identification of the items for inclusion on the draft checklist
A total of 205 items were identified from the literature review for consideration by the RAND consensus panel [11] who judged 37 to be of “high priority”. Following comparison with the CQC, CQBT and UK mandatory requirements and the review of the study team for repetition and clarity a total of 13 items were eventually included on the draft checklist.
Part B – Refinement of the draft checklist to produce the final prototype
A total of ten East Midlands (EM) and six Greater Manchester (GM) practices tested the draft checklist. A representation of the characteristics of the practices participating in the toolkit project can be found in Table 1. Within these practices a total of 38 respondents (23 from EM and 15 from GM) completed the pilot version of the checklist and accompanying questionnaire (23 GPs, four nurses, nine practice managers, and two administrative staff). An additional 25 staff from participating practices were interviewed with regard to the applicability of the PST (three GP registrars, three practice nurses, six practice managers and 13 GPs)
Table 1 Summative characteristics of participating practices [10]
|
List Size*
|
Under 18*
|
65+*
|
% Non-White**
|
Deprivation
Score (based on IMD score [21])*
|
QOF Score (2013)*
|
% Female**
|
Practice Average/SD1
|
7363/
2830
|
20.7%
3.1%
|
19.5%
3.3%
|
13.1%
15.2%
|
22.5
9.1
|
988.1
7.6
|
52.9%
3.8%
|
English
Average
|
7041*
|
20.8%*
|
16.7%*
|
13%**
|
21.5*
|
961*
|
51%**
|
Practice Median/IQR1
|
6804
4327
|
19.7%
5.9%
|
20.9%
5.8%
|
1.6%
30.9%
|
17.8
22.2
|
989.4
10.2
|
52.4%
8.0%
|
*taken from the national general practice profiles (Public Health England) www.apho.org.uk/PRACPROF/
**taken from the GP patient survery July 2014 http://practicetool.gp-patient.co.uk/practice
The practice average, standard deviation, median, and IQR use values that are weighted by the practice list size
At this stage the checklist was divided into four sections (information flow, safety information about the practice, prescribing, and use of IT systems), each with an introductory statement that was taken directly from our project taxonomy of patient safety [16].
A summary of the results from the development of the PSC can be found in Table 2 relating to one of the domains. Within each we define the domain, present the draft items, the rationale and evidence for each, the changes made as a result of the feedback collated from the questionnaires and semi-structured interviews and the related item as it appeared on the PSC which numbered nine in total.
Table 2: Development of Items for inclusion in prototype checklist
Development of draft checklist
|
Prototype Checklist
|
Description
|
Item (s)
|
Rationale
|
Evidence
|
Refinement
|
Item
|
Information flow
The practice has a systems based approach to processing incoming results* and information in to and out of the practice, which prevents human and electronic error in data handling.
*results = lab results, reports or investigations, and letters.
|
All incoming clinical information is seen by a GP in the practice to view and action before or after being filed, scanned or coded in the patient’s medical record.
|
The practice has a systems based approach to processing incoming results* and information into and out of the practice, which prevents human and electronic error in data handling [* results: lab results, reports or investigations]and letters) [11].
|
(PMCPA) [43] (Premises Records Equipment/Devices and Medicines management section) (http://www.rcgp-practiceaccreditation.org.uk
|
Wording changes - in order to allow practices to nominate appropriately trained staff to handle mail rather than just GPs
|
All incoming clinical information is seen by nominated members of the team trained (or with relevant clinical experience) to deal appropriately with this information before the information is filed in the patient’s record.
|
Where an incoming result, report or investigation requires follow-up or a diarised activity, it is recorded in the patient’s medical record and acted upon [for example follow up of blood tests such as PSA, INR etc.]
|
Where an incoming result, report or investigation requires follow-up there are systems in place to ensure it occurs.
|
Adapted from PCMPA [43] and informed by Casalino et al. [44].
|
These items were combined as they were felt to be too similar
|
Where a clinician decides it is indicated, the patient (or where appropriate the patient’s representative) is informed of abnormal investigation results in an appropriately and timely manner and this contact is documented in the patient’s record.
|
The patient (or where appropriate, families and carers) is informed of an abnormal investigation results in an appropriate and timely manner and this is documented in the patient’s record.
|
The provider has a written policy for informing patients, or where appropriate, families and carers, of the results of investigations and the policy is explained to them.
|
Adapted from PMCPA [43].
|
The practice keeps a record or log of their minor operations which will have the following information recorded; 1) date; 2) patient name; 3) procedure performed; 4) team members involved; 5) whether a specimen was sent for histology; 6) patient consent; 7) complications; 8) patient informed of result.
|
This log represents the basic safety information required about any surgery performed
|
Taken direct from PMCPA [43] (provider management), a template could easily be designed to collect this information
|
|
The practice keeps a log of minor operations
|
Safety information about the practice
The practice has a systems based approach to supplying information about safety procedures required by permanent and temporary staff.
|
Up-to-date information on the practice policies and procedures, and local facilities and services is provided to guide locums and other temporary clinical staff who work in the premises, in the form of a clinical staff handbook (hard copy).
|
There is no current legislative requirement specifically directed at trainees or temporary staff.
|
Dutch consensus process exploring safe working conditions from locum staff [45].
|
Items combined as seen as too similar.
Requirement for hard copy information was removed after discussion within our project team, considering the change to paper-light practices.
|
Up-to-date information on practice policies, procedures and local facilities/services is provided to guide all temporary clinical staff (including GP registrars).
|
There is an up-to-date office procedure manual (hard copy and/or electronic copy) covering the administrative procedures and systems for the daily running of the practice to which team members have access. These policies are discussed and agreed by team members and are reviewed at least annually.
|
There is no central policy document of safety procedures readily available to all staff.
|
Review of factors supporting successful teamwork in primary care [46].
|
|
|
Working with patients for safe prescribing
The practice has a systems based approach to working with patients to improve the safety of prescribing practices.
|
The practice works with patients to ensure medication list accuracy (medication reconciliation) upon hospital referral.
|
No such process for medicine reconciliation exists despite the potential impact on patient safety.
|
Review of reconciliation issues [47, 48].
|
Removed as seen as being beyond practice’s control.
|
|
Non-collection of prescriptions held by the practice are monitored and followed-up by the practice and medications which are not claimed by patients are a trigger for review and audit in partnership with local pharmacies.d
|
Non-collection might reflect medication error, poor compliance or other patient safety issues.
|
A study of medication reviewing in primary care27 and is also included in PMCPA [43].
|
Wording changes - in order to simplify the item.
|
Non-collection of prescriptions is monitored or followed-up and is a trigger for review and audit in partnership with local pharmacies.
|
|
Patients discharged from hospital should have a recorded follow up appointment with a member of the practice clinical team within 1 month.
|
Patients at high risk of patient safety incidents should be followed-up at risky care transitions.
|
Originally from a US process mapping study [49].
|
Clinicians believed that it was unrealistic to follow-up all of the discharges within one month so we added the word ‘vulnerable’ to this item.
|
Vulnerable patients discharged from hospital are followed-up by a member of the clinical team within 1 month.
|
IT indicators for prescribing
The practice has fit for purpose IT systems for prescribing which work with prescribers to make prescribing a safer activity.
|
The practice uses an electronic prescribing system for all prescriptions (Computerized Physician Order Entry (CPOE)).
|
Drive to implement CPOE primarily comes from its presumed benefit in reducing medical errors.
|
Evidence of CPOE can reduce medication errors [50].
|
Removed - did not allow practices enough flexibility to serve patients.
|
|
Prescribers code the indication for the drug with each prescription using the electronic prescribing system (with the exception of topical medications without active ingredients).
|
This is good practice and there is currently no legislative requirement for it to be done.
|
Canadian study of electronic coding of prescription indication [38].
|
Wording changed for clarity.
|
The indication for all repeat medications is coded within the electronic record (excluding topical preparations).
|
The practice has and uses, the most up-to-date alerting software available, routinely on all computers used for prescribing in relation to allergies and duplicates, drug-drug interactions, contraindications in terms of drug –disease, drug-age and potentially drug-lab value interactions.
|
The safety features of software systems are effective in alerting users about potential clinical hazards and errors during medication order entry.
|
Delphi study on electronic safety systems [51].
|
Felt to be imbedded in the computer systems.
|
|
All staff (including GPs) are trained to make safe use of the prescribing elements of their clinical IT systems.
|
Specific training in IT prescribing systems is not a mandatory requirement and yet is essential for all team members involved in prescribing.
|
Delphi study on electronic safety systems [51].
|
Wording changed for clarity.
|
All staff are trained to make safe use of the prescribing elements of the clinical IT system which are relevant to their role.
|
Phase Two - Testing of the prototype safe-systems checklist
Eight participating practices within North Staffordshire (NS) agreed to test the checklist
Quantitative data
Table 3 shows the percentage of practices which answered yes to final checklist items. Items with a response of ‘No’ indicate where as a practice they feel they have not addressed a checklist item and might need to make a change to its systems. The two items with the lowest percentage of ‘Yes’ responses (25% of practices did not think achieve these safety goals) were item 6 regarding the failure to monitor the non-collection of prescriptions and item 7 relating to follow-up of vulnerable patients following discharge from hospital. Several items were met by all participating practices including the appropriate handling of incoming clinical information and the timely follow-up of abnormal results.
Table 3. Percentage of “yes” answers across practices
Item Number
|
Summary description
|
% Yes answers
|
1
|
All incoming clinical information is seen by trained or clinically experienced members of staff before filing.
|
100
|
2
|
Where incoming clinical information requires follow-up this is documented in the patient’s record and acted upon.
|
87
|
3
|
Where a clinician decides it is indicated, the patient (or a suitable/ appropriate representative) is informed of abnormal investigation results and documented in the patient’s record.
|
100
|
4
|
The practice keeps a log of minor operations containing key information including,
- Date/patient’s name
- Procedure performed
- Who performed the operation and who assisted
- Any complications
|
87
|
5
|
Up-to-date information on practice policies, procedures and local facilities/services is provided to guide all temporary clinical staff (including GP registrars).
|
100
|
6
|
Non-collection of prescriptions is monitored and a trigger for review in partnership with local pharmacies.
|
75
|
7
|
Vulnerable patients discharged from hospital are followed-up by a member of the clinical team within 1 month.
|
75
|
8
|
The indication for repeat medications is coded within the electronic record.
|
87
|
9
|
Staff are trained to make safe use of the prescribing elements of the clinical IT system relevant to their role.
|
100
|
Qualitative data
We interviewed eight participants each from one of the practices that trialled the prototype checklist. Of these four were practice managers, three were practice managers, one was healthcare assistant and one was a practice nurse manager. Interviews lasted between and 37 and 13 minutes. [10]. The practices they represented were situated within a variety of socio-economic backgrounds represented using the Index of Multiple Deprivation [21] and patient list sizes from 4000 to just over 12,000 these characteristics are summarised in Table 4.
Table 4 Participant job role and practice characteristics
Practice number
|
Job role
|
Patient list size
|
Index of Multiple Deprivation (quintile)*
|
P01
|
Practice Manager
|
9,887
|
6.14 (1st)
|
P02
|
Health Care Assistant
|
6,841
|
46.02 (5th)
|
P03
|
Practice Manager
|
12,491
|
27.53 (4th)
|
P04
|
General Practitioner
|
7,851
|
21.29 (3rd)
|
P05
|
Practice Nurse (manager)
|
6,930
|
7.25 (1st)
|
P06
|
General Practitioner
|
8,299
|
33.06 (4th)
|
P07
|
General Practitioner
|
5,167
|
23.22 (4th)
|
P08
|
Practice Manager
|
8,972
|
12.15 (2nd)
|
*Where 1 is the least deprived and 5 is the most deprived
Two domains within the CFIR were relevant to our data set. The first was Intervention Characteristics and themes emerged within constructs relating to the relative advantage of using the checklist its adaptability and overall design quality. The second domain was Outer setting and within the construct of Patient needs and resources the theme relating to a lack of capacity emerged. Below we describe these emergent alongside exemplar quotes.
Table 5 Summary of CFIR Domains, Constructs and Emergent themes
CFIR Domain
|
Constructs
|
Emergent themes
|
Intervention characteristics
|
Relative advantage
|
Staff engagement
|
High level approach
|
Prescribing safety
|
Review existing systems
|
Training staff
|
Adaptability
|
Frequency of use
|
Design quality
|
Quick to complete
|
Outer setting
|
Patient needs and resources
|
Lack of capacity
|
Intervention characteristics
This domain relates to the overall design, utility and usability of an intervention [20]. A number of constructs were identified in our data namely its Relative advantage, Adaptability and Design quality.
Relative advantage
This construct describes the stakeholder’s perception of the advantage of implementing the Prototype Safe-Systems Checklist (PSC) as opposed to maintaining existing practice. Any tool or instrument designed to improve safety of care can also improve aspects of care in other respects as patient safety and quality of care are so intrinsically linked. In terms of the advantages of using the PSC a number of themes emerged; staff described how they improved patient safety directly in terms of prescribing safety and enabling the review of existing systems, but also indirectly by using it to provide a framework to discuss patient safety with the broader practice team.
Staff engagement
Participants described how using the PSC indirectly benefitted patient safety by helping engage a range of staff. Although the tool was designed to be used by a single individual frequently, its completion would or could rely on other members of the practice team, helping raise awareness of patient safety.
“So we found it on several levels a really useful tool and not least, of course, patient safety, but in terms of actually being another vehicle to encourage cross-team understanding within the practice, as well.” Practice Manager, P01
One Practice Manager felt that the document could be used to frame a discussion with GPs on whether policies and procedures were implemented as expected.
“…it’s quite straightforward, I’ll just run through everything with the GPs instead of saying ‘yes, we do this’… I mean you can have policy and procedure and no-one can follow it.” Practice Manager, P06
High level approach
The benefits of the high level approach adapted by the checklist as a way of immunising specific items against local or sporadic change were described.
“I think one of the things that’s hard … with the checklist, is… keeping it up to date as things change so fast in practice, but a lot of your sentences are quite high-level, so it means that it lasts…” Practice Manager, P03
Prescribing safety
A number of participants commented on the benefits of using the tool to improve medication safety and one practice manager felt the section on medications was the most useful.
“The medication things I thought was probably the most useful section… they say the most errors in a general practice are made on medicines…” Practice Manager, P03
The other area that the SSC appeared to be effective was at highlighting the non-collection of repeat prescriptions. One GP acknowledged how this item had raised awareness of the issue and a practice manager how it had encouraged them to discuss the issue with other members of the team
“Non-collection of prescriptions, that’s the one that we found that we weren't doing very well… because we’re moving to electronic prescribing in a couple of weeks’ time, we’ll look into that, that way…” GP, P02
“The non-collection of prescriptions was good and that did encourage me to talk to the dispensing team – “what did they do with those?” ” Practice Manager, P07
Review existing systems
It was noted, how as a whole, the PSC provided the opportunity to look again at the safety of existing systems that due to familiarity might otherwise be overlooked.
“Actually, it gives you the chance to reflect that some of the things [we do] are a system and to think, ‘Oh, yes!’ Something like mail-handling is, like so embedded …we take 500 letters in… every day, scan them in, pass them round and whatever - that, you know, you can almost forget that that is a safe system.” Practice Manager P03
Training staff
Another way in which the PSC may indirectly benefit patient safety is by its use as a training tool for clinicians in the early part of their career. One practice manager described how it presented a useful overview for inexperienced clinicians.
“One thing I thought it would be …a good training tool for, like, an overview...These things would be good for, like, GP registrars and things, like in training… it’s a good overview position.” Practice Manager, P03
Adaptability
The construct of Adaptability describes the degree to which an intervention can be tailored, refined or reinvented to better meet local needs [20]. The flexibility of the PSC in terms of how frequently it could be used emerged.
Frequency of use
There was no prescribed time interval in between using the PSC, meaning that practices could decide how often it could be used. One practice manager described how they might use the tool monthly..
“…If you’re doing it monthly, you’re more aware of the questions in your head, aren’t you, so it’ll become more of a routine. So, yes, I think it would [be monthly], in the long term.” Practice Manager, P04
Another practice manager felt it would be usefully applied every twelve months to ensure systems were operating as safely as expected.
“I think once you’ve checked through it, it might be worth just going through it on an annual basis, just to make sure that you are doing these things….” Practice Manager, P06
Design Quality
The construct of Design quality describes the perceptions of users of the quality of its design [20]. The primary design element which participants commented on was how straightforward it was to use.
Ease of use
The PSC was considered well-structured and easy to follow, which meant that it was quick and easy to use.
“I think because it is quite brief it’s quite a useful thing, just a pointer to go through it and make sure that these things are still being done as they should.” Practice Manager, P06
Outer Setting
The domain of Outer setting relates to the influence of factors external to the design of the tool and the organisation [20], and the relevant construct in our analysis related to Patient needs and resources.
Patient Needs and Resources
This describes the extent to which the practice understands and is able to meet the needs of its patients [20]. Within this construct the emergent theme concerned the lack of capacity of practices to absorb additional work streams.
Lack of capacity
One factor that may inhibit its further use was the limited capacity, in terms of time and workload in primary care. Despite not knowing the length of time it would take to use the tool, a GP at one practice asked a part-time member of staff to be responsible for the tool because of concerns over their own lack of time.
“Because we were just totally snowed under, so I knew I wouldn't have time to do this so I asked my colleague who only works part time and did that for me. So he’s… done the Safe Systems questionnaire.” – GP P02
One practice manager was positive towards the PSC but cautioned that its future implementation might depend on the ability of practices to meet the twin pressures of time and resource.
“As much as I am a big fan of this tool, I think the two key issues are finding time and, if it involves any resources, is actually finding support for those resources because that’s always challenging in this day and age.” Practice Manager, P01