We sent 24 invitations to groups or individuals to participate in the study, we conducted 12 interviews and 12 invitations were declined. Seventeen individuals took part in total, nine via individual interviews - and eight via group-interviews (table 1). Interviews were conducted between October 2014 and February 2015. Three of the individual interviewees did not give permission for their interviews to be recorded. For two interviews the recording equipment failed so in total seven interviews (two group and five individual) were recorded and transcribed. We used contemporaneous field notes from the remaining five interviews.
Table 1 – Summary of interviews
Interview
|
Number of Interviewee(s)
|
Role
|
Primary Care Clinician
|
Interview A*
|
3
|
Guideline Developer
|
No
|
Interview B
|
1
|
Guideline Developer
|
No
|
Interview C*
|
1
|
Guideline Developer
|
No
|
Interview D
|
2
|
Primary care support and quality improvement
|
Yes
|
Interview E
|
1
|
Primary care educator
|
Yes
|
Interview F
|
1
|
Primary Care Commissioner
|
Yes
|
Interview G
|
3
|
Research Charity (2 members of research funding team, one member of research policy team)
|
No
|
Interview H†
|
1
|
Chair of public funding panel
|
Yes
|
Interview I†
|
1
|
Chair of public funding panel
|
No
|
Interview J†
|
1
|
Health Technology Assessment
|
No
|
Interview K†
|
1
|
Editor - Journal publishing systematic reviews
|
No
|
Interview L†
|
1
|
Clinical director - Journal publishing systematic reviews
|
No
|
*Academic GP co lead interview
† Only field notes available for interview
Four main themes emerged from our interviews: How evidence is used; aspects of trials considered when assessing evidence; views on PRECIS-2 domains; and perceptions of pragmatic and explanatory trials.
How evidence is used
Interviewees who applied evidence from randomised trials in a clinical setting reported using evidence differently from those who produced evidence synthesis or guidelines. The latter followed a formal process starting with a systematic way of identifying evidence, critical appraisal and then production of a summary of the evidence as a review or guideline. Challenges in applying evidence came from combining the results of heterogeneous evidence and deciding whether evidence is applicable to the question of interest. In overcoming these challenges judgement has to be incorporated into the formal processes. The greatest area of judgement was around whether evidence is applicable to the question of interest.
"[the guideline development group members] make, what we call, a considered judgement on evidence from the trials, taking into account other aspects, whether it’s generalizable, whether it’s applicable." (Interview C, Guideline Developer)
For those applying evidence in a clinical setting, sources of evidence included guidelines or evidence summaries as well as less formal sources such colleagues or experts in a particular field. Due to time pressure and a need for accountability regarding decisions about patient care, the results from single studies were not commonly used to influence practice. We found a mistrust of early evidence due to the potential of trials to show greater benefit of new interventions than would be found in practice. To overcome this challenge some individuals employed a deliberate strategy of waiting before implementing new evidence. A further challenge in applying evidence was the limited resources to implement novel interventions.
"So most of the time, what you’ll find is on my computer screen [is], NICE [The National Institute for Health and Care Excellence] or CKS [Clinical knowledge summaries] open up in a separate window, that I’ll just refer to if I need to with every patient."
(Interview E, Primary care educator)
“…let’s just take the new oral anticoagulant agents … There’s been some early meta-analyses, but everybody feels the early trials were always more optimistic…
(Interview D, Primary care support and quality improvement)
…the question is, do you wish to be an early adopter, are the advantages so great that you want to take a risk or would you like to do it later.”
(Interview D, Primary care support and quality improvement)
Interviewees from the research charity were involved in advocating for change based on the results of new research and facilitating discussions amongst clinicians of issues arising in new evidence. They focused on trials funded by their charity. Rather than applying a formal process of critical appraisal they appeared to try to ensure quality by funding research meeting certain standards.
And we also use [examples of clinical trials funded by the charity] quite heavily in case studies when we’re talking to government, just as examples of how charities operate in the UK and demonstrations of exciting new things that are happening.”
(Interview D, Primary care support and quality improvement)
The funders interviewed provided perspectives on the use of evidence but from their roles as clinicians and as research funders. From the perspective of funders they advocated that research be disseminated via systematic reviews and guidelines. As clinicians their views towards evidence use were consistent with those already presented.
“Systematic reviews of all relevant research should be carried out and these should be disseminated and used to change practice. This is where organisations such as NICE can play an important role in producing guidelines based on the best evidence.”
(Interview I, Research funder)
Perceptions of pragmatic trials
Interviewees perceptions of pragmatic trials included enthusiasm, mistrust alongside limited knowledge of the term and misconceptions about its meaning. Interviewees from public research funders and journals were most enthusiastic about pragmatic trials, showing existing familiarity with the concept and positively expressing that those are the sort of trials they would be most interested in.
“We are more interested in funding pragmatic trials than explanatory and would like to see trials which were as pragmatic as possible...”
(Interview H, Research funder)
Interviewees from the research charity, involved in primary care education or from CCGs, and some interviewees involved in guideline development had little pre-existing knowledge of the term “pragmatic trials”, expressing misconceptions as to what the concept meant or it being a term they were not familiar with prior to the interview. Misconceptions about pragmatic trials included conflating the design with trials of complex interventions and with particular design features such as cluster randomisation or open label studies. It was also suggested that pragmatic trials would only be relevant to in particular clinical areas. Some interviewees also drew a distinction between pragmatic trials and randomised controlled trials.
"...to include [pragmatic trials] the guideline developers would have to be really careful to explain the difference between this and RCT [randomised controlled trial]."
(Interview C, Guideline Developer)
Amongst those who used evidence from guidelines or evidence synthesis, pragmatic trials were welcomed as they simplified the judgement around whether evidence is applicable to clinical settings. There was concern, however, that they could be significantly different from trials in the existing evidence base leading to heterogeneity which could make meta-analysis more difficult.
"Pragmatic trials may have to be considered separately in meta-analysis due to heterogeneity with other trials”
(Interview J, Health technology assessor)
“I can just imagine that people would be really thrilled to see a pragmatic trial, of diabetes or something, [set here] and that people could then use it... It would be so much better for recommendations."
(Interview C, Guideline Developer)
Pragmatic trials also carried negative connotations, sometimes being viewed as inferior to more traditional approaches to randomised trials due to perceived weaknesses.
“Pragmatic can be a dirty word when describing trials, people like to shoot at them…”
(Interview K, Editor of Journal publishing systematic reviews).
There was also concern that the degree of pragmatism of trials could be used to manipulate the systematic review process.
“During discussions of evidence it can be difficult to consider the relative pragmatism of evidence – sometimes it is used by people to try and exclude evidence that doesn’t agree with their point.”
(Interview I, Research funder)
Views on the PRECIS-2 domains (fig 1)
The population in the trials was by far the most discussed aspect of a trial’s design, with interviewees preferring trial populations to have few exclusions, including patients with comorbidities and older patients. Population is handled by PRECIS-2 over three domains: eligibility, recruitment and setting.
"So again, if you do it in your tertiary centres, then it becomes almost inapplicable in primary care, because I don’t have those resources. I don’t see all of those patients at that stage in that illness. I see them either way before or way after they’ve seen the tertiary care people. So yes I think setting is very important."
(Interview E, Primary care educator)
Aside from its part in determining the population in trials, Recruitment was an aspect of trial design not explicitly considered by most interviewees when assessing evidence. Amongst those who did consider the impact of recruitment on the relevance of trial results, there was concern that very intensive recruitment could lead to people being included in trials who would not usually present for treatment in routine practice.
“…we wouldn’t assess recruitment routinely, but it’s about whether recruitment is applicable to the question that we‘re trying to address.”
(Interview D, Guideline Developer)
"Sometimes recruitment can be too intensive and bring people into the trial who would not usually present for a condition"
(Interview I, Chair of funding panel)
The organisation domain focuses on the level of expertise and resources made available to deliver an intervention compared with what would be available in practice. Here, we identified a tension between a pragmatic and an explanatory approach. Those applying research in practice were more concerned about resourcing issues and preferred trials to test interventions that could be implemented with the limited resources available in primary care. Those from the research charity or guideline developers believed there was also a place for evidence from trials of interventions requiring resources over and above those currently available. They felt that sometimes research showed that a resource intensive intervention was effective could lead to those resources becoming available in routine care.
"The only thing in primary care that would be a limiting factor is the resources aren’t this sort of overflowing bucket"
(Interview E, Primary care educator)
"And sometimes groups will make the gold standard recommendation and that will push forward what resources are brought in"
(Interview C, Guideline Developer)
For flexibility of delivery interviewees producing evidence synthesis or guidelines favoured reduced flexibility as this allows more understanding to be gained as to what is causing any effect, makes it easier to include a trial in a meta analysis, and also reduces bias from other treatments being initiated. Reduced flexibility was also preferred as it allowed greater understanding of what the intervention being delivered in the trial actually is. For the clinicians interviewed there was an appreciation and expectation that trials would have less flexibility in the way interventions were delivered.
" After ten years of doing this I would prefer to see strict control [in how interventions are delivered] but if there are variation they need to be described properly so in attempting to make sense of this you can see what has happened."
(Interview B, Guideline Developer)
"We appreciate that you’ve got to stick to strict guidelines when you’re doing the research, otherwise it doesn’t become very accurate at the end of it. So as long as it’s not hugely different, we appreciate there is a little bit of leeway in real life, but we wouldn’t expect that in a clinical trial."
(Interview E, Primary care educator)
Poor adherence to interventions in trials was a concern as this can reduce the potential effect of a successful intervention. For some, best practice with regard to flexibility of adherence was for adherence issues to be identified prior to the trial and in the trial itself no extra steps to be taken to improve adherence. Others, notably the interviewees from the research charity, suggested knowing an intervention can be effective can lead to measures to help adherence being developed.
“Researchers shouldn’t include intense follow-up to ensure adherence. Steps should be taken to collect as much primary end point data as possible but this is separate to ensuring that people adhere to the intervention.”
(Interview I, Chair of funding panel)
“The more flexibility there, the more you’re going to actually mask any real effect because of the amount of variation... ...find out why people aren’t adhering and what can we do to try and help people adhere to that particular exercise programme, for example. So, you know, I feel that the flexibility [of adherence] should be pushed tighter."
(Interview G, Research funding and policy, Research charity)
For follow-up we identified a balance to be struck between collecting data that is useful for research and follow-up influencing participant behaviour or increasing the burden on those taking part. There was also concern raised by some interviewees that in some settings intensive follow-up could act as an additional intervention.
" ...in terms of having quite a lot follow up, you can get some really useful answers and actually you probably want to do that. But it’s thinking about the way in which you follow-up and so that you’re not actually influencing their behaviour and their any sort of clinical outcome through taking those measurements."
(Interview G, Research funding and policy, Research charity )
Where primary outcome and primary analysis was discussed all Interviewees favoured a more pragmatic approach with patient centred primary outcomes and intention to treat analysis. There was concern expressed about primary outcomes being measured at too early a time point.
Comments on applying PRECIS-2
The guideline developers, research charity and research funders considered PRECIS-2 a useful tool, with the guideline developers saying that it covered many of the areas of judgement they were required to make and those involved in research funding commenting that being able to justify design decisions across the PRECIS-2 domains would strengthen funding applications.
"we never used this instrument, it looks very helpful, we were often left to make the judgement... to what degree was it moving towards the pragmatic, and to what degree was it explanatory.
(Interview B, Guideline Developer)
"if they thought about all of them in advance and they’ve got a good reason why it’s explanatory on this and it’s pragmatic on this one, then I think that’s, that would make a strong application coming through."
(Interview G, Research funding and policy, Research charity )
Alternative uses for PRECIS-2 were also suggested including use as a teaching aid, trial reporting, and aiding judgements around applicability. It was noted that PRECIS-2 is subjective so care would have to be taken for reported PRECIS-2 scores to be justified.
“…actually in the paper they should put [PRECIS-2 wheels] in… you just want a quick summary. That might be helpful”
(Interview F, Primary Care Commissioner)
Issues raised by participants but not covered by PRECIS-2 domains
Outside of PRECIS-2 domains, interviewees raised issues around internal validity, in particular blinding where possible, and size of the trial - with larger trials preferred. Internal validity, sometimes referred to by interviewees as “quality”, was typically assessed before generalisability when developing guidelines or assessing research for funding, typically using risk of bias tools. Issues around internal validity were raised both by those involved in evidence synthesis and those applying evidence to practice.
"So if the quality is poor, that will get marked before people even think whether it’s generalizable or not."
(Interview C, Guideline Developer)
Of the factors unrelated to the design of trials, reporting of the trial was the most important issue outside PRECIS-2 domains, to be raised. Poor reporting was seen as an obstacle to using evidence in practice whereas good reporting was seen as something which could enhance the generalisability of evidence. Areas of reporting that were most important included details of what the intervention was, how the intervention was implemented in the trial and discussion of the generalizability of results. If usual care is used as a comparator it was seen as important to report in detail what usual care consisted of.
"I think we’ve found that people have not been able to use evidence before because they haven’t been clear on what the usual care has been."
(Interview C, Guideline Developer)
Other issues that were raised included patient acceptability of the intervention, whether research was carried out in collaboration with practice and whether the intervention addressed an important clinical problem either effecting a large number of people or a particular problem for a minority that has been hard to reach.