Participants’ perceptions of quality fall into two broad categories: defining characteristics and challenges. As elaborated below, comments in the former category align most closely with Harvey and Green’s [6,7] conceptualizations of quality as ‘perfection or consistency’ and quality as ‘transformation,’ and comments in the latter category align most closely with Harvey and Green’s conceptualizations of quality as ‘fitness for purpose’ and ‘value for money.’ In our analysis, Harvey and Green’s conceptualization of quality as ‘exceptional’ is removed from consideration because its focus on operationalizing high standards of students’ academic achievement in the classroom is not applicable in the production of research. Because we focus on knowledge production through research, rather than through academic processes in the classroom, we excluded the ‘exceptional’ dimension from this discussion.
Defining Characteristics
The defining characteristics of quality, as identified by our participants, were coded into the themes of ‘following scientific protocol,’ ‘committing to ethical behavior,’ ‘dedication to transparency,’ ‘knowing your field,’ and ‘research that has been deeply thought about.’ In viewing these results through the lens of Harvey and Green’s [6] framework, it became clear that individuals in our study conceptualized the defining characteristics of quality research production in ways that correspond to some extent with Harvey and Green’s conceptualizations of either perfection/consistency or transformation (See Figure 1).
Quality as Perfection or Consistency
The themes of ‘following scientific protocol,’ ‘committing to ethical behavior,’ and ‘dedication to transparency’ all align with Harvey and Green’s [6] conceptualization of quality as a process where certain specifications should be met with zero defects and where the process is done correctly the first time. By focusing on the context of scholarly knowledge production instead of institutional mechanisms, our interview data heighten awareness of individual agency, revealing a contradiction: On the one hand, individual scholars are deeply trusted to ensure the quality of the knowledge they produce, but on the other hand, conventions of scholarly publishing such as the expectation of a detailed methods section require them to document that they have behaved properly in producing knowledge. Thus, we also gain valuable insights into some key values the are pertinent in scholarly knowledge production, even across the epistemic diversity that exists among our interview sample.
Following scientific protocol. Using specific technical mechanisms to follow scientific protocol was the most frequently mentioned way of producing quality research. Some participants broadly connected quality to following scientific protocol (P02, P24), while others delved into specific mechanisms they associated with quality. Rigor was frequently mentioned (P09, P14, P16, P31, P41, P48), along with replicable methods (P02, P08, P12, P19, P25, P31, P38, P46, P48), as ways to ensure scientific protocol was being followed, as summarized by this participant: “I realize quality is a judgment call, but to me it is about...very clear methods, very clear controls, very clear descriptions of all your methods, because ultimately in science, it's supposed to be repeated” (P08). Other participants noted the importance of relying on peers to help ensure quality before journal submission (P02), modelling openness in the research process for junior scholars (P03) and having honest discussion about authorship early in the research process (P08, P09, P26, P33, P40). Participants represented in this theme represent a broad spectrum of our sample—researchers, librarians, and publishing professionals from nearly all the regions represented in our sample seemed to agree on the need for specific technical mechanisms that ensure quality in the knowledge production process.
Committing to ethical behavior. Committing to ethical behavior when no one is looking was discussed in a range of ways, from statements relating to values such as, “Just behaving, being a good citizen in the scholarly world and not doing bad things with regard to your own research” (P05) and “the moral value of honesty is essential for the scientific process to progress” (P08) to more concrete suggestions related to research practices, “not tampering with data” (P19) and “don't go and steal someone else's work” (P30). Participants touched on the need for ethical behavior in every part of the research process, from going through the “IRB [Institutional Review Board] process” (P02), to “respecting your subjects” (P10), to “present[ing] your results as it is” (P02). Ultimately, committing to ethical behavior was best summed up by this participant who said, “you have to be dedicated to the truth, period, no matter what” (P08). Across all the disciplines represented by the participants in this study, there seemed to be a sense of personal responsibility to do the right thing—and this notion of personal responsibility did not vary among participants working in different regions or professions.
Dedication to transparency. After ‘committing to ethical behavior’ and ‘technical mechanisms,’ a dedication to transparency was the next most frequently mentioned attribute of quality in scientific knowledge production. Some framed transparency as “being willing to show your work” (04) whereas others positioned it as “giving appropriate credit to the people who were involved” (P09) or mentioning specific mechanisms to increase transparency such as “preprints… because you can see how the work evolved” (P12). As one participant noted, “The more open you can be about [your processes], the more likely you are to pick up what went wrong, what went right, share it, build on it” (P21). In short, showing a dedication to transparency refers to a wide array of conventions in scholarly publishing that require a scholar to demonstrate that they have behaved properly. As with the other themes that align with perfection or consistency, participants from numerous professional and national backgrounds advocated transparency as a core marker of quality in scholarly knowledge production.
Quality as Transformation
The themes of ‘value mechanisms,’ ‘know your field,’ and ‘research that has been deeply thought about’ all align with Harvey and Green’s (1993) conceptualization of quality as transformation (See Figure 1). Specifically, participants emphasized that the goal of knowledge production is transformation, accomplished by advancing the disciplinary conversation to which one contributes. As in the previous section, interview data suggest a contradiction in the scholarly knowledge production system: scholars are trusted to behave properly and have a great deal of autonomy to make their own ethical decisions, but at the same time, scholarly discourse conventions guard against actions that would erode the quality of knowledge.
Value mechanisms. The most frequent sub-theme under the ‘value mechanism’ theme was ‘being critical of your own work,’ expressed through statements such as, “be more critical on yourself than anybody else is” (P08) and the suggestion to ask, “How can I self-evaluate more effectively?” (P12). Participants also pointed to the “moral value of honesty [as] essential for the scientific process to progress” (P08), a sub-theme that acknowledged the conundrum of needing morals in science because “science itself depends on that moral value but morality isn't science...you can't measure morality units…there's no morality meter” (P08). The final sub-theme discussed by our participants was ‘respect,’ which included mentions of respecting subjects (P10, P34, P41) along with the research environment (P10, P21) and colleagues’ work (P10). While the quotes shared here represent mostly researchers from our participant pool, in the whole corpus of data, participants from nearly every geographic region and represented discipline echoed similar perceptions of the need for strong value mechanisms to protect quality in the scholarly knowledge production process.
Know your field. Because reproducibility is considered essential to science, our participants were adamant that a scholar who knows their field will be better equipped to produce quality research, stating “Academic research… would ask you to have both knowledge of the past but also build new things for the future” (P31). In this sense, quality was couched as “fitting your study in with the rest of the field” (P08) and “a good quality article should be able to build on some sense of some people in the past and have their own new ideas that help us to understand the phenomenon or explain the phenomenon” (P31). In essence, quality research “grounds itself,” (P07) is “well fitted into the literature” (P09), and contributes to “building up incrementally” (P18) the body of knowledge. All the comments presented in this theme reflect the processual nature of research inquiries. Putting new research into conversation with the existing body of knowledge, aligning with Harvey and Green’s notion of transformation, was a core principle of quality that seemed to resonate with participants from all regions and professions in our sample.
Research that has been deeply thought about. This theme shows a dedication to the internalized process of knowledge creation, where “the researcher has to be curious on something” (P27) and “also a lot to do with a kind of passion” (P34). Additionally, there was a sense that researchers needed to take the time to produce research that “has been thought over and discussed” (P06) and that “Ultimately [the researcher] can still stand behind what [they] would have wrote then 10 years ago” (P34) so that the research “can create a dialogue with the existing literature” (P46) and continue to build upon the existing knowledge base. This notion did not appear to have regional boundaries, suggesting that individuals who pursue scholarly research as a career are dedicated to embracing deep thought processes, regardless of their ethnic background or specific regional affiliation.
Challenges
When participants discussed the challenges associated with producing quality research, their responses fell into a broad range of categories, including: ‘Producing knowledge in a global environment,’ ‘inconsistencies in evaluation methods,’ ‘getting rejected,’ ‘needing to sensationalize findings,’ ‘willful disregard of ethical practices,’ ‘valuing quantity over quality,’ and pressure to publish,’ and (See Figure 2). In contrast to the defining characteristics of quality addressed above, the challenges related to producing quality scientific knowledge align more closely with quality as ‘fitness for purpose’ or ‘value for money’ as defined by Harvey and Green [6]. Through these challenges, as identified by our participants, we begin to see the rift between institutional perceptions of quality and individual researcher conceptualizations of quality. This rift, as presented through our interview results, expands Harvey and Green’s conceptualizations, which have largely been conceived, developed, and applied from institutional, rather than individual, perspectives. Furthermore, in participants’ perceptions of challenges, we begin to see important differences among scholars working in different regions.
Quality as Fitness for Purpose
Conceptualizing quality as fitness for purpose emerges from the user’s perspective, revealing multiple, constantly changing purposes that can be at odds with each other. These conflicting purposes lead to challenges addressed by our participants in the five themes elaborated below and summarized in Figure 2.
Producing knowledge in a global environment. In this theme, participants discussed the challenges of navigating conflicting disciplinary expectations on national, regional, and global scales. A Southeast Asian academic researcher summarized this perception when he noted that “quality is associated with the Western[1] world” (P41), adding that while he was not necessarily opposed to this global view, other regions are producing high quality work that is not being recognized because “they don't meet the kind of processes that the Western publication industry is familiar with” (P41).
This same issue was also addressed by one European participant who stated, “international publishing means Western publishing, and I'm very critical against this stuff” (P35) and then offered the example of Latin America, a region this participant suggested has:
…their own research cultures, and they are very good in building their own databases and own publication networks… in Latin America, they developed a very good and working network… we can speak about Latin American research integrity because these articles meet their standards. When you are not Latin American, it will be different for you. (P35)
‘Resources’ emerged as a significant sub-theme, capturing the challenges faced by entire disciplines or scholars within a specific country (P18). For example, a researcher in an African country said she relies on a colleague in the U.S. to send her research articles because her institution does not provide access to journals she needs to cite in her research (P03). This notion of inequities in resources was also addressed by a North American researcher who asserted that even a supposedly global organization like the International Communication Association (ICA) often perpetuates biases (P34). Most participants who addressed the challenge of producing knowledge in a global environment were working in, or were originally from, regions outside the Euro-America sphere. The few Euro-American participants who mentioned the challenges of producing knowledge in a global environment tended to be publishing or consulting professionals who routinely worked with researchers or journals in non-Euro-American regions and had witnessed the challenges faced by scholars in other regions.
Inconsistencies in evaluation methods. Academia, by its very nature, encompasses all fields, meaning that one approach that works for a certain field may not work for another (P08). But a bigger problem identified by several scholars from various parts of the world, expressed succinctly by a participant from Eastern Europe, was that “there is some assessment system, but nobody cares” (P35). Another scholar from Southeast Asia noted their national system of ranking publications “only look[ed] at the ratings” (P19), before continuing to describe how this has lowered the production of quality research because so much focus has shifted to journal rankings.
Some participants also discussed the intricacies of their national standards and the influence of the government in scholarly research practices. For example, one scholar in a STEM field who has worked in Africa and Asia compared the influence of government policies in two countries, noting that in the African country there was an increase in publications in predatory journals after a new policy emphasized quantity of publications, whereas in the Asian country, “they have a ranking…, which is developed by [the National Academy of Science]…So just publishing in a predatory journal, your articles is not going to be recognized … So, people here avoid predatory journals because of that” (P10). Here again, it was primarily participants working outside the Euro-American sphere who noted the differences in evaluation systems and how it affects their ability to produce quality scholarly knowledge. This observation seemed especially salient for individuals who have worked in multiple regions.
Getting rejected. This theme captures rejection as a constant occurrence in academic life and its impact on research conduct. Although it may seem like this fear of rejection would encourage quality in knowledge production, one participant noted it may have the opposite effect, as it could lead desperate scholars to turn away from traditional, reputable publishers, and instead, “just rely on predatory journal” (P03). Getting rejected unfairly was especially salient for scholars working outside Euro-American regions, as summarized by an Eastern European scholar who stated, “when you submit your paper from Nigeria, and the same paper is submitted claiming that it is from Harvard, the referees will extremely overvalue the paper [from Harvard]” (P35). Almost all the participants who shared perceptions of unfair rejections based on national affiliation were from non-Euro-American regions.
Conversely, scholars from Euro-American or affluent Asian regions tended to couch rejections in a more positive light: “even if we didn't publish at that journal that rejected us, we used that feedback to make the article better and better” (P41), a viewpoint agreed upon by another participant who acknowledged that despite the emotional difficulty of rejection, reviewer comments often help improve the manuscript (P48). As with other themes related to quality as fitness for purpose, regional differences emerged in how the participants viewed the challenges they were facing.
Needing to sensationalize findings. This theme captures the potential of over-sensationalizing findings as a challenge to producing quality science (P15, P38)—in one participant’s words, a “preponderance of people just wanting cool findings that you just throw at whatever high-impact journal” (P45). One senior publishing professional working in Europe spoke of the need to instill a sense of personal responsibility in junior scholars “not to oversell their studies, to be honest about what their study really showed, not the things that they wished it would have shown you” (P09). A European researcher agreed with this sentiment, urging junior scholars to not succumb to “temptations to maybe make the story a bit more streamlined…, or to make it a bit more simple, a bit easier to understand” (P38).
But the trend of only publishing research that supports hypotheses or that is statistically significant is not just an issue for authors to address—it is a problem for journals and publishers as well. In the sub-theme ‘importance of null results,’ participants spoke of the additional pressure from journals to only publish work that supported hypotheses, rather than understanding that “sometimes negative results can be very important as well” (P26) and that “a lot of times an advance comes off something negative, not off something positive” (P08). One participant said that feeling pressured to sensationalize their work and not publicize negative results is “the most harmful thing probably for scientific progress… Because a null finding can be much more revealing than a finding” (P48). This theme emerged from comments throughout our participant sample, in contrast to other challenges that appear to have regional differences.
Willful disregard of ethical practices. Aside from the disciplinary and institutional pressure expressed by participants in our study, an added complexity that a handful of individuals discussed was the reasons why scholars engage in unethical research practices. Participants in the Euro-American regions tended to couch unethical behavior in softer terms, saying “people make mistakes. But there's a lot of … intentionally unethical behavior, because there's so much pressure to chase the money, and so if you're under a lot of pressure to chase money, yeah, you cook your data” (P25). Conversely, a participant from Africa chalked it up to “laziness,” sharing the practice of “faculty members here in [my country], they just pick their student's project work and modify it as a journal article and submit” noting that “sometimes they put the name of the student along, sometimes they don't” (P42). One South American participant shared a story of a colleague who knew they were submitting to a predatory journal, but did it anyway, ending the anecdote by saying:
My impression is that perhaps at that moment he was so unhappy with the system. He was so unhappy with the way reviewers were dealing with his submissions. …I think that when he found that somebody was appreciating his work …he simply was happy to have those ideas spreads through that journal. I think that sometimes they know that there is something wrong, but the fact that somebody accepted the work, … people like to have their work published. (P13)
Ultimately, despite participants from different regions giving different reasons for researchers to disregard ethical practices, there was universal agreement that the human element of honesty could fail during knowledge production, damaging the credibility of knowledge.
Quality as Value for Money
For our participants, challenges coded as ‘valuing quantity over quality’ and ‘pressure to publish’ align most closely with Harvey and Green’s conceptualization of quality as value for money (See Figure 2). As elaborated below, these themes reflect an understanding of quality as something that can be quantified and brings value to the institution.
Valuing quantity over quality. The complicated interaction between personal ethical values and institutional pressures was especially salient as participants discussed the trend in academia to value the number of publications, rather than the social benefit, practical application, or other less quantifiable markers of quality. This concept was addressed from a range of angles, including scholars from multiple regions (P02, P06, P08, P22, P24, P32, P38, P47) noting cultures of publishing in which the publication venue does not matter, summarized by one North American participant’s statement that “in some areas, unfortunately, they're mainly counting publications not thinking about quality and so you can just publish a lot and you're considered productive” (P08). An Eastern European scholar shared arguments from his students, noting when he tried to guide them towards publishing in reputable journals, “they say, ‘Oh, why should I take 100 or 1000 hours for developing a good paper, when I can just write something in low quality, but my university will accept it as international? Why should I care?’” (P35).
Others spoke about the focus on quantity from a resource perspective, noting “there are really perverse incentives in academia...with this idea of just maximizing a very poor proxy for quality, just because it's quantifiable,” an idea they continued with by saying, “quality has become…a question of numbers...the number of publications per academic” (P22).
Despite the vastly different publishing expectations across the globe, participants from multiple regions mentioned the growing emphasis on numbers of published manuscripts because of the relative ease in measuring indicators such as journal impact factor and citation counts.
Pressure to publish. The valuation of the number of publications, rather than social benefit or practical application of fewer articles has led to a culture where scholars constantly feel they are in a race to “fling [manuscripts] to a journal... rather than taking the time and the effort to really prove yourself wrong” (P45). This pressure to publish was especially salient to some participants whose national systems required them to publish before their degree could be conferred (e.g., P03, P15), placing publications as “kind of the currency of the field” (P08). The pressure to publish original research was a challenge articulated by participants from nearly all our represented regions, suggesting it is a global phenomenon in academia.
In terms of systemic challenges, the so-called ‘replication crisis’ (P47) emerged as a powerful sub-theme, with both researchers and publishers noting that placing more value on replication studies could increase the quality of research. One participant noted researchers’ wasted time because “people just keep doing the same stuff because they only have the articles published that did work” (P43). A publishing industry professional emphasized that journals should be more open to publishing replication studies (P38), a perception balanced by another participant who noted the difficulty journals face in providing all the materials scholars need to reproduce studies (P08). This challenge was expressed almost entirely from scholars in the Euro-American regions, suggesting that for scholars working in the highly competitive and often well-funded regions, they did not feel empowered to replicate studies to build on existing knowledge—rather they felt pressured to continually find new and unique ways to conduct research. Ultimately, many felt this was a disservice to the scientific method because attention was continually being diluted to new projects, rather than working on deeper understandings of specific issues.
[1] We acknowledge the problematic categorizations of East/West labeling, and have chosen to use the phrase “Euro-American” as a more accurate description of the geographic/ethnic orientation we are attempting to describe. When the label “West” and “Western” appears here and other places in the results, it reflects the actual verbiage used by participants, and thus we chose to leave their quotes intact.