The academic database search identified 2790 references (after duplicate removal), and the first five pages of the Google search identified 71 references (after duplicate removal). The search results were supplemented with 2 known relevant references from SC’s personal library that had not been identified by either the academic or Google searches. The (NNR) to identify a primary paper from the combined academic databases, Google search and SC’s personal library was 286.3 (2790 refs + 71 refs + 2 refs / 10 primary refs).
Database Comparison
Figure 2 shows the databases where primary papers were found in the searches. It also shows the databases that held these references and could have been located with ‘the optimum search’ but were not necessarily identified with our search. Seven of the ten included references were identified in the Google searches [28–34]. One of these [35] was also found in the database search. Therefore, six references were uniquely found via the Google searches. The NNR for the Google searches was 10.1 references.
Across the academic databases the CINAHL search identified two primary papers. Several database searches were ‘redundant’; HMIC did not identify any relevant references, and The Cochrane Library, EMBASE and Medline searches found two duplicates [35, 36] found in the CINAHL search. A further two primary papers were identified in SC’s personal library [37, 38]. Using our search strategies, the 10 primary papers could have been identified using three sources; CINAHL, Google and SC’s Personal Library, rather than the seven sources (Fig. 3). The NNR for the academic searches was 1395 while for the combined academic databases and Google search it was 286.3.
Reasons For Non-retrieval In Existing Searches
Figure 1 shows that some primary paper references were available in CINAHL, EMBASE, Medline and Google, but were not identified by the original searches. Table 1 lists the reasons for non-retrieval. All 10 primary references were available in Google at the time of the original search, however only 7 appeared in the first five pages of search results from the two Google searches. When the original Google searches were replicated in 2019 the 10 primary references were found within the first 12 pages of Google (i.e. the first 216 records if it is assumed there are 9 references per page on average in Google search results). This indicates that if a larger set of results (216) had been screened from the original search, all 10 primary papers would have been identified with this one source.
Table 1
Retrieval and Non-retrieval of references across databases and reasons for non-retrieval
| Google | Academic databases |
Study reference | Retrieval in Google (non-retrieval reason) | Retrieval in MEDLINE non-retrieval reason) | Retrieval in EMBASE (non-retrieval reason) | Retrieval in CINAHL (non-retrieval reason) | Retrieval in Cochrane (non-retrieval reason) |
Bell 2005 | ✓ | X | X | 3 | X |
Chapman 2014 | ✓ | X | X | X | X |
Guy 2012 | ✓ | 2,3 | 2, 3 | 3 | X |
Green 2017 | ✓ | X | X | X | X |
Fletcher 2017 | ✓ | X | 2, 3 | 3 | X |
Torra i Bou 2006 | ✓ | X | X | X | X |
Moore 2014 | ✓ | 2 | X | ✓ | ✓ |
Johansen 2014 | 1 | ✓ | ✓ | ✓ | X |
Kottner 2010 | 1 | 4 | 4 | X | X |
EPUAP/ NPUAP 2014 | 1 | X | X | X | X |
X: the reference was not stored (or indexed) in the database at time of search |
Reasons for non-retrieval when database did contain the reference |
1: not within first 5 pages of Google search |
2: No age limit indexing |
3: No terms from theory and publication types search concept |
4: In PubMed but not Ovid Medline 1946-present |
Five references were not indexed in the academic databases we searched (Table 1). These included a guideline [38], blog item [30], conference abstract [32], book chapter [34] and one journal article [37]. The journal reference record was available in PubMed and ‘Ovid Medline In-Process & Other Non-Indexed Citations’ but not the version of Ovid Medline used in the original search. Of the remaining 5 primary references, CINAHL contained 5 but only retrieved 2, EMBASE contained 4 but only retrieved 1, Ovid Medline (1946-present) contained 4 but only retrieved 1 and The Cochrane Library contained and retrieved 1. The references found in EMBASE, Medline and The Cochrane Library were duplicates of the references found within CINAHL.
Three references were not identified by the academic databases searches despite being available in at least one database [28, 29, 33] due to using the search limit ‘Adult’ or ‘Aged’ which the references had not been indexed for [29, 33]. Also search terms (index terms and free text words) used in the search concept for ‘theory and publication types’ were not present in some database indexing records [28, 29, 33].
The optimum search with the lowest NNR of 21.8 was the original Google search (Pressure ulcer risk assessment and Pressure ulcer risk assessment tools), but extending the records screened from five pages (approx. 36 unique records) per Google search to at least 12 pages (approx. 108 unique records) per Google search.
Replicability Of Google And Google Scholar Searches
The results of the replicability searches in Google where the first four pages were screened by 2 researchers working independently, were similar but not identical. For search (i) ‘Pressure Ulcer Risk Assessment’, 11 results were on the same page for each researcher but in a different order and six references were found by one but not both researchers. For search (ii) ‘Pressure Ulcer Risk Assessment Tool’, 10 results appeared in a different order for each researcher and 22 results were found by one but not both researchers.
Comparison searches were undertaken to test if Google Scholar performed better than Google Searches for identifying our ten primary papers, and to check replicability. The first five pages for each search (76 records in total) were downloaded. The search results from both researchers were exactly the same, indicating strong replicability. However, in both the ‘Pressure Ulcer Risk Assessment’ and the ‘Pressure Ulcer Risk Assessment Tools’ search, only one of the ten primary papers was identified [35].
Optimising The Search
In light of our experience in this case study we used a combined but adapted approach to help mitigate publication bias that can arise from only using one source for publications, or in the case of Google searching, an ‘Internet Research Bubble’ [39]. This comprises the existing Google search but extending to page 12 and optimising our academic database search by searching the most relevant database (CINAHL), in more precise manner by searching only in the title field and not the abstract or keywords for the phrases “pressure ulcer” “pressure ulcers” AND “risk assessment”. This would identify the same five available primary references from CINAHL with fewer abstracts needing to be screened. To demonstrate this we ran the specific phrase search in CINAHL (for the same time period of our original search) search results were limited to studies published up to May 2017, when the original CINAHL search was conducted) and 186 papers were identified with an NNR of 18.6.
The Google and CINAHL optimised searches retrieved 402 references with an NNR of 40.2, whereas the original searches retrieved (2790 + 71 + 2) 2863 references with an NNR 286.3. The researcher would save time in screening 2461 (2863–402) fewer references, and also by not developing complex search strategies and downloading records from four academic databases. If we estimate that it takes on average 2 minutes to screen each paper with 10% being screened by a second reviewer this would save approximately 90 hours of researcher time.