In order to explain the concrete implementation method, this study compares the automatic literature selection procedure with the corresponding manual procedure using ten different systematic reviews (SR1–SR10) in TCM published in the last ten years. Figure 3 illustrates the implementation details of the scheme.
2.1 Literature data collection and preprocessing
These ten systematic reviews are all related to exploring treatment effects of TCM herbs, which all report explicit inclusion and exclusion criteria and transparent search schemes. In addition, each of these reviews has a different disease type topic as follows: pediatric diarrhea (SR1) [34], diabetic peripheral neuropathy (DPN) (SR2) [35], chronic obstructive pulmonary disease stable pulmonary (COPD) (SR3) [36], simple obesity (SR4) [37], diabetic kidney disease (DKD)(SR5) [38], knee osteoarthritis (SR6-8) [39–41], chronic atrophic gastritis (SR9) [42], type 2 diabetes insulin resistance (SR10) [43].
2.1.1 Acquiring the initial literature sets
As illustrate in Fig. 3, the initial literature sets for the automated literature selection process are obtained through literature search protocols utilized in the selected systematic reviews. These reviews are predominantly based on Chinese literature published in Chinese databases, with over 90% of the relevant literature coming from these sources. As such, the automatic literature screening process was specifically designed for Chinese languages and focused solely on Chinese language articles. The retrieved results are subsequently exported to Note-express, a literature management software, to remove any duplicates by titles.
2.1.2 Convert publications into machine-readable structured texts
In order to facilitate machine analysis, the publications in PDF or picture format are converted into editable TXT format using Optical Character Recognition (OCR) identification technology [33]. Data cleaning procedures are then employed to remove irrelevant characters (e.g., ‘< p >’ and ‘\ n’) and significant identification errors. The presented procedure is updated according to the anomalies detected in the data.
Given the standardized writing pattern of RCTs, which often include the “patient source” in the ‘materials and methods’ section and the “focal disease name” in the “title” or “abstract”, articles are segmented into 11 sections (Appendix 1) to allow for more precise and efficient extraction of target information. To accomplish this, a program is written to identify the starting or ending words of each section or their near-synonyms (Appendix 2).
2.1.3 The inclusion and exclusion criteria lists: the basis for target literature selection
We first merge the inclusion criteria with the exclusion criteria of the same meaning. For example, SR1 requires the inclusion of RCTs and exclusion of uncontrolled studies; thus, the inclusive and exclusive criteria are merged because they refer to the same issue. Then, we establish a list of words to exclude for automatic exclusion according to the critical points in the exclusion criteria and their synonyms, such as ‘meta-analysis’, ‘text mining’, ‘research progress’, which are the same for all the systematic reviews.
Given that we are unable to list all the possible exclusive words for some of the exclusive criteria, we transform these criteria to inclusive criteria to avoid failure of automatic exclusion. For example, we could not exclude all the possible words related to animal experiments which served as common exclusion criteria; we set corresponding inclusion criteria with critical points such as patients, randomized method, and group name.
Based on the fusion of inclusive and exclusive criteria, we summarize nine target information that can be auto extracted from different article sections for each systematic review (Table 1). For the sentences describing the target information, we abstract critical points of information corresponding to each and enumerate all the possible synonyms for each of the descriptive critical points.
Table 1
Target fields in automated selection procedures for the ten reviews
Target fields
|
Section of extraction
|
SR1
|
SR2
|
SR3
|
SR4
|
SR5
|
SR6
|
SR7
|
SR8
|
SR9
|
SR
10
|
Disease name
|
1/2/3
|
√
|
√
|
√
|
√
|
√
|
√
|
√
|
√
|
√
|
√
|
Diagnosis standard
|
4/5
|
√
|
√
|
√
|
√
|
√
|
√
|
-
|
√
|
√
|
√
|
Study design type
|
2/5
|
√
|
√
|
√
|
√
|
√
|
√
|
√
|
√
|
√
|
√
|
Interventions
|
1/2/3/6
|
√
|
√
|
√
|
√
|
√
|
√
|
√
|
√
|
√
|
√
|
TCM syndrome type
|
2/3/5
|
-
|
√
|
-
|
√
|
-
|
-
|
-
|
-
|
-
|
-
|
Outcome indicator
|
7/8
|
-
|
√
|
-
|
√
|
√
|
√
|
-
|
√
|
-
|
√
|
Course of treatment
|
5/6
|
-
|
-
|
√
|
-
|
-
|
-
|
-
|
-
|
-
|
-
|
Cases of patients
|
5
|
-
|
-
|
√
|
-
|
-
|
-
|
-
|
-
|
-
|
-
|
Adverse reaction
|
7/8
|
-
|
-
|
√
|
√
|
√
|
√
|
-
|
√
|
-
|
-
|
Besides that, there are also have some inclusive and exclusive criteria from SR1-10 cannot be used to make automated logical judgments because they cannot be transformed into computer-recognizable data. Thus, they need to be screened manually after the automated literature selection procedure. (e.g., inability to calculate the 95% confidence interval; statistical method errors; and data errors.)
2.2 Automatic screened of structured literature data
2.2.1 literature labels setting
There are several labels we use to filter the tagged entries:
For text type fields, the articles including any designated exclusive words (e.g., meta-analysis, text mining, research progress) or excluding any designated inclusive words (e.g., type 2 diabetes, diabetes) in the title, keywords, abstracts or full-text are automatically excluded, labeled as ‘Automatic exclusion.’
For numerical type fields, we extract the sentence containing the target number from the paragraph, match phrases containing numbers and contrast the numbers with the inclusive/exclusive criteria, as shown in Fig. 4. If the article extracted numeric type fields that did not meet the inclusion criteria, labeled as ‘Automatic exclusion.’
For example, SR1 requires that the intervention in the treatment group lasts no less than eight weeks. Based on this inclusion criteria, we extract contents like ‘eight courses of treatment, seven days per course’ or ‘a total of fifty-six days of treatment’ or ‘the course of treatment for both groups were eight weeks’ and transformed them to ‘fifty-six days’ to perform the logical judgment in the ‘course of treatment’ field. If a given article has extracted numeric type fields that did not meet the inclusion criteria or met the exclusion criteria, this article would be labeled ‘Automatic exclusion.’
If the contents that cannot be made by automatic judgment were not consistent with the inclusive/exclusive criteria requirement, this article would be labeled ‘Manual exclusion.’
The remaining articles not excluded in the all-screening steps would be labeled as ‘Included.’
2.2.2 Query and retrieve in literature data
The critical points and their synonyms in each descriptive target field shown in Table 1 are automatically retrieved and extracted from the designated sections. We gradually expand the search if all the critical points of a target field and their synonyms are not located in the designated section(s). For example, when querying intervention methods, the search can be started from title and keywords, next extended to the abstract, then further extended to the study intervention in the methods section, and finally extended to the full-text search if none are extracted in the above sections. The advantage of this extraction method is that we can avoid extracting the background information in the article by extracting information from the section(s) where it is most likely to exist.
There are three types of results for retrieving: ‘extracted’, ‘manual review required’, and ‘non-extracted’. If the critical points or their synonyms for a given descriptive target field are extracted exactly, the extracted outcome would be labeled as ‘extracted’; If any of the potentially related words are extracted, we would label the outcome as ‘manual review required’; If the words or sentences are not retrieved in the whole literature, we would label the outcome as ‘non-extracted’.
2.2.3 Selected target literature processing
In the above steps, we have obtained the text and numerical information required for filtering. According to the inclusion and exclusion criteria, we set up a code form containing dictionaries and patterns for literature selection. The code form can be optimized by clinical researchers according to different screening purposes. For optimizing, clinical researchers should proofread the results marked as ‘manual review required.’ Then, for the "non-extracted" results due to the incomprehensiveness of the patterns, extraction codes would be supplemented and modified. This optimization process should continue until none of the remaining can be automatically extracted.
A matrix is created for each systematic review, including all the relevant articles and the indicator of ‘included’ or ‘excluded’ for a given reason. In this study, the stepwise exclusion procedure is designed to improve the efficiency of literature screening.