Study population
This study was conducted from September 1, 2016 to October 31, 2017 (participants are shown in Table 1). Visiting staff teachers at Chang Gung Memorial Hospital (CGMH) were enrolled in this study, and two hundred teachers in the hospital were randomly selected by computer. Those who did not consent to the assessment were excluded. The flow diagram of this study is shown in Fig. 1. All teachers and students provided their consent after understanding the study prior to their participation. A research assistant obtained the consent, and participants had the right to remove their data and consent at any time.
Table 1
Basic demographic characteristics of the teachers
Groups
|
e-book
|
PowerPoint
|
Total teachers (male %)
|
35 (48.6)
|
38 (50.0)
|
Internal Medicine
Surgery
Pediatrics
Gynecology
Emergency
< 5 years# (male %)
|
9
7
7
6
6
13 (38.4)
|
10
9
8
6
5
13 (46.2)
|
5–10 years# (male %)
|
12 (50.0)
|
14 (55.0)
|
> 10 years# (male %)
|
10 (60.0)
|
11 (54.5)
|
# X years of clinical experience * In classes taught by physicians with X years of clinical experience |
Tools: e-book or PowerPoint
Our team designed an online e-book using SimMAGIC software. The e-book contains an introduction to the basic concepts of EBM, EBM databases, database literature search skills, critical appraisal, and effectiveness evaluation methods using repeatable answer-correct quiz [20, 21] (see availability of data and materials).
A repeatable answer-correct quiz was provided to evaluate each of the learning objectives at the end of each topic. Problem sets were created for readers and used for self-assessment purposes. They were presented in a multiple-choice question format, and readers could respond by clicking to answer the question. Another traditional method (PowerPoint) was designed for use as a control group (S1 Appendix). The PowerPoint learning material was designed with the same contents as that of the e-book and also included the basic concepts of EBM, EBM databases, database literature search skills, critical appraisal methods, clinical application, and effectiveness evaluation. In the PowerPoint presentation, the questions were presented in a multiple-choice format but responses could not be elicited by clicking.
Study setting
In order to provide teachers with a better method of learning EBM and then assess their teaching effects through students feedback, a flipped classroom model was designed with an e-book or PowerPoint for teachers’ learning. To study the effect of their teaching, which was assessed by two to three students, the same three 7th year students were invited for every teacher’s teaching, but sometimes only two students were available at the particular time.
The teachers created an answerable question using the PICO model from a true case in the hospital and searched the electronic databases available at CGMH prior to the study. They attempted to consolidate the evidence in order to reach a conclusion and make a recommendation for the question. We then gave them access to the e-book or PowerPoint presentation and encouraged them to practice using it. They re-examined the information they had written by searching, consolidating evidence, and correcting the recommendation. Afterward, the assessment questionnaires shown in Table 2 were completed. The study group utilized the e-book, while the control group used the PowerPoint presentation without the e-book.
Table 2. Modified Fresno test in the e-book and PowerPoint groups (mean±SD)
Pre-test
Questions
|
e-book
|
PowerPoint
|
p value
|
Question 1. Write clinical questions based on context to search for articles to answer patient questions.
|
7.6±2.5
|
8.3±2.2
|
0.190
|
Question 2. Write three kinds of databases and explain their advantages and disadvantages.
|
8.0±2.5
|
8.0±2.3
|
0.289
|
Question 3. If you use PubMed or Medline to search for original papers, what is your search strategy? Include article categories and write the search techniques used to increase the search for the best article.
|
8.7±1.7
|
8.5±2.4
|
0.718
|
Question 4. Choose a case to explain which kind of research design is used for the study type of the question and explain your choice.
|
6.7±1.6
|
5.4±1.9
|
0.314
|
Question 5. What articles related to your problem were found? P: I: C: O:
|
9.0±2.0
|
8.6±2.6
|
0.552
|
Question 6. What are the research methods for the articles found, such as select patients, allocation, attrition, analysis, etc.?
|
8.9±1.9
|
8.6±2.4
|
0.847
|
Question 7. How are the important results expressed by the articles found, such as p-value, odds ratio or mean difference, 95% CI, effect size, etc.?
|
7.8±2.4
|
5.4±2.5
|
0.817
|
Question 8. There are articles describing the diagnosis of biliary obstruction and hepatitis patients; 200 patients have yellow gallbladder, 100 patients were later confirmed to have biliary obstruction, with 70 patients having an ultrasound gallbladder length less than 1.5 cm, and the ultrasound gallbladder length is less than 1.5 cm in 100 patients with hepatitis.
|
5.4±2.7
|
5.5±2.6
|
0.299
|
What is the sensitivity, specificity, positive predictive, negative predictive, and likelihood ratio in the diagnosis of biliary obstruction?
|
|
|
|
Question 9. We find that in our hepatitis patients treated with ursodeoxycholic acid, who were followed for one year, 40% showed a decreased liver index (AST). In the group that was not treated with ursodeoxycholic acid, 20% showed a decreased liver index (AST).
|
6.2±3.9
|
5.3±4.0
|
0.110
|
I would like to ask: What is the absolute risk reduction? What is the relative risk reduction? What is the number needed to treat?
|
|
|
|
Question 10. Children with diarrhea use probiotics. According to one report, the use of probiotics for unused probiotics reduces the number of diarrhea days, and the relative risk is 0.34 times. We would like to know if this result has statistical significance.
|
6.0±3.3
|
5.6±3.1
|
0.809
|
Please cite the number below that represents the confidence interval that indicates a statistical significance of the case: 0.1-0.6; 0.6-1.2; 1.2-2.1.
|
|
|
|
Question 11. Which research design is the best for diagnostic research? RCT? Systematic review? Cohort study? Case control study? Others?
|
6.9±4.6
|
6.1±4.7
|
0.446
|
Question 12. Which study design is the best for prognostic research? RCT? Systematic review?
|
7.0±1.4
|
6.8±1.5
|
0.183
|
Cohort study? Case control study? Others?
|
|
|
|
Post-test
Questions
|
e-book
|
PowerPoint
|
p value
|
Question 1. Write clinical questions based on context to search for articles to answer patient questions.
|
8.0±2.5
|
8.6±2.1
|
0.239
|
Question 2. Write three kinds of databases and explain their advantages and disadvantages.
|
8.0±2.5
|
8.7±2.1
|
0.239
|
Question 3. If you use PubMed or Medline to search for original papers, what is your search strategy? Include article categories and write the search techniques used to increase the search for the best article.
|
8.6±1.8
|
8.3±2.6
|
0.872
|
Question 4. Choose a case to explain which kind of research design is used for the study type of the question and explain your choice.
|
6.8±1.6
|
5.6±1.9
|
0.011*
|
Question 5. What articles related to your problem were found? P: I: C: O:
|
9.1±1.9
|
8.9±2.5
|
0.886
|
Question 6. What are the research methods for the articles found, such as select patients, allocation, attrition, analysis, etc.?
|
8.8±2.1
|
8.5±2.6
|
0.737
|
Question 7. How are the important results expressed for the articles found, such as p-value, odds ratio or mean difference, 95% CI, effect size, etc.?
|
7.5±2.5
|
5.9±2.8
|
0.023*
|
Question 8. There are articles describing the diagnosis of biliary obstruction and hepatitis patients; 200 patients have yellow gallbladder, 100 patients were later confirmed to have biliary obstruction, with 70 patients having an ultrasound gallbladder length less than 1.5 cm, and the ultrasound gallbladder length is less than 1.5 cm in 100 patients with hepatitis.
|
5.3±2.7
|
5.6±2.3
|
0.545
|
What is the sensitivity, specificity, positive predictive, negative predictive, and likelihood ratio in the diagnosis of biliary obstruction?
|
|
|
|
Question 9. We find that in our hepatitis patients treated with ursodeoxycholic acid, who were followed for one year, 40% showed a decreased liver index (AST). In the group that was not treated with ursodeoxycholic acid, 20% showed a decreased liver index (AST).
|
5.7±4.2
|
5.0±4.0
|
0.489
|
I would like to ask: What is the absolute risk reduction? What is the relative risk reduction? What is the number needed to treat?
|
|
|
|
Question 10. Children with diarrhea use probiotics. According to one report, the use of probiotics for unused probiotics reduces the number of diarrhea days, and the relative risk is 0.34 times. We would like to know if this result has statistical significance.
|
5.7±3.4
|
5.1±3.2
|
0.357
|
Please cite the number below that represents the confidence interval that indicates a statistical significance of the case: 0.1-0.6; 0.6-1.2; 1.2-2.1.
|
|
|
|
Question 11. Which research design is the best for diagnostic research? RCT? Systematic review? Cohort study? Case control study? Others?
|
6.4±4.9
|
6.3±4.9
|
0.887
|
Question 12. Which study design is the best for prognostic research? RCT? Systematic review?
|
6.9±1.3
|
6.7±1.6
|
0.814
|
Cohort study? Case control study? Others?
|
|
|
|
*p<0.05, P: patient/problem, I: intervention, C: compare, O: outcome, AST: aspartate aminotransferase
A randomized controlled trial was established with a 1:1 allocation ratio to assess the teachers before they were assigned to either the e-book (n=40) or the control group (n=40). Randomization was executed (allocation concealment) through central randomization performed by an independent randomizer (assistant). She used random assignment with a two-blocked design in the order of their entry into the study using http://www.randomizer.org.
We employed an evaluation method using a structured paper questionnaire of the Modified Fresno test (Table 2) to investigate our teachers’ EBM skills before and after the e-book or PowerPoint learning. Teachers then practiced the EBM steps to search, critically appraise, and analyze the level of evidence for the article they found, make a recommendation, and apply it to a clinical situation.
One month later, each teacher gave a lesson on EBM according to the e-book or PowerPoint content shown in the S1 Appendix.
Student assessment of the teachers’ EBM teaching effect
One month later, each teacher gave a 20 to 30-minute lesson on EBM regarding the formulation of problems, database search techniques, and literature appraisal for three students, who then assessed their teaching skills and ability. The students were provided eight statements to answer with the following options: strongly agree, agree, neither agree nor disagree, disagree, and strongly disagree. The eight statements were as follows: “I am satisfied with this lesson,” “The teaching was of high quality,” “The teacher had a good attitude,” “This was a good teaching method,” “It helped my understanding,” “I was allowed to ask unlimited questions,” “The teacher listened to my questions,” and “It aroused my interest in EBM.”
Validity and Reliability
The content validity of the EBM skill questionnaire (Table 2) was examined by three experts who had no involvement with the participants and were asked to rate each item. The final survey included only items with strong relevance. The internal consistency of all indexes was estimated using Cronbach’s α [22-24]. The questionnaire had a content validity index of 0.90 and Cronbach’s coefficient α of 0.801.
Ethical Process
This study was approved by the institutional review board of the ethics and clinical research committee of CGMH (104-9177B). All teachers and students provided their consent prior to participating in the study. To blind the statistician, the data was anonymized, and participants’ names were removed from the questionnaires before data analysis. Data storage, participant recruitment, and data collection were conducted by a research assistant who had no assessment relationship with the participants.
Statistical Analysis
The sample size analysis was calculated using power analysis and sample size (PASS) software with at least an effective sample size with 80% power at the 5% significance level (2-sided test) in each group for EBP. The evaluation of EBM skills and teaching effects were analyzed as a continuous variable and expressed as mean±standard error or mean difference. We adopted the Mann-Whitney U tests to compare teachers’ evaluation of their EBM skills and the students’ assessment of the teachers’ instruction. The teachers were divided into sub-groups according to the length of their clinical experience: less than 5 years, 5–10 years, and more than 10 years of experience. A p-value < 0.05 was considered statistically significant. The statistical analyses were performed using Statistical Package for Social Science (SPSS, version 12) software.