Background
Objective
Curricular design
Impact/effectiveness
Discussion
Limitations
Conclusion

BACKGROUND

The lecture has long been the primary teaching method for graduate medical education.¹ Because lectures can be performed with large student-to-teacher ratios, they are considered an efficient teaching method.^1-2 Effectiveness of lectures as a teaching method has been called into question due to the lack of learner engagement.^2-3 Lectures put the responsibility for learning in the hands of the teacher, who regulates both the sequence and depth of content coverage. Learner participation is limited to listening, taking notes, and asking clarifying questions.

Educators have sought to replace lectures with methods that promote active learning and longer term retention.^4-7 One such alternative, the flipped classroom, has been widely adopted across a variety of disciplines.^7-10 The premise of the flipped classroom is that learners read and study new content independently in advance of a face-to-face classroom learning session.¹¹ Content is either prescribed by the instructor or independently identified by the learner and includes online learning modules, textbooks, or journal articles. Once learners are prepared, they meet with their peers in facilitated small groups to apply newly acquired knowledge to cases or problems. The flipped classroom is student-centered. Learning is driven by the learners but guided by experienced educators.^8,11

Proponents of the flipped classroom hypothesize that it allows adult learners to integrate new knowledge with existing knowledge.^9-10 The act of covering material at their own pace prior to a meeting promotes deeper learning, longer retention and life-long learning skills. The face-to-face classroom sessions promote knowledge application, critical thinking, and peer-faculty interactions. Additionally, flipped classrooms may prepare learners for eventual information-gathering and decision-making in complex clinical settings by mimicking real-life interprofessional interactions.^13-14

Although studies of the flipped classroom are small and observational, there is growing consensus that students favor this method over the traditional lecture.^{10, 15-19} In the flipped classroom, learners use study time to build a foundation for new learning instead of spending that time reviewing lecture notes and retrofitting new knowledge with old. The flipped-classroom method not only promotes longer term retention but provides learners with cues to the depth and breadth required for use of the new knowledge in clinical application. Challenges associated with the flipped-classroom model include increased time commitment for both educators and learners, effective integration of technology, ensuring individual learner accountability, and promotion of a safe learning environment.^{10-11, 18,20}

The Accreditation Council for Graduate Medical Education (ACGME) requires that emergency medicine (EM) residency programs provide five hours of weekly didactic instruction.²¹ Residents are required to participate in 70% of these didactics. Historically, our program has fulfilled ACGME didactic requirements through weekly lectures. For the 2015–16 academic year (AY 2016), we changed our didactic format to the flipped-classroom model.

OBJECTIVE

The purpose of this study was to evaluate the outcomes of our first year of flipped-classroom instruction through comparison to preceding years of lecture instruction.

CURRICULAR DESIGN

Instructional methods

During the 2015–16 academic year, we structured our residency conference around themes covering patient presentation (e.g., chest pain, pregnancy, shortness of breath). Lectures were replaced with facilitated small-group discussions using the flipped classroom and case-based learning. Simulations and procedure sessions were also added to the conference day. Residents prepared for conference by reviewing related patient cases, and then reading recommended learning materials. Residents were also encouraged to identify and read their own learning materials. Conference time was reserved for facilitated small-group discussions about the cases, and residents were given the opportunity to apply what they learned to diagnosing and developing management plans for patient cases.

Population

We performed a cross-sectional cohort study of EM residents who entered our program between 2011 and 2016. Our average enrollment grew over this time from n=12 per entering class to n=18. The E-2011 and E-2012 cohorts (n=28) were the last two cohorts to experience only the lecture-based curriculum. The E-2013 and E-2014 cohorts experienced both lecture-based and flipped-classroom curricula (n=31). The E-2015 and E-2016 cohorts experienced only the flipped-classroom curriculum (n=36). Our institutional review board declared this exempt research.

Measurements

We compared the performance of residents who participated in the flipped classroom to those who received the lecture curriculum on the annual American Board of Emergency Medicine (ABEM) in training examination (ITE), (a standardized test normed on all residents in ACGME-accredited EM residency programs). We controlled for training level by comparing resident scores by level separately (Figure 1).

We also developed a program evaluation questionnaire to assess resident opinions of their educational experiences. The questionnaire asked residents to rate each component of the program on both quality of instruction and value to their professional development. Residents were also asked how many hours they spent with textbooks, online instruction, and journals.

Data Analysis

While controlling for level of training (i.e., interns from the new curriculum were compared to interns in the old curriculum, etc.), we used independent t-tests to compare ITE scores for residents in the lecture-based curriculum to those in the new flipped-classroom curriculum (Figure 1).

We compared program evaluation survey results between residents who experienced the final year of the lecture curriculum (Academic Year 2014–15 [AY 2015]) to those who participated in the first year of the flipped-classroom curriculum (Academic Year 2015–16 [AY 2016]). This ensured that at least two thirds of the residents had experience with both curricula and were able to make fair comparisons. To avoid Type-1 error rates, a common problem when making multiple comparisons, we redefined the p-values considered statistically significant using the Bonferroni adjustment.²²

IMPACT/EFFECTIVENESS

Table 1 shows the results of the cohort comparisons on ITE scores from independent t-tests. We observed no statistical difference on the average ITE scores between residents who participated in the lecture curriculum and those from the flipped-classroom curriculum at any of the three training levels (PGY1–3).

We received program evaluation surveys from 28 of 45 residents (62.2%) in AY 2015 and from 19 of 49 residents (38.8%) in academic year AY 2016. Twenty-seven residents were eligible to participate in both surveys; however, only nine of 27 residents (33.3%) completed both.

Program component ratings of quality and value are presented in Table 2. Program components used only in one year or the other are left blank to indicate that no statistical comparison was made. Almost all program components except for Mock Oral Boards were rated higher in terms of both quality and value by residents in the flipped classroom. However, only two components, adult and pediatric case conferences, were rated significantly higher in quality, but not value (adult case conference: t=−4.0, df=45, p≤.001, es=−1.19; pediatric case conference: t=−3.7, df=45, p=.001;es=−1.10). Cohen’s d effect sizes (es) for these comparisons are considered large.²³

Small-group methods were rated the same across program years. Although not significant, lectures were rated higher in quality and value in the flipped-classroom curriculum than they were in the lecture-based program.

Residents in the flipped-classroom curriculum reported spending significantly more hours with outside learning resources as a whole (textbooks, online learning resources, and journals combined) when compared to residents in the lecture-based curriculum (t=2.68; df=38; p=.011; es=−.852) (see Figure 2). The Cohen’s d effect size (es) for the difference in average time spent with outside learning resources (all together) is considered large.²³ When compared separately, the amount of time spent on any one type of resource was not significantly different.

DISCUSSION

The adoption of a flipped-classroom educational model for our large academic medical center’s EM residency program did not have any major impacts on traditional outcomes, such as standardized test results or program evaluations. Our findings are consistent with the published literature on use of the flipped-classroom model in health sciences education.^12,16,18

Residents in the flipped classroom reported spending significantly greater amounts of time with outside learning materials: textbooks, online learning resources, and journals. This is our most significant yet not surprising finding, since preparation for small-group discussion during class meetings is a program expectation. Residents in the flipped classroom reported spending almost double the amount of time with textbooks along with roughly 25% more time with online instruction materials and journals.

Increases in time spent with preparation materials may also explain residents’ higher quality ratings of case conferences, both pediatric and adult. We believe that because residents come prepared to discuss and apply their learning to these cases that they find these activities to be of higher quality. Residents also expressed appreciation for the inclusion of rapid-fire case presentations during case conferences.

In our flipped-classroom program, the use of self-chosen learning resources was encouraged. We believe that this is appropriate at a graduate level of medical education, since preference for different types of learning resources are likely to be varied. The Free Open Access Medical Education (FOAM) movement has provided learners with a wealth of content material presented in a variety of ways from medical education experts around the world.²⁴ The fact that we did not see a large, significant increase in the amount of time our residents spent with online instructional resources, is probably attributable to the fact that our lecture-group residents had also used these materials to supplement their education.

Increased use of FOAM resources combined with a flipped-classroom approach to weekly didactic sessions is helping students at the post-graduate level to customize their education,²⁵ while reserving valuable group time for application of knowledge to real-world scenarios under the guidance of an expert.²⁶ We expected to see higher ratings of both value and quality of most of the program components under the flipped-classroom curriculum than the lecture curriculum. However, because so few of our respondents (9 of 27) experienced both program models, we are not sure that we captured a true “curriculum change” effect. In other words, residents rated what they know, without a reference to an alternative curriculum model.

Generally, our program evaluation provided some evidence for a successful transition from a lecture-based to a flipped-classroom residency curriculum. The educational outcomes we were able to measure through standardized tests and program evaluations remained stable across the two programs. While learners in our program seemingly have responded to the flipped classroom by adopting the required preliminary learning, we are unable to confirm that the flipped-classroom model is truly superior to traditional lecture methods with regards to educational efficacy.

LIMITATIONS

Our efforts suffer a few limitations, the worst of which was incomplete program evaluation data from our residents, particularly in the second year of the study. While we reached nearly a 40% return rate from residents in that academic year, the probability of selection bias was high. We checked for selection bias and recognize that our respondents represented more PGY-1 and 3 residents.

While the ABEM ITE assesses the collective medical knowledge of resident trainees, this single, annual assessment of medical knowledge may not be sufficiently sensitive to detect the subtle differences in educational achievement obtained from two different curriculum models. While the flipped-classroom method of teaching is designed for deeper learning and longer-term retention, an annual standardized test may not be the best measure of this type of learning.

Future studies using assessment instruments more specifically designed for measuring educational efficacy between the flipped classroom and traditional lecture methods are needed. Furthermore, study designs that are effective at isolating the type of learning that occurs in classroom didactics from the type of learning that takes place in the clinical environment could contribute to further understanding the efficacy of different curriculum methods.

CONCLUSION

In the flipped-classroom program, residents spent more time with learning resources outside of the classroom. We see this as an indicator that they were investing more time with self-directed learning. The flipped-classroom program had no detectable effect on ITE scores and minimal effect on residents’ ratings of program components. Our findings are somewhat consistent with the findings of others. In summary, we believe that the flipped-classroom model is as educationally effective as traditional lecture methods and holds promise for further exploration. Additional studies with more sensitive assessment instruments are needed to identify potential differences in educational efficacy between the flipped classroom and traditional lecture methods.

Footnotes

Section Editor: Sally A. Santen, MD, PhD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Andrew M. King, MD, The Ohio State University, 750 Prior Hall, 376 W 10th 10 Avenue, Columbus, OH 43210. Email: Andrew.King3@osumc.edu. 1 / 2018; 19:11 – 17

Submission history: Revision received June 14, 2017; Submitted October 26, 2017; Accepted October 29, 2017

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Cooper AZ, Richards JB. Lectures for Adult Learners: Breaking old habits in graduate medical education. Am J Med. 2017;130(3):376-81.

2. Honan WH. Whither college lectures? Maybe right out the door. The New York Times (NY Region Ed). 2002. Available at: http://nyti.ms/1eOkCzz.

3. Hurst JW. The overlecturing and underteaching of clinical medicine. Arch Intern Med. 2004;164(15):1605-8.

4. Grasha AF. Teaching with Style: A practical guide to enhancing learning by understanding teaching and learning styles. 1996.

5. Brown PC, Roediger HL, McDaniel MA. Make it stick: The science of successful learning. 2014.

6. Hatem CJ, Searle NS, Gunderman R, et al. The educational attributes and responsibilities of effective medical educators. Acad Med. 2011;86(4):474-80.

7. Prober CG, Khan S. Medical education reimagined: a call to action. Acad Med. 2013;88(10):1407-10.

8. Gilboy MB, Heinerichs S, Pazzaglia G. Enhancing student engagement using the flipped classroom. J Nutr Educ Behav. 2015;47(1):109-14.

9. Khanova J, Roth MT, Rodgers JE, et al. Student experiences across multiple flipped courses in a single curriculum. Med Educ. 2015;49(10):1038-48.

10. Moraros J, Islam A, Yu S, et al. Flipping for success: evaluating the effectiveness of a novel teaching approach in a graduate level setting. BMC Med Educ. 2015;15:27.

11. Moffett J. Twelve tips for “flipping” the classroom. Med Teach. 2015;37(4):331-6.

12. Liebert CA, Lin DT, Mazer LM, et al. Effectiveness of the surgery core clerkship flipped classroom: a prospective cohort trial. Am J Surg. 2016;211(2):451-7.

13. Khandelwal A, Nugus P, Elkoushy MA, et al. How we made professionalism relevant to twenty-first century residents. Med Teach. 2015;37(6):538-42.

14. Hatem CJ. Teaching approaches that reflect and promote professionalism. Acad Med. 2003;78(7):709-13.

15. Mazur E. Education. Farewell, lecture?. Science. 2009;323(5910):50-1.

16. Young TP, Bailey CJ, Guptill M, et al. The flipped classroom: a modality for mixed asynchronous and synchronous learning in a residency program. West J Emerg Med. 2014;15(7):938-44.

17. Morgan H, McLean K, Chapman C, et al. The flipped classroom for medical students. Clin Teach. 2015;12(3):155-60.

18. Tan E, Brainard A, Larkin GL. Acceptability of the flipped classroom approach for in-house teaching in emergency medicine. Emerg Med Australas. 2015;27(5):453-9.

19. Rose E, Claudius I, Tabatabai R, et al. The Flipped classroom in emergency medicine using online videos with interpolated questions. J Emerg Med. 2016;51(3):284-91.

20. Whelan A, Leddy JJ, Mindra S, et al. Student perceptions of independent versus facilitated small group learning approaches to compressed medical anatomy education. Anat Sci Educ. 2016;9(1):40-51.

21. ACGME Program Requirements for Graduate Medical Education in Emergency Medicine. 2016. Available at: https://www.acgme.org/Portals/0/PFAssets/…/110_emergency_medicine_2016.pdf. Accessed March 1, 2017.

22. Bland JM, Altman DG. Statistics Notes: Multiple significance tests: the Bonferroni method. BMJ. 1995;310(21):170.

23. Cohen J. Statistical power analysis for the behavioral sciences. 1988.

24. Nickson CP, Cadogan MD. Free Open Access Medical education (FOAM) for the emergency physician. Emerg Med Australas. 2014;26(1):76-83.

25. Pierce R, Fox J. Vodcasts and active-learning exercises in a “flipped classroom” model of a renal pharmacotherapy module. Am J Pharm Educ. 2012;76(10):196.

26. Koo CL, Demps EL, Farris C, et al. Impact of flipped classroom design on student performance and perceptions in a pharmacotherapy course. Am J Pharm Educ. 2016;80(2):33.