Author	Affiliation
Timothy P. Young, MD	Loma Linda University Medical Center, Department of Emergency Medicine, Loma Linda, California
Caleb J. Bailey, MD	Loma Linda University Medical Center, Department of Emergency Medicine, Loma Linda, California
Mindi Guptill, MD	Loma Linda University Medical Center, Department of Emergency Medicine, Loma Linda, California
Andrea W. Thorp, MD	Loma Linda University Medical Center, Department of Emergency Medicine, Loma Linda, California
Tamara L. Thomas, MD	Loma Linda University Medical Center, Department of Emergency Medicine, Loma Linda, California

Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

Didactic conference scheduling presents a challenge to the leadership of residency programs in all specialties. Programs that have switched to shift-based schedules to accommodate duty-hour requirements are finding that residents’ attendance at conference is impacted.1 Emergency medicine (EM), with its inherent shift-based schedule, is no stranger to this dilemma. Not only does shift work impact EM residents’ ability to attend conference, but EM faculty are challenged with finding a balance between preparing for and delivering didactics and managing their own clinical shift schedules. Residents at our institution have commented that they would like greater participation by faculty in didactic sessions. At the same time, the implementation of the Accreditation Council for Graduate Medical Education (ACGME) milestones project in the United States has increased the importance of opportunities for face-to-face evaluation of a resident’s medical knowledge.

Over recent years, the methods and information sources that residents use to learn have changed. The availability of online material in the EM community has exploded, with residents increasingly making use of blogs, podcasts, and social media for learning.2 An entire movement, dubbed FOAM (free open-access “meducation”) has emerged in the online EM community.3 The FOAM concept promotes freely available, “sophisticated, cutting-edge learning resources that enable clinicians and students to update their knowledge and improve their understanding in a fun, motivating and time efficient way.”

The increase in online medical education material parallels one that has occurred in secondary education. The Khan academy is a non-profit organization whose website offers thousands of video lectures on multiple subjects.4,5 Secondary educators have used these and their own videos to create an alternative to traditional didactics that has been called a “flipped classroom.”6 A flipped classroom exchanges the usual process of delivering content synchronously in the form of a classroom lecture and then assigning a problem set for students to complete at home. The didactic is viewed at home asynchronously, usually in the form of a video lecture, and students use classroom time to work on the problem set in a setting where they can receive help and clarification from an expert. The concept has also been described in undergraduate,7,8 as well as graduate education.9 Advantages of the asynchronous video component include the ability of the student to pause, rewind, and fast forward the content to process information at their own pace.10,11 Suggested benefits of the in-class session include the ability of the educator to guide the application of knowledge.10,11

The need for alternatives to traditional didactics in EM has been acknowledged. The Council of Emergency Medicine Residency Directors (CORD) Academic Assembly Conference Alternative Workshop published recommendations in 2008 that included the integration of asynchronous learning activities (also called “individualized interactive instruction”) and promoted more flexibility in didactics.12 The Residency Review Committee (RRC) for EM allows for residents to use individualized interactive instruction for up to 20% of the planned educational experiences or didactics.13 However, the best method to incorporate asynchronous learning remains unclear.14 Early efforts, including sending residents tasks and receiving responses over e-mail,15 journal article discussion boards,16 institutional faculty-developed online video lectures17 and modules,18 and the adaptation of pre-existing modules,19 have had mixed results.

We trialed a flipped classroom format in our didactic schedule as a potential means to incorporate 1) an asynchronous component that could allow for more schedule flexibility and appeal to a millennial audience and 2) a synchronous, interactive component with the potential to increase residents’ interactions with faculty during our didactic sessions. Our aim was to learn about resident and faculty impressions of the sessions in order to develop them as a regular component of our formal curriculum. We present a description and evaluation of two pilot sessions below. To our knowledge, this is the first comprehensive description of a flipped classroom model in the setting of a residency training program.

METHODS

Study Design

We undertook an evaluation of two flipped classroom sessions developed for EM residents. We created a mixed-methods quantitative and qualitative questionnaire to evaluate the flipped classroom sessions. The questionnaire included a fixed-response (quantitative) component with questions meant to evaluate specific aspects of the format, as well as an open-ended (qualitative) component aimed at exploring participant perspectives in a more in-depth manner. This is known as a convergent design and offers the benefit of observing trends while obtaining more detailed responses (and thus a potentially more complete understanding of a phenomenon) from a small sample size.20 Responses for quantitative items took the form of either yes/no, fill in the blank, multiple choice, or five-point likert-type items, which were chosen for familiarity. The study was presented to our institutional review board and designated as exempt from review.

Study Setting and Population

Our EM residency is a three-year ACGME accredited program that consists of 13 residents per year, for a total of 39 residents. Our educational conferences are five-hour weekly sessions. Traditionally, this time has been filled by 50-minute lectures, separated by 10-minute breaks.

Study Protocol

The first flipped classroom session took place in December 2012. The topic was syncope and the session was scheduled for two hours. We chose a freely available online lecture about syncope created by Andy Neil and hosted on his website.21 One week prior to the session, residents were emailed a link to the video with instructions to watch it prior to conference. A reminder email was also sent out on the day prior to the session.

To facilitate the moderated discussion, we created a worksheet of 22 questions on the topic of syncope. The worksheet was a combination of open-ended, multiple-choice, and matching questions. These included questions from EM board review sources such as the Physician’s Evaluation and Educational Review in Emergency Medicine (PEER) series of question books,22 the Council of Residency Directors in Emergency Medicine (CORD-EM) question bank,23 as well as some written by residency leadership. Questions were selected to reinforce and expand on the topics presented in the video. Question stems were augmented with images of electrocardiograms (ECGs) and ultrasound stills. A corresponding key, with expanded answers and explanations for each question on the worksheet, was distributed in advance to the three faculty members moderating the small groups. The question set was not distributed to residents in advance.

On the day of the session, question sheets were distributed to the residents in attendance and the group was divided into three small groups. The three faculty moderators each led a small group discussion. Other faculty members in attendance were distributed among the groups. The small groups met at large tables in the same room. Moderators led the residents through the question set. Residents were sequentially asked to each answer a question, taking turns in an “around the table” format until every question had been discussed. The discussion points were at the discretion of the faculty member, although potential “teaching points” were suggested in the answer key. After the session was complete, residents were asked to complete an anonymous evaluation form before leaving the room. The evaluation form had no identifiers other than postgraduate year, and residents placed their own evaluations in a collection box to ensure anonymity.

For the second session in February 2013, we chose the topic of pediatric gastrointestinal presentations. Again, we chose a freely available online video on the topic.24 Residents were similarly emailed a link one week before the classroom session, with a reminder email the day prior to the session.

In response to concerns from residents in the first session about the additional time required to view the video outside of conference, we altered our plan for the second session. Residents were told that they would be excused from the last hour of conference if they attested that they had viewed the hour-long video prior to conference. They would be given the option to view the video in conference for the last hour of the day if they did not watch the video outside of conference. For this session, two members of the faculty of pediatric EM and a senior pediatric EM fellow were recruited to lead the groups. Again, a 22-item worksheet was developed, along with an answer key. Both were distributed to the faculty moderators in advance. Questions on the worksheet included images of radiographs and key physical findings.

The second session was run like the first, with several small changes. This time, the two-hour session started three hours before the end of conference. When the session was complete, residents were again asked to complete the same evaluation prior to leaving the room. Residents who signed that they had viewed the online video outside of conference were excused. For the last hour of conference, the online video was projected for residents who did not watch the video outside of conference.

To obtain the faculty perspective of the flipped classroom sessions, we surveyed the faculty participants for the two sessions. Faculty were asked to comment on advantages and disadvantages of the flipped classroom format.

Outcome Measures

Our primary outcome was the residents’ impression of the value of the session, as measured by the proportion of positive responses (“Agree” or “Strongly Agree”) to two Likert items evaluating the assigned video lectures and two Likert items evaluating the flipped classroom format. Our secondary outcomes included the proportion of residents who viewed the video prior to conference, how the residents accessed it, residents’ opinion of the ideal group size, their preference for frequency of the format in our curriculum, and the qualitative impressions of the residents and faculty regarding the advantages and weaknesses of the flipped classroom format.

Data Analysis

We compiled survey data using Excel for Mac 2011 version 14 (Microsoft Corporation, Redmond, WA). Proportions, medians and interquartile ranges were calculated and histograms were generated using Stata SE 12.1 (Statacorp, College Station, TX). We calculated 95% confidence intervals for proportions with Stata using the modified Wald technique.

RESULTS

Sixteen residents signed in for the first session, of which 14 (86%) submitted surveys. The most common year of training reported was first year (6/12, 50%). For the second session, 31 residents signed the attendance sheet, and 27 (87%) surveys were collected. The classes were more equally represented for this session, which consisted of eight first-year residents (31%), 10 second year residents (38%), and eight third-year residents (31%). Two survey respondents did not report year of training for the first session, and one did not report year of training for the second session. For the second session, three residents stayed for the last hour of conference to view the video because they had not watched it outside of conference.

The figure displays distributions for Likert item responses to questions regarding residents’ perceptions of the video as well as the small group format. For the Likert items evaluating the video lectures, 33/35 residents (94%, 95% confidence interval 80%-99%) responded that the video lecture added to their knowledge about the topic, and 33/35 residents felt that watching the video was a valuable use of their time. For the items evaluating the flipped classroom format, 36/38 residents (95%, 95% confidence interval 82%-99%) preferred the format to a traditional lecture on the topic, and 38/38 residents (100%, 95% confidence interval 89%-100%) felt that the small group session was effective in helping them learn about the topic. Most residents preferred to see the format monthly in our curriculum and chose an ideal group size of 5.5 (first session) and 7 (second session) (Table 1a and Table 1b).

Resident responses to open-ended questions regarding advantages and disadvantages of the flipped classroom are presented in Table 2. Faculty responses to the same open-ended questions are presented in Table 3.

DISCUSSION

We observed a high rate of positive responses from residents to both components of our flipped classroom model. Themes for the perceived advantages of the model in the residents’ qualitative evaluations included interactivity, active participation, access to experts, and retention of material. Faculty noted that the sessions offered the ability to assess residents’ critical thinking and provide feedback.

Figure. Distributions for likert item responses for the flipped classroom evaluation survey.
GI, gastrointestinal

Table 1a. Tabulated responses of residents to flipped classroom survey.

	Percentage answering “yes”
	Syncope session	Pediatric GI session
Did you view the entire video lecture before conference?	9/14 (64%)	23/27 (85%)
Did you watch the video in one sitting?*	7/9 (78%)	16/23 (70%)
Did you use the fast forward, pause, or replay function to enhance your understanding of the content?†	7/13 (54%)	15/25 (60%)

GI, gastrointestinal
*Only responses from those who had watched the entire video are included.
^†All responses are included.

Table 1b. Tabulated responses of residents to flipped classroom survey.

	Most frequent answer
	Syncope session	Pediatric GI session
How did you access the video?†	Tablet computer: 6/13 (46%)	Laptop computer: 10/25 (40%)
Where did you access the video?†	Home: 8/13 (62%)	Home: 20/25 (80%)
How often would you like to see the flipped classroom used in our conference?	Monthly: 8/14 (57%)	Monthly: 17/27 (63%)
What is the ideal group size?‡	5.5 (IQR 5-6)	7 (IQR 5.5-8)

GI, gastrointestinal
^†All responses are included.
^‡When the response was a range, the average value was used (ie. 5.5 for the range 5-6).

Table 2. Representative resident responses to open-ended questions of flipped classroom survey.

What are the advantages of the flipped classroom format?	What are the weakest aspects of the flipped classroom format?
Group discussion	Finding time to watch the video*
Sharing experiences	Less information covered
Interaction	Takes longer to go through material*
Critical thinking	Can get off topic
Applies more to clinical practice	Time consuming*
Allows for more effective discussion	Poor compliance with the assignment*
Allows for feedback from attendings and peers	It can get too loud in the room†
Better use of conference time	Lack of participation by some
Promotes questions	Should be “turn based” to avoid having the same people answer
Decreases anxiety	It can be hard to hear the speakers†
Reinforces learning points
Less somnolence
Active learning
Helps me remember concepts
Keeps my attention
Less boring
Small group discussion
Covers more material
Expert commentary
Discussion of different approaches to management
Easier to ask questions
Ability to ask questions in real time
More intimate
Retention of material

*Comment from first session.
†Comment from second session.

Table 3. Representative faculty responses to open-ended questions of flipped classroom survey.

What are the advantages of the flipped classroom format?	What are the weakest aspects of the flipped classroom format?
Resident/attending interaction	Not all residents are equally prepared (reidents may not have watched the video
Ability to assess residents’ understanding/competence	The quality of the discussion is dependent on the learner’s preparation
Enhanced learning by problem solving together	Need to tie to clinical application and evaluation
More time spent discussing difficult concepts	Residents get less out of it if they don’t watch the videos
Better ability to assess critical thinking
Opportunity for interactions in a less intimidating environment
Senior resident-faculty discussion can facilitate the more novice learner
The discussion that ensued “off-script”
Self-scheduling of video viewing
More interactive
Residents have better retention of material
Easier on the attendings
Ability to provide feedback

Themes for disadvantages perceived by residents included the increased time necessary to prepare for conference. These comments came from the first session, and we responded by allowing for conference “credit” if the video had been viewed asynchronously before the day of conference. We felt that this was a sustainable practice due to the RRC-EM allowance for 20% asynchronous learning, and seemed to help with the perception that additional work was being assigned. Themes for disadvantages perceived by residents in the second session centered on noise levels. We saw improved attendance at the second session, which was likely due to the first session occurring during the holidays. The improved attendance likely contributed to the increased noise level, as we held the conference in the same single room. We suspect that comments regarding lack of participation and the need to make the session “turn based” were due to faculty not adhering to our suggested protocol.

The shift in the way in which residents accessed the video from the first to second session probably reflects our decision to provide all incoming residents with iPad tablet computers, starting with our first-year class. The larger proportion of first-year residents at the first session probably accounted for the greater use of tablet computers to access the video for that session.

Our residents’ perception of the ideal group size was small. A small group size would likely help with the issue of noise level that was commented on in the second, better-attended session. It would also likely accentuate the decrease in anxiety that was cited as an advantage, and increase breadth of resident participation, which was cited as a disadvantage. To accommodate the group sizes that our residents preferred our full residency complement would need to be split into 5-7 small groups. Our residents felt that the ideal frequency of implementation of the flipped classroom in our curriculum was monthly, which would entail the development of 36 unique in order to create a non-repeating three-year curriculum. We were expecting a lower preferred frequency due to the newness of the format, and felt this was a positive endorsement for the format.

The RRC-EM has the following requirements for individualized interactive instruction: the program director must monitor resident participation; there must be an evaluation component; there must be faculty oversight; and the activity must be monitored for effectiveness.13 The model as we have incorporated it provides the ability to monitor resident participation in the synchronous classroom session. Faculty provide oversight by selecting the content, creating the worksheet, and leading the small group discussions. We plan to monitor the format for effectiveness through evaluations of the sessions and by monitoring residents’ performance on in-service exams after its incorporation into our curriculum.

We feel that faculty exposure to residents’ patterns of critical thinking during the small group sessions is a strength of the format. When errors in critical thinking are found, residents can be given the opportunity to relearn critical concepts with faculty guidance. We anticipate that this will provide us with a robust platform for faculty to evaluate residents’ medical knowledge that will complement evaluations done on clinical shifts. Our faculty commented that senior-level conversations during the sessions facilitated the more novice learners. The classroom small group sessions may allow residents with greater mastery of a topic to take more of a leadership and teaching role in discussions. This may offer faculty a potentially elusive opportunity to evaluate upper-level milestones.

From our experience with these two sessions, the best topics to choose for an EM flipped classroom are those that are broad and complex enough to sustain in-conference discussion, as well as those that lend themselves well to visual stimuli such as ECGs, clinical photos and radiographic images. We have also found that a rate limiting step can be finding a high quality video on such a topic, and suggest that as an early criterion. If the schedule permits (ours did not), it may be beneficial to schedule the “optional” hour in which the video is shown as the first hour of conference, so that those who have not watched the video can come at the usual conference start time and those who have not can come at the start of the flipped classroom session. This option would theoretically allow for more residents to have seen the video prior to the synchronous session.

LIMITATIONS

We report on a small sample size of residents from one institution, which limits the generalizability of our results. We evaluated an additional pilot session in an effort to increase our sample size. While this allowed us to obtain evaluations from a larger pool of residents, repeated measurements also likely biased our results toward the perspectives of residents who are more diligent with conference attendance. While familiar to respondents and easily quantifiable, Likert-type items limit dimensionality of responses, which likely did not capture the full breadth of resident attitudes. We hoped to mitigate this with the addition of open-ended questions. Our pilot sessions did not look at performance indicators or patient care outcomes to evaluate the success of the flipped classroom format. This would be an important future step that could be accomplished by a pre- and post-test comparing the flipped classroom to a traditional lecture format.

Future directions

For future sessions, we plan to place the small groups in separate rooms to address issues with the noise level generated by discussion. We plan to recruit more faculty to lead groups for future sessions in order to decrease group size. We are excited about the possibility of re-using the sessions in the future, allowing a broader base of faculty to participate in didactics without onerous preparation. As we proceed with the implementation of the ACGME milestones, we plan to incorporate faculty evaluations of residents’ performance in the sessions as a means to provide data to our clinical competency committee regarding residents’ progress. We are currently developing an online format for the flipped classroom worksheet that will allow for embedded video and audio to augment the problem set.

CONCLUSION

Our flipped classroom model was positively received by EM residents. Residents preferred a small group size and favored frequent use of the format in our curriculum. The flipped classroom represents one modality that programs may use to incorporate a mixture of asynchronous and interactive synchronous learning and provide additional opportunities to evaluate residents.

Footnotes

Supervising Section Editor: Cecylia Kelley, DO

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Timothy P. Young, MD, Loma Linda University Medical Center, 11234 Anderson Street, A108, Loma Linda, California 92354. Email: tpyoung@llu.edu.

Submission history: Submitted August 14, 2014; Revision received October 3, 2014; Accepted October 4, 2014

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. The authors disclosed none.

REFERENCES

1. Burns J. Critical care in the age of the duty hour regulations: circadian-based scheduling, standardized handoffs, and the flipped classroom? Crit Care Med. 2012;40:3305-06. [PubMed]

2. Mallin M, Schlein S, Doctor S, et al. A survey of the current utilization of asynchronous education among emergency medicine residents in the United States. Acad Med. 2014;89:598-601. [Pubmed]

3. Nickson C. FOAM. Available at: http://lifeinthefastlane.com/foam/. Accessed Jun 28, 2013.

4. O’Donnell JF. The Khan Academy: a great opportunity for cancer education. J Cancer Educ. 2012;27:395-6. [PubMed]

5. Parslow GR. Commentary: the Khan academy and the day-night flipped classroom. Biochem Mol Biol Educ. 2012;40:337-8. [PubMed]

6. Bergmann J, Sams A. Flip Your Classroom: Reach Every Student in Every Class Every Day. Washington, DC: International Society for Technology in Education; 2012.

7. Gannod GC, Burge JE, Helmick MT. Using the inverted classroom to teach software engineering. Proceedings of the 30th international conference on software engineering, Leipzig, Germany. 2008.

8. Lage MJ, Platt GJ, Treglia M. Inverting the Classroom: A Gateway to Creating an Inclusive Learning Environment. J Econ Educ. 2000;Winter:30-43.

9. Ferreri SP, O’Connor SK. Redesign of a large lecture course into a small-group learning course. Am J Pharm Educ. 2013;77:13. [PubMed]

10.Horn M. The Transformational Potential of Flipped Classrooms: Different strokes for different folks. Education Next. 2013;Summer:78-9.

11. Seven things you should know about flipped classrooms: Educause. 2012; Available at: http://net.educause.edu/ir/library/pdf/ELI7081.pdf. Accessed Jul 16, 2013.

12. Sadosty AT, Goyal DG, Gene Hern H, et al. Alternatives to the conference status quo: summary recommendations from the 2008 CORD Academic Assembly Conference Alternatives workgroup. Acad Emerg Med. 2009;16Suppl2:S25-31. [PubMed]

13. Frequently Asked Questions: Emergency Medicine. Accreditation Council for Graduate Medical Education Review Committee for Emergency Medicine 2012; Available at: http://www.acgme.org/acgmeweb/Portals/0/PDFs/FAQ/110_Emergeny_Medicine_FAQs.pdf.

14. Reiter DA, Lakoff DJ, Trueger NS, et al. Individual interactive instruction: an innovative enhancement to resident education. Ann Emerg Med. 2013;61:110-13. [PubMed]

15. Ashton A, Bhati R. The use of an asynchronous learning network for senior house officers in emergency medicine. Emerg Med J. 2007;24:427-428. [PubMed]

16. Stoneking LR, Grall KH, Min AA, et al. Online research article discussion board to increase knowledge translation during emergency medicine residency. Adv Med Educ Pract. 2013;4:17-21. [PubMed]

17. Burnette K, Ramundo M, Stevenson M, et al. Evaluation of a web-based asynchronous pediatric emergency medicine learning tool for residents and medical students. Acad Emerg Med. 2009;16Suppl2:S46-50. [PubMed]

18. Chang TP, Pham PK, Sobolewski B, et al. Pediatric emergency medicine asynchronous e-learning: a multicenter randomized controlled solomon four-group study. Acad Emerg Med. 2014;21:912-919. [PubMed]

19. Gisondi MA, Lu DW, Yen M, et al. Adaptation of EPEC-EM Curriculum in a Residency with Asynchronous Learning. West J Emerg Med. 2010;11:491-9. [PubMed]

20. Cremwell JK, Plano-Clark VL. Choosing a mixed methods design. In: Designing and Conducting Mixed Methods Research. 2nd ed. Thousand Oaks CA: Sage;2011:53-106.

21. Neil A. Blackouts, Syncope. 2012; Available at: http://emergencymedicineireland.com/2012/08/blackouts-and-syncope/. Accessed Jun 28, 2013.

22. Wagner MJ. PEER VIII: Physician’s Evaluation and Educational Review in Emergency Medicine: American College of Emergency Physicians; 2011.

23. www.cordtests.org. Council of Emergency Medicine Residency Directors. 2013; Available at: http://www.cordtests.org. Accessed Jun 28, 2013.

24. Rudd S. Pediatric GI Emergencies. 2013;Available at: http://cmedownload.com/lecture/pediatric-gi-emergencies-free-lecture-no-cme-available. Accessed Jun 28, 2013.