Author	Affiliation
Molly Estes, MD	Loma Linda University, Department of Emergency Medicine, Loma Linda, California
Puja Gopal, MD	University of Chicago, Department of Emergency Medicine, Chicago, Illinois
Jeffrey N. Siegelman, MD	Emory University, Department of Emergency Medicine, Atlanta, Georgia
John Bailitz, MD	Northwestern University Emergency Medicine, Feinberg School of Medicine, Chicago, Illinois
Michael Gottlieb, MD	Rush University Medical Center, Department of Emergency Medicine, Chicago, Illinois

Background
Appraisal of the literature
Current uses of iii in emergency medicine
Best practice recommendations
Cautions of implementation
Best practice recommendations
Options for iii activities
Best practice recommendations
Limitations
Conclusion

BACKGROUND

One of the most recent trends in medical education is the transition from traditional didactics (i.e., lecture-based classroom teaching) to online learning modules, collectively referred to as asynchronous learning. Over the last several years, asynchronous learning has been shown to be a successful learning style for many learners. For example, Liu and colleagues performed a meta-analysis of what the authors termed “blended learning” (i.e., the combination of traditional teaching methods with asynchronous learning) throughout all health professional learners. Their review found that blended learning consistently performed better than no intervention and that it did not perform inferiorly to traditional “non-blended” learning.¹ A host of additional data exists, demonstrating that learners prefer smaller learning environments² and that these methods can address the challenge of teaching physician self-assessment and fostering the practice of lifelong learning.³

It is known that there is a broad range of the use of asynchronous learning across the field of medicine. Looking specifically at resident training, a survey of internal medicine program directors (PD) revealed that out of the 214 responding programs, 71.5% used asynchronous learning sometimes, somewhat often, or very often.⁴ Examples of asynchronous learning curricula can be found in nearly every medical area and specialty, from a pediatrics gastroenterology subspecialty rotation,⁵ microsurgery competencies in plastic surgery,⁶ and radiology residents receiving more real-time feedback on radiographic reads,⁷ to journal club for general surgery.⁸ There are examples for the training of fellows⁹ and faculty.¹⁰ There are even examples of all learners, laypeople and medical professionals, participating in a basic life support class¹¹ and for interprofessional learners from all levels of training and fields participating in teamwork training.¹²

The early 2000s to 2010s saw a unique challenge to residency programs specifically as Free Open Access Meducation (FOAM) resources increased exponentially.¹³ Anecdotal evidence at that time suggested that residents were using these resources for their own asynchronous education, with or without residency program oversight. Programs faced the decision to either begin vetting and incorporating these resources into their curricula or to maintain a more traditional didactic approach. Questions were raised whether time spent in asynchronous learning could even be counted as part of Accreditation Council for Graduate Medical Education (ACGME) required didactic time.

In 2008, the Council of Emergency Medicine Residency Directors (CORD) in conjunction with a task force from the Residency Review Committee for Emergency Medicine (RRC-EM) set out to critically evaluate the ACGME EM Program Requirements specifically pertaining to educational conferences. One of the suggestions from that task force was for residency programs to actively consider incorporating asynchronous learning as an educational tool.³ Not long after the task force’s recommendations, the RRC-EM published criteria allowing up to 20% of conference didactic time to be spent in asynchronous learning, which was renamed individualized interactive instruction (III).¹⁴ A subsequent publication from the same group further defined specific requirements of a valid III program (see section on “Cautions of Implementation”).¹⁵

Since then, there has been increasing research into how and which aspects of EM residency teaching can be transitioned to III.¹⁶ Some programs have applauded it as the way of the future,^17-18 while others have advised caution in implementation.^19-20 Multiple ideas have been published on how to incorporate III such as flipped classroom,²¹ journal article discussion boards,²² or a series of varied online learning tasks.²³ Comprehensive databases have emerged offering vetted sources, centralized information, and access to experts.²⁴

Surveys have shown extensive utilization of III among residents,²⁵ as well as significant incorporation into EM training programs.^26-27 A survey by Waxman and colleagues in 2014 showed that 63% of programs were incorporating III into residency training; however, they noted there were significant variations in the structure of the curricula. Of the 37% that were not using III, 71% had concerns related to the understanding and implementation of III within the ACGME/RRC-EM criteria.²⁶ The purpose of this article is to provide a review of the current literature on III and best practices recommendations for programs to consider as they refine their already-existing III curricula or implement a curriculum for the first time.

APPRAISAL OF THE LITERATURE

This article is the second in a series of best practice reviews from the CORD Best Practices Subcommittee. The first three authors performed a search of PubMed for articles published from inception to March 31, 2018, using the same keywords “asynchronous learning” and “individualized interactive instruction.” Bibliographies of all relevant articles were reviewed for additional studies. The search authors screened articles to evaluate for any that addressed the specific topics of implementation and utilization of III curricula within the field of EM.

The search yielded a total of 664 articles, of which 19 were deemed to be directly relevant to EM and for inclusion in this review. When supporting data were not available, recommendations were made based upon the authors’ combined experience and consensus opinion. Prior to submission, the manuscript was reviewed by the entire CORD Best Practices Subcommittee. It was additionally posted to the CORD website for two weeks for general feedback and review from the entire CORD community.

CURRENT USES OF III IN EMERGENCY MEDICINE

In 2015, the CORD III Task Force performed an updated survey of PDs of ACGME-accredited EM residency programs on their current use of III (unpublished data). Of the 77 unique programs that responded (approximately 46% response rate), 74% reported incorporating III into their programs. More four-year format programs used III than three-year programs (91% compared to 67%). Programs implementing III were divided among those who offered either four or five hours of synchronized didactics weekly, or some variation thereof. Of those who reported not using III, the most cited rationale was an unclear definition of what constituted III. Other programs were concerned about compliance or the resources required for implementation. Offerings for III credit were quite diverse. Many programs offered online learning modules, FOAM resources, and board review sessions for III credit. Some used simulation, journal club, and attendance at national or regional meetings. This survey shows that although there is a high rate of utilization of III among programs, there still remains a wide variation in qualifying activities.

While there is a significant amount of literature on the importance and acceptance of III as a learning tool, no standard or consensus method of implementation currently exists in EM. In addition, there is a dearth of information (only the single survey as described above) in the published literature as to how individual EM residency programs specifically implement III. And there is significant variation among programs based on qualitative preliminary surveys. Some research even suggests that III may not be an adequate replacement for all of the didactics in a traditional curriculum, specifically for novice learners, concerns namely being their ability to identify specific knowledge gaps and their need to have adequate expert oversight to ensure true knowledge acquisition and retention.²⁰ Several publications in recent years highlight examples of how EM residency programs nationwide have and are using III; some selected examples are discussed below.

Wray and colleagues implemented an III curriculum in 2013 and measured the effect on in-training exam (ITE) scores. Faculty and chief residents created four modules per month, each designed to be completed in less than one hour. Educational content included journal articles, audio and video lectures, podcasts, links to FOAM resources, and modules linked to quizzes. Residents were required to complete these modules, and their progress was monitored in addition to ITE scores. The group found that despite the decrease in traditional conference hours, time now allotted to III, there was no negative impact on resident ITE scores.²⁸

Pensa and colleagues created a digital course for residents in 2014 and surveyed residents to assess satisfaction. The program educational material was curated by faculty from various FOAM/digital resources, and participation was optional. The modules included an assignment page with the content; a discussion page, which was a mandatory component of the module and allowed for learners to post queries and for faculty members to answer questions; and a multiple-choice quiz page for assessment. Thirty-three of 48 residents participated in the survey in the first year and appeared overall to find the course useful, although there were significant variations in time spent participating in the course both among residents as well as faculty. The biggest barrier to participation identified by residents was lack of time.²⁹

Kornegay and colleagues developed an III curriculum implemented during the 2011–2012 academic year. Faculty members identified gaps in the pre-existing synchronous curriculum and topics better suited for independent learning and then developed a web-based platform consisting of curated content and an evaluation component, namely a reflective writing assignment or quiz. Of responding residents, about 80% were satisfied, very satisfied, or extremely satisfied with the new modality. The group also analyzed conference attendance and ITE scores and found that postgraduate year (PGY)-1 resident attendance rate significantly improved from the prior year (85% vs 62% mean), although other curricular changes in the program (e.g., small group-based learning, interactive case-based conferences, and changes in off-service rotations) may have also enhanced participation. There was no statistically significant difference in mean ITE scores pre- and post-intervention. Faculty reported a time commitment of about four to eight hours per month, which was comparable to the time spent to prepare one hour of instruction for weekly conference pre-intervention.³⁰

Kothari and colleagues designed an III curriculum based on Academic Life in Emergency Medicine (ALiEM)’s popular Approved Instructional Resources (AIR) series. The AIR series curates FOAM content from the top 50 open-access EM and critical-care blog and podcast sites, provides associated core teaching points and multiple-choice questions for residents, and tracks resident participation to provide residency PDs with resident progress.²⁴ Kothari and colleagues then implemented a second component to their III curriculum, which consisted of two high-impact journal articles selected by faculty on a monthly basis. The group found that introduction of the III did not negatively affect residency educational conference; attendance across all PGY levels was comparable to the year before.³¹

Other innovative strategies and formats to implement III in EM have been centered upon discrete, focused topic areas within the larger EM curriculum, such as pediatrics,^32-33 palliative and end-of-life care,³⁴ and disaster medicine.³⁵ Commonalities exist among these examples, namely facilitators’ deliberate choosing of either a specific asynchronous learning program or a specific topic to be taught using asynchronous learning depending on their program’s needs.

BEST PRACTICE RECOMMENDATIONS

 

III should be used cautiously with the novice learner.

 

When deciding to develop or implement an III curriculum, first identify gaps in the current curriculum or those topics that may be best transitioned to an III format. This is likely to vary between programs.

 

A combination of available III (e.g., online blogs, podcasts, and journal articles) seems to attract a greater number of residents to participate, likely as this variety addresses a broader span of individual learning preferences.

 

Transition to III does not seem to negatively affect resident ITE scores or weekly conference attendance rates.

 

CAUTIONS OF IMPLEMENTATION

The ACGME policy statement on the use of III within EM residency education is very strict as to the criteria that must be met for an activity to be considered III. Given that up to 20% (one out of every five hours) of previously considered core curriculum time can now be spent as III, there may be a natural inclination among programs to begin to cut back on planned, traditional educational activities. This is a fallacy, and there are several ways that implementation of III can go wrong (Table).¹⁵ Below are listed some common pitfalls encountered when implementing III.

Independent Reading and Use of Question Banks

The ACGME places particular emphasis on any potential III being a planned activity that is tailored for the individual’s level of learning. Resident-directed reading is not considered a planned activity. Additionally, independent use of a question bank is not directed to the individual’s particular needs, even if the astute resident is choosing specific topics to review. Faculty may choose a specific reading or set of questions to include as a part of III, but these by themselves do not qualify.

Resident Attestations of Completion

An attestation of completion of an III activity is not considered to be adequate enough to prove resident participation. There must be a separate, tangible source of evaluation. Tracking quiz completion/participation after an online module or required reading would provide ample proof of activity completion, just as a sign-in sheet before a simulation does the same.

Audio, Video, or Podcasts

These learning methods are considered to be passive learning, and use of them alone does not qualify as III. However, they can be combined with other learning modalities, such as a particular question set from an online question bank, to include an active component.

Monitoring for Effectiveness

At the time of implementation of the chosen curricula, PDs must have a plan for how they will go about tracking the effectiveness of the III program. This can take many different forms: use of periodic review quizzes; objective clinical performance; test scores on the ITE, etc. However, this type of evaluation must be planned over several generations of residents to account for individual class variation and ensure the III program itself is not causing knowledge gaps. Regular check-ins with residents to ensure their continued perspective of the curricula as beneficial are also recommended.

BEST PRACTICE RECOMMENDATIONS

 

Before designing or implementing an III curriculum, carefully review the ACGME criteria to ensure compliance.

 

Resident-driven use of question banks does not meet III criteria.

 

An attestation of completion does not meet III criteria for participation.

 

Use of passive learning methods alone (e.g., podcasts) does not meet III criteria.

 

Regular curriculum assessment is essential to ensure adequate instructional merit and continued benefit to resident learning.

 

OPTIONS FOR III ACTIVITIES

Several best practices have emerged from surveying EM PDs who have implemented III, both with respect to high quality, effective educational programming and compliance with RRC-EM regulations.²⁷

Simulation

Simulation activities easily satisfy the requirements of III. They can provide an individual resident the opportunity for self-directed work on a particular area of improvement with direct faculty supervision and immediate feedback. These work best when a resident identifies a particular case, topic, or procedure on which he or she would like to focus.

Online Resources

A wealth of freely accessible material is available for III learning via podcasts, blogs, and online modules. PDs need to creatively consider how they will allow for the use of such material for III while maintain compliance with RRC regulations. Additionally, faculty must take care to appropriately vet all resources to ensure credibility and academic rigor.^36–37 Perhaps the most widely adopted single resource is the ALIEM-AIR Series,²⁴ which (as of its 2016 publication) has been implemented in 65 programs. This group rigorously selects the highest quality online resources, as judged by EM faculty, provides a quiz for an evaluative component, and allows for online discussion. Individual PDs are able to monitor both the modules as well as their residents’ participation. Other best practices include discussion sessions with a faculty lead about a particular podcast or blog post.

National/Regional Conferences

Attendance at specialty society meetings offers many learning opportunities. To rise to the level of III and meet the criteria set forth by the ACGME, programs have instituted a number of policies for such activities. Monitoring participation and faculty oversight are key areas of concern, and can be addressed by checking in with faculty who are also attending or presenting at a particular session. Some programs require discussion or written assignments following the session or conference.

Question Banks

Many question banks are available online and in print for residents’ use in preparing for standardized tests. While answering questions alone does not meet criteria for III (see “Cautions of Implementation” above), reviewing specific questions missed or themes with a faculty member would be acceptable.

Other Opportunities

Multiple other activities are in use in EM programs for III including journal clubs, research and teaching activities, oral boards practice, and many others.

BEST PRACTICE RECOMMENDATIONS

 

When designing an III curriculum, many options for learning activities are available to be included: simulation, online resources, national/regional conferences, question banks with faculty oversight, etc.

 

When choosing online sources, take care to ensure credibility and academic rigor, scoring methods exist and can be used to assess these factors.

 

LIMITATIONS

While all attempts have been made to create an inclusive review of the current use of III in EM residency education, limitations must be acknowledged. In the identification of pertinent articles for inclusion, although multiple search terms were used and bibliographies cross-referenced, it is possible that some articles may not have been identified by the current review. We chose articles based on their primary relevance to the field of EM; thus, our analysis was not intended to be an expansive review of the history of the use of III or its current use in other medical fields or specialties. In the absence of data, every effort was made to make conservative recommendations based on the authors’ experience and expertise as educators in the field of EM and, although a potentially limiting factor, these opinions were available for review by the entire CORD Best Practices Subcommittee prior to publication.

The primary limitation to this data analysis was the relative paucity of data available on the direct implementation or utilization of full III curricula within EM residency programs. Multiple sources have supplied information pertaining to the use of specific, topic-based curricula, but few show analysis of a more extensive use of III as might pertain to what can be considered a core curriculum.

CONCLUSION

This article provides a review of the literature currently available on the implementation and use of III in emergency medicine residency education. It can be said conclusively that III has been proven to be an accepted part of modern residency education. Preliminary data suggest that III may very well augment resident learning without negatively affecting standardized testing scores or resident participation in other traditional didactics. Care must be given to choose the appropriate learning level of the resident and ensure ACGME compliance with curricular activities. However, despite multiple sources of curricula options, there remains a paucity of information regarding the effectiveness of specific III as it pertains to resident knowledge acquisition and retention. More research is needed to further refine what we determine to be gold standard III modalities. Until then, it is the authors’ intention that readers will be more aware of the ACGME guidelines and the III options that exist in order to avoid the potential pitfalls of implementation at their home institutions.

 

CORD Best Practice Committee 2017–18

 

Michael Gottlieb, MD – Co-Chair

Rush University Medical Center

 

John Bailitz, MD – Co-Chair

Northwestern University, Feinberg School of Medicine

 

Jeremy Branzetti, MD

New York University/Bellevue

 

Richard Byyny, MD

Denver Health Medical Center

 

Molly Estes, MD

Loma Linda University

 

Puja Gopal, MD

University of Illinois in Chicago

 

Albert Kim, MD

Washington University in Saint Louis

 

Andrew King, MD

The Ohio State University

 

Melissa Parsons, MD

University of Florida – Jacksonville

 

Footnotes

Section Editor: Jeffrey Druck, MD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Molly Estes, MD, Loma Linda University Medical Center, Department of Emergency Medicine, 11234 Anderson Street, MC-A890A, Loma Linda, CA 92354. Email: mollykestes@gmail.com. 3 / 2019; 20:363 – 368

Submission history: Revision received July 24, 2018; Submitted December 5, 2018; Accepted December 14, 2018

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Liu Q, Peng W, Zhang F, et al. The effectiveness of blended learning in health professions: systematic review and meta-analysis. J Med Internet Res. 2016;18(1):e2.

2. Graffam B. Active learning in medical education: strategies for beginning implementation. Med Teach. 2007;29(1):38-42.

3. Sadosty AT, Goyal DG, Gene Hern H, et al. Alternatives to the conference status quo: summary recommendations from the 2008 CORD Academic Assembly Conference Alternatives Workgroup. Acad Emerg Med. 2009;16(Suppl 2):S25-31.

4. Wittich CM, Agrawal A, Cook DA, et al. E-learning in graduate medical education: survey of residency program directors. BMC Med Educ. 2017;17(1):114.

5. Feist M, Ciccarelli M, McFerron BA, et al. Methods and effects of a case-based pediatric gastroenterology online curriculum. J Pediatr Gastroenterol Nutr. 2013;56(2):161-5.

6. Satterwhite T, Son J, Carey J, et al. Microsurgery education in residency training: validating an online curriculum. Ann Plast Surg. 2012;68(4):410-4.

7. Choi HH, Clark J, Jay AK, et al. Minimizing barriers in learning for on-call radiology residents’ end-to-end web-based resident feedback system. J Digit Imaging. 2018;31(1):117-23.

8. Hammond J, Whalen T. The electronic journal club: an asynchronous problem-based learning technique within work-hour constraints. Curr Surg. 2006;63(6):441-3.

9. Inra JA, Pelletier S, Kumar NL, et al. An active learning curriculum improves fellows’ knowledge and faculty teaching skills. Adv Med Educ Pract. 2017;8:359-64.

10. Bucklen KA, Carlson DW, Shah N, et al. Development of a pediatric hospitalist curriculum to promote faculty development, teaching excellence, and evidence-based care. Hosp Pediatr. 2014;4(6):387-92.

11. Hsieh MJ, Bhanji F, Chiang WC, et al. Comparing the effect of self-instruction with that of traditional instruction in basic life support courses-A systematic review. Resuscitation. 2016;108:8-19.

12. Umoren RA, Poore JA, Sweigart L, et al. TeamSTEPPS Virtual Teams: Interactive Virtual Team Training and Practice for Health Professional Learners. Creat Nurs. 2017;23(3):184-91.

13. Cadogan M, Thoma B, Chan TM, et al. Free Open Access Meducation (FOAM): the rise of emergency medicine and critical care blogs and podcasts (2002–2013). Emerg Med J. 2014;31(e1):e76-7.

14. ACGME Program Requirements for Graduate Medical Education in Emergency Medicine. 2012. Available at: https://www.acgme.org/Specialties/Program-Requirements-and-FAQs-and-Applications/pfcatid/7/Emergency%20Medicine. Accessed on Mar 1, 2018.

15. Frequently Asked Questions: Emergency Medicine. Available at: https://docplayer.net/17206018-Frequently-asked-questions-emergency-medicine-review-committee-for-emergency-medicine-acgme.html. Accessed on Mar 1, 2018.

16. Toohey SL, Wray A, Wiechmann W, et al. Ten tips for engaging the millennial learner and moving an emergency medicine residency curriculum into the 21st century. West J Emerg Med. 2016;17(3):337-43.

17. Reiter DA, Lakoff DJ, Trueger NS, et al. Individualized interactive instruction: an innovative enhancement to resident education. Ann Emerg Med. 2013;61(1):110-3.

18. Pourmand A, Lucas R, Nouraie M. Asynchronous web-based learning, a practical method to enhance teaching in emergency medicine. Telemed J E Health. 2013;19(3):169-72.

19. Brydges R, Dubrowski A, Regehr G. A new concept of unsupervised learning: directed self-guided learning in the health professions. Acad Med. 2010;85(10 Suppl):S49-55.

20. Jordan J, Jalali A, Clarke S, et al. Asynchronous vs didactic education: it’s too early to throw in the towel on tradition. BMC Med Educ. 2013;13:105.

21. Young TP, Bailey CJ, Guptill M, et al. The flipped classroom: a modality for mixed asynchronous and synchronous learning in a residency program. West J Emerg Med. 2014;15(7):938-44.

22. Stoneking LR, Grall KH, Min AA, et al. Online research article discussion board to increase knowledge translation during emergency medicine residency. Adv Med Educ Pract. 2013;4:17-21.

23. Ashton A, Bhati R. The use of an asynchronous learning network for senior house officers in emergency medicine. Emerg Med J. 2007;24(6):427-8.

24. Lin M, Joshi N, Grock A, et al. Approved instructional resources series: a national initiative to identify quality emergency medicine blog and podcast content for resident education. J Grad Med Educ. 2016;8(2):219-25.

25. Mallin M, Schlein S, Doctor S, et al. A survey of the current utilization of asynchronous education among emergency medicine residents in the United States. Acad Med. 2014;89(4):598-601.

26. Waxman M, Jiang J, Sawtelle S. Utilization of independent individualized instruction (III) in United States emergency medicine residency programs: results of the Council of Emergency Medicine Residency Directors (CORD-EM) III Task Force Survey. West J Emerg Med. 2014;15(5.1).

27. Best Practices in Individualized Interactive Instruction. 2015. Available at: https://cordemblog.wordpress.com/2016/06/09/cord-individualized-interactive-instruction-task-force-update-best-practices-in-iii/. Accessed March 1, 2018.

28. Wray A, Bennett K, Boysen-Osborn M. Efficacy of an asynchronous electronic curriculum in emergency medicine education in the United States. J Educ Eval Health Prof. 2017;14:29.

29. Pensa G, Smith J, McAteer K. Calling all curators: a novel approach to individualized interactive instruction. West J Emerg Med. 2018;19(1):169-71.

30. Kornegay J, Leone K, Wallner C, et al. Development and implementation of an asynchronous emergency medicine residency curriculum using a web-based platform. Intern Emerg Med. 2016;11(8):1115-20.

31. Kothari A, Breaud A, Manasco A, et al. Resident attendance at weekly conferences after implementation of an optional asynchronous learning curriculum. Int J Med Educ. 2017;8:163-4.

32. Burnette K, Ramundo M, Stevenson M, et al. Evaluation of a web-based asynchronous pediatric emergency medicine learning tool for residents and medical students. Acad Emerg Med. 2009;16(Suppl 2):S46-50.

33. Chang TP, Pham PK, Sobolewski B, et al. Pediatric emergency medicine asynchronous e-learning: a multicenter randomized controlled Solomon four-group study. Acad Emerg Med. 2014;21(8):912-9.

34. Gisondi MA, Lu DW, Yen M, et al. Adaptation of EPEC-EM curriculum in a residency with asynchronous learning. West J Emerg Med. 2010;11(5):491-9.