A brief history of competencies, milestones and the nas
Milestones and em training
Where next?

A Brief History of Competencies, Milestones and the NAS

The NAS was part of the educational community’s response to improve graduate medical education (GME).³ The NAS is designed to help achieve the original vision of the Outcomes Project that was officially launched in 2001, based on the six general competencies formally approved by the Accreditation Council for GME (ACGME) and the American Board of Medical Specialties (ABMS) in 1999.⁴ However, programs struggled to implement an outcomes-based approach and operationalize the competencies, new concepts to many medical educators, into meaningful changes in curriculum and assessment.

One reason for this struggle is the lack of shared mental models, or frame of reference, regarding the competencies among programs and clinical faculty. There were several reasons for this struggle. First, the ACGME/ABMS general competencies were defined in conceptual terms that were often hard to translate into practice. Second, some of the competencies, especially practice-based learning and improvement and systems-based practice, were new concepts altogether. Third, work-based assessment methods were either unavailable or not well aligned with the purpose and goals of the competencies.

The Milestones were developed collaboratively by each specialty to create the core blueprint, or roadmap, of the discipline in narrative, developmental language. In other words, they have helped to describe the competencies in more understandable language. The Milestones serve as a framework to inform and guide curriculum, choice of assessment methods and instruments, and assessment judgments by the clinical competency committee.⁵ Milestones also begin to move us away from an over-reliance on the quantification of competence, traditionally represented by numeric rating scales, toward a more qualitative, descriptive approach.

There are two important caveats. First, milestones do not define the totality of any discipline, including EM. Milestones are key elements of a larger “whole” of clinical competence. Second, substantial professional judgment, on the part of the faculty, is critical in the overall assessment of readiness for clinical practice. Informed judgment, based on multiple assessments through various forms of observation, is a cornerstone of a competency-based system.⁶ Using Milestones to guide and perform systematic measurement holds promise to enhance our ability to assure the public of the effectiveness of GME to prepare physicians for practice.

Milestones and EM Training

EM was one of the early adopters in the NAS, being just one of five specialties to report both mid-year and end-year Milestones data in the 2013–14 academic year. Several aspects of the EM approach to developing and implementing the Milestones are noteworthy. First, the Milestones are grounded in the “Model of the Clinical Practice of EM” (EM Model). The EM Model consists of three core components: 1) individual conditions; 2) physician tasks; and 3) acuity levels. The knowledge, skills and attitudes comprising the current EM Model were informed by a national survey of 9,740 physicians in 2007 regarding EM practice.⁷ Second, the American Board of EM (ABEM) engaged over 60% of EM residencies as part of a national validation study. Participating program directors essentially took the EM Milestones for a “test drive” and provided the EM Milestones working group with feedback on the Milestone placements and descriptions. In fact, of the final 227 Milestones included in the 23 subcompetencies, 46 Milestones were reassigned different performance levels based on the program feedback.^2,7

The EM approach to Milestone development appears to have had an important impact. The initial validity study of the EM Milestones reported in year one shows very promising results. Factor analysis of the national Milestones data revealed a three-factor structure concordant with the three component EM Model of practice.² In addition, reliability coefficients for the Milestones were robust.² My hypothesis for these early positive findings brings me back to one of the initial purposes of the Milestones: to create shared mental models of the general competencies not only within EM residencies, but equally important between EM residencies across the country.⁷ EM will likely stand as an exemplar for other specialties in how to build national standards for judging EM residents. In essence, the EM developmental approach is akin to a nationally-based performance dimension training (PDT) exercise. PDT is an established approach to helping to improve performance evaluations by getting all evaluators “on the same page.”^6,8 By incorporating empiric evidence into EM certification design along with robust involvement of educators and program directors, the EM educational community has already likely made substantial progress in creating shared mental models of EM training and assessment.

Where Next?

The NAS is built on a foundational principle of continuous quality improvement. In the United States and most of the world, education and healthcare systems are experiencing significant change and disruption. We must continue to move away from a “static/stable” view of education and clinical care to one that is dynamic and constantly evolving. There are several implications for GME. First, changes in educational programs must become better integrated with the changes occurring in healthcare delivery and systems. Care of patients and populations is a dynamic, integrated process. As the frontline specialty of the healthcare system experiencing this disruption, EM is well positioned to lead and inform educational redesign. Second, work-based assessments will continue to grow in importance and prominence. One example of a useful technique in EM is end-of-shift encounter cards.⁹ When used properly, encounter cards can enhance the quality of assessment and feedback. EM also leads the way in the development of chart stimulated recall (CSR), a validated method to assess clinical reasoning of actual patient care.⁹ While not currently in widespread use, CSR and other performance-based methods (e.g. clinical indicators and patient experience) represent the next frontier in work-based assessment for EM training.

Third, the current Milestones are truly version 1.0; as in all continuous quality improvement processes some amount of change and revision will be needed. The processes used by the EM community to create version 1.0 will be invaluable to the larger educational community and capturing the detail behind these processes will be important. Finally, we want to ensure the Milestones do not create overly reductionistic assessments and curricula. Residency education will be most effective when the output is a whole physician who effectively integrates all competencies, however defined, into his or her practice. For example the EM community has developed entrustable professional activities (EPAs) to further help operationalize the competencies and milestones.^10,11 EPAs hold promise to help enhance curriculum and assessment, using milestones as “building blocks” for each EPA. Our collective goal is to produce physicians who can successfully enter unsupervised practice and continue their trajectory toward expertise and mastery. Competencies, Milestones and in the near future EPAs can serve as meaningful frameworks to help produce a talented, whole physician.

In conclusion, here is a request to the EM community: The ACGME, the ABEM and the dozens of talented EM faculty who volunteered their time, expertise, and wisdom to advance the Milestones, need and welcome constructive feedback to continually improve the NAS and Milestones. Milestones are tools to facilitate and promote innovation and continuous improvement in GME in the Unites States, but they are not yet fully realized and will require changes and adjustments. We are entering a period of transformation that requires collectivism among all the key stakeholders and that can feel, like any change, uncomfortable. Only by working together through dialogue and across organizations can the full potential of outcomes-based medical education be realized. The EM educational community has clearly taken up this charge.

Footnotes

Section Editor: Jeffrey Love, MD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Eric S. Holmboe, MD, Accreditation Council for Graduate Medical Education, 515 North State Street, Chicago, IL 60611. Email: eholmboe@acgme.org. 11 / 2015; 16:801 – 803

Submission history: Revision received August 11, 2015; Accepted August 11, 2015

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. The authors disclosed that Dr. Eric S. Holmboe receives royalties from Mosby-Elsevier for a textbook on assessment and serves on the boards of Medbiquitous and NBME.

REFERENCES

1. ACEP Emergency Visits Up Since Implementation of ACA. Available at: http://newsroom.acep.org/ACEP-Emergency-Visits-Up-Since-Implementation-of-ACA. Accessed May 24, 2015.

2. Beeson M, Holmboe E, Korte R, et al. Initial Validity Analysis of the Emergency Medicine Milestones. Acad Emerg Med. 2015.

3. Nasca TJ, Philibert I, Brigham T, et al. The next GME accreditation system—Rationale and benefits. N Engl J Med. 2012;366:1051-6.

4. Batalden P, Leach D, Swing S, et al. General competencies and accreditation in graduate medical education. Health Affairs (Millwood). 2002;21:103-11.

5. Holmboe ES. Realizing the promise of competency-based medical education. Acad Med. 2015;90:411-3.

6. Holmboe ES, Sherbino J, Long DM, et al. The role of assessment in competency-based medical education. Med Teach. 2010;32:676-82.

7. Beeson MS, Carter WA, Christopher TA, et al. The development of the emergency medicine milestones. Acad Emerg Med. 2013;20:724-9.

8. Woehr DJ, Huffcutt AI. Rater training for performance appraisal: a quantitative review. J Occup Org Psych. 1994;67:189-205.

9. Sherbino J, Bandiera G, Frank JR. Assessing competence in emergency medicine trainees: an overview of effective methodologies. CJEM. 2008;10:365-71.

10. Ten Cate O. AM last page: what entrustable professional activities add to a competency-based curriculum. Acad Med. 2014;89:691.

11. Beeson MS, Warrington S, Bradford-Saffles A, et al. Entrustable professional activities: making sense of the emergency medicine milestones. J Emerg Med. 2014;47:441-52.