Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

A recent shift to focus on “value” in healthcare, often defined as health outcomes achieved per dollar spent, has emerged in response to persistently rising costs over decades.
¹
Recent events have highlighted the cost of emergency care in the national spotlight, including federal legislation on surprise billing, insurer denials of claims for emergency department (ED) visits without a final emergent diagnosis, and regulations on payments for air ambulance transports.
<sup>2

–

5</sup>
Consistent with these developments, current Accreditation Council for Graduate Medical Education (ACGME) guidelines state that “residents must demonstrate competence in … incorporating considerations of value, equity, cost awareness, delivery and payment, and risk-benefit analysis in patient and/or population-based care as appropriate.”
⁶

Despite the current ACGME guidelines and increasing demands for high-value care (HVC), the appropriate educational content and instructional methods have not been clearly established. Moriates and colleagues delineated 21 HVC competencies with beginning, proficient, and expert levels through an iterative process led by a multidisciplinary committee.
⁷
While rigorous and expert-led, this approach did not include a resident-focused needs assessment, and subsequent needs evaluations have been limited to surveys of internal medicine or pediatrics residents at a single site.
<sup>8

–

10</sup>
Similarly, evaluation of proposed internal medicine or pediatrics resident curricula have been limited to single-site pre-/post-surveys, with one study also including post-implementation focus groups.
<sup>10

–

13</sup>

Within emergency medicine (EM), HVC and health economics educational resources are limited, as a 2010 systematic review of cost-effectiveness curricula identified only a single EM curriculum focused on the Ottawa ankle rules. Since that review, two additional contributions that we are aware of include 1) the Emergency Medicine Residents’ Association Residents’ Advocacy Handbook addressing policy-related topics in a textbook-like format and 2) a cost-conscious care curriculum developed by Lin and Laskowski at a single site in New York (personal communication, L. Laskowski).
<sup>14

,

15</sup>
There is a paucity of formal, resident-focused needs assessments across specialties, particularly in EM. Our objective was to perform a targeted needs assessment to assess EM residents’ needs and interests in HVC and preferences for instructional modality.

METHODS

Study Design

As part of a curriculum development process, we performed a problem identification and targeted needs assessment for EM residents, corresponding to Kern’s six-step approach to curricular development.
¹⁶
To achieve our objective, we conducted a qualitative, exploratory study using conventional content analysis. This method allowed us to critically examine the participant responses to identify common categories and elucidate themes. Our secondary objective to determine preferences for instructional modality included a collection of respondents’ self-assessments using a survey questionnaire. We obtained institutional review board (IRB) approval for all study procedures.

Setting and Participant Selection

The setting was a single Midwestern United States EM residency program with 56 total residents. Two physician authors were residents at the time of the data collection phase of the project (BHL, SKM). Recruitment of a convenience sample of eight EM residents was performed via email by one of the authors (SKM) to the remaining 54 residents. Nine residents responded. (One resident could not participate due to scheduling constraints.) No participant terminated their participation during the focus group.

Data Collection Procedures

We obtained documentation of informed consent prior to study procedures. A semi-structured interview guide for focus groups was primarily authored by a single author (BHL) and reviewed sequentially by additional authors for revision of content and phrasing (SKM, BP). The interview guide is included as Appendix 1. Focus groups were co-led by two physician authors who were residents at the time (BHL, SKM) following the interview guide. Both focus groups were audio recorded and subsequently transcribed. No field notes were made, nor were transcripts returned to participants for comment. The focus groups occurred during September 2020 in a medical school conference room with no other person present aside from focus group leaders and participants. After the focus group discussion was complete, participants independently completed a survey questionnaire using Likert scale and rank order questions on paper (Appendix 2). Each focus group included four participants with at least one intern (postgraduate year [PGY] 1) in each group. In total, the first group included one PGY-1, one PGY-2, and two PGY-3 residents; the second focus group included two PGY-1 and two PGY-4 residents. Each focus group lasted between 75–85 minutes. No repeat interviews were completed. Participants received a $15 gift card for compensation, consistent with IRB guidelines.

Data Analysis

The transcripts were reviewed and conventional content analysis with line-by-line coding was completed by two independent coders (BHL, SKM). Using an open coding technique, important statements were identified (generally termed “the first cut”).
¹⁷
Codes were developed in vivo and did not reference previous literature. (They are depicted in a coding tree in Appendix 3.) Significant redundancy in codes was identified, which was felt to be consistent with thematic saturation.
¹⁸
The analysis team came together with a third reviewer (BP) to categorize, refine, and cluster important statements, and subsequent themes and domains emerged. We used the consolidated criteria for reporting qualitative research (COREQ) as reporting guidelines (Appendix 5).
¹⁹
Descriptive statistics were performed in Microsoft Excel for questionnaire data, and we used Word (Microsoft, Redmond, WA) for transcripts and coding documentation. The use of independent coders and a team of three to categorize and develop themes enhanced credibility, and investigator triangulation aided confirmability of the results.
¹⁸

Reflexivity Statement

Reflexivity of the research team included recognition that the focus group leaders and coders were known to the participants and identified their respective specific interests in HVC/health economics (BHL) and medical education (SKM) to the participants as part of the introduction. The focus group leaders identified as male (BHL) and female (SKM). BHL and SKM were residents at the time of the study. BP provided training to BHL and SKM regarding techniques in semi-structured, focus group facilitation; BHL had limited prior experience with focus group facilitation. A methodological limitation is that the same residents comprised the focus groups and completed survey questionnaires; survey questionnaire results may have been influenced by the preceding focus group discussion, although all questionnaires were completed independently by all participants without additional discussion.

RESULTS

A total of eight residents participated. With respect to the importance of education about HVC topics, residents endorsed the relevance of HVC topics to the resident physician (7/8, [88%]) and the importance of a HVC curriculum (8/8, [100%]) (Appendix 4, Figure 2). We identified two overarching themes: 1) experience with and perception of HVC; and 2) desired education and curricular design. For each overarching theme, component sub-themes summarized clusters of resident comments for which we include representative comments and (if identified) participant recommendations.

Overarching Theme 1: Experience with and Perception of High-value Care

Sub-theme 1: Understanding of high-value care focuses on diagnosis and decision-making.

Residents most frequently associated HVC with the activities that facilitate diagnosis and decision-making in the ED. For example, when asked whether they had a general definition for or had heard of the phrase “high-value care,” one resident highlighted using the ED evaluation to “appropriately figure out what is going on with this patient and decide where to send them” (Resident #1, PGY-1). In this understanding, residents believe care activities are high value if they allow the clinician to make a diagnosis or disposition. Less commonly, other residents mentioned aspects of HVC such as resource use, stewardship (citing a specific example of a cost-savings initiative related to the use of combat gauze [Resident #7, PGY-4]), and the concept of cost-benefit analysis: “clinical decision rules that … reduce unnecessary head CTs, not only from a radiation perspective, but also from a cost-savings perspective” (Resident #8, PGY-4).

Sub-theme 2: Concern about patient costs.

In the focus group discussion, residents voiced uncertainty due to varying patient insurance reimbursement of care provided in the ED and concerns surrounding high patient costs, in large part due to a self-identified lack of knowledge. Because of this knowledge gap, residents felt inadequately prepared to have conversations surrounding cost and insurance coverage with patients. One of the participants recalled a patient encounter in which the resident felt uninformed to address the patient’s reaction after the resident disclosed the presence of a new mass concerning for cancer:

How much is this going to cost me? How am I going to pay for this?’[and] I didn’t know the answer. … It’d be nice if I actually had some data … like you’re uninsured, it’s ok, because it’s going to be like this for the financial plan, if you’re insured, this is what happens. I have no clue.”(Resident #5, PGY-3)

Other residents stated that they were unaware of the costs of commonly ordered diagnostics and therapeutics in the ED. They described being concerned and unaware of the financial and social ramifications of care activities on patients’ lives outside of the hospital, and they particularly worried about the impact on patients’ ability to engage with recommended outpatient care: “It’s how much the patient gets charged that would actually matter from a social determinants of health perspective” (Resident #2, PGY-1). Residents particularly cited feeling challenged by shared decision-making discussions when patients had financial concerns.

Sub-theme 3: Conflict between internal beliefs and external expectations.

Residents noted that there may be a conflict between a physician’s personal beliefs and the external expectations and pressures they face. Some external expectations, such as those from systems-level “hurdles” placed in the electronic health record-ordering interface, are explicitly identifiable for residents: “I try to order [intravenous acetaminophen] all the time. IT takes you through, you have to go through all these questions because they’re trying to keep me from ordering [it]. … I know they’re trying to keep me from ordering it, but I’m going to keep on ordering it” (Resident #8, PGY-4). Other external expectations are perceived to be implicit within the medical community: “Even though we talk about in an academic setting, or in a boardroom, it’s OK to have a miss from a statistical perspective, I think culturally that’s not acceptable. … It’s just not playing out in the real world, in my opinion, accepting that there is a miss rate” (Resident #4, PGY-2).

Residents particularly highlighted that patients are a source of external expectations and recognized that patients may view cost, quality, and value of care differently from the emergency physician. This difference in perception may lead to a disconnect in expectations: “Value can really be in the eyes of the beholder … makes me think about what I think might be the best thing for the patient may not be at all the same as what the patient values” (Resident #6, PGY-3). Moreover, the conflict between internal beliefs and external expectations can overshadow attempts to prioritize HVC. A context cited for this conflict were ED visits of patients who commonly frequent the ED. For these patients, the lack of community resources for patients can be frustrating and render a learner feeling helpless or unable to provide holistic patient care. For these patients, trainees noted feeling a disconnect between the care they felt expected to provide and the care they desired to provide.

Sub theme 4: Approach to high-value care changes with increasing clinical experience.

Residents shared anecdotes that demonstrate how the definition of and approach to optimize HVC changes with increasing clinical experience. One junior resident highlighted “wanting to know” as motivation for ordering testing: “I’m as curious as [the patients] are, to be honest; so I want to know that this patient is perhaps a presentation of [a specific diagnosis]” (Resident #4, PGY-2). Similarly, as one non-intern resident reflected:

“And honestly, that’s something that comes with time – like if you told me as an intern I could order a million-dollar test and get the answer that I need, I would 100% do it because it’s easy, I’ll be right, and I can help the patient. But as you practice medicine you realize … if you have a million-dollar test to answer if it’s GERD … it’s not going to change your management … As I’m progressing through residency I get more and more curious, and I’m more willing to accept information about [HVC]” (Resident #5, PGY-3)

Sub-theme 5: Slow-moving, political discussion around high-value care in medicine.

In general, residents describe themselves as loosely aware of the political, academic, financial, and clinical implications of national discussions on HVC topics for future emergency physicians. For example, “How you determine value? I remember back when Obama was still around and in office, I remember that was a big discussion, you know—what is real value and who determines that? That’s sort of a black box” (Resident #8, PGY-4). Another resident reflected,

“There is always chatter out there in the … political and insurance world. And I’m not sure I know where like the landmark policy or … guiding foundation is for that conversation. So, certainly, outside there is a feeling that there is always this chatter happening” (Resident #4, PGY-2).

When asked about proposed physician reimbursement models currently undergoing federal regulatory review, most residents did not know what those future policies entailed. In addition, many residents reported not being well versed in current reimbursement models, although non-intern residents reported more interest in current reimbursement information.

Overarching Theme 2: Desired Education and Curricular Design

Sub-theme 1: Limited medical education on health economics and high-value care.

When asked about their prior training in health economics and HVC topics, all residents noted minimal to no prior exposure during their medical training. In the survey questionnaire, all participants (8/8 [100%]) either strongly disagreed or disagreed with the statement “I feel confident that I know the cost of the care that I provide to patients in the emergency department” (Appendix 4). Much of the prior exposure described by residents was comprised of brief and infrequent didactic-based discussions that were described as leading to limited information retention and limited application to clinical practice.

Beyond this, they voiced the belief that there were few opportunities for knowledge acquisition due to lack of available resources, particularly with respect to prices and costs of healthcare activities. Residents were not familiar with hospital-based or nationally based resources that could assist with day-to-day clinical healthcare questions such as patient cost: “I think hospitals are mandated to have some sort of list, master list, of how much things costs, but it’s also super hard to find … I have no idea where I would find that information” (Resident #7, PGY-4).

Sub-theme 2: Motivated to receive training on high-value care and health economics.

The EM residents identified themselves as frontline healthcare workers. In their role, they interact directly with the community and patients with diverse backgrounds, particularly individuals facing financial barriers to accessing care. Because of this unique position in the medical field, residents believe that financial and insurance pressures may underlie patients’ utilization of the ED and that clinicians should therefore understand these factors. One resident reflected

“I think when you … look … at healthcare as a gestalt, people are seeing primary care [clinicians] less and less and relying on the ED more for primary care. And assuming that that trend continues … I think as an emergency physician it is important to know those things [healthcare economics topics] because of that reason, just the utilization of the ED in general” (Resident #1, PGY-1).

One resident also noted that the lack of health economics knowledge can put emergency physicians at a disadvantage in influencing and leading systems-based practice:

“I think not understanding [HVC and health economics topics] takes away a lot of our power to be a leader and makes us more pawns carrying out someone else’s vision of how medicine should be practiced” (Resident #6, PGY-3).

Residents recognized the importance of and need for further training on HVC topics to understand the impact that their decision-making has on patients and the healthcare system.

Sub-theme 3: Desire for discussion-based format for high-value care curriculum.

Residents were asked what the optimal format for HVC curriculum would be for residency-level learners, and the majority were in support of a discussion-based format.

“I like the idea of the case-based, small-group discussion. Especially when you have attendings there, and you have varied learner levels, and I kind of like that because you get varied sorts of inputs and that’s interesting. And I just feel like this sort of stuff, these sorts of topics, are best, for me, explored verbally” (Resident #8, PGY-4).

One resident noted that because this is not common knowledge among emergency clinicians, involving a content expert would be critical to a successful curriculum:

“Another part of incorporating this, is who is the content expert…. [HVC care is] a topic that … a typical academic [emergency] physician would [not] know about. It almost needs to be a collaboration… [someone] with health economic interest and knowledge and someone with an education background, too, to figure out how to incorporate this” (Resident #7, PGY-4).

Consistent with this qualitative theme, the highest percentage of residents ranked modalities with the opportunity for interactive small-group discussion highly, whether as online apps or in person, on the survey questionnaire (Figure 1).

Sub-theme 4: Curriculum targeted to their level of training.

While most of the residents recognized the need for a formal HVC curriculum during medical training, there was variation in when they thought this curriculum should be introduced at the residency-training level. The PGY-1 and PGY-2 residents voiced desire to focus on clinical knowledge acquisition in lieu of health economics topics:

“As an intern, I’d rather be more towards the clinical aspect of things right now … I don’t think I’ve developed that skill enough to want to sacrifice one of those journal clubs for health economics. I think as a later resident, I’d be on board …” (Resident #1, PGY-1).

“My initial thought was that I would want something clinically relevant because I feel like [I am] earlier in training and just trying to build that foundation …” (Resident #4, PGY-2).

A non-intern resident noted “I feel like as I’m progressing through residency, I get more and more curious and I’m more willing to accept information about that stuff [HVC]” (Resident #5, PGY-3). In reply to an intern indicating the topic of “[relative value units] and physician-associated income … wouldn’t appeal or apply to me right now when I would just forget it” (Resident #3, PGY1), Resident #7 (a PGY-4) reflected that non-intern residents would be interested due to personal decision-making: “I would say the PGY-3s because some of the PGYs would start signing contracts in the summer”

An interesting perspective raised by one of the non-intern residents was the potential to negatively influence junior learners’ practice patterns if topics of HVC were introduced too early in medical training:

“I wonder from an education mission side, could you influence early trainees’… practice patterns because of knowledge of this. And I don’t want that to happen … you need to see where you fall in that spectrum to develop your practice pattern. And I wonder if you find out that a test costs this amount of money, maybe you won’t get to fully explore that spectrum and develop your own point on that spectrum” (Resident #7, PGY-4).

Lastly, a couple of residents voiced concern about the integration of a novel curriculum in an EM training program given that EM’s scope of practice already addresses many adjacent disciplines:

“We’re all kind of in agreement that a baseline level of understanding you should have … but as far as about data and literature … I’d kind of reserve that for people that have an interest in it, similar to how we do with other things, like sports medicine” (Resident #3, PGY-1).

“You have so many things to learn. Not only clinically, but also our non-clinical curriculum … is pretty impressive, so it’s tough [to] add a whole other curriculum” (Resident #8, PGY-4).

DISCUSSION

Residents recognized the importance of learning HVC principles for application in both patient care and to inform systems-based practice; however, they felt inadequately trained on the topic. Our needs assessment identified two main themes to inform EM-specific curricula addressing HVC topics: resident experience with and perception of HVC, and desired education and curricular design.

Consistent with studies in other disciplines and settings, the residents reported limited confidence in their knowledge of basic HVC principles, and the financial impacts of cost of care for individual patients and the healthcare system as a whole.
<sup>9

,

10

,

13</sup>
Sub-themes 1 (understanding of HVC focuses on diagnosis and decision-making) and 2 (resident concerns about patient costs) in this study were consistent with themes from focus groups completed with general pediatrics residents at two centers of “how an intervention changes management” and “thinking about the cost as a harm.”
¹⁰

Residents stated that early on in their training, HVC knowledge gaps are related to patient costs, patient insurance reimbursement, cost-benefit analysis, and resource stewardship. Later, self-identified knowledge gaps emerging as non-intern learners were primarily related to physician reimbursement. A review of the literature, including prior work within pediatrics and internal medicine, suggested no prior evidence of resident knowledge or interest varying by experience level; if validated in additional settings, such variations with learner experience would provide valuable guidance in the design of educational curricula.

The resident participants stated their lack of formal training in and basic knowledge of HVC was a barrier to providing high-value emergency care. They also reported limited awareness of national health policy yet were less interested in a detailed understanding of these topics. This finding suggests that a specialized elective may be better suited to education regarding health policy topics that do not directly tie into day-to-day emergency care, as in the example described by Greysen and colleagues.
²¹
Finally, the participants also indicated the need for more education on system-wide reimbursement and HVC policies. To meet this need, prior national-level survey data from internal medicine residents and program directors suggests that institutional support for both HVC faculty development and provision of physician cost-of-care performance data are associated with an increase in resident reports of education on HVC.
²²

Unanticipated aspects of HVC that were viewed as learner obstacles included dynamic conflicts between internal learner beliefs and external expectations and the variability in value perception between patients and clinicians. These issues may complicate residents’ perception of and implementation of HVC in the clinical setting; addressing these issues within HVC education is critical to avoid unintentional creation of anxiety, or even moral distress, in the training environment. In an intern-targeted curriculum in internal medicine, Hom and colleagues also discussed resident-perceived barriers surrounding intra-team, interdisciplinary, and patient and family dynamics and how they complicate understanding and implementation of HVC principles at an early learner stage.
¹⁴
Thus, future curricula will need to focus both on foundational knowledge dissemination and techniques on how to approach the above barriers.

An additional unexpected barrier raised by residents in the focus group was the concern that the existing EM training curriculum does not have the capacity to incorporate HVC; and, therefore, HVC training may not fit as a core element. While not addressed in these focus groups, a future direction for work in this area should include evaluation of how residents would weigh HVC training compared to other curricular elements and whether there would be opportunity to make potential “tradeoffs.”

In terms of curricular design and format, themes emerged to optimize not only knowledge acquisition and understanding, but also timing during the residency training program. The resident participants were in support of an expert-led, discussion-based curriculum to learn the principles of HVC, consistent with the experiences of Hom and colleagues.
¹³
These findings also coincide with those of Stammen et al in their systematic review, concluding that reflective practice through feedback and group discussions incentivize physicians to think critically about medical decisions.
¹⁹
The residents also suggested that HVC topics should be targeted more toward non-intern residents who have mastered proficiency in basic clinical knowledge and skills and would be able to apply these new principles with more purpose than their junior counterparts, although some earlier knowledge base to supplement formative experiential growth throughout residency may be beneficial. They did voice concern that the introduction of HVC too early in residency could jeopardize early learners’ practice pattern development.

LIMITATIONS

There are several limitations to consider with regard to our study. First, this study reflects a sample of residents from a single-center, large academic hospital and may not be applicable to all academic- and community-based training programs. Because it was a single-center study, we could not distinguish how three-year programs or four-year programs with different approaches to resident progression or “seniority” would differ from the findings identified here. Second, only a small subset of program residents participated in either focus group, leading to the possibility of selection bias with regard to the participants who volunteered to discuss their thoughts on HVC. These residents may have had a particular interest in medical education or HVC that may not be applicable to all EM residents across the country. The small subset of participating residents also likely limited the number of available perspectives to be collected and inform thematic saturation.

Third, the study included a mix of junior and senior residents. While the study allowed for a rich spectrum of experience to inform previous exposure to HVC principles, it may not have been as impactful as evaluating the perspectives of the most experienced residents in a program who had nearly completed the entire program curriculum and could identify areas for nuanced improvement. Fourth, while the use of focus groups (rather than one-on-one interviews) allowed emergent discussion between participants, the presence of peers may have led some participants to avoid making statements due to fear of being perceived as controversial. Fifth, due to transitions in roles, member checking could not be performed. While our study adds a critically necessary needs assessment to the current body of literature, further and more rigorous studies that include a larger number of residency programs and participating residents are needed to verify these findings to accurately inform future EM curricula.

CONCLUSION

Our targeted needs assessment indicates that residents currently face gaps in knowledge of high-value care topics pertaining to the medical care that they provide and may benefit from additional training during residency. Residents interviewed in this study identified several perceived barriers to understanding HVC, but they consistently expressed interest in a formal curriculum to address these challenges. We found a preference for interactive, small-group discussion-based formats with content adjusted by level of clinical training.

Footnotes

Section Editors: Sara Krzyzaniak, MD, and Jeffrey Love, MD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Bennett H. Lane, MD, MS, University of Cincinnati College of Medicine, Department of Emergency Medicine, 231 Albert Sabin Way ML 0769, Cincinnati, OH 45267-0769. Email: lanebt@ucmail.uc.edu
01 / 2024; 25:43 – 50

Submission history: Revision received December 13, 2022; Submitted October 3, 2023; Accepted November 3, 2023

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. This work was supported by internal departmental funding from a designated Resident Research Fund. No author reports any related conflicts of interest. Bennett H. Lane reports receiving research grant support from the Society for Academic Emergency Medicine Foundation for investigator-initiated work on interhospital transfers by air. Bennett H. Lane also reports an equity interest and consulting fees from TriAxia Health for unrelated work.

REFERENCES

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.