Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

With emergency departments (ED) open 24/7, most emergency physicians work some night shifts. Unfortunately, prior data has shown that night shift work is associated with increased risk of a variety of medical conditions
<sup>1

–

5</sup>
and motor vehicle collisions after those shifts.
⁶
Additionally, emergency physicians working at night may have to deal with reduced support staff, tired patients, and fewer available consultants. Moreover, multiple prior studies have demonstrated that while on night shift, cognitive performance declines.
<sup>7

,

8</sup>

Despite the unique challenges of night shifts, emergency physicians are generally held to the same standards on night shifts as they are on day shifts, and one way they are now assessed is by patient satisfaction scores. Indeed, patient satisfaction has become an increasingly important part of healthcare in large part because of the incentives initiated by the Affordable Care Act in 2010
⁹
; now, both institution and physician payment are sometimes based on patient satisfaction scores.
¹⁰

Prior studies have shown that certain factors including shorter ED length of stay (LOS),
<sup>11

–

13</sup>
older patient age,
¹⁴
and good communication
¹⁵
are associated with better ED patient satisfaction scores. Two prior studies have investigated the relationship between treatment during night shifts and patient satisfaction scores.
<sup>14

,

16</sup>
One found no statistically significant association,
¹⁴
while another found that physicians who worked fewer night shifts had higher patient satisfaction scores.
¹⁶
Given the conflicting evidence to date and the increasing emphasis on patient satisfaction, we felt that additional study was warranted to assess the relationship between night shift work in the ED and patient satisfaction.

Our primary objective in this study was to determine whether patients who are cared for during night shifts provide lower patient satisfaction scores than those cared for during day shifts, using the real-world satisfaction data. Secondarily, we sought to determine whether other demographic and clinical characteristics are associated with ED patient satisfaction scores.

METHODS

Study Design and Setting

We performed a cross-sectional analysis of ED patient satisfaction scores from patients who presented to a single hospital system in the State of Florida in the southeastern United States from January 1–December 31, 2022. Specifically, we performed a secondary analysis of a previously collected dataset of satisfaction scores, and we performed a chart review to supplement that data. We followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines. The study was approved by the Mount Sinai Medical Center Institutional Review Board. This study received no external funding.

Our hospital system has a tertiary care, community teaching hospital located in Miami Beach, Florida, as well as a freestanding ED located in Hialeah, Florida, (freestanding ED #1) and a freestanding ED located in Aventura, Florida, (freestanding ED #2). The main hospital’s ED had 56,005 visits during 2022, while freestanding ED #1 had 37,932 visits and freestanding ED #2 had 19,635 visits. Emergency medicine residents work shifts only at the main hospital’s ED. Advanced practice practitioners (APP) work shifts at all three facilities. Shift times are shown in Table 1. In 2022, three emergency physicians only worked only night shifts, and one physician worked only day shifts. Some attending physicians and APPs only worked at one facility; others worked at two or all three.

Selection of Participants

In 2022, NRC Health (Lincoln, NE) administered our patient satisfaction surveys and tracked satisfaction data. Surveys were sent by both text message and email to all patients who left the ED. All patients who completed the NRC Health ED patient satisfaction survey in 2022 were eligible for inclusion in this study. Admitted patients were not sent surveys and were excluded from analysis.

Measurements

NRC Health keeps a database with the responses from satisfaction surveys and demographic information about the patients who complete the surveys. In our hospital system, currently, individual physician-level patient satisfaction scores are tracked and assessed using these data, but compensation is not dependent upon them. We generated a report from NRC Health’s data for all patients who completed a satisfaction survey during 2022 and then used a random number generator to create a sample of 1,000 patient encounters for analysis. For each of these patient encounters, two medical students transferred patient responses and available demographic information into a spreadsheet in Microsoft Excel v16.79.1 (Microsoft Corp, Redmond, WA). In particular, NRC Health provided us with the following data for each encounter: the date of the visit; the facility; the name of the physician or APP; method of patient response (email or text message); and the patient’s age, sex, race, medical record number, address, marital status, and preferred language. The two medical students who abstracted these data points had no role in the abstraction of the other data discussed below.

Next, we created a separate spreadsheet with additional clinical information for each of the 1.000 patient encounters using our electronic health record system (EHR) (Epic Systems Corporation, Madison, WI). Six abstractors (three emergency medicine residents, two emergency attendings, and one nurse practitioner) performed manual chart review to determine the patient’s ethnicity, mode of arrival to the ED, times of arrival and departure, Emergency Severity Index (ESI) score,
¹⁷
disposition, clinician who discharged the patient (and their supervising attending, if applicable), and whether or not each of the following was performed during the patient encounter: resident participation; sign-out; blood test; advanced imaging; in-person consultant evaluation, consultation by phone (only); opioid pain medicine administration; and prescription provided.

In general, manual chart review followed the methods suggested by Kaji et al.
¹⁸
The abstractors who performed manual chart review were blinded from the satisfaction data. None of the abstractors or investigators have been a nocturnist. The abstractors filled in 15 columns in the spreadsheet with the data points above. They were trained on proper data abstraction by the principal investigator (TZ), and they followed a data dictionary that explicitly defined the variables and explained where to find them in the EHR. The data dictionary is included as an appendix, which provides detailed definitions of all variables. The definitions of a few important variables are also defined here as follows:

We considered patients to have arrived during the day shift if they arrived in the ED between 7 am–7 pm and to the night shift if they arrived between 7 pm–7 am. We chose these definitions because many physician and nursing shifts follow these time schedules in our system (Table 1). We also divided patients into the time of year they came to the ED by standard quarters.

The type of clinician (physician or APP) who evaluated the patient primarily was determined based on the name of the clinician on the survey as per NRC Health data. For example, a patient was considered to have been seen primarily by an APP if the APP was the person listed on the satisfaction survey. As mentioned above, we also manually recorded the name of the clinician (and their supervising attending) who discharged the patient for each patient encounter. For patients who were not signed out, the discharging clinician (or supervising attending) was fully consistent with the listed name on the surveys. However, for patients who were signed out, sometimes the initial clinician who treated the patient was listed on the survey and sometimes a subsequent one was. Since administrators assess the satisfaction data based on the name of the clinician on the surveys, we used the name of the clinician on the survey as the primary treating clinician.

Patients who left the ED before being evaluated by a physician or APP could still be included in the study if they completed a satisfaction survey but were considered to have not been seen by a physician or APP. All six abstractors obtained the data for a group of the same 50 patients to allow for an assessment of the inter-rater reliability. We calculated the free-marginal kappa for the two variables that we considered to be the most difficult to abstract: sign-out, and in-person evaluation by consultant.

After completing data collection, the principal investigator (TZ) merged the spreadsheets with the satisfaction data and the clinical data, and the data was analyzed as described below.

Outcomes

In 2022, our administration considered the most important question on the satisfaction surveys to be: “How likely is it that you would recommend [facility name] to a friend or colleague?” (from 0–10). A patient who provided a score of 9 or 10 was considered to be a “promoter”; a score of 7 or 8 was considered “passive”; and a score of 0–6 was considered to be a “detractor.” The percentage of promoters minus the percentage of detractors is deemed the “net promoter score,” which is used to measure overall satisfaction in healthcare as well as in other businesses.
<sup>19

,

20</sup>
Our primary outcome was the percentage of completed patient satisfaction surveys that qualified as promoters. Secondarily, we determined the net promoter score and the adjusted odds ratios for being a promoter for several demographic and clinical variables.

Analysis

Based on a preliminary analysis of NRC Health data, we anticipated that there would be approximately twice as many completed surveys from patients who arrived during the day shift vs the night. Additionally, we knew that approximately 75% of our completed surveys in 2022 qualified as promoters. Based on gestalt, we hypothesized that the percentage of promoters from the day shift would be eight points higher than the night shift. To test our hypothesis with an alpha of 0.05 and power 0.8, we required responses from 957 patients. We rounded this up to 1,000 and chose that as our sample size.

For our primary analysis, we compared the percentage of promoters for patient encounters in which the patient arrived during the day shift compared to night shift. We made the unadjusted comparison our primary analysis since that is how patient satisfaction scores are being used to assess emergency physicians in our hospital system. We also compared the demographic and clinical characteristics of the patient encounters for the two groups. We determined normality with the Shapiro-Wilk test. For normal distributions, we compared the means of groups using t-tests. For non-normal distributions, we compared medians using the median test. We used the Fisher exact test to compare categorical variables.

Secondarily, given that a patient who arrives during the end of a day shift might be mostly treated by the night staff (or vice versa), we also analyzed patients based on the time of ED departure. In other words, we considered night shift patients to be those who departed between 7 pm – 7 am. Lastly, to further isolate the night-time hours when people are generally sleeping, we divided patients by arrival time into three eight-hour epochs: 6 am – 2 pm (day), 2 pm – 10 pm (swing), and 10 pm – 6 am (night).

Lastly, we performed a multivariable logistic regression analysis with “promoter” (yes or no) as the dependent variable. Based on prior data,
<sup>11

–

14

,

16

,

21

–

23</sup>
we included the following variables in our model: ED LOS (continuous); ED site (categorical); elderly (age > 65) (binary); pediatric (binary); race (White or not); ethnicity (Hispanic or non-Hispanic), health insurance type (no insurance, commercial, or government/other); non-English speaking (binary); Emergency Severity Index (ESI) (2 or 3 vs 4 or 5); and advanced imaging performed (binary). Based on investigator hypothesis, we also chose to include the following as covariates: quarter of the year (1, 2, 3, or 4); married (binary); seen by a resident (binary); seen primarily by an APP (binary); arrival by ambulance (binary); and blood test performed (binary). We also hypothesized that consultant evaluations would be associated with better satisfaction scores, but these occurred too rarely in our dataset to be included in the regression analysis. We used the Hosmer-Lemeshow statistic to assess goodness of fit of the regression model.

Data was aggregated in Excel and analyzed in R Studio v2023.03.0 (RStudio PBC, Boston, MA). Using two-sided hypothesis tests, we considered P < 0.05 to be statistically significant.

Missing Data

In a few cases, race and ethnicity were not recorded. This was handled as follows: Patients who were documented as having White or Caucasian race were considered to be “White.” Patients who were documented as Black or African American were considered to be “Black.” Patients documented as Asian, American Indian, multiracial, other, or for whom race was not documented were considered neither White nor Black. Similarly, if a patient was documented as “Hispanic,” their ethnicity was considered to be “Hispanic.” If they were documented to be non-Hispanic or if their ethnicity was not documented, they were considered “non-Hispanic.”

RESULTS

Overall

As shown in the Figure, 6,375 satisfaction surveys were completed in 2022, and we randomly selected 1,000 for analysis. Of these, 824 patients responded by text message and 176 responded by email. Our data included surveys evaluating 44 different attending physicians and 18 APPs. There were no missing data points, except for five patients for whom no race and ethnicity were recorded. Inter-rater reliability for the two assessed variables was almost perfect with free-marginal kappa 0.98 (95% confidence interval [CI] 0.94–1.0) for sign-out and 0.98 (95% CI 0.9–1.0) for in-person evaluation by a consultant. Overall, 75.3% of patient encounters qualified as promoters, and net promoter score was 57.8.

Night vs Day Shifts

In total, 663 (66.3%) patients arrived during the day shift, and 337 (33.7%) arrived during the night shift. Table 2 shows a comparison of characteristics of these two groups. Notably, the groups were not balanced on several characteristics including age, race, marital status, insurance, type of clinicians involved, ESI, and advanced imaging performed. Regarding the primary outcome, 525 (79.2%) of those who arrived during the day shift were promoters compared to 228 (67.7%) of those who arrived during the night shift, an absolute difference between groups of 11.5% (95% CI 5.7–17.4%), P  < 0.001. The net promoter score for the day shift was 64.9 and 44.0 for the night shift. Data stratified by facility are shown in Table 3.

When redefining day shift by departure time, there were 492 day-shift patients and 508 night-shift patients. Of those, 396 (80.5%) and 357 (70.3%) were promoters for the day and night shift, respectively, a difference of 10.2% (95% CI 4.9–15.6%), P < 0.001.

When analyzing the data by eight-hour epochs, 307 (80.2%) of 383 patients who arrived between 6 am–2 pm were promoters. Meanwhile, 339 (75.5%) of 449 patients who arrived between 2 pm–10 pm were promoters, and 107 (63.7%) of 168 who arrived between 10 pm–6 am were promoters. Combining the eight-hour day and swing shifts together, 77.6% of surveys were promoters, which is 13.9% (95% CI 6.2–21.8%) higher than the eight-hour night shift group, P < 0.001.

Nineteen completed surveys came from patients seen by one of our three nocturnists. Of those, 12 (63.2%) were promoters. Additionally, the one physician who only worked day shifts had 24 completed surveys, of which 21 (87.5%) were promoters. Excluding the combined 43 encounters from that physician and the three nocturnists made it such that 78.8% of patients who arrived between 7 am–7 pm were promoters and 68.2% of the patients who arrived between 7 pm–7 am were promoters, a difference of 10.6% (95% CI 4.6–16.6%), P =  < 0.001.

Multivariable Regression Analysis

On multivariable analysis, arrival during the night shift had a statistically significant association with a lower chance that the patient would be a promoter, with adjusted odds ratio 0.60 (95% CI 0.43–0.84), P = 0.003. Other than night-shift arrival, no other variables were associated with a reduced chance of being a promoter. On the other hand, elderly patients (age > 65) and non-English speaking patients had positive associations with being a promoter (Table 4).

DISCUSSION

In this cross-sectional study, we found that ED patients who arrive or depart during night shift are less likely to be promoters as compared to day-shift patients. Notably, the patient population that completed the satisfaction surveys and arrived during a night shift was substantially different than those patients who arrived during day shift. Considering this and other intuitive challenges of night shifts, unadjusted comparisons of physicians who work different ratios of day and night shifts on any number of metrics are likely to be compromised. However, in our study even after adjusting for several differences between day and night shifts, we still found an association between night-shift arrival and lower patient satisfaction scores.

Prior data on this subject has been mixed. One prior study evaluated the relationship between night shifts and ED patient satisfaction scores using data from 2009–2013 from a single ED and did not show a significant association.
¹⁴
On the other hand, a prior large study using data from 42 facilities from 2012–2015 found that physicians who worked fewer night shifts had higher patient satisfaction scores.
¹⁶
Both of these studies attempted to assess a large number of physician, facility, and operational factors that might affect patient satisfaction scores. A relative strength of our study was that it was a more targeted and granular assessment specifically of night- vs day-shift patient satisfaction scores.

Our study was not designed to specifically assess the associations of other variables with patient satisfaction scores, but we will briefly review the secondary findings. In this regard, our results were largely consistent with previous work, including our findings that elderly patients and non-English speaking patients are more likely to provide high satisfaction scores.
<sup>14

,

22</sup>
Prior data has been mixed with regard to the association of LOS and patient satisfaction.
<sup>11

–

13

,

16

,

24

,

25</sup>
We failed to find an association between ED LOS and patient satisfaction (Table 4), which is consistent with previous work that has reported that perceived LOS is more important than actual LOS.
<sup>24

,

25</sup>

Overall, although empathy and communication are important contributors to patient satisfaction
<sup>15

,

22

,

24</sup>
that an emergency clinician can mostly control, there are many factors that they cannot. Our data and previous demonstrate that night shift work, the patient population,
¹⁴
and the facility
<sup>16

,

23</sup>
all influence patient satisfaction scores. Considering also that the response rate for ED satisfaction surveys is so low (<10% in our system and similar in many others
²⁶
) and that only discharged patients are sent surveys, we recommend against the use of patient satisfaction scores to determine payment for emergency clinicians.

LIMITATIONS

This study had several limitations. First, our data comes from a single hospital system that has fairly high patient satisfaction scores; so, our results may not be applicable to other EDs. Additionally, given the retrospective and observational nature of the study, there could have been some unmeasured confounders that could explain the differences in patient satisfaction between the day and night shifts. Namely, while physicians usually work both day and night shifts, nursing staff and support staff more typically work only days or only nights. Therefore, differences in staffing might explain the differences in satisfaction scores. Moreover, prior studies have demonstrated that communication is an important factor in ED patient satisfaction scores,
<sup>22

,

27</sup>
but given the design of this study, it was not possible to assess the quality of communication.

Next, our data did not have the granularity to adjust for patient volume for each shift, which could have impacted patient satisfaction. However, given that the median ED LOS was similar in the day- and night-shift groups, we doubt that differences in patient volume would explain the lower satisfaction scores by night-shift patients. Lastly, the low response rate to patient satisfaction surveys is a limitation in that survey responses are likely substantially influenced by selection bias, but we do not consider this a limitation specific to our study because our goal was to compare the real-world patient satisfaction scores from day- vs night-shift patients (with current survey techniques). Our results thus provide a comparison of the data that is actually being used to assess clinicians’ performance on patient satisfaction.

CONCLUSION

In this cross-sectional study, night-shift arrival to the ED was associated with a statistically significant lower chance that the patient would be a promoter on satisfaction surveys. Given this finding and previous data suggesting that other issues beyond the physician’s control heavily influence satisfaction scores, facility factors, patient characteristics, and operational factors (including the time of the ED visit) should be considered when assessing patient satisfaction scores.

Footnotes

Section Editor: Laura Walker, MD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Tony Zitek, MD, Mount Sinai Medical Center, Department of Emergency Medicine, 4300 Alton Rd., Miami Beach, FL 33140. Email: zitek10@gmail.com
11 / 2024; 25:929 – 937

Submission history: Revision received March 11, 2024; Submitted August 29, 2024; Accepted August 30, 2024

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

25.

26.

27.

Share on Facebook

Tweet

Share via email