Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

The Internet has become an important source of health information for patients. According to the most recent Pew Internet and American Life Project national survey (2013), 81% of adults use the Internet and 72% looked up online health information in the preceding year.¹ Although many of these online health searches may be more general or related to an already-diagnosed condition or planned treatment, 35% of Americans reported looking online specifically to determine what medical condition they may have; 46% of those online diagnosers reported that the information they found online led them to think they needed medical attention.¹

Within the context of emergency medicine, previous studies asked patients if the health information they found online made them more likely seek care in the emergency department (ED). Among ED patients with Internet access, estimated rates of Internet searches prior to ED presentation varied from 15.1% to 47%.^2-4 Many companies and health systems have produced online “symptom-checking” websites to harness these searches and attempt to improve self-triage, with variable success.⁵ On a population level, healthcare website traffic measurements have been used to forecast ED visit volume.⁶ Similarly, epidemiologic trends for certain conditions such as influenza correlated well with Internet searches for related symptoms.⁷ These prior studies suggest that patient Internet use affects patient concerns, and impacts their choice to seek medical care.

What remains unexplored in the current literature is exactly what individual patients are searching prior to their ED visit. We believe it is important for the emergency physician (EP) to understand what the patient is seeking with an Internet search because an awareness of these patient concerns may inform the conversations and counseling in the ED. What types of information are patients seeking when they turn to the Internet and how does their ultimate diagnosis relate to their original search? We sought to answer these questions through a qualitative analysis.

METHODS

This analysis is part of a larger prospective survey study focused on how patients use the Internet and their primary physician for health information prior to an ED visit.⁸ This study uses qualitative methods to further analyze the responses of participants who conducted an Internet search prior to visiting the ED (data from primary study reported separately).

Participants and Procedure

Data collection occurred at an urban academic medical center (>88,000 annual patient visits) with patients enrolled from May 23, 2014, to July 21, 2014. Trained research assistants (RAs) enrolled patients on weekdays 9am–9pm and Saturday 9am–5pm, based on RA availability. All adult patients (age >17) were eligible. The larger study specifically investigated differences in access to health information between adult and geriatric patients. Therefore, there was intentional oversampling of the geriatric population to achieve a balance between geriatric and adult patients; sample-size calculations targeted a total enrollment of 720 participants based on the primary outcome of the larger study. The exclusion criteria included an inability to complete the written survey for any reason (e.g., physical impairment, clinical condition, language barrier).

Population Health Research Capsule

What do we already know about this issue?

The Internet is an important source of health information, and prior studies estimate up to half of ED patients search the Internet prior to ED presentation.

What was the research question?

We sought to describe what types of information patients are searching for in their pre-ED Internet searches.

What was the major finding of the study?

Patients searched for symptoms more often than diagnoses. Correlation between search and ED diagnosis was poor.

How does this improve population health?

Many discharged patients have symptom-based diagnoses (similar to pre-ED symptom-searches). Discussing the lack of a formal diagnosis may be warranted.

Participants provided written informed consent. RAs administered the survey on paper, and it took approximately five minutes to complete. Participants received compensation with a $5 gift card. RAs later entered data into REDCap, a secure, web-based application designed to support data capture for research. The institutional review board approved all study procedures.

Survey Measures

This qualitative analysis includes the subset of patients from the larger study who answered “yes” when asked, “Before coming to the emergency room today, did you search the Internet about your current symptoms or condition?” As a follow-up question, patients listed the search terms entered and the Internet site(s) visited to look up the information. The top six most frequently accessed health websites were listed, as well as a free-text space for “other” websites.⁹ Additionally, the survey contained questions regarding demographic information (age, sex, race/ethnicity), socioeconomic information (education, household income), access to a primary care physician, and questions related to number of devices owned capable of accessing the Internet. Following survey completion, we extracted additional patient-specific clinical data from the electronic medical record (EMR), including triage acuity, chief complaint, ED disposition (admission, discharge), and ED discharge diagnosis.

Qualitative Measures and Analysis

Three qualitative metrics are reported. First, based on a review of the literature, we developed an a priori coding schema to categorize the Internet search terms based on content.^10-12 The categories included search terms related to a symptom, a specific diagnosis, treatment options, anatomy, processes of care or physicians, or “other.” Table 2 contains a detailed definition of each category.

The second qualitative analysis phase investigated the relationship between the patients’ initial search term and their final ED diagnosis for those patients who had searched either a symptom or a diagnosis. Many discharge diagnoses in the ED are, in fact, “symptom-based” (e.g., chest pain) as opposed to a more “formal” diagnosis (e.g., myocardial infarction). Therefore, we divided final diagnoses into two large groups: symptom-based diagnoses and formal diagnoses. We considered an ED discharge diagnosis a symptom-based diagnosis if it met one of two criteria: 1) ICD-9 code range 780–799 (symptoms, signs and ill-defined conditions) (e.g., malaise, abdominal pain, fever, rash); or 2) it named an anatomical body part followed by pain (e.g., ankle pain, wrist pain). Therefore, every encounter received a designation in one of these two categories, either the more general symptom-based diagnosis group or the more specific formal diagnosis group.

After the designation of symptom-based or formal diagnosis, we assigned the relationship between the search term and diagnosis to one of four categories of search/diagnosis combinations representing different “trajectories.” If patients had more than one search term listed, the analysis used the most specific search term. (Diagnosis searches were rated as being more specific than symptom searches.) This analysis excluded patients who had only used search terms related to treatment, anatomy, ED processes, or “other.” A patient’s trajectory from pre-ED presentation Internet search to post-ED-care doctor-assigned discharge diagnosis was defined as flat trajectory—symptoms if they searched for a symptom and received a final diagnosis of a symptom. It was defined as flat trajectory—diagnosis if they searched for a diagnosis and received a final formal diagnosis. Patients’ diagnosis search term accuracy was recorded. If a patient searched for a symptom and ultimately received a formal diagnosis, they were categorized as having general to specific trajectory, whereas if they searched for a specific diagnosis by name and left the ED with a symptom-based diagnosis they were categorized as having a specific to general trajectory.

The two qualitative analyses described above examine the bookends of the visit, the initial search and the final diagnosis. An additional analysis conducted describes an intermediate step in the process—the chief complaint. Although it may be of interest to know how the patient’s search influenced the wording of his/her presenting complaint, the concordance between search term and chief complaint was not examined because of concerns that chief complaints were potentially influenced by nurse interpretation. The EMR allowed for free-text entry of the chief complaint or for selection from a drop-down menu by the triage nurse; therefore, the chief complaint may not have fully captured the patient’s concern at the time of presentation. For example, “I’m having chest pain, I’m worried it is a heart attack” may have been recorded as “chest pain.” This limitation did not allow for use of the chief complaint as a proxy measure for how the Internet search may have influenced the patient’s statement of his complaint. However, of interest from the physician perspective, we assessed the concordance between the chief complaint and the final diagnosis. Even allowing for nursing influence on the chief complaint, the chief complaint recorded in the record was the first introduction that the physician had to the patient, and therefore we assessed the concordance between the complaint and the final diagnosis. Table 4 defines the concordance scale and provides examples.

Two coders analyzed all cases independently. A kappa analysis for a 10% random subsample of cases ensured reliability prior to coding the entire sample. The coders reconciled all disagreements through discussion and selected a final code through consensus. Frequencies are reported for demographic characteristics and for each of the codes. The Fischer’s exact test assessed the association between demographic variables and Internet search terms. All analyses were conducted using STATA software version 13.1 (StataCorp, College Station, TX). We could not pre-determine optimal power or estimate an effect size for this study because the qualitative analysis was exploratory and not testing a quantifiable hypothesis.

RESULTS

Of the 723 participants, 177 (24%) who completed the larger study searched the Internet prior to ED presentation. Seven participants had incomplete data, resulting in a final sample of 170 (see Figure). The participants had a mean age of 47 years (standard deviation 18.2) and slightly more than half were female (58.6%). The vast majority owned at least one device capable of Internet access (98.2%) (Table 1).

In our sample, 32% (N=55) reported using more than one search term, resulting in a total of 243 search terms. When conducting their Internet searches, the majority of search terms focused on symptoms (54.7%) rather than a diagnosis by name (31.7%) (Table 2). Participants accessed a variety of websites to gather information; 58% of the sample reported searching on WebMD, followed by 40% using the Mayo Clinic website. Although not a formal option on the survey instrument, 37 patients (21.8%) wrote in the “other” category that they simply conducted a Google search and looked at the top hits.

Overall, 56% of the sample left the ED with a formal diagnosis and the remaining 44% received a symptom-based diagnosis upon discharge. The second qualitative analysis excluded 13 patients because they only searched for treatment, anatomy, ED processes or “other,” resulting in a sample of 157 patients. Looking specifically at the relationship between pre-ED presentation Internet search terms and final ED diagnosis, the largest grouping of patients appeared in the flat trajectory-symptoms category (34%, 95% confidence interval [CI] for proportions: [27%–41%]). These patients searched for a symptom and were discharged (or admitted) with a diagnosis of a symptom. Approximately a fifth of patients (22%, 95% CI [16%–28%]) had a general-to-specific trajectory. For those categorized as flat trajectory-diagnosis, 20 patients (13%, 95% CI [8%–18%]) had perfect accuracy in their Internet search (having searched for a diagnosis and received the same, correct final diagnosis). Although this trajectory was flat, it was accurate. In contrast, 33 patients (21%, 95% CI [15%–27%]) searched for a diagnosis and received a different formal diagnosis. Finally, 16 patients (10%, 95% CI [5%–15%]) had a specific-to-general trajectory wherein they searched for a specific diagnosis and left the ED with a symptom-based diagnosis (Table 3). Among all of the patients who searched for a diagnosis by name (n=69), 23% received a symptom-based final diagnosis, 48% received a different detailed final diagnosis, and only 29% received the diagnosis that they had searched.

In nearly two-thirds of cases, the chief complaint and final diagnosis showed near or complete concordance (Table 4). This does not, however, imply that a formal diagnosis was made in all of these cases. For example, a chief complaint of “chest pain” and a final diagnosis (symptom-based) of “Chest Pain, ICD-9: 786.5” was considered complete concordance; yet no definitive cause of the pain was identified (despite numerous causes likely being ruled out).

There was no relationship between patient age (younger adult versus geriatric), gender, or education level and the category of search term used. Patients who reported talking to their primary care provider prior to presentation did not have a different distribution of search terms than those who did not talk to (or did not identify) a primary care provider (data not shown).

DISCUSSION

We characterized ED patients’ pre-visit Internet search terms using a qualitative approach and looked at the relationship between these search terms and the patients’ final diagnoses. We found the majority of patients searched online for symptoms rather than for specific diagnoses. Previous studies using web-analytics of Internet queries similarly noted that the majority of searches focused on symptoms.¹¹ However, in contrast to studies in other settings, in this sample of ED patients very few searches focused on treatment. For example, in a sample of orthopedic patients 21% sought information about treatment, and in a sample of melanoma patients 96% sought information about treatment.^12,13 The higher ratio of treatment-related searches in the outpatient clinic and specialty context contrasts with our ED data. However, our report is unique in illuminating the frequency and nature of Internet search strategies that may serve as the genesis of the decision to seek unscheduled ED care. Additionally, understanding these symptom-based searches may help EPs address concerns that arise after Internet searches.

Previous data from the Pew Internet and American Life Project highlighted that 41% of “online diagnosers” say that a medical professional confirmed their suspicions, whereas 18% said a medical professional did not agree or offered a different opinion.¹ We did not specifically query patients about their leading diagnosis post-search; however, in our sample only 11.8% of patients originally searched a diagnosis that perfectly matched their final diagnosis. In contrast, a much higher rate of patients displayed complete (32.3%) or near (30%) concordance between their chief complaint and their discharge diagnosis. This finding may be because the chief complaint represented the patient’s post-search leading diagnosis; however, this metric was likely also influenced by nurse entry of the chief complaint, and the frequent use of symptom-based discharge diagnoses.

The correlation between Google-searched diagnosis and ED diagnosis was poor. One explanation is patients accessing misinformation on the Internet; however, patients’ limited medical knowledge and hypersensitivity to dangerous or deadly diagnoses (e.g., heart attack, stroke) may drive this poor correlation. We are currently in an era of medicine with unprecedented attention to patient satisfaction, and as such, matching patient expectations with experience is necessary for the EP to ensure a satisfied patient. If EPs ask patients about Internet searches and concerns that arose from those searches, the physicians can directly address patients’ concerns and highlight how those concerns have been ruled out. This process may help to match patient experience to their expectations and may ultimately improve patient satisfaction. Future studies measuring the impact of EPs asking about Internet searches, directly addressing patients’ concerns after searching the Internet, and the impact on patient satisfaction are warranted.

Many of the patients in our sample departed the ED with a similar level of specificity in terms of diagnosis as when they presented. EPs see value in ED encounters that do not result in a formal diagnosis. Such visits serve many functions, such as excluding life-threating causes of the symptoms, providing reassurance to patients regarding the severity of illness, and the urgency with which to seek future care. However, for patients seeking a diagnosis these visits (and the associated physicians) may be viewed negatively because the patient’s ultimate question (what is wrong with me?) was not definitively answered. Armed with the knowledge garnered from this study, EPs can better explain to patients the value of an ED visit by asking about Internet searches and addressing the concerning diagnoses patients encountered after searching for their symptoms online.

The lack of a “formal” diagnosis is a frequent occurrence in the ED; a recent study reported that at least 37% of discharged ED patients do not receive a pathologic diagnosis.¹⁴ Faced with this uncertainty, patients often experience fear and anxiety that negatively influences their mental and physical health in the post-discharge period.^15,16 We did not follow patients after their visit to learn about post-visit Internet searches. However, it seems likely that the lack of a formal diagnosis mentioned above could also be associated with post-visit Internet searches. Bell et al. evaluated the factors patients named as prompting a post-visit Internet search (not specific to ED patients) and found that patients were more likely to use the Internet post-visit when their anxiety was high and their trust in the physician was low.¹⁷

Interestingly, irrespective of their search terms and concordance, patients used a variety of websites to gather medical information. Although the survey specifically asked patients which destination sites they used to gather information, many volunteered that they used Google to start their search. A similar pattern exists in other settings as well with an estimated eight out of ten health-related Internet searches starting at a search engine such as Google, Yahoo! or Bing.¹ Interestingly, evaluation of the content of the “top hits” on Google, Yahoo! and Bing searches with respect to critical symptoms that would prompt an acute evaluation revealed that a minority of sites contained a clear set of critical symptoms or recommendations for further care.¹⁸ These metrics make it easy to criticize such websites for not clearly defining symptoms that warrant emergent evaluation. At the same time, it is difficult to imagine how an online list of symptoms could appropriately capture the nuanced combination of patient risk factors, presenting symptoms, and physical examination (as well as years of clinical experience) that allow physicians to accurately diagnose and risk-stratify patients.

LIMITATIONS

This was a small sample of patients from a single site, containing English-speaking participants with generally high income, education and ease of access to the Internet. Data collection occurred over an eight-week timeframe on a convenience sample basis during daytime hours and limited to patients who were not too severely ill to participate. Although the decision to omit individuals unable to read the survey may have introduced sampling bias, we believe this to be minimal as we were asking patients about an activity that requires basic writing and reading skills (namely, typing a search term into the Internet and reading the results of the search). These factors limit the generalizability.

We only present data on patients who answered “yes” to performing an Internet search prior to arrival. We did not investigate why patients did not perform an Internet search if they answered “no” and therefore cannot comment on whether this lack of search was related to their clinical condition, Internet access or trust in the Internet. Additionally, there are limitations inherent in the metrics. The patient search terms were based on self-report and are subject to recall bias. In some cases, a prior healthcare encounter (either an established diagnosis or another same-day encounter via phone or in-person) likely influenced the search terms and chief complaint. For example, one patient searched online for “kidney infection” and their chief complaint was “kidney infection, seen yesterday.” Nurse interpretation, as noted above, also potentially influenced the recording of the chief complaints. Although this limitation prevented us from conducting an analysis of the relationship between search terms and chief complaint, it is an accurate reflection of the working environment of the ED.

Finally, the use of ICD-9 codes is also potentially flawed. In our system the ED attending or resident physician enters the diagnosis into the EMR at either the time of ED note completion or the time of disposition. Variable amounts of information may be available depending on the timing and could result in a less specific ICD-9 (e.g., viral syndrome instead of influenza). Since the time of data collection, ICD coding has advanced to the currently used ICD-10 coding system that contains 155,000 codes and procedures compared to only 17,000 codes in the ICD-9 system.¹⁹ The expansion of the coding system may have resulted in more specificity in ED discharge diagnoses; however, this topic requires further study.

CONCLUSION

A quarter of our sample reported using the Internet prior to their ED visit and approximately half used a symptom-based approach for their search strategy. Similarly, nearly half of these patients left the ED with a symptom-based (or non-pathologic) diagnosis. When patients did search for a specific diagnosis, only 29% searched for the diagnosis they eventually received. Physicians who discharge patients with a symptom-based diagnosis may benefit from understanding that patients had similar symptom-based searches prior to coming to the ED, and more fully explain how the ED workup has ruled out specific diagnoses patients were concerned about after an Internet search and changed the treatment plan prior to discharge. Such conversations may address the fear and anxiety that other studies have reported being associated with diagnostic uncertainty at the time of discharge.

Footnotes

Section Editor: David Thompson, MD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Danielle M. McCarthy, MD, M., Northwestern University, Feinberg School of Medicine, Department of Emergency Medicine, 211 E. Ontario St, Suite 200, Chicago, IL 60611. Email: d-mccarthy2@northwestern.edu. 8 / 2017; 18:928 – 936

Submission history: Revision received March 2, 2017; Submitted April 20, 2017; Accepted May 18, 2017

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. Data collection for this project was funded by a CMS Contract (Site PI: Dresden). The sponsor had no role in the study design, data collection, analysis or interpretation of data, or the writing of the manuscript.

REFERENCES

1. Fox S, Duggan M. Health Online 2013. Internet and American Life Project. 2013. Available at: http://pewinternet.org/Reports/2013/Health-online.aspx. Accessed May 23, 2017.

2. Broadwater-Hollifield C, Richey P, Podolsky S, et al. Potential influence of Internet health resources on patients presenting to the emergency department. Ann Emerg Med. 2012;60(4S):S134.

3. Pourmand A, Sikka N. Online health information impacts patients’ decisions to seek emergency department care. West J Emerg Med. 2011;12(2):174-7.

4. Sullivan AF, Ginde AA, Weiner SG, et al. Multicenter study of internet use by emergency department patients in Boston. Ann Emerg Med. 2009;54(3):S19.

5. Semigran HL, Linder JA, Gidengil C, et al. Evaluation of symptom checkers for self diagnosis and triage: audit study. BMJ. 2015;351:h3480.

6. Ekstrom A, Kurland L, Farrokhnia N, et al. Forecasting emergency department visits using internet data. Ann Emerg Med. 2015;65(4):436-42 e431.

7. Thompson LH, Malik MT, Gumel A, et al. Emergency department and ‘Google flu trends’ data as syndromic surveillance indicators for seasonal influenza. Epidemiol Infect. 2014;142(11):2397-2405.

8. Scott GN, McCarthy DM, Aldeen AZ, et al. Use of online health information by geriatric and adult ED patients: access, understanding and trust. Acad Emerg Med. 2017.

9. Top 15 Most Popular Health Websites. 2014. Available at: http://www.ebizmba.com/articles/health-websites. Accessed May 30, 2014.

10. Eysenbach G, Kohler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ. 2002;324(7337):573-7.

11. Jadhav A, Andrews D, Fiksdal A, et al. Comparative analysis of online health queries originating from personal computers and smart devices on a consumer health information portal. J Med Internet Res. 2014;16(7):e160.

12. Shuyler KS, Knight KM. What are patients seeking when they turn to the Internet? Qualitative content analysis of questions asked by visitors to an orthopaedics Web site. J Med Internet Res. 2003;5(4):e24.

13. Hamilton SN, Scali EP, Yu I, et al. Sifting through it all: characterizing melanoma patients’ utilization of the Internet as an information source. J Cancer Educ. 2015;30(3):580-4.

14. Wen LS, Espinola JA, Kosowsky JM, et al. Do emergency department patients receive a pathological diagnosis? A nationally-representative sample. West J Emerg Med. 2015;16(1):50-4.

15. Rising KL, Hudgins A, Reigle M, et al. “I’m just a patient”: fear and uncertainty as drivers of emergency department use in patients with chronic disease. Ann Emerg Med. 2016;68(5):536-43.

16. Rising KL, Padrez KA, O’Brien M, et al. Return visits to the emergency department: the patient perspective. Ann Emerg Med. 2015;65(4):377-86 e373.

17. Bell RA, Hu X, Orrange SE, et al. Lingering questions and doubts: online information-seeking of support forum members following their medical visits. Patient Educ Couns. 2011;85(3):525-8.

18. North F, Ward WJ, Varkey P, et al. Should you search the Internet for information about your acute symptom?. Telemed J E Health. 2012;18(3):213-8.

19. HHS Proposes Adoption of ICD-10 Code Sets and Updated Electronic Transaction Standards. 2008. Available at: http://www.dhhs.gov/news/press/2008pres/08/20080815a.html. Accessed February 27, 2017.