Author	Affiliation
Leslie S. Zun, MD	Mount Sinai Hospital, Department of Emergency Medicine, Chicago, Illinois
Lavonne Downey, PhD	Roosevelt University, School of Policy Studies, Chicago, Illinois
Susan Brown, MD	Mount Sinai Hospital, Department of Emergency Medicine, Chicago, Illinois

INTRODUCTION

Procuring information on a variety of healthcare concerns utilizing online resources has become increasingly commonplace in the United States. According to the most recent census data, 70 million US households report having at least 1 computer with internet access.¹ In a 2005 survey conducted by the National Institutes of Health (NIH), 60% of respondents reported accessing health information for their own edification or that of a family member utilizing the internet.² Predictably, popular health Web sites, such as NIH.gov and WebMD, receive between 6 and 12 million visitors per month.³ The availability of such information in the privacy of one’s own home ostensibly provides patients a means to become better informed and to accurately guide their own medical care.

Though the internet represents a virtually inexhaustible resource of healthcare information readily accessed by millions of increasingly technologically- and medically-savvy consumers, prior studies have determined the accessibility and accuracy of this information to be lacking. One such study conducted in 2002 found that attempting to return relevant healthcare information by using search engines with simple search terms is not efficient and that information returned is often incomplete or inconsistent. Further, the majority of sites reviewed were written at a 12th grade reading level, rendering medical information and recommendations inaccessible to less-educated consumers.⁴Such negative evaluations of medical information available on the World Wide Web were commonplace; a systematic review conducted in 2002 synthesizing data from 79 studies demonstrated that the quality of healthcare information on the internet was a problem. Fully 55 of the 79 studies compiled returned negative results, whereas only 7 provided positive assessments of available online health information.⁵ Unfortunately, each of these studies used different methods to evaluate the quality of online health information.

In an attempt to improve and standardize healthcare information on the Web, bodies such as the Health on the Net (HON) Foundation developed accreditation guidelines and provide a seal of approval for those sites that comply. HON is a Swiss-based organization established in 1996, which “promotes and guides the deployment of useful and reliable online health information, and its appropriate and efficient use.”⁶ HONcode accreditation can be obtained free of charge for those publishers that voluntarily submit their sites for review and abide by the HON principles. HON certification is based on agreeing to comply with 16 items stated in their Web page as well as on the principles of authoritative, complementarity, privacy, attribution, justifiability, transparency, financial disclosure, and advertising policy. Compliance is monitored with periodic review and through a user complaints department accessed on the HON Web site.⁶ Despite the laudable goal of the HON organization, evaluation of accuracy for those Web sites bearing the HONcode seal has been mixed.^7–12 The other credentialing organization, Utilization Review Accreditation Commission (URAC), considers itself an impartial organization that reviews a company’s operations to ensure that the company is conducting business in a manner consistent with national standards of healthcare.¹³The URAC standards were developed by a committee of experts that represent the diverse healthcare community. It is stated that these standards can be found on their Web page after contacting their marketing department.

Reliable and accurate Web-based health information can be extremely valuable and desirable when applied to emergency medical diagnoses. If a patient is able to correctly identify risk factors and symptoms of potential medical emergencies as well as the appropriate time to seek out medical attention, intervention can be streamlined and care can be improved. Published in a 2004 study by Zun et al,¹⁴ “Accuracy of emergency medical information on the Web” sought to evaluate the completeness and accuracy of emergency medical information available on the internet by comparing information available on popular health Web sites concerning 4 common medical emergencies to standard information developed from medical associations. It was determined that none of the sites evaluated were complete. Furthermore, multiple sites contained questionable and even dangerous content or recommendations.¹⁴ The following were a few examples of this concern: HealthWorld Online featured an article about bathing in cool water and taking health supplements immediately after the onset of a stroke. WebMD stated, “If the child’s temperature is over 103.5 degrees Fahrenheit 1 to 2 hours after giving medication for fever, place the child in a tub of lukewarm water up to the navel.”¹⁴ Both Yahoo!Health and MedlinePlus have articles stating that parents should bring their child to the emergency department if the fever reaches 103°F.

By nature, Web-based content is mutable and constantly transformed since publishers have the ability to edit and expand the web content in real time. Thus, the content of any given Web site may be altered at any time. With this study, we seek to build upon our 2004 findings by determining whether the completeness and accuracy of emergency medical information available online has improved over time. Further, we will evaluate the correlation between Web site accuracy and common credentialing criteria as well as the correlation between accuracy and Web site popularity as determined by traffic.

METHODS

As with the prior study, the top 15 healthcare Web pages, as determined by internet traffic (Appendix 1; online only), were reviewed between February 4, 2008, and February 29, 2008.¹⁵Those sites included in this study contained medical information for a variety of common disease conditions intended for use by the general public. Web sites of medical associations and specialty societies were excluded if they catered primarily to healthcare professionals or researchers. We compared the top sites in 2002 with those sites in 2008. Sites available in both time periods were separately evaluated for improvement in content over time.

By using the same development process as that in the 2004 study by Zun et al,¹⁴ standard checklists were re-created to evaluate medical content on each of the Web sites for 4 common emergency department diagnoses: myocardial infarct (MI), stroke, influenza, and febrile child. Standard checklists were created from the latest information provided by American Stroke Association, American Heart Association, NIH, and American College of Emergency Physicians (with sample found in Appendix 2; online only).^16–19 These checklists encompassed symptoms and risk factors for each of these topics as well as recommendations for obtaining medical care. Checklist items ranged 44 items for MI to 21 items for influenza (see Appendices 3–8; online only). Any incorrect or dangerous information or recommendations were listed separately on the basis of generally accepted medical practice and were reviewed by the study team. The study team consisted of a seasoned emergency physician, a PhD in health policy, and a medical student. Credentialing checklist items were composed of 5 criteria: review by HON or URAC, author and reviewer citation, author’s degree, links to specialty societies, and listing of contact information for author and reviewer. The Flesch-Kincaid reading level test was used to determine the reading level of the Web pages.²⁰

Each Web site was evaluated for descriptive information, completeness, and accuracy. Descriptive information, including reading level, search method, presence and type of advertising, availability in languages other than English, and date of last update, was gathered for each of the Web sites. No limit was placed on the number of articles or links viewed within each Web site; however, only information available within the site itself was utilized. Links to other health information Web sites were not included in the search. Completeness was defined as containing the greatest number of checklist items, whereas accuracy was based on the absence of incorrect or dangerous information or recommendations. Data was sorted for total medical checklist items, certification and credentialing, and medical items by topic. Descriptives, frequencies, and correlation between credentialing and Web site accuracy and popularity were calculated by using SPSS (version 14, Chicago, Illinois). This study was approved by the institutional review board as exempt.

RESULTS

Three of the 15 top sites were excluded. Drugs.com did not contain general medical information and catered specifically to questions regarding prescription drugs. KidsHealth.com covered only those topics specific to children. Finally, RealAge.com provided information on dieting tips and general preventative medicine and did not address specific health conditions. The 4 sites reviewed in the 2004 study that remained among the top 15 most popular in 2008 were as follows: WebMD (http://www.webmd.com), MayoClinic.com (http://www.mayoclinic.com), Yahoo!Health (http://Health.yahoo.com), and Intelihealth (http://www.intelihealth.com). Four of the prior sites were no longer found in 2008: Americas Doctor (http://www.americasdoctor.com), allHealth.com (http://www.allhealth.com), DrKoop.com (http://www.drkoop.com), and thriveOnline (http://www.thriveonline.oxygen.com).

Sites were evaluated for completeness of medical information by utilizing a total of 123 gold-standard checklist items that encompassed 4 topics: MI, stroke, influenza, and febrile child (Table 1). Completeness of sites ranged from 46% to 80% of total checklist items found. The mean percentage of items found was 55, and the median value was 73%. Two sites, MSN Health and Yahoo!Health, contained the greatest amount of medical information; 98 of 123 checklist items were found for each site. Healthology.com was the least complete Web site, containing 57 of 123 items. All Web sites but one, Healthology.com, contained greater than 50% of aggregated checklist items, and the majority (ie, 7 of 12) contained greater than 70%.

Analysis of the Web site medical content for each of the 4 emergency medicine diagnoses demonstrated that sites were most complete regarding stroke, with a median of 89.5% of items found. Four Web sites—WebMD, MSN Health, AOL Health, and Quality Health—contained 31 of the 32 gold-standard checklist items for stroke (Table 2). All Web sites contained greater that 50% of stroke checklist items, and completeness ranged from a low of 69% (NIH) to a high of 97% (WebMD, MSN, AOL, and Quality Health).

MI was the next most complete medical topic, with a median of 70% of gold-standard items found (Table 3). Completeness of information regarding MI ranged from a low of 18 (41%) of 44 checklist items for Healthology.com to a high of 36 (82%) of 44 items for Yahoo!Health (Table 3). Eleven of the 12 Web sites contained greater than 50% of items, but the majority (7 of 12) had at least 70% of the gold-standard items.

Medical information for the topic of a febrile child ranged from a dismal 3 (12%) of 26 gold-standard items found on Healthology.com to 23 (88%) of 26 items found on MSN Health (Table 4). The median percentage of items found was 58%, and 8 of 12 sites contained greater than 50% of gold-standard items for the topic.

 

Reviewed Web sites were still less complete on the topic of influenza, with a median of only 57% of checklist items found. That said, all but 1 of the Web sites contained greater than 50% of the gold-standard items. The number of items found ranged from 9 (43%) of 21 for RxList.com to 15 (71%) of 21 items for Yahoo!Health and Intelihealth (Table 5).

Web sites were evaluated for 5 credentialing criteria (Table 6). WebMD contained the greatest number, with 5 of 5 credentialing questions answered in the affirmative; Quality Health had the least, with 0 of 5. The median percentage of items found was 60% (ie, 3 of 5). No significant correlation was found between credentialing and completeness of site (correlation coefficient = −0.385) or credentialing and site popularity (correlation coefficient = 0.184).

Data from 4 sites—WebMD, MayoClinic, Intelihealth, and Yahoo!Health—were collected in the prior study by using similar gold-standard checklists. This information was compared with current results to determine if any of these 4 sites have made improvements to content in the past 6 years. One obvious improvement is that none of the 12 sites evaluated for this study contain any examples of inaccurate information or dangerous recommendations. In 2002, 6 such instances were found. As compared with the prior study, completeness of medical information as measured by the mean percentage of medical checklist items found within these 4 sites has improved significantly (t test P = 0.01). In 2002, only 45.5% of the standard items could be found on Intelihealth, whereas 63% were found in 2008 (Table 7). Yahoo!Health made the most significant strides, improving from 41.3% of checklist items in 2002 to 80% in 2008.

The Flesch-Kincaid reading level of Web sites reviewed ranged from 7th grade (Quality Health) to 12th grade (AOL Health, Intelihealth, Rx List.com, and Healthology.com); the median level was 8.8 (Table 8). The majority of Web sites utilized either alphabetical menus or search engines. Search engines were consistently less likely to return relevant information than alphabetical menus; however, in all cases, a relevant article was found for each topic within 1 page of results.

Nine of the 12 Web sites contained advertising of various forms, typically for multiple products or services. Three sites, NIH.gov, MayoClinic.com, and Intelihealth, did not contain ads. Of those with advertising, 8 advertised pharmaceuticals, 7 advertised vitamins, 5 advertised food products, 3 advertised books, 2 advertised television programs, and 1 site, MedicineNet.com advertised acupuncture.

Two of the 11 Web sites reviewed—MSN Health and NIH.gov—were also available in Spanish language versions. All sites were updated between 2002 and 2008. One site, QualityHealth, did not specify when information was last updated.

DISCUSSION

This study indicates a distinct improvement in online emergency medical information available to consumers on the Web between 2002 and 2008; 11 of 12 Web sites evaluated contained greater than 50% of standard items, and 7 contained greater than 70%. In 2002, only 2 of 12 of the Web sites reviewed could boast greater than 50% of aggregated medical information. Similarly, none of the sites reviewed in 2008 contained inaccurate or dangerous information, including those 4 Web sites evaluated for individual improvement over time. This indicates improved medical content on a Web site–to–Web site basis as well as improvement of online information as a whole. In addition, Yahoo!Health, MayoClinic.com, Intelihealth, and WebMD all demonstrated improvement in medical content between 2002 and 2008.

That said, the process of retrieving emergency medical information on the Web is far from perfect. Though much of the standard information was available within each of the Web sites reviewed, accessing it was often less than straightforward. For many sites, utilizing search engines to locate information led to sifting through multiple irrelevant articles. Furthermore, it was often the case that information was dispersed amongst several articles and required the use of multiple internal links. NIH.gov and MayoClinic.com were in the minority of sites that attempted to present information on a given topic within 1 to 2 concise articles. For a patient seeking information in an emergency, time wasted by searching could have dire consequences.

On average, the Web sites evaluated for this study were written at the reading level of a high school freshman. Though such a literacy level may still exclude a significant portion of the population from accessing online health information, it does represent a trend toward presenting complex information in a simplified manner so that it can be understood by a greater number of Americans. Unfortunately, only 2 Web sites, NIH.gov and MSN Health, were available in Spanish-language versions. Because Spanish is the second most commonly spoken language in the United States, such a deficiency may be denying access to valuable information to a large subset of the population.

This study found no correlation between credentialing checklist items and accuracy or site popularity, which indicates that the credentialing process is of limited value to both consumers and purveyors of health information on the Web. However, the omission of any patently false or dangerous information from sites evaluated may indicate that the primary impetus of review by organizations such as HON is to remove these items rather than ensure that each site contains all possible information on a topic.

Future studies expanding the breadth of Web sites and medical diagnoses would be beneficial, as would measuring the amount of time required for the average user to retrieve emergency medical information. Also, detailing what effect credentialing and review has on Web site content and whether or not consumers view credentialing to be an indicator of reliability may be useful. Finally, research into the effect of literacy and language restrictions on the utilization of online health resources as well as on the published content of Web sites would provide additional insight.

LIMITATIONS

This study was limited to 12 Web sites and 4 specific emergency medicine diagnoses and was by no means a comprehensive evaluation of online health information as a whole. A single investigator evaluated Web site content; as a result, interrater reliability could not be measured. No restrictions were placed on the amount of time spent or number of articles viewed in order to complete checklists; thus, the information obtained by the reviewer may not be representative of an actual patient seeking emergency medical information. Furthermore, medical content checklist items were merely recorded as present or absent and were not weighted for importance. This study did not use differing raters who examined the same information. This could have caused a bias, and future studies should include numerous reviewers to enable an interrater reliability indicator. There was no assessment of the accuracy of the physicians’ most popular Web pages. Also, this study did not examine the impact of the presence of advertising on each Web site. This could affect how users viewed the information.

CONCLUSION

This study indicates that the completeness and accuracy of online emergency medical information available to the general public has improved over the past 6 years. The majority of popular health Web sites evaluated contained greater than 70% of aggregated medical information on 4 common emergency department diagnoses, and 4 sites examined that had similar criteria in both 2002 and 2008 demonstrated significant advancement. It would appear that online healthcare resources have made strides toward becoming a reliable source of reference for patients seeking emergency medical information.

Footnotes

Supervising Section Editor: Shahram Lotfipour, MD, MPH
Submission history: Submitted June 3, 2009; Revision received October 19, 2009; Accepted October 21, 2010
Reprints available through open access at http://escholarship.org/uc/uciem_westjem
DOI: 10.5811/westjem.2010.10.1607

Address for Correspondence: Leslie S. Zun, MD
Mount Sinai Hospital, Department of Emergency Medicine, 1501 S California, Rm F 1310, Chicago, IL 60608
E-mail: leslie.zun@sinai.org

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources, and financial or management relationships that could be perceived as potential sources of bias. The authors disclosed none.

REFERENCES

1. US Census Bureau. Computer and internet use in the United States. US Census Bureau Web site.2008. Available at: http://www.census.gov/population/www/socdemo/computer.html. Accessed February 19.

2. Rutten LF, Moser RP, Beckjord EB, et al. Cancer communication: health Information national trends survey. Washington, DC: National Cancer Institute; 2007.

3. EBusiness News and Information. eBizMBA Inc. 2008. 20 most popular health Web sites, November 2007. Web site. Available at: http://www.ebizmba.com/articles/health.html. Accessed February 19.

4. Berland GK, Elliott MN, Moales LS, et al. Health Information on the internet: accessibility, quality, and readability in English and Spanish. JAMA. 2001;;285:2612–2621. [PubMed]

5. Eysenbach G, Powell J, Kuss O. ER: Empirical studies assessing the quality of health information for consumers on the World Wide Web. JAMA. 2002;;287:2691–2700. [PubMed]

6. Health on the Net Foundation. Health on the Net Foundation Web site. 2011. The services offered by HON. Available at: https://www.hon.ch. Accessed July 6.

7. Fallis D, Fricke M. Indicators of accuracy of consumer health information on the internet: a study of indicators relating to information for managing fever in children in the home. J Am Med Inform Assoc. 2002;;9:402–403. [PMC free article] [PubMed]

8. Kim P, Eng TR, Deering MJ, et al. Published criteria for evaluating health related Web sites. BMJ.1999;;318:647–649. [PMC free article] [PubMed]

9. Baur C, Deering MJ. Proposed frameworks to improve the quality of health Web sites. MedGenMed.2000;;2:E35. [PubMed]

10. Croft DR, Peterson MW. An evaluation of quality and content of asthma education on the World Wide Web. Chest. 2002;;121:1301–1307. [PubMed]

11. Nater T, Boyer C. Debate about evaluation and monitoring of sites carrying the HON logo. J Med Internet Res. 2000;;2:E13. [PMC free article] [PubMed]

12. Khazaal Y, Chatton A, Cochand S, et al. Quality of Web based information on cocaine addiction.Patient Educ Couns. 2008;;72:336–341. [PubMed]

13. URAC. General questions about URAC accreditation. Utilization Review Accreditation Commission Web site. 2008. Available at: http://www.urac.org/accreditation. Accessed October 16.

14. Zun LS, Blume DN, Lester J, et al. Accuracy of emergency medical information on the Web. Am J Emerg Med. 2004;;22:94–97. [PubMed]

15. Consumer Reports WebWatch and Health Improvement Institute. 2008. Top 20 health information sites, ranked by traffic. HealthRatings Web site. Available at:http://www.healthratings.org/2007/index.htm. Accessed February 6, 2008 to February 22.

16. American Heart Association. Heart attack and stroke warning signs. American Heart Association Web site. 2008. Available at: http://www.americanheart.org/presenter.jhtml?identifier=3053. Accessed February 6.

17. American Stroke Association. Stroke warning signs and about stroke. American Stroke Association Web site. 2008. Available at: http://www.americanstrokeassociation.org. Accessed February 6.

18. National Institutes of Health. Flu (influenza). US Department of Health and Human Services Web site. 2008. Available at: http://www3.niaid.nih.gov/healthscience/healthtopics/Flu/default.htm. Accessed February 6.

19. American College of Emergency Physicians. Children and fevers: what parents should know, and when your child has an emergency. American College of Emergency Physicians Web site. 2008. Available at: http://www.acep.org. Accessed February 6.

20. Walsch TM, Volsko TA. Readability assessment of internet-based consumer health. Respir Care.2008;;53:1310–1315. [PubMed]