Author	Affiliation
Michelle Lin, MD	University of California, San Francisco, Department of Emergency Medicine, San Francisco, California KidsCareEverywhere – Vietnam, Berkeley, California
Trevor N. Brooks, BS	University of California, San Francisco, Department of Emergency Medicine, San Francisco, California
Alex C. Miller, BA	KidsCareEverywhere – Vietnam, Berkeley, California
Jamie L. Sharp, BA	KidsCareEverywhere – Vietnam, Berkeley, California
Le Thanh Hai, MD, PhD	National Hospital of Pediatrics, Hanoi, Vietnam
Tu Nguyen, MD	National Hospital of Pediatrics, Hanoi, Vietnam
Daniel R. Kievlan, MD	University of California, San Francisco, Department of Emergency Medicine, San Francisco, California
Robert M. Rodriguez, MD	University of California, San Francisco, Department of Emergency Medicine, San Francisco, California
Ronald A. Dieckmann, MD, MPH	University of California, San Francisco, Department of Emergency Medicine, San Francisco, California KidsCareEverywhere – Vietnam, Berkeley, California University of California, San Francisco, Department of Pediatrics, San Francisco, California

Introduction
Methods
Results
Discussion
Limitations
Conclusion

Abstract

INTRODUCTION

The United Nations’ Millennium Development Goals campaign is focused on pediatric health care and mortality rates as one of its 8 major goals.¹ This is directly aligned with other global health agencies, such as the World Health Organization, United Nations, and Health Information for All by 2015 (HIFA2015), which have identified the lack of access to current medical information as a significant barrier to effective patient care in the developing world.²–⁴

Health Information Technology (HIT) is a potentially transformational solution towards universal access for healthcare providers in the developing world. It offers under-resourced public health systems and providers with highly scalable, efficient approaches to medical education and training, as well as instant access to evidence-based literature for patient care decision making. More specifically, clinical decision support (CDS) software, a form of HIT, provides information and/or knowledge-based systems, which guide medical decision-making. This might include dosing calculators, management algorithms, computerized order sets, and other tools that help the clinician care for patients in real-time.⁵,⁶ A commonly known CDS software is UpToDate. CDS systems have several unique characteristics that distinguish this type of knowledge system from traditional, hard-copy medical references: immediacy of access to various digital platforms; portability to the point-of-care of the patient; availability of a constantly updated knowledge database; an interactive user interface; and a search engine for rapid navigation to desired information. Some forms of CDS software also offer multimedia tools. PEMSoft (Pediatric Emergency Medicine Software; Brisbane, Australia) is such an English-based CDS software that is designed for the acute and emergent care of children.

In response to urgent requests by the global health community for improved pediatric care in the developing world, KidsCareEverywhere, a nonprofit public charity and non-governmental organization, began a pilot program to train physicians on the use of PEMSoft in Hanoi, Vietnam in 2007. This aligned with the Vietnam Ministry of Health’s vision for improved pediatric emergency care in Vietnam,⁷ which was emphasized as a national priority at the 2009 Vietnam Pediatric Emergency Medicine Conference. After a highly encouraging 3-year trial, leaders from KidsCareEverywhere and from the Vietnamese National Hospital of Pediatrics in Hanoi sought to evaluate the impact of PEMSoft in improving Vietnamese physician knowledge acquisition on a broader scale throughout Vietnam. Although access to CDS systems has been shown in developed countries to improve knowledge acquisition, processes of care, and patient management decisions,⁸–¹⁴ this has not been well studied in more resource-poor countries such as Vietnam.¹⁵,¹⁶

We hypothesized that physicians in both urban and rural Vietnam hospitals, after only an 80-minute CDS training session, would improve their scores on a written clinical exam testing common pediatric emergencies. Additionally, we sought to determine whether subgroups showed greater improvement than others to help plan for future software training sessions in other developing countries.

METHODS

Study design

A multicenter, prospective, pretest-posttest study was conducted in 11 geographically diverse Vietnamese hospitals, selected by Hanoi hospital leaders. These included 6 urban (Blood Transfusion Center, Da Nang Central Hospital, Hue Central Hospital, National Hospital of Pediatrics, Ho Chi Minh City Children’s Hospital #1 and #2) and 5 rural (Dong Nai Pediatric Hospital, Hoa Binh Provincial Hospital, Nghe An Provincial Hospital, Thai Binh Pediatric Hospital, Tien Giang Provincial Hospital) hospitals. We evaluated the impact of a donated CDS software (PEMSoft) on physician performance on a written clinical examination from November 2010 to April 2011. This study was approved by the National Hospital of Pediatrics.

Selection of participants

The study participants were a convenience sample of volunteer Vietnamese physicians. Hospital leaders from each of the 11 study sites invited volunteer participants from their hospital and local affiliated sites, who had not used PEMSoft before, to attend a PEMSoft software training session. They were told to bring their own laptop and any medical references that they use currently for patient care. After an “open book” pretest using native medical references, each participant installed a free subscription of PEMSoft onto their personal laptop. Participants were excluded from the study if they were medical students or did not submit both their pretest and posttest exams.

Interventions

PEMSoft is a desktop-based, digital reference tool containing approximately 800 different pediatric conditions in its database. A highlighted piece of the software during the training session included the interactive resuscitation module for critical care conditions. For example in this module, the user can type in either the patient’s length or age, and the software provides an estimated lean body weight. Subsequently weight-specific drug doses and equipment sizes along with key pearls and pitfalls for various resuscitation algorithms are displayed to guide patient management plans and minimize calculation errors.

Two English-speaking, non-medically trained KidsCareEverywhere interns (TB, AM) dually conducted the 11 training sessions with each lasting 3 hours ^{(Table 1)}. The actual study intervention lasted only 80 minutes, which consisted of a 20-minute software training video (in Vietnamese) and a 60-minute scripted discussion in the medical management of simulated cases (in English and Vietnamese). A Vietnamese translator was used at all of the sessions.

Methods and measurements

Two 15-question, multiple-choice exams (Tests A and B) were initially created, revised, and translated into Vietnamese. Both exams were mapped similarly to include questions about diagnosis, treatment, equipment sizing, and medication doses for common pediatric advanced life support scenarios, such as sepsis, seizures, and trauma. To access for face validity, these exams were used at pilot training events held at 5 hospitals (Ben Tre Hospital, Hue School of Medicine and Pharmacy, Ho Chi Min City Children’s Hospital #1, National Hospital of Pediatrics, and Thai Binh Hospital) in October 2011. The final version of the tests used for this study incorporated participants’ test performance and feedback.

A sample question is as follows:

An 8-year-old boy is brought to the hospital after a bee sting to the face. The child is poorly responsive. He has pale skin with hives, rapid breathing, and audible wheezing. The HR is 180/min, BP cannot be obtained, RR 75/min, T 37, SaO2 80%. After opening the airway and giving 100% oxygen, you cannot start an IV. What is the next step?

Albuterol by nebulizer

Dopamine 30 microgram/kg IM

Epinephrine 0.26 mg IM

Intubation

Normal saline orally

At the beginning of the training session, each participant was assigned a unique, sequential identification number and given a brief survey about demographic and self-reported proficiencies in English and computers. This identification number was used, instead of the participant name, to maintain anonymity, when distributing the pretests and posttests. Participants with an odd-numbered identifier received Test A, while those with an even-numbered identifier received Test B as a pretest. Based on the alternating test assignments, no participant was seated next to another participant with the same test to minimize sharing information.

The pretests were “open-book” such that the participants were allowed to use any written or electronic reference materials to answer the clinical questions. After the 80-minute educational study intervention using a crossed design, participants were then given the other exam as a posttest at the conclusion of the training session. All of the participants navigated PEMSoft on their personal laptop to answer questions in the posttest.

The same two KidsCareEverywhere interns distributed and collected the exams during each 3-hour training event. Training sessions were standardized and scripted for all testing sites to minimize instructor variability. The interns subsequently entered each participant’s demographic information, survey data, and exam scores into a private Google Docs database document, which was de-identified and converted into a spreadsheet in Microsoft Excel (Version 14.2.4 for Macintosh, Redmond, WA, USA). We performed analysis natively in Excel and via StatPlus:Mac (AnalystSoft, Inc., Version 2009, Alexandria, VA, USA).

Outcomes

The primary outcome measure was defined a priori as the mean percentage difference in individual physician scores between the pretest and posttest. Secondary outcome measures included the mean percentage change in scores in relation to physician comfort level with written English, spoken English, and computer use.

Analysis

For the primary outcome, we calculated the mean percentage difference and 95% confidence interval in individual physician scores between pretest and posttest using a paired, two-tailed t-test, with an alpha of 0.05 and a power of 80%. We calculated subgroup analyses using the same method to determine if the following predictor variables impacted the participants’ change in test scores: participant practice setting, number of years in practice, and 5-point Likert scale comfort levels with written English and proficiency with computers. We considered subjects, who scored 11 or higher (out of a maximum of 15) on the pretest, to meet a satisfactory pretest knowledge threshold and excluded them from posttest improvement analysis.

RESULTS

A total of 234 participants from 11 hospitals were initially enrolled into the study. Three participants were excluded because they were medical students, and 23 were excluded because they did not submit either a pretest or posttest. Five participants met the predetermined pretest knowledge threshold and were also excluded from subsequent analysis, yielding 203 subjects for analysis. Of these, Tests A and B were used as the pretest for 102 and 101 subjects, respectively. Furthermore, 41% of the participants were male and 66% practiced in an urban location. The median age was 33 years (interquartile range 27–42 years) and the median number of years of clinical experience was 5.5 years (interquartile range 2–14 years). The participants’ self-reported comfort level with written English, verbal English, and computer proficiency on a 5-point Likert scale were 3.0 (95% CI: 2.8–3.1), 2.6 (95% CI: 2.5–2.7), and 3.5 (95% CI: 3.4–3.7), respectively, where a score of 1 represented “very uncomfortable” and 5 represented “very comfortable.”

Figure 1 shows a histogram of absolute change in scores from pretest to posttest. The mean pretest and posttest percentage scores were 37% (95% CI: 35–38%) and 70% (95% CI: 68–72%), respectively, with p<0.0001. The mean percentage change in scores from pretest to posttest was 33% (95% CI: 30–36%), which represents an 89% performance improvement over baseline.

Subgroup analyses of the mean score percentage changes based on physician practice setting, years of clinical experience, and comfort levels with written English and computer proficiency showed that all subgroups equivalently improved their test scores (data not shown). We noted a trend towards higher pretest and posttest scores for those physicians in urban practices compared to rural practices, although both groups had similar improvements in scores. Also, we noted a trend towards a higher pretest score for physicians who reported that they were “very comfortable” with written English, but their posttest scores were similar to the other physicians, resulting in a lower mean score improvement.

DISCUSSION

To our knowledge, this is the first study demonstrating improved performance on clinical test scores by Vietnamese physicians using an English-based CDS program. The subjects scored significantly higher on their posttest than their pretest across all 11 hospital study sites, despite knowing English only as a second language. Subgroup analyses demonstrated that participants universally performed better on the posttest using the CDS program, regardless of practice setting, years of clinical experience, and self-reported comfort levels with written English and computer proficiency. These encouraging results suggest that CDS technologies, such as PEMSoft, may be broadly impactful in other resource-poor countries, independent of these variables.

We did note, however, that both the pretest and posttest scores for rural physicians were slightly lower than those for urban physicians. We hypothesize that this phenomenon was related to a lesser degree of familiarity with medical English comprehension, based on the observations of the study interns. To reduce such obstacles, key parts of the software could be translated into Vietnamese. Our results may actually significantly underestimate the potential impact of CDS technologies, because the content is written in English.

There were several strengths in our study. First, the study was prospective in design and therefore limited the biases inherent in retrospective studies. Second, the analysis has significant statistical power, because it employed a multicenter approach, yielding a large sample size of 203 physician participants. Third, the test questions were revised and improved for clarity following pilot training sessions at 5 hospitals in the month preceding the study period. Fourth, the pretest-posttest design controlled for bias and possible differences in difficulty between the two tests. Fifth, instructor bias was minimized because each of the software training sessions was standardized, using training videos and scripted lectures, and given by the same paired team of study interns. Last, all written instructional materials – which included the registration, pretest, posttest, and training documents – were presented in both English and Vietnamese. Providing instructions in Vietnamese optimized participant comprehension on how to access and use the English-only software content.

Our encouraging study results show that CDS technologies may be a feasible solution for improved access to current medical knowledge in Vietnam, specifically for acute and emergent pediatric conditions. Future research efforts should focus on validation of the test instrument, assessment of patient outcomes using CDS technologies, and more objective assessment of the impact of English proficiency. If English language proficiency is found to be a major obstacle in knowledge acquisition, priorities might focus on translating PEMSoft into the country’s native language.

The ultimate goal, a daunting challenge, will be to assess the impact of CDS on actual patient outcomes in Vietnam. Kirkpatrick’s model for learner evaluation suggests a roadmap (Figure 2). The first step in this model requires studying the learners’ reactions to the current training tool. This was initially evaluated in 2010, based on a survey conducted at Danang Hospital, Vietnam, after a pilot software training session by KidsCareEverywhere. Learners universally scored the CDS program highly on all features – including software speed, ease of navigation, medical accuracy, image resolution, value, and quality. The second step includes objective assessment of the learners’ ability to improve their medical knowledge, which was accomplished in this study. The third level in the Kirkpatrick pyramid scheme involves evaluation of physician behavior in the use of CDS at the point-of-care. This evaluation could involve assessing whether the learned knowledge translates into measurable behaviors. Indirectly, one could measure frequency of provider usage of the CDS program in the clinical area to assess acceptability, feasibility and sustainability. During the study, the interns noted that desktop computers were available in hospitals, but they were either used for clerical work or located away from the clinical area, such as in libraries. Thus to enhance CDS use, the software might be converted into a web-based or native mobile application for use on smartphone devices, which are widely used by physicians in Vietnam (Dr. Tam Bui, personal communication, August 6, 2012). The fourth Kirkpatrick level would evaluate the impact of CDS on objective patient care indicators.¹⁷,¹⁸

Although multiple studies have already evaluated the advantages of CDS systems in developed countries on each of the first 3 Kirkpatrick tiers, studies assessing the fourth tier of patient outcomes show that actual patient outcomes may or may not be improved by CDS technologies. ¹³,¹⁴ One theory might be that healthcare facilities in developed countries have redundancies and resources in place to maintain a high quality standard of patient care, independent of CDS systems. In resource-poor countries, however, fewer safety nets exist and standards of patient care are often lower. Poor patient outcomes may actually be linked to access to evidence-based health information and practice guidelines. These health systems may benefit significantly from a CDS system. Although early studies using CDS tools in developing countries have demonstrated variable results,¹⁶,¹⁹ our results suggest that PEMSoft might play an impactful role not only in Vietnam’s goal to improve pediatric emergency care but also in HIFA2015’s mission for improved universal health information access in developing countries.

LIMITATIONS

There were several limitations in our study. First, we excluded 26 of the initial 234 study participants, because they were either medical students or did not complete both the pretest and posttest exams. It is possible that these participants may have eliminated themselves because of anticipated poor exam scores; these exclusions, if added into the study pool, may have lowered the mean percentage improvement in the cohort. Declaration of anonymity and nondisclosure were included in the study introductions to minimize this possible confounder. Second, although the research project was conceived and designed by the authors (ML, RD), who are PEMSoft contributors, it is possible that their involvement may have introduced bias. For this reason, only the study interns managed the entire training and data collection processes in Vietnam. Third, because participant enrollment in the CDS training sessions was based on convenience sampling, our findings may not be externally valid for the entire Vietnamese physician population. The study participants may have self-selected themselves because of greater familiarity with computer software and English. And lastly, the current study does not distinguish among different CDS tools. PEMSoft was selected for this project, but the external validity of these results for other products is not known.

CONCLUSION

We demonstrated that Vietnamese physicians could successfully navigate and use a computerized CDS tool written in English, as measured by improved performance on a written clinical exam testing knowledge on pediatric emergencies. These preliminary results suggest that CDS technology can improve pediatric emergency care in Vietnam by bringing current medical literature and reference guides to the bedside. CDS systems may offer a highly scalable, sustainable, and potentially transformative tool in pediatric emergency care in resource-poor environments.

Footnotes

Address for Correspondence: Michelle Lin, MD. San Francisco General Hospital, Department of Emergency Medicine, 1001 Potrero Avenue, Suite 1E21, San Francisco, CA 94110. Email: Michelle.Lin@emergency.ucsf.edu.

Submission history: Revision received November 14, 2012; Submitted March 21, 2013; Accepted April 2, 2013

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. The authors disclosed none.

REFERENCES

1 Keating EM, Chock M, Fischer PR Big hopes for the children of the world: a review of the Millennium Development Goals. Ann Trop Paediatr. 2011; 31:287-295

2 Health Information For All 2015 (HIFA2015) website. Available at:http://www.hifa2015.org/about/glossary. Accessed August 5, 2012.

3 Smith R, Koehlmoos TP Provision of health information for all. BMJ. 2011; 342:d4151

4 Godlee F, Pakenham-Walsh N, Ncayiyana D Can we achieve health information for all by 2015?. Lancet. 2004; 364:295-300

5 Kim GR, Lehmann CU Council on Clinical Information Technology. Pediatric aspects of inpatient health information technology systems. Pediatrics. 2008; 122:e1287-1296

6 Lyman JA, Cohn WF, Bloomrosen M Clinical decision support: progress and opportunities. J Am Med Inform Assoc. 2010; 17:487-492

7 Young S, Hutchinson A, Nguyen VT Teaching paediatric resuscitation skills in a developing country: introduction of the Advanced Paediatric Life Support course into Vietnam. Emerg Med Australas. 2008; 20:271-275

8 Blackman D, Cifu A, Levinson W Can an electronic database help busy physicians answer clinical questions?. J Gen Intern Med. 2002; 17:220

9 Kawamoto K, Houlihan CA, Balas EA Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005; 330:765

10 Low D, Clark N, Soar J A randomised control trial to determine if use of the iResus© application on a smart phone improves the performance of an advanced life support provider in a simulated medical emergency. Anaesthesia. 2011; 66:255-262

11 Isaac T, Zheng J, Jha A Use of UpToDate and outcomes in US hospitals. J Hosp Med. 2012; 7:85-90

12 Lucas BP, Evans AT, Reilly BM The impact of evidence on physicians’ inpatient treatment decisions. J Gen Intern Med. 2004; 19:402-409

13 Sahota N, Lloyd R, Ramakrishna A CCDSS Systematic Review Team. Computerized clinical decision support systems for acute care management: a decision-maker-researcher partnership systematic review of effects on process of care and patient outcomes. Implement Sci. 2011; 6:91

14 Garg AX, Adhikari NK, McDonald H Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005; 293:1223-1238

15 Tolmie CJ, du Plessis JP The use of knowledge-based systems in medicine in developing countries: a luxury or a necessity?. Methods Inf Med. 1997; 36:154-159

16 Tomasi E, Facchini LA, Maia MF Health information technology in primary health care in developing countries: a literature review. Bull World Health Organ. 2004; 82:867-874

17 Hutchinson L Evaluating and researching the effectiveness of educational interventions. Br Med J. 1999; 318:1267-1269

18 Kirkpatrick DL Evaluation of training. Training and development handbook: A guide to human resource development. 1976;

19 Chhanabhai PN, Holt A The disparity information and communication technology for developing countries has in the delivery of healthcare information. Open Med Inform J. 2010; 4:195-201