Author	Affiliation
Cynthia A. Brandt, MD, MPH	VA Connecticut Healthcare System, West Haven, Connecticut; Yale School of Medicine, Department of Emergency Medicine, New Haven, Connecticut
T. Elizabeth Workman, PhD	The George Washington University, Biomedical Informatics Center, Washington, District of Columbia; VA Medical Center, Washington, District of Columbia
Melissa M. Farmer, PhD	Center for the Study of Healthcare Innovation, Implementation & Policy (CSHIIP), VA Greater Los Angeles Healthcare System, Los Angeles, California
Kathleen M. Akgün, MD, MS	VA Connecticut Healthcare System, West Haven, Connecticut; Yale School of Medicine, Department of Internal Medicine, New Haven, Connecticut
Erica A. Abel, PhD	VA Connecticut Healthcare System, West Haven, Connecticut; Yale School of Medicine, Department of Psychiatry, New Haven, Connecticut
Melissa Skanderson, MSW	VA Connecticut Healthcare System, West Haven, Connecticut
Bevanne Bean-Mayberry, MD, MHS	Center for the Study of Healthcare Innovation, Implementation & Policy (CSHIIP), VA Greater Los Angeles Healthcare System, Los Angeles, California; UCLA David Geffen School of Medicine, Department of Medicine, Los Angeles, California
Qing Zeng-Treitler, PhD	The George Washington University, Biomedical Informatics Center, Washington, District of Columbia; VA Medical Center, Washington, District of Columbia
Maryann Mason, PhD	Northwestern University, Department of Emergency Medicine, Chicago, Illinois
Lori A. Bastian, MD, MPH	VA Connecticut Healthcare System, West Haven, Connecticut; Yale School of Medicine, Department of Internal Medicine, New Haven, Connecticut
Joseph L. Goulet, PhD	VA Connecticut Healthcare System, West Haven, Connecticut; Yale School of Medicine, Department of Emergency Medicine, New Haven, Connecticut
Lori A. Post, PhD	Northwestern University, Department of Emergency Medicine, Chicago, Illinois; Northwestern University, Department of Geriatric Medicine, Chicago, Illinois

Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

In 2020 42% of United States (US) households reported owning a firearm.¹ Firearms in the home increase risk of violent events,^2-5 and is a significant threat to public health. Nearly half (44.9%) of all US military veterans own a firearm, with ownership reportedly higher among males (47.2%).⁶ Veterans are at disproportionate risk for suicide,⁷ accounting for 20% of suicide deaths despite constituting 13% of the US population. Firearms are involved in 67% of suicides among veterans compared with 50% of the general public.⁸

While access to firearms is associated with increased risk for injury and death, safe firearm storage is associated with decreased risk.^9,10 Public health advocates recommend strategies to restrict access to lethal means as a suicide prevention strategy.¹¹ For firearms these processes include safe storage measures such as gun safes, gun locks, storage of ammunition and guns separately, and storage of guns unloaded and locked. ¹² Members of the military tend to store firearms unsafely with 45.2% reporting they store firearms both loaded and unlocked, and an additional 33% store firearms either loaded or unlocked.^13,14

Screening veterans for firearms ownership and safe storage is needed to prevent unnecessary injuries and deaths. Healthcare providers are in a position to screen and counsel patients on safe firearm storage.⁴ Counseling on health and safety is a well-established healthcare practice; there are guidelines for screening and counseling in many areas of health including healthy eating, physical activity, mental health, and injury prevention.¹⁵ While firearm-related injuries and deaths are a public health problem, particularly in the US,¹⁶ a minority of physicians report engaging in firearm counseling.¹⁷ Despite several groups having recommended both targeted and universal screening for firearm access,^17-24 there are no current national guidelines for screening in primary care, urgent care or emergency care settings even though gun safety is associated with lower risk of injuries and death.^25,26

To understand how current practice may be adapted, more information on the frequency with which healthcare providers document firearm screening is needed. In this study we present results of Veterans Health Administration (VHA) healthcare providers’ documentation of firearm access screening in electronic health record (EHR) notes among VHA patients in outpatient primary care, urgent care, and emergency department (ED) settings.

METHODS

The study is a cross-sectional examination of the frequency of documentation of screening for veterans’ access to firearms across several healthcare settings using natural language processing (NLP), which refers to automatic computational processing of human language.²⁷ The study was approved by the Veterans Administration Connecticut Healthcare System Institutional Review Board.

 

Population Health Research Capsule

What do we already know about this issue?

United States Veterans are more likely to own a firearm and to be at risk for firearm injuries and death than civilian populations.

What was the research question?

Our study aimed to determine how frequently VA healthcare providers document firearm access screening.

What was the major finding of the study?

Documentation of firearm access for Veterans by healthcare providers was low but higher in men than women.

How does this improve population health?

Identifying barriers and facilitators to help healthcare providers increase screening for firearms and counsel safe storage could support prevention efforts.

 

The study sample included men and women veterans from a national, post-9–11 cohort^28,29 during their first year of VHA healthcare, defined by the presence of at least one primary care visit from 2012–2017. We obtained data on demographic and veteran military service characteristics from the Defense Manpower Data Center–Contingency Tracking System Deployment File, provided to the VHA from the US Department of Defense. Variables included age, gender, race/ethnicity, marital status, education, rank (e.g., officer, enlisted), military branch (e.g., Army, Marine Corps), and deployment dates. VHA visit information came from EHR data extracted from the Corporate Data Warehouse (CDW). The CDW includes information on healthcare utilization, pharmacy, laboratory, vital signs, coded diagnostic and procedural data (International Classification of Diseases, 9th and 10th revisions, Clinical Modification [ICD-9-CM and ICD-10-CM]) and Current Procedural Terminology (associated with all VHA inpatient and outpatient encounters.^30,31

We identified comorbid conditions using ICD-9 and ICD-10 coded diagnoses defined by ≥2 outpatient (on separate days) or ≥1 inpatient code for the condition. This methodology has been used for the identification of psychiatric disorders in administrative data³² and human immunodeficiency virus in Medicaid data.³³ Diagnostic code groupings were previously validated.³⁴ Major mental health diagnoses included post-traumatic stress disorders (PTSD), major depressive disorders, alcohol use disorders, and substance use disorders.

Natural Langauge Processing Tool Development and Performance for Firearm Access Identification

Firearm Taxonomy

For the information extraction process, we developed a coding manual for chart review and a taxonomy for firearms for annotation. A taxonomy was created by by searching existing vocabularies (NCBIO, UMLS, SnoMed) and the literature for published ontologies used for guns, gunlock, and firearms. A Cochrane review on gunshot wounds contained terms such as trauma* or injur* or penetrat* or wound*or perforat* or stab* or gunshot or shot, and the Medical Subject Headings (MeSH) database included the following: “Wounds, Gunshot”[Mesh]) OR “Wounds, Penetrating”[Mesh:NoExp]) OR“Multiple Trauma”[Mesh])) OR “polytrauma.” This list of terms was supplemented with over 120 candidate terms and phrases contained in a national database³⁵ on firearm homicides. We then reviewed and narrowed the phrases down to 27 (shown in Table 1) deemed relevant by VHA clinicians.

Annotation

We identified 2,584,607 notes with one or more of the phrases, and annotated 1856 text snippets randomly selected from notes that contained any of the search terms. Each snippet contains a 35-word span before and after a firearm-relevant phrase. The annotation classifications for firearm access were the following: positive (ability to determine that the veteran had current access to at least one firearm); negative (language that the veteran did not have current access to any firearms); and ambiguous (there was insufficient evidence for either a positive or negative classification from the note – an example might be that the veteran owned a firearm but it was somewhere else). Each snippet was annotated by two of the authors and disagreement adjudicated by their consensus. An inter-annotator agreement was calculated. The annotated snippets served as the reference standard in training and testing.

Features

We used n-grams as features. In clinical text, unigrams are single words, and bigrams are two words that occur in a sequence. For example, in the phrase “patient owns a shotgun” the unique unigrams are patient, owns, a, and shotgun. In the same phrase, patient_owns, owns_a, a_shotgun are unique bigrams. Alpha or numeric tokens (discrete words and numbers) were counted in the unigrams and bigrams. The features included unique unigrams with a frequency greater than 34, and unique bigrams in the annotation spans with a frequency greater than four. These threshholds are empirically chosen to filter out the less prevalent n-grams and reduce overfitting. The training features for the model (for each document) consisted of binary indications of the presence of each of the identified unigrams and bigrams, along with the offset location of the keyphrase in the snippet.

Training and Testing

We used the annotated snippets to train a random forest model with 200 estimators or trees. The random forest model maximum depth was set to 15, with maximum features automatically determined by the model and the gini split criterion. Hyperparameters were determined through gridsearch and other testing. We split the 1856 text snippets into 85% for training and 15% for testing. The model performance was measured by accuracy.

Validation

For validation, we annotated an additional 238 clinical notes on the note (instead of the snippet) level, with 175 negtive and 63 positive for firearm access. The random forest model was applied to these notes, based on the snippet identified in each document. Figure 1 below is a schematic of this process.

Statistical Analysis

We conducted all statistical analyses using SAS software version 9.4 (SAS Institute, Cary, NC). Baseline characteristics of veterans include frequency (percentages) and means (± standard deviations) or median (interquartile range), and differences by age, race, ethnicity and gender were examined using chi-squared test or Student’s t test, as appropriate. We used a multivariable logistic regression model to assess firearm mention and adjust for potential confounding based on the literature. Among those with firearm mention, a logistic model was run to assess firearm access. We examined model fit using quasi-information criterion and residual plots. Hypotheses were tested at a two-sided significance level of α = 0.05.

RESULTS

The Cohen kappa score measuring inter-annotator agreement among the review team members identifying screening documentation incidents was 80%. On the testing dataset (15%), the accuracy was 81.0%. On the final validation dataset, the random forest model achieved 81.9% accuracy, 90.9% specificity, 57.1% sensitivity, and positive predictive value of 69.2% in classifying the 238 test notes. Table 1 demonstrates the frequency of the most common firearm-related terms within the VHA text notes. These counts are non-distinct by patient but demonstrate the breadth of terms used in clinical notes by providers; many of the highly specific terms were present in notes as historical and exposure events for PTSD documentation, and/or noise exposure (out of 105,316,004 outpatient care notes).

We included data during the first year in VHA care for 762,953 veterans in the analytic sample. Table 2 demonstrates the frequency of documentation of access to firearms and other guns by clinicians within one year of entry into VHA healthcare. The mention of any firearm within a clinical note for veterans was 9.8% of men, 7.1% of women, and 6.5% in veterans over 50 years of age. Among the small number of veterans with any firearm term present, only 24% were classified as positive for access to a firearm (24.7% of men and 20.9% of women [data not otherwise shown]). Prevalence patterns by race of any mention/positive access were similar, with the highest rates among Whites (9.6% mention and 26.3% access [data not otherwise shown]). Documentation of firearms was higher in veterans with higher numbers of mental health visits, emergency and urgent care visits than primary care. Documentation of firearms did not vary regardless of the number of primary care visits (data not shown). After adjustment for demographics, utilization, and comorbidities, significant differences in documentation of access remained by age, gender, and among veterans with major depression or PTSD diagnoses.

DISCUSSION

Results demonstrate documentation of firearm access in clinical notes for less than 10% of contemporary veterans within the first year of enrollment in VHA healthcare, and that nearly one quarter of those with documentation were identified as having access to a firearm. There was a significantly lower rate of documented access for women veterans, despite data that show high rates of both men and women veterans who live in homes with firearms, and increasing rates of fiream-related suicides among women veterans.^6,36 While documentation does not always equate with conversations between providers and patients, the low frequency of documented patient-provider interactions seen in this population suggests that there is a clear opportunity to increase initiation of conversations about firearm access and safety. Barriers to implementation of firearm screening and safety counseling include provider uncertainty about the effectiveness of firearm screening, provider uncertainty about the legality of asking about firearm ownership, and provider unfamiliarity with firearms. Further, provider unfamiliarity with lethal means restriction as a firearm suicide prevention strategy may prohibit uptake of screening and counseling.^17,23,37-39 These barriers indicate a need for increased training of healthcare providers on firearm screening and safety counseling and normalizing the opportunities to discuss firearms in a population that has higher rates of firearm ownership and use.

Discussions must be acceptable to providers and to patients for it to be effective. Roszko and colleagues’ review of 53 studies of non-veteran clinician firearm attitudes and practices found that positive attitudes toward firearm discussions were higher than actual documented discussions, with low firearm discussions across all disciplines.¹⁷ This is encouraging in that it could indicate healthcare providers may be willing to undergo training in initiating and carrying out these discussions, although it remains to be seen whether attitudes differ among VA providers.

While providers may have positive attitudes toward firearm screening and counseling, recent studies show mixed support by gun owners and veterans for healthcare provider initiation of gun safety conversations.^11,40 This suggests that while providers may be willing to initiate these discussions, it is not clear that patients will welcome or participate in them if initiated. Such conversations will need to be clearly delineated as prevention oriented for gun-owning citiziens and families with specific, evidence-based practices such as the following: Homes with locked guns are less likely to have unintentional or self-inflicted injuries with firearms or deaths.⁴¹

Perhaps related to the reasons specified above, the evidence for the effectiveness of this firearm safety conversation in the clinical setting is mixed.^42,43 For this reason, appropriate, acceptable communication must be used and evaluated to maximize the impact and inform the knowledge base of these efforts in the clinical setting.^44,45 However, a recent epidemiologic review indicates that counseling combined with safety-device provision can impact safe storage in the community.⁴¹ Promising strategies include following the guiding principles of shared decision-making, with providers stating neutral risks and protective factors related to gun safety and involving gun owners in the development of messaging.^46,47 These neutral risks can be culled from the US Centers for Disease Control and Prevention data as simple facts for patients to understand and acknowledge as part of their gun ownership responsibilities.⁴⁸

Specific to the VHA, an appropriate clinical response to the public health problem of firearm suicide in the veteran population is needed. Further research within the VHA is needed to determine the healthcare setting(s) and provider types most appropriate for firearm screening and counseling interventions. This step will require a participatory approach among health services and informatics researchers to improve the feasibility, acceptability, relevance, and sustainability of interventions.^49-52 In addition, research is needed to determine the modality and intervention format (electronic, face to face, written) that are most effective for each of the key domains in firearm injury research. Data on the moderators of acceptability and effectiveness (demographics, political views, comorbidities, etc.) of screening and interventions from the veteran and provider perspectives are needed. Only then can researchers begin to measure the short- and longer-term outcomes of such interventions and policies. While this approach is clearly specific to the clinical context and persons involved for veteran prevention with firearms, prevention is likely best on the frontlines of care and where repeated encounters occur with trust-building relationships. Thus, primary care, mental health and ED settings/providers may need to partner with the health services and health informatics researchers to fully address the scope of this need and develop interventions that fit the veteran patients and the VHA system. Equipped with information and curiosity, clinicians can engage their veteran patients as part of routine care, instead of urgent or emergent care, and the health services and health informatics teams can inform us about which methods are most feasible and impactful for veteran quality of life and provider use and sustainability.

LIMITATIONS

Given retrospective studies may introduce sampling bias^53, we included the entire population, not a sample. The results of the NLP algorithm were limited for the first year of entry into VHA healthcare for years 2012–2017, which might underestimate firearm documentation. The identification of firearm documentation for patterns such as temporal changes, variations in types of providers and provider settings, and other patient characteristics will be explored in future work. For example, in this sample there were increases by year (from 3% in 2012 to 21% in 2017). Further research is needed to help explain this increase.

CONCLUSION

Natural language processing methods are able to determine the prevalence of documented firearm screening and safety counseling across a large population of US military veterans. We identified low prevalence of firearm access screening documentation and believe that further investigation into facilitators and barriers is necessary. This work should inform the process for development of systemwide practices to reduce firearm suicide and injury among US veterans, a large group at elevated risk.

Footnotes

Section Editor: Chadd K. Kraus, DO, DrPH, MPH

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Cynthia A. Brandt, MD, MPH, Yale School of Medicine, Department of Emergency Medicine, 464 Congress Ave., Suite 26, New Haven, CT 06519-1315. Email: Cynthia.brandt@yale.edu. 5 / 2021; 22:525 – 532

Submission history: Revision received December 16, 2020; Submitted March 31, 2021; Accepted April 1, 2021

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare. The views expressed in this paper are those of the authors and do not necessarily represent the views of the United Stated Department of Veterans Affairs, the United States Government, or the affiliated academic institutions.

REFERENCES

1. Gun ownership in the U.S. 1972–2020. 2020. Available at: https://www.statista.com/statistics/249740/percentage-of-households-in-the-united-states-owning-a-firearm/. Accessed December 13, 2020.

2. Dempsey CL, Benedek DM, Zuromski KL, et al. Association of firearm ownership, use, accessibility, and storage practices with suicide risk among US Army soldiers. JAMA Netw Open. 2019;2(6):e195383.

3. Anglemyer A, Horvath T, Rutherford G. The accessibility of firearms and risk for suicide and homicide victimization among household members: a systematic review and meta-analysis. Ann Intern Med. 2014;160(2):101-10.

4. Berrigan J, Azrael D, Hemenway D, Miller M. Firearms training and storage practices among US gun owners: a nationally representative study. Inj Prev. 2019;25(Suppl 1):i31-8.

5. Miller M, Hemenway D, Azrael D. State-level homicide victimization rates in the US in relation to survey measures of household firearm ownership, 2001–2003. Soc Sci Med. 2007;64(3):656-64.

6. Cleveland EC, Azrael D, Simonetti JA, Miller M. Firearm ownership among American veterans: findings from the 2015 National Firearm Survey. Inj Epidemiol. 2017;4(1):33.

7. Adams J, van Dahlen B. Preventing suicide in the United States. Public Health Rep. 2020;136(1):3-5.

8. Waliski A, Matthieu MM, Townsend JC, McGaugh J, Kirchner J. Understanding veteran suicide by firearm. J Veterans Stud. 2017;2(2):91-109.

9. Monuteaux MC, Azrael D, Miller M. Association of increased safe household firearm storage with firearm suicide and unintentional death among US youths. JAMA Pediatr. 2019;173(7):657-62.

10. Mann JJ, Michel CA. Prevention of firearm suicide in the United States: what works and what is possible. Am J Psychiatry. 2016;173(10):969-79.

11. Betz ME, Miller M, Barber C, et al. Lethal means access and assessment among suicidal emergency department patients. Depress Anxiety. 2016;33:502-511.

12. What qualifies as a secure gun storage or safety device?. 2015. Available at: https://www.atf.gov/firearms/qa/what-qualifies-secure-gun-storage-or-safety-device. Accessed December 13, 2020.

13. Anestis MD, Bandel SL, Butterworth SE, et al. Suicide risk and firearm ownership and storage behavior in a large military sample. Psychiatry Res. 2020;291:113277.

14. Simonetti JA, Azrael D, Miller M. Firearm storage practices and risk perceptions among a nationally representative sample of U.S. Veterans with and without self-harm risk factors. Suicide Life Threat Behav. 2019;49(3):653-64.

15. Health maintenance and counseling. Am Fam Physician. 2020. Available at: https://www.aafp.org/afp/topicModules/viewTopicModule.htm?topicModuleId=64#0. Accessed December 13, 2020.

16. Grinshteyn E, Hemenway D. Violent death rates: the US compared with other high-income OECD countries, 2010. Am J Med. 2016;129(3):266-73.

17. Roszko PJ, Ameli J, Carter PM, Cunningham RM, Ranney ML. Clinician attitudes, screening practices, and interventions to reduce firearm-related injury. Epidemiol Rev. 2016;38(1):87-110.

18. Betz ME, Knoepke CE, Siry B, et al. ‘Lock to Live’: development of a firearm storage decision aid to enhance lethal means counselling and prevent suicide. Inj Prev. 2019;25(Suppl 1):i18-i24.

19. Feldman MD. Guns, doctors and public health. J Gen Intern Med. 2016;31(10):1109-10.

20. Parent B. Physicians asking patients about guns: promoting patient safety, respecting patient Rights. J Gen Intern Med. 2016;31(10):1242-5.

21. Ranney ML, Fletcher J, Alter H, et al. A consensus-driven agenda for emergency medicine firearm injury prevention research. Ann Emerg Med. 2017;69(2):227-40.

22. Weiss RT. Removing the “Silencer”: Coverage and protection of physician speech under the First Amendment. Duke Law J. 2016;65(4):801-42.

23. Wintemute GJ, Betz ME, Ranney ML. Physicians, patients, and firearms. Ann Intern Med. 2016;165(12):893.

24. Wintemute GJ, Betz ME, Ranney ML. Yes, you can: physicians, patients, and firearms. Ann Intern Med. 2016;165(3):205-13.

25. Jakupcak M, Varra EM. Treating Iraq and Afghanistan war veterans with PTSD who are at high risk for suicide. Cogn Behav Pract. 2011;18(1):85-97.

26. Simonetti JA, Azrael D, Rowhani-Rahbar A, Miller M. Firearm storage practices among American veterans. Am J Prev Med. 2018;55(4):445-4.

27. Chowdhury GG. Natural language processing. Annual Review of Information Science and Technology. 2003;37(1):51-89.

28. Haskell SG, Brandt C, Burg M, et al. Incident Cardiovascular risk factors among men and women veterans after return from deployment. Med Care. 2017;55(11):948-955.

29. Haskell SG, Mattocks K, Goulet JL, et al. The burden of illness in the first year home: Do male and female VA users differ in health conditions and healthcare utilization?. Womens Health Issues. 2011;21(1):92-7.

30. Nelson JP, Pederson LL, Lewis J. Tobacco use in the Army: illuminating patterns, practices, and options for treatment. Mil Med. 2009;174(2):162-9.

31. McGinnis KA, Brandt CA, Skanderson M, et al. Validating smoking data from the Veteran’s Affairs Health Factors dataset, an electronic data source. Nicotine Tob Res. 2011;13(12):1233-9.

32. Lurie N, Popkin M, Dysken M, Moscovice I, Finch M. Accuracy of diagnoses of schizophrenia in Medicaid claims. Hosp Community Psychiatry. 1992;43(1):69-71.

33. Walkup JT, Wei W, Sambamoorthi U, Crystal S. Sensitivity of an AIDS case-finding algorithm: Who are we missing?. Med Care. 2004;42(8):756-63.

34. Clinical Classifications Software (CCS) for ICD-9-CM. CCS Classification software website for healthcare cost and utilization project (HCUP). Available at: http://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp. Accessed May 3, 2013.

35. Anestis MD. Advancing Suicide Prevention Through a Focus on Firearm Safety. Am J Public Health. 2017;107(11):1701-2.

36. 2019 National Veteran Suicide Prevention Annual Report. 2019.

37. Betz ME, Miller M, Barber C, et al. Lethal means restriction for suicide prevention: beliefs and behaviors of emergency department providers. Depress Anxiety. 2013;30(10):1013-20.

38. Johnson-Young EA, McDonald D, Burrell T, et al. Understanding pediatric residents’ communication decisions regarding anticipatory guidance about firearms. J Health Commun. 2020;25(3):243-50.

39. Runyan CW, Brooks-Russell A, Betz ME. Points of influence for lethal means counseling and safe gun storage practices. J Public Health Manag Pract. 2019;25(1):86-9.

40. Firearm injuries becoming more severe. APHA News Releases. 2017. Available at: https://www.apha.org/news-and-media/news-releases/apha-news-releases/2017/am-research-release-firearm-injury. Accessed October 14, 2020.

41. Rowhani-Rahbar A, Simonetti JA, Rivara FP. Effectiveness of interventions to promote safe firearm storage. Epidemiol Rev. 2016;38(1):111-24.

42. Kruesi MJ, Grossman J, Pennington JM, Woodward PJ, Duda D, Hirsch JG. Suicide and violence prevention: parent education in the emergency department. J Am Acad Child Adolesc Psychiatry. 1999;38(3):250-5.

43. Brent DA, Baugher M, Birmaher B, Kolko DJ, Bridge J. Compliance with recommendations to remove firearms in families participating in a clinical trial for adolescent depression. J Am Acad Child Adolesc Psychiatry. 2000;39(10):1220-6.

44. Barber C, Frank E, Demicco R. Reducing suicides through partnerships between health professionals and gun owner groups-beyond docs vs Glocks. JAMA Intern Med. 2017;177(1):5-6.

45. Barber C, Hemenway D, Miller M. How physicians can reduce suicide-without changing anyone’s mental health. Am J Med. 2016;129(10):1016-7.

46. Betz ME, Wintemute GJ. Physician counseling on firearm safety: a new kind of cultural competence. JAMA. 2015;314(5):449-50.

47. Betz M. How to talk about guns and suicide. Podcast. 2017. Available at: https://www.tedxmilehigh.com/emmy-betz/. Accessed December 14, 2020.

48. Firearm Violence Prevention. 2020. Available at: https://www.cdc.gov/violenceprevention/firearms/fastfact.html. Accessed December 15, 2020.

49. Unertl KM, Schaefbauer CL, Campbell TR, et al. Integrating community-based participatory research and informatics approaches to improve the engagement and health of underserved populations. J Am Med Inform Assoc. 2016;23(1):60-73.

50. Millery M, Aguirre AN, Kukafka R. Does a community-engaged health informatics platform facilitate resource connectivity? An evaluation framework. AMIA Annu Symp Proc. 2017:1292-301.

51. Millery M, Ramos W, Lien C, Aguirre AN, Kukafka R. Design of a community-engaged health informatics platform with an architecture of participation. AMIA Annu Symp Proc. 2015:905-14.

52. Wang KH, Hambleton I, Linnander E, et al. Towards reducing health information inequities in the Caribbean: our experience building a participatory health informatics project. Ethn Dis. 2020;30(Suppl 1):193-202.

53. Jager KJ, Tripepi G, Chesnaye NC, et al. Where to look for the most frequent biases?. Nephrology (Carlton). 2020;25(6):435-41.