Author	Affiliation
Matthew G. Malone, MD	The Ohio State University College of Medicine, Department of Emergency Medicine, Columbus, Ohio
Michelle M. Carney, MD	University of Michigan Medical School, Department of Emergency Medicine, Ann Arbor, Michigan; University of Michigan Medical School, Department of Pediatrics, Ann Arbor, Michigan
Joseph B. House, MD	University of Michigan Medical School, Department of Emergency Medicine, Ann Arbor, Michigan; University of Michigan Medical School, Department of Pediatrics, Ann Arbor, Michigan
James A. Cranford, PhD	University of Michigan Medical School, Department of Psychiatry, Ann Arbor, Michigan
Sally A. Santen, MD, PhD	University of Michigan Medical School, Department of Emergency Medicine, Ann Arbor, Michigan

Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

Student evaluations are paramount to faculty both administratively and academically. Evaluations have been used as data to inform the decision for promotion and tenure in higher education for years.¹ By comparing data of faculty obtained through trainee evaluations, individual educator performance can be measured. Equally as important is the ability for faculty to grow as educators by internalizing feedback from evaluations; celebrating accomplishments and providing a substrate for areas in which growth is necessary.² Using student evaluations in this way employs the social constructivist model; faculty use feedback from students for professional development and reflective improvement. Knowledge and behavior are built through interaction and feedback from others.³

Online evaluations have become an increasingly popular method of obtaining evaluation data.⁴ Previous work has identified significant advantages to the online evaluation model, which include potentially significant cost savings, improved turnaround time, greater elaboration afforded by typed responses and convenience for students to respond without using valuable class time.^5,6 In addition, online evaluations are often the preferred method by students.⁷

However, online methods of evaluation are not without disadvantage. It is well established that converting from paper evaluations to an online evaluation system results in lower response rates, which in turn can lead to increased bias and less valuable data.^8,9

Online evaluations may be more convenient, but literature is lacking in how to motivate students and trainees to complete online evaluations. To bridge this gap, the authors tried to make the completion of evaluations a tit-for-tat situation. If the students wanted to see their evaluations during the rotation and prior to receiving their grade, they must complete an evaluation for their supervisor. In this way, the motivation comes from an internal need for feedback. The objective of this study was to identify an increase in the number of faculty and resident evaluations completed by students rotating through their Emergency Medicine clerkship following the implementation of a tit-for-tat incentive strategy. The authors hypothesize that a tit-for-tat strategy whereby students had to complete a resident or faculty evaluation in order to view their student assessment completed by their resident or faculty preceptor would increase the total number of preceptor evaluations.

METHODS

This study was a retrospective cohort study of medical student evaluations of faculty and resident preceptors before and after the implementation of a tit-for-tat method to increase the number of total evaluations completed. The (blinded) Institutional Review Board approved the study.

As part of the required fourth-year Emergency Medicine clerkship at (blinded), students were asked to fill out evaluations of residents and faculty members with whom they worked. These evaluations were encouraged but voluntary. Prior to the 2014–2015 academic year, students would receive online assessments from faculty and resident preceptors. Similarly, faculty and residents would receive online evaluations from medical students with whom they worked. These evaluations of faculty and residents were blinded and aggregated so that the preceptor could not identify the medicals student.

Beginning in the 2014–2015 academic year, a tit-for-tat strategy was employed whereby each student would receive an online push notification that an assessment from a resident or attending physician had been completed and in order to view this assessment, the medical student had to complete an evaluation of the resident or attending physician in order to see the assessment of their performance. The assessment of the medical student by resident or faculty thus became un-blinded to ensure that the students knew which evaluations to complete in order to view their own assessment from the preceptor. This method was thought to not introduce bias, as the preceptor’s assessment of the student could not be viewed until the student submitted the evaluation of that preceptor. Further, the evaluations completed by medical students remained blinded, such that the preceptor could not identify the medicals student evaluator. The authors could not identify any other changes in the evaluation process that would confound the results. This strategy was employed to increase the total number of resident and faculty evaluations completed by medical students.

 

Population Health Research Capsule

What do we already know about this issue?

Methods to increase online survey and evaluation response rates have been applied in the commercial and undergraduate education literatu e, but rarely studied with medical students.

What was the research question?

Does an incentive strategy increase the response rate of preceptor evaluations completed by medical students?

What was the major finding of the study?

There was an increase in evaluations completed by medical students following the implementation of an incentive strategy.

How does this improve population health?

More feedback to educators hopefully leads to better educators, educational materials and methods. Better education leads to better health care providers and healthier populations.

 

In both the control cohort and the tit-for-tat cohort, all evaluations were submitted by three weeks after the completion of the rotation. Once the grade was assigned, 3–4 weeks after the clerkship, students could view all of their assessments of performance, as it was felt to be unfair to completely withhold feedback information.

The total number of student evaluations of both resident and faculty was recorded from 2014–16 following the implementation of tit-for-tat, as well as from 2012–2014, which was used as a control. A Chi-squared analysis was performed to demonstrate a statistically significant increase in the number and proportion of medical students who chose to fill out evaluations. The mean number of evaluations completed per student per academic year and the mean number of evaluations received per preceptor per academic year were also calculated and compared.

RESULTS

In the control cohort, 201 of the 306 rotating medical students completed a total of 1101 of evaluations of faculty and resident preceptors. In the tit-for-tat cohort, 307 of the 334 rotating medical students completed a total of 2736 of evaluations of faculty and residents (Table 1). In the control cohort, 64.0% of rotating students completed at least one evaluation. In the tit-for-tat cohort, 91.3% of rotating students completed at least one evaluation. A Chi-squared analysis was performed and there was a statistically significant increase in student participation in completing evaluations following the implementation of tit for tat, x² (1) = 69.8, p < .05.

The mean number of evaluations completed per student was calculated from 2012–2016 to control for the variation in number of medical student rotators between academic years. An independent samples t-tests was performed to demonstrate a statistically significant increase from the control cohort to the tit-for-tat cohort. The mean number of evaluations completed per student was 3.60 (SD = 3.959) in the control cohort. The mean increased to 8.19 (SD = 3.791) evaluations completed per student in the tit-for-tat cohort, which was statistically significant with p-value <0.001. This statistically significant increase in evaluations completed per student is maintained for both faculty and residents when calculated independently (Table 2).

In addition, the mean number of evaluations received per preceptor was also calculated to control for variation in the number of resident and faculty between academic years. The mean number of evaluations received per preceptor increased from 3.77 (SD = 2.743) in the control cohort to 7.52 (SD = 5.599) evaluations per preceptor in tit-for-tat cohort, which was statistically significant with p-value <0.001. Again, this increase is maintained for both faculty and residents when calculated independently (Table 3).

DISCUSSION

The term “tit-for-tat” is an English saying dating to 1556 meaning an equivalent to an action given in return.¹⁰ While this often carries a negative connotation, such as a blow for blow retaliation, tit-for-tat has also been used to describe positive symbiotic relationships, such as reciprocal altruism.¹¹ In the student-preceptor relationship, the responsibility of the preceptor is to provide feedback; likewise it is the student’s role to reciprocate. Faculty evaluations are used to recognize and reward excellence as well as to identify outliers in performance and provide feedback to facilitate reflective improvement.

It is well established in the current literature that converting from paper evaluations to an online evaluation system results in lower response rates.⁸ Further, it has been suggested that response rates are themselves a critical indicator of both student and faculty engagement in the course.⁴ With lower response rates, the potential for error in any survey increases and in turn, the reliability of the data tends to weaken as response rates decline.⁹ It has also been shown that respondents and non-respondents to evaluations differ. For instance, students are more likely to complete evaluations in courses where students have specific interest in the subject and poor performing students complete fewer evaluations.¹² As non-response rates increase, the likelihood that non-respondents’ opinions differ from respondents’ opinions increases.⁹ Therefore, low response rates are more likely to result in bias.

Some previous work has been done on ways to increase response rates to online evaluations by using a variety of methods. These methods include teachers making a concerted effort to promote the online evaluation, faculty providing students with information on the use of their feedback and entering student participants in a drawing for a cash prize.¹³ Small up-front gifts and conditional incentives have also been shown to increase response rates.¹⁴ In a review article, Nulty offered a set of twelve best practices for increasing response rates to online surveys and suggests that these method are additive (table 4). While some techniques demonstrate an increase in evaluation response, they are often dependent on the enthusiasm of the faculty and response rates decline as time passes and enthusiasm wanes.¹⁵

While not supported by the literature, there is concern that making evaluations mandatory may affect evaluation quality. In addition, most methods that have been shown to increase response rates require input or effort by the preceptor or by an administrator. The benefit of the tit-for-tat method is it uses an automated system completely independent of additional input or effort and it accomplishes an increase in response rates through incentives. Therefore, this study supports the hypotheses that a tit-for-tat incentive strategy does increase the total number of preceptor evaluations submitted by medical students without making evaluation submission mandatory.

LIMITATIONS

Unfortunately, the evaluation system used did not track demographic data. The demographics of each study year are approximately similar to the graduating class of each academic year, but due to the presence of away rotators, the demographics cannot be calculated accurately. Differences in demographics could exist between the students in each academic year and possibly skew the results. Further, we also have no data on the number of student assessments completed by preceptors. It is unclear if there were fluctuations in the number of student assessments between academic years or what affect those fluctuations would have if they exist. During this time there were minor increases in the number of faculty and residents, it is unclear what effect this may have had. In addition, our data also shows that that average evaluation score of the emergency medicine rotation steadily increased over the course of the data collection period. It is possible that the increased popularity could have contributed to the response rates of the evaluation. As previously discussed, students are more likely to complete evaluations in subjects of personal interest. It should also be noted that this study demonstrated that the increase occurred after the intervention and concluding a causal relationships from this before and after study has limitations. Finally, the authors have no data on the quality of the evaluations. It is possible that the additional submitted evaluations differ in usefulness of comments and thus have affect the utility of the intervention.

CONCLUSION

A significantly increased in the number of evaluations submitted by students rotating through their emergency medicine clerkship was observed following the implementation of the tit-for-tat method. This served as an effective tool to increase the number of evaluations completed by students and the proportion of students that completed evaluations. Further work should be done to identify any affect of the tit-for-tat method on the quality of evaluations and better understand additional methods of increasing evaluation response rates and assess if these methods are summative.

Footnotes

Section Editor: Jonathan Fisher, MD, MPH

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Matthew Malone, MD, Ohio State University College of Medicine, Department of Emergency Medicine, 791 Prior Hall/376 W 10^th Avenue, Columbus, Oh 43210. Email: malone.154@osu.edu. 1 / 2018; 19:75 – 79

Submission history: Revision received June 17, 2017; Submitted August 24, 2017; Accepted September 11, 2017

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Layne BH, DeCristoforo JR, McGinty D. Electronic versus Traditional Student Ratings of Instruction. Research in Higher Educ. 1999;40(2):221-32.

2. Centra JA. Reflective Faculty Evaluation: Enhancing Teaching and Determining Faculty Effectiveness. The Jossey-Bass Higher and Adult Educ Series: ERIC. 1993.

3. Rust C, O’Donovan B, Price M. A social constructivist assessment process model: how the research literature shows us this could be best practice. Assessment & Evaluation in Higher Education. 2005;30(3):231-40.

4. Anderson J, Brown G, Spaeth S. Online student evaluations and response rates reconsidered. Innovate: J Online Educ. 2006;2(6):5.

5. Bothell TW, Henderson T. Do online ratings of instruction make sense?. New Directions for Teaching and Learning. 2003;2003:69-79.

6. Sorenson DL, Reiner C. Charting the uncharted seas of online student ratings of instruction. New Directions for Teaching and Learning. 2003;2003:1-24.

7. Avery RJ, Bryant WK, Mathios A, et al. Electronic course evaluations: Does an online delivery system influence student evaluations?. J Economic Educ. 2006;37:21-37.

8. Dommeyer CJ, Baum P, Hanna RW. College students’ attitudes toward methods of collecting teaching evaluations: In-class versus on-line. J Educ for Business. 2002;78(1):11-5.

9. Groves RM, Fowler FJ, Couper MP, et al. Survey methodology. 2011.

10. Dictionary Entry: tit for tat. 2015. Available at: www.merriam-webster.com/dictionary/tit%20for%20tat. Accessed March 25, 2017.

11. Milinski M, David K, Kettler R. Tit for tat: sticklebacks (Gasterosteus aculeatus) ‘trusting’ a cooperating partner. Behavioral Ecology. 1990;1(1):7-11.

12. Adams MJ, Umbach PD. Nonresponse and online student evaluations of teaching: Understanding the influence of salience, fatigue, and academic environments. Research in Higher Education. 2012;53(5):576-91.

13. Ballantyne C. Online evaluations of teaching: An examination of current practice and considerations for the future. New Directions for Teaching and Learning. 2003;2003:103-12.

14. Balajti I, Daragó L, Ádány R, et al. College students’ response rate to an incentivized combination of postal and web-based health survey. Eval Health Prof. 2010;33(2):164-76.

15. Cummings R, Ballantyne C. Student feedback on teaching: Online! On target. 1999.