Qualitative Study of Fourth-Year Medical Students Experiences in Assessing Program Fit Through Virtual Interviews

doi:10.21203/rs.3.rs-4926539/v1

Background

Limited literature exists on the perception of virtual interviewing by medical students in a post-pandemic environment. The current study was conducted to observe the subjective experiences of medical students participating in virtual interviewing and how the ability to assess cultural fit is impacted.

Methods

Recently matched fourth-year medical students were interviewed utilizing a phenomenological approach. Participants were asked to rate ability to assess three common metrics of culture fit of a program on a Likert scale and elaborate on their score. Culture fitness metrics included: How much the program cares about trainees, how satisfied trainees are with their program, and how well trainees get along with each other.

Results

Fourteen fourth-year medical students participated in the study, including specialty fields of Orthopedics, Neurosurgery, General Surgery, Med-Peds, Psychiatry, Internal Medicine, and Family Medicine. Gender identity of the participants included 57.1% (n = 8) female and 42.9% (n = 6) male. Applicants were asked if they preferred virtual or in-person interviewing, with 92.9% (n = 13) preferring virtual interviewing prior to the application cycle and 46.2% (n = 6) of those participants changing their preference to in-person interviews following the application cycle. Average participant scores for the ability to assess how much the program seemed to care about trainees, how satisfied the current trainees were with their program, and how well the trainees got along were 3.4 (Range 2-4.5), 3.0 (Range 1-4.5), 3.1 (Range 1–5) respectively. Most participants reported their scores for ability to assess culture and fit would have increased for in-person interviews.

Conclusion

Applicants in this small qualitative study reported difficulty assessing program culture through virtual interviewing, with some participants preferring in-person interviewing following the interview season. Programs should consider adding an in-person component to interview day, especially for specialties with extended training timelines.

Virtual Interviewing

Fit

Medical Students

Residency

Residency interviews underwent a cultural reset in 2020 to account for stay-at-home orders during the beginning of the COVID-19 pandemic¹. Virtual interviews replaced traditional in-person interviews and have since remained in place despite loosening travel restrictions^1,2. Applicants have largely viewed virtual interviewing as favorable given the flexibility, cost savings, and comfort of doing interviews at home³. Although virtual interviews have generally been well received, the virtual setting limits applicants' ability to assess cultural fit given the lack of body language, environmental observations, and geographic details that are not easily communicated through virtual interviewing⁴. Assessing program fit or cultural, social, and academic alignment with individual goals and personality has been vital in how fourth-year medical students rank residency programs for match⁵. Prior studies have shown that a significant challenge of virtual interviewing for applicants is the loss of ability to assess program culture and interpersonal relationships^5,6.

Prior research have demonstrated students have limited ability to assess culture, with most of these studies primarily based on surveys^1,3,5,7,8. While obtaining quantitative data on a large cohort of medical students is essential, it is also crucial to identify what students are personally experiencing. Few studies have reported the first-hand experiences of medical students through qualitative interviewing⁶. Moreover, since 2020, new strategies to improve virtual interviewing have been introduced including second-look weekends and multi-stage interviewing². Additional study into the narrative impact of virtual interviewing on fit assessment is needed to appropriately assess the holistic benefit of virtual interviewing in the present day. We conducted a phenomenological-focused survey on fourth-year medical students who completed virtual interviewing in 2023–2024. We hypothesize that applicants will report difficulty assessing program culture through virtual interviewing.

IRB approval was obtained from institutional review board, need for signed consent was waived given deidentified nature of the data. Inclusion criteria included all fourth-year medical students at a single institution participating in the 2023–2024 ERAS cycle. Students were recruited from two class-wide emails requesting participation. We utilized a phenomenological approach to assess reported perspectives of medical students. An interview template based on prior studies' critical components of assessing program fit was created and included as supplemental material 1. The general structure for the interview included asking participants to rate their ability to evaluate different aspects of program fit through virtual interviewing on a scale of one to five, with five representing being able to assess very easily and one being very difficult to determine. Participants were then asked to elaborate on their scores. The three questions that participants were asked were how much the program cares about its trainees, how satisfied they are with their program, and how well they got along. The interviewer was allowed to ask follow-up questions outside the bounds of the interview template to expand on a participant's perception.

Participants were verbally consented and participated in one 20 to 50-minute interview. The lead author conducted all interviews and transcribed recordings using Zoom Version 2.0.6⁹. All interviews took place between March 2024 – May 2024. Recordings were de-identified to ensure participant privacy, and author AW listened to all interviews following initial recording and transcription to ensure the accuracy of the record. Analysis of personal perceptions and biases before and throughout data analysis was conducted to reduce bias¹⁰.

Quantitative data from Likert score ratings were summarized in Microsoft Excel using mean and ranges given the small number of participants. Comparative analysis and inductive analysis were applied through a phenomenological approach to identify the cognition of fourth-year medical students around virtual interviewing¹¹. Transcriptions were analyzed for similar patterns of thought, and were grouped together¹². Groups were further sorted into overarching themes. Word clouds of responses to fit metrics were generated in MATLAB Version 2020a (Natick, MA: The Math Works, Inc., 2020).

A recruitment email was sent to the fourth-year medical student class list, including 181 medical students, 14 (7.2%) of whom responded. Demographic and specialty characteristics are described in Table 1, with 53.8% (n = 7) self-identifying as female and including students applying to eight different specialties. Mean virtual interview participation was12.5 virtual (Range: 5–28), with 85.7% (n = 12) of students participating in additional on-site activities as documented in Table 2. Of note, 92.9% (n = 13) participants initially preferred virtual interviewing over in-person interviewing; however, following the interview season, only 50% (n = 7) participants preferred virtual over in-person interviewing.

Table 1. Demographics of Study Cohort

Demographics
N = 14	n (%)
Self-Identified Gender
Female	8 (57.1)
Male	6 (42.9)
Non-Binary	0 (0)
Specialty
General Surgery	3 (21.4)
Internal Medicine	3 (21.4)
Medicine-Pediatrics	2 (14.3)
Psychiatry	2 (14.3)
Family Medicine	1 (7.1)
Anesthesia	1 (7.1)
Orthopedics	1 (7.1)
Neurosurgery	1 (7.1)

Table 2. Interview Characteristics

Interview Characteristics
N = 14	n (%)
In-Person and Virtual Participation	13 (92.9)
Initial Preference
Virtual	13 (92.9)
In Person	1 (7.1)
Post-Interview Season Preference
Virtual	7 (50.0)
In Person	6 (42.9)
Ambivalent	1 (7.1)

Participants were asked about their ability to judge components of program culture fit, detailed in Table 3, using a Likert Scale from 1–5. A score of 1 represented very difficult assessment of metric, while 5 represented very easy assessment of metric. Participants rated their ability to assess how trainees got along with the greatest range at an average score of 3.11 (Range: 1–5). Participants, on average, rated ability to assess how much the program seemed to care about trainees and how satisfied trainees are with their program a 3.38 (Range: 2-4.5) and 3.0 (Range: 1-4.5), respectively. When asked how their scores would have changed had their interview been in-person rather than virtual, 78.6% (n = 11) of participants said their scores would have increased, 7.1% (n = 1) said scores would decrease and 14.2% (n = 2) said scores would remain the same.

Table 3

Likert Scores of Ability to Assess Fit in a Virtual Environment
Likert Scores of Ability to Assess Fit in a Virtual Environment
Program Culture Fit	Mean (SD), Range
- Overall ability to assess culture in virtual interviewing	3.6 (0.62), 2.5-4
- How much the program seemed to care	3.4 (0.68), 2-4.5
- How satisfied are trainees with program	3.0 (1.09), 1-4.5
- How well do trainees get along	3.1 (1.11), 1–5
Program Logistics/Academics	3.1 (1.4), 1–5

Participant Likes and Dislikes of Virtual Interviewing

Participants were more supportive of virtual interviewing than opposed. Three main benefits were commonly noted by participants when they were asked what they liked about virtual interviewing. All participants cited cost-savings as a benefit of virtual interviewing. Participants could participate in a more significant number of interviews because they saved money on travel accommodations. The median cost budgeted out for interviews was 200 dollars, with the median amount spent being 700 dollars. One applicant noted that virtual interviewing allowed him to apply to programs that he might not initially apply to a given geographic location. Participants also cited flexibility in scheduling as a benefit to virtual interviewing, as scheduling interviews strategically during the fourth year of medical school was easier because students did not have to factor time required to travel. A third benefit of virtual interviewing was the convenience factor of virtual interviewing. Participants did not have to worry about finding parking for the interview and navigating complex hospital campuses on interview day, which alleviated stress. One participant also noted that virtual interviewing offered convenience for program residents and faculty, allowing them to "pop in" to interviews throughout the day with less disruption to their daily schedule. A fourth, less commonly noted benefit of virtual interviewing was the ability to showcase an applicant's personality by exhibiting items related to the applicant's hobbies, family, or interests outside of medicine behind the applicant's virtual interview.

Virtual interviewing did limit the applicant's ability to observe interpersonal relationships. Many participants noted that body language and simple interpersonal interactions such as "holding the door for people" were crucial in assessing a program's cultural norms and behaviors. One applicant noted that technical difficulties of using Zoom, such as cutting off other speakers due to lag, may contribute to an awkward interview environment and limit natural conversation. Assessment of program environment suffered in a virtual setting, which limited applicants' abilities to assess program culture, with one participant citing that "you don't get to see the physical place where you might be working at [and] you don't get a feel for the staff as much [and] kind of catch a passing …feel for the residents, but even then, … being in a zoom room …you don't get to seeing them in the true environment."

Notably, some participates noted that there are inherent limitations in assessing program culture due to the nature of interviewing itself: applicants and programs tend to showcase their strengths and diminish negative characteristics during interview days. Regardless of the setting in which interviews happen, there would be difficulty in ascertaining a true reflection of the program culture. Additionally, participants stated that they were concerned about possible selection bias of residents/trainees participating in interview days where residents with more positive experiences might be selected to participate over other trainees: "Very few people [will] tell you bad things, and if there are people that [will] tell you bad things, they probably got screened out before they were asked to come talk to you." This tension is further exacerbated in a virtual environment where applicants might not have the opportunity to interact with residents and “read in between the lines”. Participants states that in-person interviews might increase opportunities for informal conversations during walks to different locations or when smaller groups can break off from the scrutiny of a larger crowd.

Participant Assessment of Culture Fit

Participants were asked about their ability to assess three critical components of judging program cultural fit, including how much the program cares about trainees, how well trainees got along, and how satisfied trainees are with their program. Word clouds of participant responses to each of the three questions is demonstrated in Fig. 1. Participants approached questions regarding their ability to assess program culture through external and internal observations. External observations included interpersonal behavior between faculty and residents and tangible resources implemented for resident and applicant benefit. Internal observations included personal accounts of the program culture by residents or alumni, emphasizing the resident's emotional state. Examples of internal and external observations are listed in Table 4. Most participants would then assess the quality of these observations based on truthfulness. For example, one participant demonstrated how he might interpret the statements of trainees: "[Residents] won't say they're unhappy because they want you to come, and usually the residents that do show up for some of the meetings are the ones that are rested. So, they're either in a lighter rotation or don't have kids." After assessing the quality of these observations, participants would then compare their standards with observations to determine culture fit. Quality of observations substantially impacted how positively the program culture was viewed, with less transparent programs being viewed more negatively or ambivalently. One participant noted higher scores: "If they included a program overview during the interview session, I would give them a 4 or 5, [because] I feel like these people take care of their residents. If they didn't include it, sort of neutral…2 or 3."

Table 4

Examples of External and Internal Observations Made by Participants
Examples of External and Internal Observations Made by Participants
External Observation Examples	Internal Observation Examples
"how… the attendings treat the residents and medical students, and how the coresidents are with each other."	"I would call [trainees/alumni] or email them or text them to get a more honest perspective about how they felt…"
"I would emphasize the organization of the interview day itself, which spoke a lot about their culture to me and how much they cared."	"I relied on the [pre-interview] social to, I guess, talk more about what [trainees] really enjoyed about their program and what they would like changed about their program, as well as what changes have been implemented."
"Most programs talk about wellness and things like benefits… Sick leave vacation things like that and like. Social events that they put on for their residents."	"Though like one-on-one interviews with multiple people, it gives you a sense of culture."
"You can see them joking. You can see what kind of jokes they make, and then you sort of see how they bounce their humor, like topic, or the question how they answer it."	"It's hard to get a sense for the roots of a program on the interview day, and that's when you kinda ask around the people who've been through that program."

How well are applicants able to assess how much a program cares about its trainees?

How well participants could judge how much a program cares about its trainees were based mainly on external observations of how mentorship, feedback, career development, and academic discipline were approached. Participants who believe that the nature of interviews creates inherent barriers also stated that how the program director presented themselves was an important indicator of how much the program cares about the trainees: "You see all of them [program directors] at the interviews. You could see their enthusiasm and how much they paired just visually. I was able to assess that better…" Additionally, participants noted that the structure of the interview day itself impacted how much they perceived the program cares, with well-organized and thoughtful schedules reflecting more positive feelings toward that program's culture. In addition to external observations, participants would ask strategic questions to tease out genuine internal observations from residents, such as asking how resident feedback is implemented in the program. Asking neutral questions such as these allowed residents to answer truthfully, allowing applicants to tease out some critical internal observations.

How well are applicants able to assess how satisfied trainees are with their programs?

This component was assessed through mostly internal observations reported by trainees. Participants noted difficulty trusting what residents reported, with one participant stating: "I felt like we often had to push people kind of to talk about the negative things or the things they didn't like as much. Usually, people weren't as upfront…" Participants also noted that programs would incentivize residents to participate in interview day "so it was hard to tell whether the residents were there because they felt supported and wanted to express that, or they wanted the incentives that came along with being there." Participants who felt like trainees were relatively straightforward and honest suggested that increased experience with interviews and the confidence to ask more direct questions such as "How do you feel in your program? What is a weakness in your program? What is something you wish they would change about your program? Do you feel burnt out?" improved ability to assess how satisfied trainees are with their program. External observations that participants utilized to determine how satisfied trainees included observations about the amount of vacation time, call schedule, childcare, parking, and meal stipends.

How well did trainees get along with each other?

How well trainees got along was primarily assessed through external behavior observations during social gatherings. Most participants noted that seeing multiple residents log on from one virtual conference call indicated positive collegiality. Rather than asking trainees directly what they thought of their peers, most participants preferred to "be quiet and pay attention to [residents]." More informal settings allowed applicants greater ability to judge interpersonal relationships, with one participant stating that "seeing residents in a non-judgmental setting like that was helpful." When trainees logged on to the video interview from separate computers, it was very difficult for participants to assess how well trainees got along with each other.

In this qualitative analysis of fourth-year medical students' experience with assessing program culture fit through virtual interviewing, participants reported that virtual interviewing impacts applicants' ability to evaluate program culture. Most participants initially preferred virtual interviewing: however, they preferred in-person interviews following virtual interviewing.

All participants noted that cost saving was a benefit of virtual interviewing. The median cost spent on virtual interviewing in the study was 700 dollars, significantly lower than the cost paid by applicants during years with standard in-person interviews¹³. Cost saving may be important defining accessibility of interview participation for applicants of lower socio-economic status¹⁴. Additionally, because applicants did not have to travel for interviews, many noted greater flexibility in scheduling interviews and decreased stress, allowing for greater comfort on the interview day. This sentiment is similarly reported in other studies^6,7. However, participants noted that virtual interviewing severely limited the ability to get an accurate feel for what life would look like as a trainee in the program, and conversely may limit the interviewer’s ability to judge the applicant¹. This was especially concerning for participants applying to surgical specialties, which requires longer training. Participants noted that there is difficulty imagining what life would look like as a trainee without walking through the hospital and engaging with the program trainees in person. Participants who attended second looks, away rotations, or independently traveled to the program location commented that these in-person events were essential to differentiating fit between equivocal programs. On-site supplements to virtual interviewing strengthened initial assessments and offered perspective that changed some participant’s minds.

Culture Fit

Applicants commonly observe three metrics when assessing program fit: how much the program cares about its trainees, how well they get along with each other, and how satisfied trainees are with their program⁵. Participants interviewed in the current study utilized external and internal observations based on these metrics to assess program fit. Metrics that relied heavily on external observations included how well trainees get along and how much the program cares about its trainees. Participants were more likely to trust external observations as these were more directly reported. How satisfied trainees were with their program required reliance on internal observations derived from what trainees shared during interview day. Likert scores for how well participants could assess these metrics were lowest for how satisfied trainees were with their program. Internal observations were highly scrutinized as many participants cited either selection bias of the trainees involved in the program or hesitance to speak on negative experiences in coloring their ability to assess these observations.

Some participants noted that interviewing is inherently manufactured as both parties, program, and applicant, highlight positive characteristics and minimize negative ones. Virtual interviewing exacerbates this tension as natural interpersonal behavior and interacting with the physical environment of the program campus are limited in a virtual environment. All participants except for two noted that their ability to assess metrics of program culture would increase had their interviews been in person, with almost half of participants who preferred virtual interviewing initially changing preference to in-person interviews. In contrast, a prior study in complex surgical oncology applicants reported that applicants initially preferred in-person interviews¹⁵. This contrast may be due to differences between level of training of the two groups of participants, with medical students prioritizing program culture over specific training details which are more easily evaluated through virtual means.

Participants in the current study were asked to give feedback on virtual interviewing. The most common feedback given was to increase opportunities for applicants to engage informally with program trainees to mitigate interviewer hesitation to speak negatively about the program and be more "themselves." Putting applicants in contact with program alumni can also help applicants get a more accurate reflection of the program as alumni have lower stakes and may offer more truthful answers. Other feedback included logistical recommendations such as limiting the amount of on-camera time during the interview day, and instead distributing information prior to interview day to be reviewed independently. Prior large survey studies have determined that websites were important for applicants to learn about training details, and should be thoroughly updated to improve program outreach¹⁶. Additionally, offering opportunities to participate in on-site activities can significantly strengthen the ability of applicants to assess program fit based on the opinions reported in this study.

This was a small sample of medical students from a single institution, limiting the generalizability of this study. Although many specialties were represented, a greater rate of participants were participating in surgical specialties as compared to the national average¹⁷, which may skew results which may skew results to preferring in-person interviews as location is more important given the length of training.

Applicants in this qualitative study reported difficulty assessing program culture through virtual interviewing, with some participants preferring on-site interviews following the interview season. Programs should consider adding an in-person component to interview day, especially for specialties with longer training requirements.

Ethics approval and consent to participate: Approval for this study was obtained from the UNC- Chapel Hill IRB with written signed consent requirement waived

Consent for publication: Not Applicable

Availability of data and material: The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Competing interest: The authors declare that they have no competing interests

Funding: The authors declare that they have no competing interests

Authors' contributions: AW contributed to conceptualization, IRB writing, data collection and manuscript writing. LP contributed to conceptualization of the study and manuscript editing of this study. All authors read and approved the final manuscript

Acknowledgements: No other acknowledgements to disclose

Author’s Information: AW is a 4^th year medical student during the period of this study and LP served in the role as General Surgery Career Goal Advisor.

Jacob Shreffler, Shreffler J, Melissa, Platt, et al. Planning virtual residency interviews as a result of COVID-19: insight from residency applicants and physicians conducting interviews. Postgrad Med J Published online. 2021. 10.1136/postgradmedj-2020-139182.
Wong TY, Thomas Y, Wong JJ, Huang, et al. Adapting to the Era of Virtual Recruitment: Radiology Departmental Website Response to COVID-19 and Portrayal of the Resident Experience. Acad Radiol. 2022;29(5):771–8. 10.1016/j.acra.2021.11.023.
Mallory Davis, Davis MG, Mary R, Haas et al. Zooming In Versus Flying Out: Virtual Residency Interviews in the Era of COVID-19. 2020;4(4):443–6. 10.1002/aet2.10486
Rozenshtein A, Rozenshtein A, Brent, Griffith, et al. Residency Match During the COVID-19 Pandemic: The Clear and Present Danger of the Remote Interview. J Am Coll Radiol. 2020;18(3):438–41.
Jorge GZárate, Rodriguez J, García Rodríguez, Connie Y, Gan, et al. Applicants’ perception of fit to residency programmes in the video-interview era: A large multidisciplinary survey. Medical Education. Published online January. 2022;10. 10.1111/medu.14729.
Lee E, Terhaar S, Shakhtour L, et al. Virtual Residency Interviews during the COVID-19 Pandemic: The Applicant’s Perspective. South Med J. 2022;115(9):698–706. 10.14423/SMJ.0000000000001442.
Cynthia Czawlytko, Czawlytko C, Elana B, Smith et al. The Effect of Virtual Interviews and Social Media on Applicant Decision-Making During The 2020–2021 Resident Match Cycle. Academic Radiology. Published online July 6, 2021. 10.1016/j.acra.2021.05.028
Ravinder Bamba, Bamba R, Bamba R, et al. Virtual Interviews for the Independent Plastic Surgery Match: A Modern Convenience or a Modern Misrepresentation? J Surg Educ. 2020;78(2):612–21. 10.1016/j.jsurg.2020.07.038.
Barbu CM. cbarbu/R-package-zoom. Published online June 4, 2020. Accessed July 28, 2024. https://github.com/cbarbu/R-package-zoom
Neubauer BE, Witkop CT, Varpio L. How phenomenology can help us learn from the experiences of others. Perspect Med Educ. 2019;8(2):90–7. 10.1007/s40037-019-0509-2.
Cristancho SM, Goldszmidt M, Lingard L, Watling C. Qualitative research essentials for medical education. Singap Med J. 2018;59(12):622–7. 10.11622/smedj.2018093.
Creswell JW, Poth CN. Qualitative Inquiry and Research Design: Choosing Among Five Approaches. SAGE; 2016.
Fogel HA, Liskutin TE, Wu K, Nystrom L, Martin B, Schiff A. The Economic Burden of Residency Interviews on Applicants. Iowa Orthop J. 2018;38:9–15.
Lee AL, Andrew H, Lee, Lee AH, et al. I dream of Gini: Quantifying inequality in otolaryngology residency interviews. Laryngoscope. 2019;129(3):627–33. 10.1002/lary.27521.
Jennifer F, Tseng, Tseng JF, Jennifer Tseng. How Has COVID-19 Affected the Costs of the Surgical Fellowship Interview Process? J Surg Educ. 2020;77(5):999–1004. 10.1016/j.jsurg.2020.05.018.
Benjamin T, Jarman, Jarman BT, Amit J, et al. Factors and Influences That Determine the Choices of Surgery Residency Applicants. J Surg Educ. 2015;72(6). 10.1016/j.jsurg.2015.05.017.
ERAS® Statistics. AAMC. Accessed July 28. 2024. https://www.aamc.org/data-reports/data/eras-statistics-data

No competing interests reported.

Qualitative Study of Fourth-Year Medical Students Experiences in Assessing Program Fit Through Virtual Interviews

Status:

Version 1

Abstract

Background

Methods

Results

Conclusion

Figures

Introduction

Methods

Results

Participant Likes and Dislikes of Virtual Interviewing

Participant Assessment of Culture Fit

How well are applicants able to assess how much a program cares about its trainees?

How well are applicants able to assess how satisfied trainees are with their programs?

How well did trainees get along with each other?

Discussion

Culture Fit

Conclusion

Declarations

References

Additional Declarations

Status:

Version 1