Education Research: High-Fidelity Simulation to Evaluate Diagnostic Reasoning Reveals Failure to Detect Viral Encephalitis in Medical Trainees

Setting and Study Design

This is a prospective, single-center simulation study performed between February 2018 and January 2022. It was conducted by 3 neurointensivists (M.B.P., W.-T.W.C., N.A.M.). Each simulation was captured by video for review by the raters. Participants ranging from neurology subinterns to attending physicians voluntarily participated individually in the study. They did not receive any information about the case before the simulation. Before participation, they completed a questionnaire regarding demographics, level of training, and prior experience with the management of multiple neurologic emergencies including meningoencephalitis and status epilepticus. Participants completed a pretest regarding the management of neurologic emergencies, including but not limited to meningoencephalitis and status epilepticus. They were allowed to use smartphones if they wished during the simulation. The simulation took between 20 and 30 minutes to complete. After the simulation clinical scenario was completed, they debriefed with a neurointensivist for an additional 20–30 minutes. The debriefing was structured following the Debriefing with Good Judgment approach to invoke reflective practice leading to behavioral change and performance improvement.¹¹ Simulation took place on an afternoon when trainees were rotating in the neurocritical care unit. They were excused from clinical duties, and consequently, the simulation took place within their duty hours. We told participants that their performance did not have any effect on their grade or evaluation for the rotation. They did not receive any financial compensation for participation in the study.

Clinical Simulation Case and Trainee Assessment Development

We previously published development of the clinical simulation case and critical actions checklist using a modified Delphi method.¹² The clinical case, critical actions checklist, and rating scale can be found in the supplementary material of our prior publication.¹² We chose the items on the critical action checklist based on the Neurocritical Care Society's Emergency Neurologic Life Support protocols, cross-referenced with guidelines from the Epilepsy Foundation/American Epilepsy Society,¹³ the Infectious Disease Society of America,^14,15 and the Neurocritical Care Society.¹⁶ A board-certified neurologist with certification in neurocritical care (N.A.M.), an additional board-certified neurologist with certification in neurocritical care at a different institution (D.P.L.), and a board-certified neurologist with board certification in epilepsy and clinical neurophysiology (C.A.G.) reviewed the clinical scenario and checklist. The final list checklist represented a consensus among the three.

Simulator and Simulation Environment

In a prebriefing, the simulation operator oriented the participants to the simulation environment and SimMan. We asked them to verbalize any components of the neurologic examination they wished to perform and to verbalize any orders or diagnostics. Diagnostics including laboratory results, neuroimaging, and EEG findings were displayed on a monitor in the simulation room. If participants wished to place a consult, they did so through telephone request in the simulation room. The simulation operator acted as the consultant. Consultant responses were not scripted; however, consistency of response was ensured through intermittent viewing of cases for standardization. If they were able to independently perform a procedure such as intubation, we asked them to perform that skill, otherwise they were asked to consult a service to perform any procedure outside of their skill set. A nurse embedded participant accompanied participants through the entirety of the simulation scenario. The nurse had direct communication with the simulation operator using an earpiece.

We used the SimMan 3G manikin (Laerdal, Wappinger Falls, NY). SimMan 3G can depict both neurologic and physiologic signs including pupillary constriction, voice through an internal speaker, respiratory patterns, and seizures. Although SimMan is not able to perform the entirety and nuances of a neurologic examination, participants can obtain these findings by asking the nurse embedded participant at bedside whether extraocular movements, the motor examination, and reflexes are intact. A bedside monitor displays telemetry, arterial blood pressure, and oxygen saturation. These parameters can be altered by the simulation operators.

The simulation room represented a patient room on a medical/surgical floor. It included all equipment needed to perform the clinical scenario. Medications included IV fluids, rapid sequence intubation medications, benzodiazepines, antiseizure medications, anesthetic drips, thiamine, and dextrose. Although medications were available in the simulation room, they were not visible to participants. Airway equipment included nasal cannula, non-rebreather mask, bag valve mask, laryngoscope, endotracheal tubes (ETTs), Bougie, and a ventilator. Other equipment included a lumbar puncture tray and EEG machine.

Clinical Scenario

Participants performed individually by responding to a rapid response on a medical/surgical floor for a 54-year-old woman with a history of alcohol/heroin use and remote seizures who was prescribed levetiracetam who was initially brought into the emergency department for altered mental status. Per report from her boyfriend, she “hadn't been herself” for several days. She complained of headaches, reported continued alcohol and drug use, and denied missing doses of levetiracetam. The nurse at bedside witnessed the patient have a generalized tonic-clonic seizure for about 1 minute followed by somnolence. The patient's medical history also included hypertension, diabetes mellitus, and remote history of seizure. Her medications included lisinopril, metformin, and levetiracetam 500 mg twice daily.

The initial phase of the case required participants to appropriately workup altered mental status with seizure. During the initial workup, participants were expected to gather histories for seizures and alcohol and drug use disorders. They were expected to check the patient's temperature (she was febrile) and evaluate for meningismus. If obtained, lumbar puncture showed lymphocytic pleocytosis with xanthochromia indicative of viral encephalitis. EEG showed right temporal discharges evolving into seizures. As the case progresses, the patient suffered recurrent seizures and ultimately refractory status epilepticus requiring intubation as well as escalation of antiseizure medications to anesthetic infusion for seizure control. Successful management of the case required recognition that the patient had an underlying meningoencephalitis, recognition of both convulsive seizures and nonconvulsive status epilepticus, appropriate treatment of presumed bacterial and viral meningoencephalitis, appropriate dosing of antiseizure medications and anesthetic infusions for status epilepticus, and identifying basic EEG characteristics for status epilepticus and burst suppression.

Outcomes

The primary outcome was the sum score of 25 critical action items performed by trainees only (Table 2). When actions required completion of a proximal action (for instance, administration of acyclovir required consideration of a diagnosis of viral encephalitis), participants were not credited for either action. Empirical treatment of a condition was taken as evidence of consideration of a diagnosis. The secondary outcome was a global rating scale of trainee performance ranging from 1 to 5 with 5 indicating superior performance. Critical actions were rated by 2 independent coders, M.B.P. and N.A.M. (both certified in neurology and neurocritical care) using videotaped recordings of each participant's simulation. Both coders were aware of participants' level of training.

Validity Evidence

We previously provided validity evidence for the development of the cases and checklist action items in the domains of content evidence and response process, according to Messick's framework.¹² We aimed to provide further validity evidence to support our findings in the domains of content evidence, internal structure, and relationship to other variables.¹⁷ After simulation, participants completed a questionnaire indicating how realistic they felt the scenario to be along with their level of engagement using a 7-point Likert scale ranging from “not at all” to “extremely” to provide content evidence. For internal structure, we calculated inter-rater reliability of both sum scores of critical action items and global rating scales. Regarding relationship with other variables, we assessed sum score of critical action items by level of training, training background, self-rated experience, and multichoice pretest performance assessing knowledge of general neurology, neurocritical care, and general critical care principles. We grouped the critical action items into 3 categories: diagnostic workup critical actions, assessment and interpretation critical actions, and management critical actions. Diagnostic workup critical actions included ordering intracranial imaging, reviewing complete blood count and basic metabolic panel, obtaining point of care glucose, ordering a toxicology screen and levetiracetam level, obtaining blood cultures, lumbar puncture, and EEG. Assessment critical action items included verbalizing a differential diagnosis, recognizing viral encephalitis as the most likely diagnosis based on cerebrospinal fluid results, and recognizing nonconvulsive seizures on EEG without assistance. Management critical action items included administering thiamine, appropriately dosed benzodiazepine, appropriately dosed second-line agent, dexamethasone, and empiric antibiotics and antivirals, recognizing a need for intubation at the appropriate time, preoxygenating before intubation, confirming ETT placement, and starting an anesthetic drip at the appropriate time at proper dosing and titrating the anesthetic drip until either seizure cessation or burst suppression.

We stratified participants into 4 levels to compare level of training. Level 1 included neurology subinterns and neurosurgery interns (who are not involved in consults for altered mental status and management of status epilepticus and lack any critical care training). Level 2 participants included postgraduate year (PGY)-2 neurology residents (who routinely perform consults for altered mental status, seizure, and status epilepticus at our institution), surgical critical care fellows, and medicine-trained critical care fellows (who average 1 month of neurocritical care experience). Level 3 participants included PGY-3 and PGY-4 neurology residents (who have 2 or more years of experience performing altered mental status, seizure, and status epilepticus consults as well as 2 or more months of experience in the neurocritical care unit) and emergency medicine-trained critical care fellows (who have training in the initial management of patients presenting with altered mental status, seizure, and status epilepticus and have at least 1 month experience in the neurocritical care unit). Level 4 participants included neurocritical care fellows and attending physicians in neurocritical care (who were all board-certified in neurology).

We evaluated performance on critical action items relating to airway management including preoxygenation, intubating the patient at the correct time in the clinical scenario, and assessing ETT placement using chest x-ray between participants with critical care training and those without. We hypothesized that participants with critical care training would perform better on these items. We also evaluated performance between participants with a training background in neurology and those without a training background in neurology on status epilepticus-related action items including administration of appropriately dosed benzodiazepine, administration of an appropriately dosed second-line agent, initiation of an anesthetic drip, recognition of nonconvulsive status epilepticus on continuous video EEG, and titration of the anesthetic drip to seizure control or burst suppression. We hypothesized that participants with training in neurology would perform better on status epilepticus-related action items.

Statistical Analysis

We reported descriptive statistics as mean (SD) for continuous variables and counts and frequencies for categorical variables. Based on a previous study in which we found trainees to perform 68% of critical action items,¹⁸ we felt adequately powered with a sample size of 59 trainees (1 − β = 0.8, α = 0.05). We performed an analysis of variance for univariate comparisons between participants' levels of training with post hoc comparison between training levels using the Scheffe test given unequal sample sizes between groups and the increased sensitivity of this post hoc test for comparisons involving more than 2 means. We measured effect size using the coefficient of determination, R². Using a t test, we compared performance of participants trained in critical care and those without critical care training on airway-related critical action items. We also used a t test to compare performance of participants with a background in neurology and those without a background in neurology on status epilepticus-related critical action items. Using Pearson correlation, we assessed participants' performance with self-rated experience and performance on a subset of questions focused on meningoencephalitis and status epilepticus from a written, multiple-choice test testing diagnosis and management of all neurologic emergencies as well as self-rated experience. Agreement among the raters was assessed using the intraclass correlation (ICC) with an ICC value of greater than 0.75 considered as excellent. The results were considered statistically significant if the p < 0.05.

All analyses were performed using IBM SPSS Statistics 27. The reporting format is in accordance with the guidelines established by the Strengthening the Reporting of Observational Studies in Epidemiology study and the extended guidelines for health care simulation research.^19,20

Standard Protocol Approvals, Registration, and Patient Consents

This study was approved by the University of Maryland Institutional Review Board, which waived the need for informed consent.

Data Availability Statement

On reasonable request, the data that support the findings of this study are available from the corresponding author, N.A.M.