Table 1

List of Studies With Level 3 or 4 on Kirkpatrick Framework Training Evaluation

Author	Study design	Education innovation	Comparison group	Outcomes	Results	Kirkpatrick assessment
Kelly et al.¹⁴	Investigation, interventional study	Simulation	N/A	Residents' self-perceived ability to effectively counsel patients and families before and after the exercise was compared. Trends in time to initiate IV thrombolysis for stroke were investigated before and after the intervention.	Residents' preparedness to perform counseling on acute stroke improved, though not significant (p > 0.05). Mean door-to-thrombolytic treatment for acute stroke was significantly improved in the 6 mo following simulation exercises compared with the previous 18 mo (54 vs 69 min, p = 0.01).	4
Mehta et al.¹⁵	Investigation, interventional study	Simulation	N/A	Resident performance pre- and post-intervention, critical action checklist completion, and door-to-needle time for IV-tPA.	Simulation training independently predicted reduction in door-to-needle time by 9.64 min (95% CI −15.28 to −4.01, p < 0.001) after controlling for age, night/day shift, work week vs weekend, and blood pressure at presentation.	4
Richardson et al.¹⁶	Interventional study	Simulation	Post- vs pre-simulation door-to-needle times	A mannequin was used and controlled by simulation laboratory personnel while participants were videotaped as they performed the scenario. Incorporated into the debriefing of the scenario was education of door-to-needle goals, stroke alert process, and a discussion of role expectations.	Postintervention, the door-to-needle average decreased to 39.8 min (p = 0.0794). Before the education, the goal of administering alteplase in less than 60 min occurred 80% of the time and following education the goal was met 89% of the time.	4
Dwortezky et al.¹⁷	Interventional, mixed-methods design	Simulation	N/A	Used a questionnaire assessing baseline knowledge and attitudes regarding seizure management. Calculated interobserver reliability of the checklist for consistency among the raters.	Training led to sustainable improvement in performance in the actual EMU. Knowledge in seizure management was significantly improved following the program. Interrater agreement was moderate to high.	3
Gordon et al.¹⁸	Interventional, comparative study	eLearning	Control facilities vs intervention facilities	The primary outcome variables were the percentage of residents receiving antipsychotic medications and the percentage who were physically restrained.	Residents in facilities were 75% less likely to be physically restrained and 17% less likely to be prescribed antipsychotic medication compared with residents in control facilities over the 18-mo intervention period (OR 0.25, p = 0.05) and (OR 0.83, p = 0.07)	3
Sivakumar et al.¹⁹	Interventional study	eLearning	N/A	Attainment of skill sets is tracked through self-reported case logs and professionalism. Resident and faculty surveys are conducted for feedback and system change.	Resident attendance and participation had significantly increased after the intervention.	3
Stork et al.²⁰	Investigational, pilot study	Simulation	N/A	Entrance and exit surveys were administered to measure changes in knowledge, skills, and attitudes. The use of rtPA for AIS was tracked before and after the intervention.	All trainees reported significantly increased confidence in acute stroke patient selection and management with rtPA, including its potential complications. IV rtPA use increased by 75% in the 7 mo after workshop and increase persisted at 13 mo.	3
Zaika et al.²¹	Interventional, investigational study	3D modeling, simulation	N/A	Participants were assessed on their procedural pace, coiling quantity and quality, and perforation rates. Spatial ability was assessed using a mental rotations test and used in the performance analysis	Individuals were able to perform the procedure faster after 6 sessions, reducing their average time from 42 to 24 min. The coil success rate improved from 82% to 88%, and the coil packing rate remained consistent at 30% throughout testing.	3

Abbreviations: 3D = 3 dimensional; AIS = acute ischemic stroke; EMU = epilepsy monitoring unit; N/A = not applicable; OR = odds ratio; rtPA = recombinant tPA; tPA = tissue plasminogen activator.
First author, neuroscience topic, participants, comparison group in design, study outcomes, and results of the studies who met Kirkpatrick training evaluation Level 4 (3 studies) and Level 3 (5 studies) criteria.