Curriculum Innovation: Design and Implementation of Synchronous and Asynchronous Curricula to Enhance Residents' EEG Knowledge and Experience

Synchronous EEG Course

In 2017–2018, 2 of the authors, D.J.W. and K.S., discussed proposals with the AES to develop EEG-based educational content for neurology trainees. Further discussion identified a need for both synchronous and asynchronous educational resources. The AES established a workgroup. The initial workgroup members (D.J.W., K.S., A.F., J.J.M., and R.M.G.) were identified by the course codirectors as individuals with expertise in teaching EEG to residents and consisted of 5 practicing adult and pediatric epileptologists and electroencephalographers. The workgroup was later increased to 8 members (N.A., J.P., and Victoria Wong), through informal recruitment by the codirectors and committee members, for a final total of 2 pediatric and 6 adult EEG experts. Most of the committee members were in current academic practice or had prior experience in training residents in EEG. The workgroup started meeting in January 2019 with initial steps focused on creating a broad overview of the course structure.

Online Asynchronous EEG Curriculum

The development of the online asynchronous EEG learning curriculum (the American Epilepsy Society EEG Learning Curriculum [AESELC]) started in 2020, and the full curriculum was launched in March 2021. The AESELC was designed to be stand-alone, modular, structured, and self-paced. The AESELC development process was similar to the synchronous course. The structure and content outline was developed by consensus using a similar process to the synchronous course and again based on local EEG curricula and the EEG milestones. The content was created by 20 faculty members, including workgroup members, faculty from the synchronous course, and 5 additional invited faculty with experience in EEG education.

The AESELC is intended for a wider audience. It was divided into 9 major modules. Each module has several submodules consisting of a mini-lecture, 5 multiple-choice self-assessment items, and a structured EEG (self-) assessment tool (SEAT). The mini-lectures consist of slide sets for the learner to follow without a facilitator. The MCQ items were created by the faculty member responsible for the mini-lecture and then reviewed by 3 faculty members (A.F., J.J.M., and J.P.) for accuracy and quality. Each SEAT was linked directly to the preceding submodule (e.g., a module on neonatal EEG would be followed by a neonatal EEG SEAT) and consisted of images of a 1- to 2-minute segment of EEG recording in at least 2 different montages. Each faculty member then created an “answer key” based on a standard coding sheet, using a similar process to a previously published study by 2 of the committee members (J.P. and D.J.W.).¹¹ The coding sheet included descriptions of state, background features, technical apsects, and both epileptic and nonepileptic abnormalities. Each coding sheet was subsequently reviewed by an independent committee of 3 of the authors (A.F., J.P., and D.J.W.), and the answer key was finalized by consensus. As learners progressed through each submodule, they were provided with immediate feedback on the MCQ items (including the breakdown of responses by other learners) and on how their SEAT responses matched with the answer key. An example of the SEAT format is shown in eAppendix 1 (links.lww.com/NE9/A53). In total, there are 9 courses with a total of 40 submodules with 200 MCQs, 40 lectures, and 40 SEATs.

For the online asynchronous curriculum, we used several principles of multimedia learning, including minimizing extraneous materials in the slides (Coherence principle), allowing learners to control the pace of the learning, chunking the activities into smaller segments (Segmenting principle), and including advance organizers and overviews of the curricular structure (Signaling principle).¹² For the assessment, the multiple-choice items were developed by content experts according to the best practices of the National Board of Medical Examiners.¹³ They included immediate feedback and references for further reading to engage the retrieval effect in learning.¹⁴

Program Evaluation

Our program evaluation process was designed to evaluate our ability to achieve the objectives of enhancing existing EEG training, increasing resident interest in EEG and improving participants' knowledge.

To evaluate engagement and interest, we sent postcourse surveys to each resident participant and faculty member. Respondents rated their agreement on a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree) with items that focused on course design, learning, and career/professional variables. We asked faculty to rate their agreement with items focused on the mini-lectures, small group activities, and quizzes. The survey items varied slightly from year to year, so the analysis was performed only on items that remained unchanged between years.

We used survey data to modify subsequent synchronous courses and inform the transition to the virtual course format in 2020. We reviewed the qualitative comments from faculty and resident surveys identifying recurring themes in feedback.

For knowledge assessment in the synchronous course, we used a 24-item MCQ examination as a learner assessment method precourse and postcourse. Each of the faculty members was asked to write 3 multiple-choice items related to their mini-lecture content, so that the content outline of the assessment tool matched the overall composition of the pretest and posttest. Two workgroup members edited the items and constructed the final test on Qualtrics (2019 course), which was later migrated to Survey Monkey (2020 and 2021 courses). The same test was administered precourse and postcourse. The pretest was completed before the start of the EEG course, and the posttest was administered immediately after the didactic elements of the course ended.

Evaluation of engagement and utilization for the online asynchronous EEG curriculum included analysis of number of registered users and number of users who completed the entire curriculum. As a marker of usership of the individual submodules, we tracked the number of users who completed each posttest. As a quality measure, we also calculated the difficulty index for each posttest item (provided by the learning management system; Oasis LMS, Chicago, IL). To evaluate the achievement of primary program objectives, individuals who completed the entire course completed a postcourse survey focusing on the effectiveness of the format and the appropriateness of the format for level of learner.

Statistical Analyses

All statistical calculations were performed using SPSS version 28.0.0.0 for Mac. Comparisons between pretest and posttest scores were performed using the Wilcoxon signed rank test. Comparisons of test scores between years were performed using single-factor analysis of variance. Comparisons of resident and faculty survey responses were performed using the Pearson χ² test.

Standard Protocol Approvals, Registrations, and Patient Consents

Our program evaluation methods were reviewed by a representative of the institutional review board (IRB) of Thomas Jefferson University and were determined not to represent IRB-regulated human research on the basis that the surveys and assessments focused on factual information about the learning process and did not gather additional personal information (i.e., information that is not otherwise publicly available) about living individuals. Confidentiality was maintained, and individual performance on pretests and posttests was not released.

Data Availability

Anonymized data not published within this article or in the supplemental materials will be made available to qualified investigators on request.

Resident Selection and Participation

The first in-person synchronous course was held in 2019 during the AES Annual Meeting in Baltimore, MD. The second course was delivered virtually in 2020, and the third course returned to an in-person format at the 2021 AES Annual Meeting in Chicago, IL.

Twenty resident participants were selected from 124, 95, and 130 nominations in 2019, 2020, and 2021, respectively. Of the 20 chosen residents, 20, 17, and 19 attended in 2019, 2020, and 2021, respectively (with 1 person attending only day 2 in 2019).

For the virtual course in 2020, the total duration of prerecorded lectures was 250 minutes, but the median time participants watched the content was 122.2 minutes (ranging from 2.4 to 247.2 minutes). Some residents also participated via mobile devices, limiting the adequate review of EEG samples.

Knowledge Assessment

The results of pretests and posttests are shown in Figure 1. Among participants over all 3 years, there was an increase in the median score from 60% (14.5/24) to 75% (18/24; p < 0.001). There was a significant increase in scores after course completion for the in-person courses in 2019 and 2021, but no significant change for the 2020 virtual course. There was no significant difference in pretest or posttest scores between years of administration of the course.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1 Box and Whisker Plot Indicating Participant Scores on the 24-Item Pretests and Posttests for the Synchronous Resident EEG Course, 2019–2021

The horizontal lines inside each box indicate the median scores for each iteration of the test, and the whiskers indicate the full range of scores. p Value was calculated using a Wilcoxon signed rank test. Only participants who had complete data for both the pretest and posttest were included in the analysis.

Survey Evaluation of Effectiveness of the Courses

The results of the resident postcourse survey are summarized in Table 2. The completion rate of the resident survey was 65% (13/20) in 2019, 76% (13/17) in 2020, and 84% (16/19) in 2021. Regarding our primary outcomes, most of the residents agreed that the course was a helpful addition to EEG training at the home institution and increased interest in EEG. Most of the respondents also agreed or strongly agreed that the course was well organized, had a good mixture of activities, and would be recommended to others. The in-person courses (2019 and 2021) had higher overall ratings than the virtual course (2020) for most elements of course design and learning outcomes.

The results of the faculty postcourse survey are summarized in Table 3. The completion rate was 80% (12/15) in 2019, 73% (11/15) in 2020, and 60% in (9/15) in 2021. The ratings for the in-person courses were overall higher than those for the virtual course. These differences reached statistical significance for the categories of the mixture of activities, level of engagement, and the number of small group activities.

We found several recurring themes from the qualitative comments in the resident and faculty surveys (Table 4), including approval of the overall course design, especially the pacing and balance of short lectures with interactive small group activities. Participants also highlighted the benefit of networking opportunities. The participants noted that the virtual format did not work as well as the in-person format, including factors such as the assignment of videos for the mini-lectures not being as conducive to learning as the integration of mini-lectures during the in-person course.

Asynchronous Curriculum Evaluation

As of August 2023, 5,880 individuals had registered for the AESELC. The largest groups of registrants were residents/fellows (24%) and neurologists (21%). Other participants included students, other types of physicians, nurse practitioners, physician assistants, and EEG technologists. Thirteen percent of users did not indicate an occupation or level of training.

As a measure of engagement and utilization for the asynchronous curriculum, Figure 2 shows the number of users who had completed the posttest for each of the individual submodules, grouped by course. More than 2,300 individuals completed the first submodule, but this number dropped to approximately 1,000 by the end of the first course. Each successive course had a lower number of individuals completing the first submodule, and through each of courses 2 through 5, retention of learners typically dropped by half. Learner retention was better in the later courses, but only 200–300 learners started each of these courses. The mean difficulty index (percent correct) for the posttest items was 0.74 (range 0.14–0.99), indicating good overall performance on the posttests.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2 Number of Users Who Completed the Posttest for Each Submodule in the American Epilepsy Society EEG Learning Curriculum

The submodules are grouped by course number, and the courses are organized in the order they appear in the curriculum.

A total of 55 individuals who completed the entirety of the course by April 2022 were asked to complete a postcourse survey. The respondents generally agreed the format was effective for the presentation of the EEG material (86% agree or strongly agree). The respondents agreed the course was most appropriate for the level of a resident (29, 53%) or fellow (16, 29%). As a proxy for improved knowledge, respondents overwhelmingly agreed that the course improved their ability to identify various components of the EEG from 78% (coma patterns) to 95% (focal slowing and spikes and sharp waves). Most (78%) agreed that the AESELC was as effective or more effective than other EEG learning modalities they had used and that they would recommend this course to others (91%). According to the unstructured comments, strengths included the accessibility of the course, the comprehensive nature of the course, and the novel SEAT. Weaknesses included a desire for the lectures to have an accompanying voice-over and a concern that the course may be difficult for individuals without any EEG experience. When asked for suggestions for improvement, 36 of 55 (66%) said nothing to improve, and the most common recommendation was to add voice-overs for the lectures (n = 5; 9%).

Synchronous Courses

The AES Resident EEG course was perceived by residents to be well designed and organized, appropriate to training level and a helpful addition to EEG training at residents' home institution. Individual components of the course, including the mini-lectures, small group activities, and pretests and posttests were perceived to be well balanced, of high quality, and relevant to learning. There was a modest but significant improvement in performance on an EEG knowledge assessment for the 2 in-person iterations of the course, but not for the virtual version. Overall, resident and faculty responses for the in-person versions of the EEG course were more positive than those for the virtual version. The face-to-face interaction during the in-person course also facilitated networking and further learning during informal discussions, and these opportunities were highly rated by both faculty and residents.

The virtual EEG course in 2020 still provided some immersion and engagement. A potential benefit of the virtual format is the ability to present to a larger audience because there are no travel costs and there was more flexibility for scheduling. However, despite the higher costs, the in-person format was preferred in relation to all major objectives, including learning, engagement, and networking. There may be several reasons for these findings. Residents may have had less protected time to participate and engage in these sessions, including potential increased local distractions and competing responsibilities. In addition, this course took place less than a year into the COVID-19 pandemic, at a time of substantial disruption in the clinical experiences of residents. It is possible that the competing personal and professional demands of the pandemic introduced distractions unrelated to the virtual format. Many residents did not watch the recorded lectures before the small group activities, and this may have contributed to the perceptions among residents and faculty of a lower level of preparation and engagement. While all residents were new to the course during the virtual year, the faculty had all experienced the synchronous course the year earlier, allowing them to form relationships with the faculty group and deliver the material in person at least once. This may have contributed to the fact that faculty members had an overall more positive impression of the virtual synchronous course than the resident participants.

Online Asynchronous Curriculum

Although different from synchronous EEG courses, the online curriculum provides wider self-directed access to EEG content. Limitations include the lack of engagement and networking opportunities compared with the synchronous format. While many users completed the first 1–2 courses, only a small percentage completed the entire curriculum. This low long-term retention may suggest that the linear structure of the course is a weakness. There may be a missed opportunity to engage learners in “core” elements of EEG early in the course, leaving some technical aspects (e.g., history of EEG, physiology, and technology) to later in the course, for more committed learners. In addition, it could be possible to create “tracks” based on learner type (e.g., pediatric vs adult neurology residents, technologists, students, etc).

The development and implementation of the AESELC had unique challenges. The biggest challenge has been technologic. Unlike the synchronous course, there is no readily available mechanism for the online delivery of electronic EEG recordings for dynamic review and interpretation. The “mini-lectures” in the AESELC consist of static slides with some animation but without voice-over. This was chosen to ensure the self-navigable structure, but at the same time, it compromises some aspects of effective multimedia learning. More sophisticated learning management systems may offer better delivery mechanisms. The variability in styles of interpretation of EEG was also a challenge in the development of the SEAT. The reference interpretation used in the curriculum was achieved by consensus, but that may not always be possible for all elements of EEG.

Assessment and Program Evaluation

There are inherent limitations in the pretest/posttest format of learner assessment.¹⁵ The pretests and posttests were identical, so some of the improvement could be attributed to the effects of repeated retrieval rather than learning, although the learners did not get any feedback after completing the pretest. In addition, the EEG images in the test were static and are much simpler than the range of findings that are encountered in actual practice. However, learner assessment was not the principal objective of the EEG courses. As outlined earlier, our goals also included providing tools to enhance EEG training that occurs in residents' home programs and to develop curricula that are engaging and used by a broad range of learners. The need for more detailed outcome assessment instruments was identified with the experience gathered with the design and implementation of the EEG courses. Challenges include the need for more substantial resources to collect and analyze the data of more complex instruments of learner assessment, including longitudinal follow-up. From the program evaluation perspective, information on how the learner uses the knowledge obtained in the course would provide valuable data to modify the course to meet learners' needs better and understand the different contexts of EEG teaching and learning across various institutions.

EEG Courses in the Context of Residency EEG Education

Currently, each residency program must provide its own EEG teaching, leading to a high degree of variability in experiences. Digital tools can allow for sharing of learning resources between institutions (including those developed for the EEG course), reducing the amount of effort that local faculty have to spend on the most fundamental concepts so that local teaching time is engaged in helping residents hone their skills in an authentic clinical setting.¹⁶ Based on our experiences, the small group sessions were essential to the learning process and were very well received by both faculty and residents in postcourse surveys. This highlights an opportunity to integrate these instructional methods into resident EEG learning experiences at home institutions. These experiences were also the seed of the interactive components of the online asynchronous EEG curriculum, including a short quiz and a SEAT at the end of each module to replicate the immediate feedback that occurred during the synchronous course.

No course will be able to replace hands-on EEG reading.¹⁷ It will be up to the residency program and the local experts at each site to partner with the learner to apply the knowledge gained in the course to the real-world clinical setting. As such, developing a standardized national course does not replace the need for local experts nor does it eliminate practice variability.

Future Directions

We have shown evidence of the feasibility, effectiveness, and acceptability of tools to augment EEG learning at home institutions, and our goal is to ensure that these tools are used by the broadest range of learners. For the synchronous course, the in-person iterations were more effective than the virtual version, but they are resource-intensive and will always have a limited scope. We plan to continue the in-person course, expanding attendance to 40 residents per year and focusing on individuals with an interest in epilepsy and EEG. In this way, we can leverage the effectiveness of this course in increasing interest in EEG/epilepsy careers and providing networking opportunities for high-interest learners.

The online asynchronous EEG curriculum has broader reach, but our analysis shows less engagement, with few individuals completing every module. Early modules are highly used, and there is an opportunity to provide a road map for resident use of this course, with designated core and optional components and “tracks” for different learners. We could also educate residency directors on how to integrate these modules into their formal curricula, possibly providing residents with protected time to complete them. Similar issues with course completion have been seen in other online asynchronous curricula for neurology residents, although because of a self-paced curriculum, learners will complete components at all times of day and at vastly different rates; some completing all components in a few hours while others spread the experience over several months.¹⁸ The asynchronous curriculum is simply a tool and could be better used with the engagement of stakeholders in resident EEG education at individual institutions. Ultimately, there is the opportunity to curate this and other resources into a set of consensus-based didactics and proposed rotation structures for resident EEG education.