Effectiveness of a virtual program for OSCE preparation during COVID-19: a descriptive and repeated cross-sectional study among nursing students

The OSCE is an accepted strategy for assessing clinical competencies of nursing students. Harden et al. [14] described the strategy as series of stations, in each of which the students are asked to carry out a procedure which involved demonstration various clinical competencies. OSCE benefits include greater objectivity, increased consistency of experience between students, reduced risk of examiner bias, a broader range of clinical skills tested, a high level of reliability and validity, and increased motivation for learning [20, 29]. At the same time, some limitations of the OSCE method are also mentioned in the literature, such as students’ stress, which could adversely affect their performance,complexity of the process; faculty time; high cost; difficulty in ensuring the confidentiality of the stations between different cohort of student, etc. [29]. There is a consensus among healthcare educators that it is essential to preserve OSCE during social distancing situations such as COVID-19, by adopting new performance strategies.

COVID-19 restrictions posed a major challenge for faculty to preserve OSCE and supply students with adequate preparation for the clinical examination. While some academic centers have decided to cancel the OSCE during the pandemic [30], other nursing schools reported various solutions that enable them to adjust to the COVID-19 restrictions. In several centers, where students were allowed to be physically engaged in clinical education, an in-person examination was conducted under strict limitations (e.g., [31]. Where in-person meetings were impossible, some nursing educational institutions replaced OSCE with a virtual examination [4, 11, 23].

Educators who have conducted virtual OSCEs find the Zoom application to be a suitable platform for implementation, especially the breakout-rooms feature, which creates a safe learning environment for examination and feedback [18]. Major et al. [24] reported that users considered it a promising strategy, which can be turned into telemedicine. Others call to consider the use of remote platforms for clinical training, even post‐pandemic [4, 23].

Upon reviewing results of virtual OSCEs described in the literature, students and examiners expressed overall satisfaction with the method [23], and most of the improvements needed were to overcome technical difficulties [18]. Regarding the effectiveness of the virtual exams, Arrogante et al. [4, 23] noted that no significant differences were found between scores of traditional OSCEs conducted in recent years and those of virtual OSCEs performed thus far during the COVID-19 pandemic. However, they reported difficulties in assessing clinical hands-on skills. A notable disadvantage mentioned was that students cannot practice manual skills during the test andwere required to describe their clinical skills by verbalizing to faculty, i.e., what they would perform if the treatment was in-person.

In summary, there are several gaps in the knowledge and understanding regarding strategies to maintain hands-on virtual OSCE, as well as its effectiveness in assessing clinical practices. As the OSCE usually assesses students' clinical skills proficiency, it is essential to find ways that will assist faculty to evaluate actual clinical performance via virtual examination. Reports on virtual OSCEs also highlight the importance of the preparation phase for both students and staff [24].

The OSCE format used in the Department of Nursing at Ben-Gurion University of the Negev, is a “Multi station OSCE” [29], and is used as a final exam in the "Critical Clinical Thinking and the Nursing Process" course, part of the second-year baccalaureate curriculum. There are usually six OSCE stations, each taking 10–12 min, with a 2-min break between them. The clinical skills evaluated are taking medical history, nursing assessment for a holistic approach, clinical-manual skills, patient-education skills, critical thinking, and clinical decision-making principles. All stations are directly observed by faculty members of the Department of Nursing, who are also part of the teaching team. Evaluation during the exam is based on binary structured checklists and a global rating scale [20]. Every year, as a part of the students’ preparation for the OSCE, an in-person preparation program with two clinical stations is conducted. This program aims to demonstrate an OSCE station and to enable students to experience this test.

The OSCE preparation virtual program was designed for second-year baccalaureate nursing students, based on the INACSL Standards of Best Practice: Simulation^SM [15]. The program was conducted on January 18, 2021, during a total lockdown imposed by the pandemic. Due to the lockdown restrictions, and in accordance with the university lockdown rules, students were not allowed to come to the campus for the preparation program, but were required to take the final OSCE in-person. To meet these restrictions, we designed a similar virtual OSCE preparation program via Zoom (instead of the traditional in-person preparation program), which was directed to prepare students for the in-person OSCE.

Program design and procedure:

Two scenario-based OSCE stations were developed and designed to be as similar as possible to the real (hands-on) exam. Appropriate checklists were adjusted for enabling the evaluation of students' performance via Zoom. Table 1 presents the scenario details of each station.

To enable the trial of clinical skills, an equipment kit had been organized and delivered to the students a week before the program began. The kit included syringes, needles, tubes, a fluid bag, IV connection set, micro enema, water for injection (WFI) ampoules, and disposable protective equipment (e.g., gloves and apron). The students were instructed to keep the equipment kit closed until the program began. For practice needs, students were instructed to prepare a disposable drinking bottle with a narrow mouth, which was used to represent the patient.

A few days before the program began, preparatory Zoom meetings were held to inform students and teachers of the program schedules, goals, and methods, and instruct them on how they should prepare. Students and teachers received the schedule, and Zoom links for each of the stations were sent following the preparatory meeting.

Each student practiced at two virtual OSCE stations via Zoom. Each station session lasted 10 min. The student entered the first Zoom meeting, where a teacher was waiting for her/him. At the beginning of each session, the scenario was presented to the student. During the time at the station, the student was required to collect relevant clinical information from the patient’s medical records, assess the patient, decide on the appropriate care, present the patient to the doctor and ask for a medical order, perform the medical order (a specific clinical procedure), and evaluate the intervention results. The teacher shared all the required materials on the screen. The student practiced manual skills on the disposable drinking bottle using the equipment kit. After 10 min, the teacher announced the end of the session and gave the student a short debriefing. Then, the student left the first station Zoom meeting and proceeded to the second station, in which another case was presented. Figure 1 illustrates a student's path through the sessions.

Using a structured checklist, a teacher evaluated the student's performance at each station. As this was a preparatory program for OSCE, students were given a 10-min personal debriefing at the end of each station by the teacher who observed their performance. The debriefing reflected what went well during the session, and what skills require additional training. Following the virtual sessions, students' satisfaction was evaluated using quantitative and qualitative methods.

The study leans on Kirkpatrick and Kirkpatrick's [22] four-level evaluation model for educational learning outcomes. This model is commonly used for evaluating the effectiveness of health professionals’ educational programs, and especially useful in helping evaluators identify learner outcomes [12]. In this study, we assessed the virtual program's effectiveness through the first two levels of Kirkpatrick and Kirkpatrick's model [22]: Level – evaluation of participants’ reaction (in terms of participants' satisfaction to the educational program,Level 2 – evaluation of the learning generated by the program, demonstrating the influence of the educational program on participants’ knowledge, attitudes, and/or skills.

The study aims were (1) to assess nursing students’ satisfaction with the virtual OSCE preparation program, and (2) to explore the learning outcomes of the virtual program by comparing the OSCE scores obtained following virtual preparation to those obtained following in-person preparation. According to this theoretical model we hypothesized that:

The study used a descriptive and repeated cross-sectional design.

Participants were second-year baccalaureate nursing students in the Department of Nursing at Ben-Gurion University of the Negev between 2017 and 2021. Inclusion criteria were participation in the virtual OSCE preparation program conducted in 2021, or in the in-person OSCE preparation in 2017–2020.

Participants’ satisfaction with the virtual preparation program was measured among students who had participated in the program (n = 82), using a single item (as suggested by [5] from the general evaluation survey for the course: "I am satisfied with the virtual OSCE preparation program.” Answers were rated on a 4-point Likert scale (1 – strongly disagree 2 – disagree, 3 – agree, 4 – strongly agree). Additionally, an open-ended question gave participants the opportunity to express their subjective, personal experience with the program.

In order to evaluate learning outcomes, we compared the OSCE scores following the virtual program in 2021 to those achieved following in-person preparation in 2017–2020 (N = 419). The score of an OSCE station is based on binary structured checklists, including about 30 expected clinical behaviors to be performed at each station, followed by a global rating scale, assessing the quality of student’s performance [20]. Calculating an OSCE station score is based on combining the results of the binary structured checklist, 90%; and the global-rating scale, 10%. The final OSCE score is calculated as the mean score of all OSCE stations. We compared the final OSCE scores means, and the mean scores of four selected OSCE stations that were found to be similar all over the study duration: patient education, IV administration, respiratory assessment, and pain management. Content validity of the checklists was measured by at least two faculty members, who are experts in fundamental of nursing education. Demographic information for each year was retrieved from the administrative unit at the Department of Nursing.

The study was approved by the Head of the Nursing Department, and by the Institutional Review Board at the Faculty of Health Sciences of Ben-Gurion University of the Negev (request #35–2021).

Analyses were conducted using IBM Statistic SPSS software version 26. Descriptive statistics were calculated to explore participants' profiles and study scores. Chi-square tests examined differences among students. A multivariate analysis of variance (MANOVA) with Scheffe post-hoc analysis was performed to compare the effect of the pre-pandemic and during pandemic final OSCE scores. Analysis of the open-ended question was used to strengthen descriptive statistics regarding participants’ satisfaction.

The study was conducted on 419 s-year baccalaureate nursing students between 2017 and 2021. Table 2 presents the age mean and standard deviation, and the gender frequency of participants over the study period. No significant differences in students' characteristics were found over the years.

Evaluation of students’ satisfaction from the virtual preparation program revealed that 79 (96.3%) students ranked their satisfaction with the virtual program. Most of them were strongly satisfied or satisfied with the virtual program (n = 49 (62%) and n = 21 (26.5%), respectively), while only nine students (11.5%) were not satisfied. Students' reflections in the open question were generally positive, and expressed high satisfaction from the program, for example:

The practice was just great! Very good teaching and preparation, and I came out with a great feeling.… The attention to small details, the preparations, the equipment, the professional guidance, the educational level. I feel much better prepared for the exam. There really are no words to say except a big thank you!

The kits that were delivered, and the training that was performed (even though it was via Zoom)… it was evident that a lot of thinking and a lot of effort was invested despite all the COVID-19 restrictions.

I wanted to say thank you for your creative thinking and willingness to help and find creative solutions during this complex period.

Evaluation of the learning outcomes of the virtual program as reflected in the OSCE scores revealed that as hypothesized, no significant differences were found in the OSCE scores following the virtual preparation program conducted in 2021 and the scores following the traditional in-person programs conducted in 2017–2020. Figure 2 presents the final OSCE scores over the years. As depicted, no differences were found between scores in 2018–2021. The only significant difference was between the score of 2017 and the scores achieved in all other years of the study (F (4, 414) = 36.666, p < 0.001).

Exploring the scores of the four selected stations, revealed un-specific trends. The results show a significant effect of the year of study, V = 0.825, F(20, 1652) = 21.467, p < 0.001.. However, post-hoc analysis revealed that these differences did not rise specifically following the virtual preparation program. Indeed, the 2021 scores were found to be similar to other scores during the study. Figure 3 presents the mean score prevalence of the four stations over the years. For example, in the IV administration station, students achieved the highest scores in 2021 similarly to 2020 (F (4, 414) = 64.679, p < 0.001), while in the patient education station, the 2021 scores showed no significant differences from those of 2020, 2018, and 2017. The significant difference in this station stems from the high score in 2019 (F (4, 414) = 8.398, p < 0.001). Detailed OSCE scores with MANOVA results are presented in Table 3.

The assessment of students’ satisfaction is crucial in integrating new strategies into the nursing curriculum. Trainees’ satisfaction is the basic stage of program-evaluation models [22], enabling feedback for the program developers. Furthermore, Doménech-Betoret et al. [9] highlighted that academic satisfaction and performance are positively associated with student’s self-efficacy, which is essential for academic success. Focusing on the learning methodology, Kim and Park [21] showed that nursing students’ satisfaction with distance e-learning mediated the influence of learning flow and learning outcomes (defined as the final goal when evaluating education). Students who are satisfied with the learning program may be better engaged in learning and show greater involvement [19]. Creating new educational programs during the pandemic in a short time is challenging, and evaluating students’ satisfaction is warranted, especially if schools are considering using virtual programs beyond the pandemic.

Evaluating students’ satisfaction is also essential for the faculty, as greater satisfaction may be associated with higher trust, thus leading to better relationship between students and faculty [26]. Furthermore, while students’ negative feedback on a new educational program generated negative feelings among faculty (e.g., disappointment and frustration), students’ positive feedback that reflects great satisfaction may generate positive feelings thus enhancing faculty motivation and commitment to create more innovative programs that meet students’ needs.

In our nursing department, the “Multi-station OSCE” method [29] has been used since 2017. The results of this study provide a perspective for the entire study period, enabling an analysis of learning outcomes reflected by OSCE performance. The present study is consistent with other studies conducted during COVID-19 [4, 17, 23, 27], emphasizing the challenges in maintaining clinical practice and evaluation – the core of nursing education [25]. The described virtual preparation program was unique in that it succeeded in preparing students for an in-person OSCE. In addition, our findings are corresponding with other studies that found no significance differences between virtual and in person the nursing education (e.g. [37].

Thus, we suggest that virtual clinical learning may be integrated into traditional programs, without harming students’ competencies. Although the virtual preparation program was developed specifically for the COVID-19 lockdown, following this study, similar programs could be designed and be used in the nursing curricula post-pandemic both to address other situations of limited access to clinical practice, and to expand students’ possibilities to acquire and improve competencies during their studies.

Although this virtual preparation program was developed for solving the problems imposed by the lockdowns during the pandemic, virtual programs may have broad implications for the nursing and medical educational system: First, virtual teaching has high availability, time, and location flexibility [8], with an increased potential of strengthening students’ clinical competencies [38]. Second, considering the shortage of clinical training resources [33], virtual programs may partially replace clinical rotation’s hour requirements. Hence, further evaluation of the effectiveness of this replacement is needed. Third, these alternative teaching methods enable the continuity of clinical teaching and practicing when the primary mode is not available (for any reason), thus possibly strengthening the flexibility and resilience of the nursing educational system [30]. Fourth, virtual clinical practices enable students to acquire new skills of using technology in the nursing arena. As telehealth became highly common during the pandemic and is expected to thrive in the future [16], additional learning outcome of the virtual training may be that students learn how to deliver healthcare on a virtual platform effectively (e.g., patient-nurse communication, patient teaching and consultation). Following former studies, advancing the integration of clinical simulation requires supportive policy, and could not be achieved by individual efforts [1]. Park and Yu [28] highlighted that formal standards are needed to improve the overall flow of nursing education, including exposure to various methods' applications, training the trainers and the importance of trained nursing faculty.

The results presented in this study have several limitations. First, this study was conducted in one nursing educational center; further studies may expand the study sampling. In addition, sample size was not determined since sampling was based on a retrospective data. Second, OSCE scores may be influenced by students and faculty characteristics that were not considered in this study. However, the duration of this study decreases this possible bias. Third, this study measured the general satisfaction via a single item used to assess students’ satisfaction as suggested by Assunção & Pimenta [5]. Further studies may focus on other satisfaction domains.