Introduction
Most health professional pre-registration courses now make use of simulation-based education (SBE) as an evidence-based educational method to practise clinical skills [1‐3]. Clinical simulation environments are immersive and multisensory, promoting the development of psychomotor skills and executive functioning [1]. Simulation became particularly valuable as an education method for meeting the challenges posed by the need for virtual learning during the COVID 19 pandemic [4]. Traditional simulations are highly successful but currently not scalable due to increasing demand, student numbers and access limitations [5]. Few students can actively participate in an immersive simulation, with the majority being inactive observers [5]. The ability to provide increased opportunities for all students to be involved in authentic work-based learning experiences is essential for students to become effective and competent practitioners.
Virtual Reality (VR) technology is an innovative and emerging technology that is increasingly being used in health professions education. VR simulations allow for active learning experiences that are interactive, authentic, standardised, and safe [6‐8]. Three-dimensional VR simulations where there is a perception of being physically present in the virtual world [9, 10], are seen as being highly immersive [11]. Wearing VR glasses or head mounted displays, adds to the visceral feeling of being in the simulated world [10]. Once the VR scenarios have been created, they can be re-used multiple times for a variety of learners [6], and can be accessed by students anywhere at any time [7]. They allow students to experiment with different outcomes, practise the scenarios, and experience elevated risk events without compromising safety [7]. Limitations to VR include lack of immersion and realism [12]. VR scenarios can also be expensive to develop and time-intensive [6, 7, 13]. VR requires a fast and reliable internet service, computer skills, technology resources and appropriate training and orientation prior to commencement [7].
Anzeige
Whilst there is some evidence for the use of VR simulation (VRS) in student nursing education to increase students’ knowledge and positive perceptions of their learning experience [8, 14‐16], there is limited literature on the economic viability of VRS. JasperVR was a fully immersive virtual reality education program, developed through a collaborative consortium, that immersed students’ senses (vision, hearing, and motion) in a 3D world for a range of common clinical scenarios. By using the VR headset and by means of gaze control, students were able to make decisions that in turn determined the future trajectory of each patient they interacted with.
The aim of this study was to evaluate learner outcomes of traditional SBE compared to a series of fully immersive VRS scenarios, for vocational and higher education nursing students at a training and further education institute in Melbourne, Australia.
Research questions:
1.
What was the impact of VRS compared to SBE on students’ knowledge of, confidence in, and motivation to learn about managing common clinical conditions?
2.
What are the reactions of students in relation to the usability, efficiency, and effectiveness of VRS?
3.
What is the impact of VRS compared to SBE on the number of nursing students with an immersive simulation learning experience?
4.
Is the use of JasperVR a feasible and economically viable education method?
Anzeige
Materials and methods
Using a mixed methods quasi-experimental design, this study compared educational outcomes from traditional simulation using simulated participants with VRS. Ethical approval was obtained from Monash University Human Research Ethics Committee, Project ID: 19235. Written informed consent was obtained from the participants of the study.
Materials
JasperVR was a uniquely developed VR-based education platform called VirtualU, that was developed through a collaborative consortium. Using 360-degree video and sound technology, the JasperVR VirtualU software application captured the variations and potential outcomes of a range of common clinical scenarios using simulated participants. By using the VR headset and by means of gaze control, students were able to select pre-determined actions, that in turn determined the future trajectory of each patient they interacted with. Feedback on patient outcomes following choices made by the student, was provided via visualisation of the different pathways. Debriefing and discussion took place after each simulation.
Participants
A representative sample of consenting students enrolled in the Bachelor and Diploma of Nursing courses at a Training and Further Education Institute between July 2019 and June 2020, participated in the study. Consenting students consisted of 7 cohorts (Table 1). Students were pre-allocated to tutorial groups (n = 5–6), which were randomly assigned by an external organisation, to either the control or intervention groups. Students who did not consent to participate in the research remained in a non-intervention group and received standard teaching.
Facilitators
For the duration of the research a core group of experienced facilitators (n = 3) were used for both the control and intervention groups. Facilitators received a 1-hour training session on how to debrief and were provided with a debriefing guide. Debriefing was based on the Promoting Excellence and Reflective Learning in Simulation (PEARLS) framework [17].
Simulation scenarios (VRS and traditional simulation)
Module 1: The verbally aggressive patient, Module 2: The deteriorating patient, Module 3: The patient with cognitive impairment, Module 4: Palliative and end of life care.
Interventions
Control group (traditional simulation)
students participated in standard teaching activities in the Diploma and Bachelor of Nursing, including lectures (large group), tutorials (small group), clinical skills laboratories, role-plays, and four face to face large group immersive simulation scenarios using simulated participants, in the Simulation Centre.
Intervention group (VRS)
students participated in the same standard teaching activities as the control group but received the four JasperVR modules as an alternative to traditional face to face large group immersive simulation scenarios.
Implementation
Healthcare simulation standards of best practice were utilised for both the intervention and control scenarios [18]. Whilst, pre-briefing and debriefing was delivered separately to the control and intervention groups, the intervention and control groups received the same pre-briefing and debriefing according to the PEARLS framework [17].
Pre-briefing for each module
An experienced and trained facilitator provided an overview and briefing to control or intervention students on the module’s content, key learning points and its application to clinical practice.
Debriefing
Control or intervention students received a scheduled debriefing session for each module from one of the experienced facilitators. The debrief provided an opportunity for students to explore the content in more depth, discuss important clinical issues, ask questions, and clarify important concepts.
Delivery of VRS
Intervention students were provided with a JasperVR handbook and VR headset. At the commencement of the first session students downloaded the required mobile software application, VirtualU, onto their mobile phone, using the education institute’s Wi-Fi. Students were provided technical assistance from the project managers for the duration of the research study.
Each VRS module had two options:
a.
Free roam: available within or external to classroom time, allowed students to navigate their way through the module scenarios, make decisions about clinical practice and reflect on critical issues that arose. They were able to repeatedly view the scenarios in preparation for clinical placement.
b.
Mastery videos: embedded into the program and included videos of clinical experts performing a range of skills to demonstrate exemplar performance of best practice. Students were able to identify key features of a good performance to help prepare them for clinical placement. In addition to offering students the opportunity to view the mastery videos in their own time, teachers could select to use this option in class to highlight aspects of high-quality performance.
Data collection
Multiple methods of data collection were used (Table 1).
Table 1
Protocol for between group comparisons on specific modules
Stage | Comparison Group | Intervention Group |
|---|---|---|
Pre – Teaching | Student Survey 1 (Pre-test) • Student characteristics • Knowledge test • Knowledge scale • Motivation to learn scale • Self-efficacy confidence scale | Student Survey 1 (Pre-test) • Student characteristics • Knowledge test • Knowledge scale • Motivation to learn scale • Self-efficacy confidence scale |
Semester | Traditional teaching including simulation | Traditional teaching plus JasperVR Software analytics |
Post Teaching (Final Week of Semester) | Survey 2 (Post-test 1) • Knowledge test • Knowledge scale • Motivation to learn scale • Preparation for clinical placement scale • Self-efficacy confidence scale • Views and ratings on the simulation learning experience • Clinical assessment | Survey 2 (Post-test 1) • Knowledge test • Knowledge scale • Motivation to learn scale • Self-efficacy confidence scale • Preparation for clinical placement scale • Views and ratings on the JasperVR learning experience • System Usability Scale • Clinical assessment |
Focus Group Interview | Focus group interview OR Individual telephone interviews | |
Post Clinical Placement | Survey 3 (Post-test 1) • Knowledge test • Knowledge scale • Motivation to learn scale • Preparation for clinical work scale • Self-efficacy confidence scale • Views on clinical placement experience | Survey 3 (Post-test 1) • Knowledge test • Knowledge scale • Motivation to learn scale • Preparation for clinical work scale • Self-efficacy confidence scale • Views on clinical placement experience |
Economic Evaluation | End of Semester | |
Anzeige
Surveys
The study was based on a pre and post-test design, elaborated through a mixed methods research approach. Data was collected via surveys at multiple time points.
Pre-test (Survey 1): Week 2–3 of semester prior to simulation week.Post-test 1 (Survey 2): Final week of semester.Post-test 2 (Survey 3): Post clinical placement.Students completed surveys either online or paper based. Pre-test surveys included quantitative measures of knowledge (multiple choice questions), a self-reported knowledge scale (7-point Likert scale from ‘not at all knowledgeable’ to ‘extremely knowledgeable’), a motivation to learn scale (7-point Likert scale from ‘not at all motivated’ to ‘extremely motivated’), and a self-efficacy in learning scale. The self-efficacy in learning scale consisted of a range of 10–13 items with students indicating their level of confidence to perform skills for dealing with patients for each of the modules on a 5-point Likert scale from ‘not at all confident’ to ‘extremely confident’. Questions were based on the learning objectives for the Bachelor and Diploma of Nursing curriculum in relation to each topic. The items were designed collaboratively between researchers and the faculty teaching teams. They were checked for authenticity and content validity through peer review. Post-test surveys repeated these measures and added views and ratings on the learning experience (simulation or virtual reality). The items for the views on the learning experience were based on McCausland et al’s (2004) survey evaluating student experiences of simulation in undergraduate nursing [19] and used a 5-point Likert-scale from ‘strongly disagree’ to ‘strongly agree’. Post-test surveys for the VR intervention students also included the System Usability Scale (SUS). The SUS is a 10-item scale giving a global view of subjective assessments of usability using a 5-point Likert Scale [20]. The SUS has demonstrated robust psychometric properties and measured the overall usability of the VirtualU software application.
Focus group interviews
semi-structured interviews were conducted with intervention students that focused on the students’ views about their general learning experiences with JasperVR, the lessons learnt and what could be improved.
Student clinical assessment
Students were asked to consent to using a de-identified course assessment of an Objective Structured Clinical Observation (OSCE), which they undertook at the conclusion of each semester, as a measure of the impact on clinical learning outcomes on actual performance related to specific clinical content of each module. These assessments were undertaken with simulated participants and assessments were directly aligned to the specific module content.
Economic evaluation
a cost benefit analysis was conducted to compare the cost and benefits of JasperVR with traditional SBE. The cost of immersive, mannequin and simulated patient-based simulation was calculated per student and a comparison made.
Data analysis
Quantitative data
Using SPSS (IBM SPSS Statistics for Windows, Version 21.0), descriptive statistics identified demographic data. The chi-squared test was used to ensure that the demographics of the intervention and control groups were sufficiently equivalent. The remainder of the quantitative analysis used inferential statistics to test for systematic differences between outcomes for the control and intervention groups, and systematic differences within the intervention groups across the three time points (for 2019 groups). This was based on the application of several quantitative instruments. Statistical controls were performed using a two-way repeated measures ANOVA.
Qualitative data
Qualitative data from the open-ended questions in the surveys were transcribed verbatim. Quantitative data were analysed and reported thematically. Themes reflected intervention participants’ accounts of their learning experience with VRS and characterised perceptions that were relevant to the research questions [21, 22]. One researcher developed descriptive codes of the responses to each of the open-ended questions on the surveys. A different researcher reviewed the initial descriptive codes, and the two researchers discussed and finalised the thematic framework for analysis. Data management and analysis were assisted by MS Excel.
Results
The overall cohort for the comparison study comprised 675 students, from the aggregation of seven distinct teaching cohorts (Cohorts 1 to 7). Due to the COVID-19 global pandemic in 2020, the implementation of control groups and cross over groups was not able to be achieved and all students were allocated into intervention groups to receive the JasperVR learning experience. This was due to all simulations being cancelled in Semester 1, 2020. Furthermore, it was not possible to implement any Survey 3s and OSCEs in 2020 due to the overwhelming number of cancellations of clinical placements and clinical assessments for students. This resulted in 282 students in the traditional simulation control group and 393 in the VR intervention group. Table 2 describes the control and intervention groups resolved in terms of: Cohorts (1 to 7); course of study (BN or DN); Age Range (18–25 years and 26–50 + years); and Gender.
Table 2
Demographics of the overall cohort. The Pearson Chi-Square test indicated no statistically significant difference between the Control and Intervention groups based on: contribution from the different teaching cohorts (c2 = 10.4, df = 6, p = 0.108), course of study (c2 = 3.22, df = 1, p = 0.073), student status (c2 = 0.042, df = 1, p = 0.837), distribution of the age ranges (c2 = 0.043, df = 1, p = 0.836), or gender (c2 = 2.91, df = 1, p = 0.088)
Cohort | Control (Sim) | Interven. (VR) | Total N | Total % |
|---|---|---|---|---|
Cohort 1 | 45 | 66 | 111 | 16.4 |
Cohort 2 | 29 | 31 | 60 | 8.9 |
Cohort 3 | 42 | 56 | 98 | 14.5 |
Cohort 4 | 23 | 54 | 77 | 11.4 |
Cohort 5 | 91 | 99 | 190 | 28.1 |
Cohort 6 | 34 | 63 | 97 | 14.4 |
Cohort 7 | 18 | 24 | 42 | 6.2 |
Total | 282 | 393 | 675 | 100.0 |
BN or DN | Control (Sim) | Interven. (VR) | Total N | Total % |
BN | 178 | 221 | 399 | 59.1 |
DN | 104 | 172 | 276 | 40.9 |
Total | 282 | 393 | 675 | 100.0 |
Age Range | Control (Sim) | Interven. (VR) | Total N | Total % |
18–25 yr | 180 | 263 | 443 | 65.6 |
26–50 + yr | 90 | 127 | 217 | 32.1 |
Missing | 12 | 3 | 15 | 2.2 |
Total | 282 | 393 | 675 | 100.0 |
Gender | Control (Sim) | Intervention (VR) | Total N | Total % |
Male | 62 | 70 | 132 | 19.6 |
Female | 203 | 320 | 523 | 77.5 |
Other | 2 | 0 | 2 | 0.3 |
Missing | 15 | 3 | 18 | 2.7 |
Total | 282 | 393 | 675 | 100.0 |
Simulation participation
Approximately 95% of students actively participated in the VR scenarios. Approximately 3–5% of students were not able to participate due to incompatible phones and therefore were transferred to the traditional simulation control group. Approximately 15% of students in the traditional simulation control group actively participated in the simulation scenarios with 85% observers, which is standard practice at the participating institute.
Anzeige
Knowledge test results
Pre-intervention there was nothing to suggest any systematic difference between the knowledge test results of the control group and of the intervention group. For the post-test (survey 2), 15 out of 17 t-tests returned positive results at the level of p < 0.001, suggesting that the intervention group performed better than the control group in the knowledge test (Table 3).
In survey 3, there was no significant difference between the knowledge test results of the control group and of the intervention group. The results indicate that any difference in knowledge was not maintained post clinical placement.
Table 3
Knowledge Test, Questions Q1 – Q10 responses pooled to give Total Score/10. Descriptive statistics (columns 4–7; N, Mean, StdDev and 95% Confidence Interval) for Control and Intervention Groups
Selection Property | Group | Descriptive Statistics | t-test for Equality of Means | ||||||
|---|---|---|---|---|---|---|---|---|---|
N | Mean | Std. Dev. | 95% C.I. | t | df | Sig. | |||
All | All Cases | Control (Sim) | 734 | 5.75 | 2.15 | 0.159 | -1.4 | 1598.4 | 0.169 |
Intervention (VR) | 1115 | 5.89 | 2.21 | 0.132 | |||||
Module | Module 1 | Control (Sim) | 157 | 7.85 | 1.49 | 0.238 | -1.8 | 279.8 | 0.080 |
Intervention (VR) | 267 | 8.10 | 1.23 | 0.150 | |||||
Module 2 | Control (Sim) | 232 | 5.21 | 1.48 | 0.194 | -0.9 | 527.7 | 0.361 | |
Intervention (VR) | 327 | 5.33 | 1.65 | 0.182 | |||||
Module 3 | Control (Sim) | 245 | 4.36 | 1.87 | 0.240 | 0.9 | 484.9 | 0.382 | |
Intervention (VR) | 364 | 4.23 | 1.68 | 0.176 | |||||
Module 4 | Control (Sim) | 100 | 7.09 | 1.62 | 0.324 | -0.2 | 219.3 | 0.813 | |
Intervention (VR) | 157 | 7.14 | 1.71 | 0.273 | |||||
Survey 2, Section B, Q1-10, total score/ 10 (cohorts 1–6) | |||||||||
All | All Cases | Control (Sim) | 511 | 6.09 | 2.24 | 0.20 | -9.1 | 928.7 | < 0.001** |
Intervention (VR) | 810 | 7.16 | 1.84 | 0.13 | |||||
Module | Module 1 | Control (Sim) | 129 | 7.73 | 1.60 | 0.28 | -4.3 | 194.8 | < 0.001** |
Intervention (VR) | 223 | 8.41 | 1.07 | 0.14 | |||||
Module 2 | Control (Sim) | 129 | 5.68 | 1.78 | 0.31 | -6.7 | 235.8 | < 0.001** | |
Intervention (VR) | 217 | 6.93 | 1.51 | 0.21 | |||||
Module 3 | Control (Sim) | 183 | 4.92 | 2.14 | 0.32 | -3.9 | 346.2 | < 0.001** | |
Intervention (VR) | 239 | 5.69 | 1.75 | 0.23 | |||||
Module 4 | Control (Sim) | 70 | 6.84 | 2.21 | 0.53 | -4.4 | 99.3 | < 0.001** | |
Intervention (VR) | 131 | 8.13 | 1.40 | 0.24 | |||||
Survey 3, Section B, Q1-Q10 (cohorts 3, 4 and 6) | |||||||||
All | All Cases | Control (Sim) | 270 | 5.88 | 2.68 | 0.33 | -0.6 | 570.9 | 0.541 |
Intervention (VR) | 442 | 6.00 | 2.69 | 0.26 | |||||
Module 1 | Control (Sim) | 75 | 8.24 | 1.00 | 0.23 | -1.3 | 155.7 | 0.188 | |
Module | Intervention (VR) | 120 | 8.43 | 0.99 | 0.18 | ||||
Module 2 | Control (Sim) | 72 | 5.94 | 1.91 | 0.45 | -1.1 | 136.0 | 0.257 | |
Intervention (VR) | 121 | 6.26 | 1.71 | 0.31 | |||||
Module 3 | Control (Sim) | 75 | 2.55 | 1.31 | 0.30 | 0.4 | 135.8 | 0.710 | |
Intervention (VR) | 121 | 2.48 | 1.09 | 0.20 | |||||
Module 4 | Control (Sim) | 48 | 7.29 | 1.61 | 0.46 | -0.1 | 104.2 | 0.945 | |
Intervention (VR) | 80 | 7.31 | 1.72 | 0.38 | |||||
Students’ self-perceived knowledge and motivation
The results show immediately post intervention, the intervention group’s ratings for self-perceived knowledge, motivation and preparedness for clinical placement were more positive than the control group at the p < 0.01 level (Table 4). The differences between the control and interventions groups were not maintained post clinical placement.
Anzeige
Table 4
Students’ self-perceived knowledge, motivation and preparedness for clinical placement or clinical practice. Descriptive statistics and Independent Samples t-test (All Cohorts. All Modules pooled. Surveys 2 and 3)
Survey 2 Question | Group | Descriptive Statistics | t-test for Equality of Means | |||||
|---|---|---|---|---|---|---|---|---|
N | Mean | Std. Dev. | 95% C.I. | t | df | Sig. | ||
KNOWLEDGE | Control (Sim) | 493 | 4.57 | 1.495 | 0.134 | -9.99 | 903.3 | < 0.001** |
Intervention (VR) | 786 | 5.37 | 1.240 | 0.088 | ||||
MOTIVATION | Control (Sim) | 493 | 5.73 | 1.134 | 0.102 | -6.04 | 968.0 | < 0.001** |
Intervention (VR) | 786 | 6.11 | 1.027 | 0.074 | ||||
PREPAREDNESS (for clinical placement) | Control (Sim) | 493 | 5.03 | 1.163 | 0.104 | -6.57 | 1018.1 | < 0.001** |
Intervention (VR) | 786 | 5.46 | 1.124 | 0.080 | ||||
Survey 3 Question | Group | Descriptive Statistics | t-test for Equality of Means | |||||
N | Mean | Std. Dev. | 95% C.I. | t | df | Sig. | ||
KNOWLEDGE | Control (Sim) | 268 | 5.10 | 1.097 | 0.134 | -0.45 | 515.35 | 0.653 |
Intervention (VR) | 439 | 5.14 | 0.978 | 0.094 | ||||
MOTIVATION | Control (Sim) | 268 | 6.00 | 1.067 | 0.130 | 0.532 | 543.143 | 0.595 |
Intervention (VR) | 440 | 5.96 | 1.017 | 0.096 | ||||
PREPAREDNESS (for clinical practice) | Control (Sim) | 245 | 5.28 | 1.161 | 0.148 | 0.48 | 464.156 | 0.632 |
Intervention (VR) | 401 | 5.23 | 1.017 | 0.102 | ||||
Self-efficacy in learning
Immediately post intervention, the intervention group’s self-efficacy in learning was more positive than the control group’s for modules 3 and 4 and the p < 0.01 level. However, differences between the intervention and control groups were not maintained post clinical placement (Table 5).
Table 5
Self-efficacy in learning, pooled responses to give Total Score/100, results resolved according to module number (1–4). Descriptive statistics (columns 4–7; N, Mean, StdDev and 95% Confidence Interval) for All Cohorts (i.e., both BN Cohorts and DN Cohorts), for Control and Intervention Groups. Independent Samples t-test comparing the mean values for the Control and Intervention groups. ** indicates significance at the p < 0.01 level
Selection Property | Group | Descriptive Statistics | t-test for Equality of Means | ||||||
|---|---|---|---|---|---|---|---|---|---|
N | Mean | Std. Dev. | 95% C.I. | t | df | Sig. | |||
Survey 2 | Module 1 | Control (Sim) | 121 | 81.00 | 10.05 | 1.83 | -1.3 | 260.2 | 0.183 |
Intervention (VR) | 217 | 82.56 | 10.63 | 1.44 | |||||
Module 2 | Control (Sim) | 130 | 78.78 | 10.57 | 1.85 | -0.6 | 309.3 | 0.545 | |
Intervention (VR) | 217 | 79.55 | 12.65 | 1.72 | |||||
Module 3 | Control (Sim) | 182 | 70.04 | 14.96 | 2.22 | -6.4 | 352.3 | < 0.001** | |
Intervention (VR) | 238 | 78.91 | 12.63 | 1.64 | |||||
Module 4 | Control (Sim) | 71 | 76.08 | 14.00 | 3.32 | -4.1 | 117.1 | < 0.001** | |
Intervention (VR) | 131 | 83.97 | 10.94 | 1.91 | |||||
Survey 3 | Module 1 | Control (Sim) | 75 | 81.29 | 9.77 | 2.26 | -0.2 | 166.3 | 0.821 |
Intervention (VR) | 120 | 81.63 | 10.55 | 1.93 | |||||
Module 2 | Control (Sim) | 72 | 76.00 | 12.27 | 2.89 | 0.3 | 160.6 | 0.787 | |
Intervention (VR) | 121 | 75.48 | 13.48 | 2.45 | |||||
Module 3 | Control (Sim) | 73 | 72.05 | 14.89 | 3.49 | -1.1 | 137.3 | 0.256 | |
Intervention (VR) | 121 | 74.46 | 13.14 | 2.39 | |||||
Module 4 | Control (Sim) | 48 | 75.00 | 12.09 | 3.49 | -1.1 | 100.3 | 0.289 | |
Intervention (VR) | 78 | 77.37 | 12.20 | 2.76 | |||||
Analysis of OSCE data
OSCE data was obtained for 478 students (control n = 177, intervention n = 301). The Pearson Chi-Square test (c2 = 0.267, df = 2, p = 0.875) indicated there was no statistically significant difference between the control and intervention groups.
For Modules 1 and 2 there was no significant difference between the Mean OSCE score (as a %) for the control and intervention groups. However, for Module 3 the Mean OSCE score for the intervention group was significantly greater than that for the control group, at the p < 0.01 level. When all three Modules were pooled together, while the OSCE score was greater for the intervention group than for the control group, the difference was not significant at the p < 0.05 level (Table 6).
Table 6
Mean OSCE Score (as %, StdDev and 95% Confidence Interval) for all cases (n = 478), for Control and Intervention Groups
Module Number | Group | Descriptive Statistics | t-test for Equality of Means | |||||
|---|---|---|---|---|---|---|---|---|
N | Mean Score % | Std. Dev. | 95% C.I. | t | df | Sig. | ||
Module 1 | Control (Sim) | 30 | 79.8 | 16.6 | 6.06 | 0.726 | 43.3 | 0.472 |
Intervention (VR) | 51 | 77.3 | 10.7 | 2.98 | ||||
Module 2 | Control (Sim) | 99 | 70.0 | 11.1 | 2.23 | − 0.790 | 241.9 | 0.431 |
Intervention (VR) | 162 | 71.3 | 14.0 | 2.19 | ||||
Module 3 | Control (Sim) | 48 | 49.1 | 17.7 | 5.10 | -3.606 | 94.3 | 0.000** |
Intervention (VR) | 88 | 60.4 | 17.2 | 3.66 | ||||
Pooled Modules | Control (Sim) | 177 | 66.0 | 17.8 | 2.67 | -1.931 | 332.3 | 0.054 |
Intervention (VR) | 301 | 69.1 | 15.7 | 1.80 | ||||
Views about the module (survey 2)
All t-tests comparing mean values for the control and intervention groups for views on the modules returned positive results at the level of p < 0.01, with three of them at the level of p < 0.001 (Table 7).
Table 7
Views about the module, responses pooled to give Total Score/50, (cohorts 1–6 -no data cohort 7)
Selection Property | Group | Descriptive Statistics | t-test for Equality of Means | ||||||
|---|---|---|---|---|---|---|---|---|---|
N | Mean | Std. Dev. | 95% C.I. | t | df | Sig. | |||
All | All Cases | Control (Sim) | 382 | 40.37 | 6.59 | 0.67 | -8.9 | 665.4 | < 0.001** |
Intervention (VR) | 791 | 43.88 | 5.71 | 0.41 | |||||
Module | Module 1 | Control (Sim) | 77 | 41.34 | 6.41 | 1.46 | -2.8 | 125.6 | 0.005** |
Intervention (VR) | 218 | 43.72 | 5.98 | 0.81 | |||||
Module 2 | Control (Sim) | 125 | 41.32 | 5.22 | 0.93 | -3.9 | 272.0 | < 0.001** | |
Intervention (VR) | 205 | 43.67 | 5.48 | 0.77 | |||||
Module 3 | Control (Sim) | 127 | 39.65 | 6.89 | 1.22 | -5.1 | 222.8 | < 0.001** | |
Intervention (VR) | 237 | 43.35 | 5.80 | 0.75 | |||||
Module 4 | Control (Sim) | 53 | 38.42 | 8.32 | 2.28 | -5.7 | 69.1 | < 0.001** | |
Intervention (VR) | 131 | 45.44 | 5.21 | 0.91 | |||||
Qualitative analysis
Several common themes were identified across the 4 modules for intervention students’ responses on views about the learning experience with JasperVR (Table 8). Students reported that they found the VRS scenarios realistic. They valued the ability to be able to practise the various scenarios multiple times to embed their learning and highlighted the importance of being able to make mistakes without fear of impacting care. The VRS scenarios were perceived to be less stressful and intimidating than the usual simulation environment. The value of developing critical thinking from the VRS scenarios was highlighted and the development of the necessary knowledge and skills to be able to manage similar situations.
While student responses were mostly positive and enthusiastic about the VR scenarios, they did express some reservations. Occasionally the VR headsets were reported to be uncomfortable to wear. Whilst some students found the VRS scenarios less intimidating than a normal simulation, a small number of students described their desire to be more physically involved, rather than watching and choosing options. A small number of students also reflected that their clinical reasoning skills would have benefitted from there being more than one correct choice of response, learning diverse ways to manage situations, or having multiple scenarios for each module. Further detail on students’ views on what they enjoyed the most and least for each module are detailed in appendix 1.
Table 8
Qualitative data for intervention students’ views on Jasper VR.
Module | Theme | Quote |
|---|---|---|
Enjoyed the most | ||
Common feedback (all modules) | Realism | Felt like a real situation and it helped me to handle any situation in the future. The scenario was realistic, and the module was able to give you a chance to reflect on the options you have chosen to manage the situation. Use of different locations and scenarios made the virtual experience more realistic |
Rehearsal, ability to make mistakes | The information that was delivered was relevant. I loved the systematic delivery of information. I could choose wrong answers and know how the patient would react. … I liked that it told you what you did right and what you didn’t. You get to see the different things that can happen if you don’t choose the right choice which is educational. The existence of ‘mastery mode’ -the best solution for the situation and how a professional would deal with the situation. Choosing the wrong choices leads you to well performed scenes. Actors took the time to show all paths | |
Reduced stress and increased confidence compared to a normal SIM | I was not there physically to witness the scene. It was less intimidating. The VR was fantastic compared to the usual physical sim. The VR environment was student-friendly, did not cause any anxiety and therefore, helped learn a lot. I was able to maintain calm and well settled behaviour due to confidence build up through our Jasper activities | |
Ability to practise the modules multiple times | Not being stressed out having the opportunity to practise as many times as I want. | |
Development of critical thinking | The fact that I felt well immersed into the environment of the simulation. A proper handover was delivered and gave me direction of how I could manage the patient. The VR project gave … me options to choose from which promoted higher order critical thinking and reasoning skills to be used in practice within the simulated scenario | |
Enjoyed the least | ||
Common feedback (all modules) | Technical: VR headset | The VR headsets did not work well, was better able to watch using mobile view mode. Using the VR-GX device. Using normal view on my phone was more enjoyable. It was difficult with wearing glasses |
Scenario: Lack of interaction | The thing I least enjoyed was not being able to interact with the patient and the situation. Not having the chance to do skills by my own hand. Having to stand on a corner and watch the whole situation. I know it’s hard, but I wanted to be more involved Break up sections with more interactive parts to better take in all the information | |
Scenario: Development of clinical reasoning | Wish there were more questions and options to choose and participate. More options that could potentially be helpful, not just one right way | |
Scenario: Revision of information presented | Add multiple choice questions after the scenario to recap the covered information presented in the scenario. | |
Additional scenarios required in different settings | Maybe in a hospital setting with triggers that will cause patient to escalate (e.g., postponed surgery, etc.) Provide multiple video situations because everyone’s anger/aggression is different. Provide video of angry patient in room, as that situation is what we will most likely experience. I still want to learn and want content of deteriorating patient condition. Need more depth about this topic in regard to VR simulation. | |
System usability scale
The mean scores of the pooled usability scale are very complimentary of the usability of the system, particularly its ‘ease of use’ (Q3) and its property of being ‘easy to learn quickly’ (Q7) (Table 9).
Table 9
System Usability Scale responses. Descriptive statistics (N, mean value, standard deviation and 95% confidence interval) for the pooled cohort (N = 306, Cohorts 1,2,3,4,5 and 6 combined) responses
Question # | Question Text | Cohort | N | Mean | Std. Dev. | 95% C.I. |
|---|---|---|---|---|---|---|
Q1 | I think that I would like to use this system frequently | All | 305 | 4.06 | 0.97 | 0.11 |
Q2 | I found the system unnecessarily complex | All | 306 | 2.11 | 1.19 | 0.14 |
Q3 | I thought the system was easy to use | All | 306 | 4.22 | 0.92 | 0.10 |
Q4 | I think that I would need the support of a technical person to be able to use this system | All | 306 | 2.12 | 1.25 | 0.14 |
Q5 | I found the various functions in this system were well integrated | All | 305 | 3.95 | 0.93 | 0.11 |
Q6 | I thought there was too much inconsistency in this system | All | 306 | 2.02 | 1.06 | 0.12 |
Q7 | I would imagine that most people would learn to use this system very quickly | All | 305 | 4.19 | 0.87 | 0.10 |
Q8 | I found the system very cumbersome to use | All | 300 | 2.53 | 1.25 | 0.14 |
Q9 | I felt very confident using the system | All | 306 | 4.29 | 0.82 | 0.09 |
Q10 | I needed to learn a lot of things before I could get going with this system | All | 306 | 2.27 | 1.28 | 0.15 |
These findings were supported by the qualitative data in the surveys. Intervention students appreciated the ability to re-visit the VR scenarios as often as they liked and in their own time:
I really enjoyed being part of JasperVRand I felt it was a really nice way to learn the situation… best thing was I can rewatch it as many times as I want.This was a great way to complete a simulation under the circumstances and a great way to continue this in the future as we have the luxury of completing it as many times as we will.
Intervention students also found the VRS scenarios less intimidating than traditional simulation and liked the ability to be able to make mistakes and see the consequences of those choices:
I really enjoy doing VR. I find it interesting and less stressful than doing a simulation with your whole class watching you and in front of actors/actresses. I also like how you can see what would happen if you chose the wrong choice in the situation and that you can do the VR as much as you want.
Occasionally students reported difficulties with the VR headsets:
I like the way that we could revisit the SIM whenever we wanted to, but I found controlling the SIM with the headwear hard to manage.
Feasibility and economic viability of JasperVR compared to SBE
Table 10 depicts the cost of developing and delivering the VR and SBE for both the Bachelor and Diploma of Nursing groups. The total cost of delivering a single module of JasperVR each year was $3,350. The cost of delivering a single SBE scenario (not necessarily the same) in nursing for both courses each year was $18,670.
Table 10
The cost of delivering the VR and SBE for the Bachelor and Diploma of Nursing groups
JasperVR (costs per individual scenario) | Bachelor of Nursing | Diploma of Nursing | |||||||
|---|---|---|---|---|---|---|---|---|---|
One-off work | Item | Hrs | Cost | Item | Hrs | Cost | Item | Hrs | Cost |
Script development and review | 160 | $8,000 | Script development | 20 | $1,000 | Script development | 20 | $1,000 | |
Rehearsals | 40 | $2,000 | Document review | 12 | $600 | Document review | 8 | $400 | |
Filming (1 day) crew | $3,000 | ||||||||
Filming (1 day) actors | 64 | $2,560 | |||||||
Filming (1 day) teachers | 10 | $500 | |||||||
Document update | 10 | $500 | |||||||
Software development | 160 | $8,000 | |||||||
Testing and QA | 160 | $8,000 | |||||||
Project development overhead | 60 | $3,000 | |||||||
Totals | 664 | $35,560 | 32 | $1600 | 28 | $1400 | |||
On-going use | Item | Hrs | Cost | Item | Hrs | Cost | Item | Hrs | Cost |
Software license | $2,000 | Preparations | 23 | $1,150 | Preparations | 16 | $800 | ||
Admin preparation | 15 | $750 | Sim day – teachers | 48 | $2,400 | Sim day – teachers | 32 | $1,600 | |
Pre-brief students | 4 | $200 | Sim day – actors | 21 | $840 | Sim day – actors | 32 | $1,280 | |
Debrief students | 4 | $200 | Sim day – others (sim techs) | 10 | $500 | Sim day – others (sim techs) | 10 | $500 | |
Technical support | 4 | $200 | Additional remedial sim day (depending on group size | 102 | $5,100 | Additional remedial sim day (depending on group size | 190 | $4,500 | |
Totals | $3,350 | 204 | $9,990 | 180 | $8,680 | ||||
Discussion
Our study demonstrated that VRS provided authentic and positive learning experiences for nursing students. Participants indicated that they found the VR scenarios realistic, immersive, and aided in the development of their clinical reasoning. Authenticity in VR scenarios is important for preparation for the reality of clinical practice [25]. Our outcomes also support the view that VRS can be successfully employed to teach explicit behavioural skills such as teamwork, and decision-making [25, 26]. The VR scenarios catered for greater student numbers when compared to traditional simulation. All VRS students were fully immersed in the scenarios and able to take an active learning role the decision-making process. JasperVR was found to be a sustainable and cost-effective alternative to SBE.
Overall, this study found VRS to effective for improving student knowledge and performance, which is supported by other studies on the effects of VRS in nursing education [8, 14‐16, 27]. Several factors can increase the cognitive load in VR scenarios compared to SBE. VR learners may initially need to process a large amount of sensory information due to a highly immersive experience, engagement of multiple senses simultaneously (for example, vision and hearing), mastery of controllers to interact with the virtual environment, and understanding and navigating 3D space. Our findings also reinforce the effectiveness of immersive VRS for improved cognitive load in nursing education, and that VRS is an effective teaching tool [9, 26].
VRS students’ OSCE scores for module 3, the patient with cognitive impairment, were significantly greater than for traditional simulation at the end of semester, although this difference was not maintained post clinical placement. For this scenario, students were particularly complimentary about the communication and collaborative skills amongst the interprofessional team and being able to practise skills such as recording the handover and assessing the patient. Active learning methods have previously been found to facilitate the development of logical reasoning [28] and reflective thinking [29]. The opportunity to perform similar skills in future VR scenarios may contribute to their further success. The significant differences in knowledge test scores between the VRS and traditional simulation groups immediately post intervention were not maintained. VR scenarios have the advantage of being available for users to access at any time point with easy repetition of scenarios, [30], and therefore VR students in may need to be reminded to re-visit these to further consolidate their learning and achieve proficiency.
Students appreciated the opportunity the VR scenarios provided to develop clinical competencies in authentic clinical experiences and see new perspectives, whilst maintaining their own safety and that of their patients [9]. VR simulations can offer a high level of realism and immersion compared to traditional SBE, with the learning process accelerated as students can experience scenarios that closely mimic real-life situations. The use of VRS in nursing education has been found to influence positive learning outcomes, such as stimulating interactivity and motivation amongst participants [25, 31, 32]. Sim et al. (2022) found that VRS enhances delivery of content related to patient care management (Sim et al., 2022). VR students valued access to the mastery mode to learn strategies like de-escalation techniques for dealing with an aggressive patient. Enabling VR students to practise skills through repetition, may allow them to master the skills more efficiently and achieve proficiency, versus SBE which has limitations in terms of repetition. VRS has been found to be particularly useful in situations such as dealing with aggressive patients as it protects the health professional’s safety and allows them to take a ‘trial and error’ approach to learning how to respond in these situations [33].
Whilst a small number of students highlighted a lack of interaction in the scenarios and being unable to practise the clinical skills in person, others reported the value of a fully immersive environment, such as being able to assess the deteriorating patient and recording the handover. Simulating the management of an acutely deteriorating patient has previously been successfully implemented in a nursing curriculum, with participants finding the simulation to be realistic and prepared them for clinical practice [34]. VR students also appreciated the opportunity for easy repetition of scenarios, experiencing the consequences of making wrong decisions, and the value of this for their learning. Through the VRS, nursing students could develop greater self-awareness and modify their reactions to a situation [33]. The importance of VRS for learning non-clinical skills by being able to ‘get it wrong to get it right’ was also highlighted in a study of undergraduate nursing students [25]. VR students found participating in the scenarios less intimidating than a normal simulation and appreciated the psychological safety involved in the learning [32]. This feedback echoed the findings of other VR studies that reported students using VR were less anxious than control groups when performing in the real world [35, 36]. Students also appreciated the ability to make mistakes in the virtual environment without fear of consequences [32].
Whilst the VR scenarios provided immediate feedback to the students and allowed them to correct their mistakes in real time, some students commented that there was only one correct option in the scenarios. They felt it would be more beneficial if the scenarios allowed a greater range of choices for patient management to more reflect the real world. Another study involving a VR scenario depicting unnecessary patient demand for antibiotics from a general practitioner, also found some scepticism amongst participants that VR technology could reflect the diversity and complexity of patient responses [33]. A system of learning for health professionals that involves menu-based actions, may mean that the student does not develop critical clinical reasoning skills [37]. VR simulations can be customised to meet the needs of learners and to meet specific learning objectives, which should be considered when developing further VR scenarios.
Participants in this study were generally positive about the usability of the VR technology, which echoes the findings of other immersive VRS nursing studies [11, 38]. Usability in terms of ease of use and users’ level of satisfaction are important characteristics of learning using virtual reality [7].
When considering the economic viability of VRS, the cost effectiveness of VR versus SBE can vary depending on several factors, including initial investment, maintenance, scalability, and accessibility. A previous study found the cost-utility ratio of virtual simulation (US$1.08) to be lower compared to mannequin-based simulation (US$3.62) [39]. In this study, the VR required significant upfront costs for software, hardware and development and was therefore more costly to develop when compared to standard SBE. This concurs with the findings of Liaw et al. (2018) who found that funding was important for the development and evaluation of virtual worlds in nursing education, due to the high outlay costs during design and development phases [40]. The VR simulations may also require ongoing software updates and maintenance costs, whilst the SBE may require periodic equipment replacement. However, our economic evaluation found the long-term delivery costs of VR to be significantly reduced due to less on campus teaching time and more independent learning for students. As reported by Pottle (2019) the costs of simulation are difficult to define, vary widely between institutions and are frequently under-reported [41]. Therefore, further studies are required which evaluate the cost effectiveness of VR compared to SBE for nursing students [30]. Our study found VR simulations required more time initially for learners to become comfortable with the technology and to set up the VR equipment, when compared to SBE. However, the virtual simulations are more scalable, allowing a larger number of students to be served simultaneously and to be used in further studies. VR simulations can be accessed remotely, offering flexibility for students, and making them more convenient for students who may not have easy access to physical simulation labs. This can save time and eliminate the need for travel, compared to SBE which usually requires students to be physically present in a specific location. As a result of COVID-19 in 2020, all learning with JasperVR moved to remote delivery. This resulted in an even further reduction in the hours associated with briefing and debriefing.
Limitations
This study focused on undergraduate students and therefore further research is required to explore its application across the healthcare professional continuum. Some students felt their clinical judgement would benefit from having additional options rather than only one correct way of doing things. In future research, students could be given the opportunity to be included in the development of scenarios to gain their perspectives. There should be repeated exposure to similar situations to gain confidence in appropriately responding in real life. Future work could focus on students’ comments related to limited opportunities for hands on work in the VR scenarios, reduced patient interactions, and how these findings translate in the clinical setting. Due to the COVID-19 global pandemic in 2020, the implementation of control groups and cross over groups was not able to be achieved and all students were allocated into intervention groups to receive the JasperVR learning experience. This was due to all simulations being cancelled in Semester 1, 2020. Furthermore, it was not possible to implement any Survey 3s and clinical assessments in 2020 due to the overwhelming number of cancellations of clinical placements and clinical assessments for students. This resulted in many more students in the intervention group than the control group.
Conclusions
Through a collaborative content and software development process, a sophisticated, scalable, highly usable, and authentic learning experience was created for pre-licensure nursing students. JasperVR enabled increased numbers of students to actively participate in an immersive simulated learning environment at their own pace, in their own time and venue. JasperVR fostered critical thinking and decision making and provided an efficient, cost effective and sustainable platform of learning for future nursing students. The VR simulations offered advantages in terms of immersive, repeatable, and feedback-rich experiences. JasperVR provided an efficient, cost effective and sustainable platform for learning for future nursing students.
Acknowledgements
The authors acknowledge: Swinburne University as a major collaborative partner, Karen Hall and Ronen Vansover for Project management, CurveTomorrow for software development, Ruth Callandar for VR implementation, teams of supporting clinicians that assisted in script writing of all scenarios, Michael Esler for statistical analysis, teaching staff of the Diploma and Bachelor of Nursing, and participating students.
Declarations
Institutional review board statement
The study was conducted in accordance with the Declaration of Helsinki and ethical approval was obtained from Monash University Human Research Ethics Committee, Project ID: 19235.
Informed consent
Informed consent was obtained from all subjects involved in the study.
Conflict of interest
The authors declare no conflict of interest.
Consent for publication
Not applicable.
Competing interests
The authors declare no competing interests.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
