Exploring intensive care nurses’ team performance in a simulation-based emergency situation, − expert raters’ assessments versus self-assessments: an explorative study

The study used an explorative design based on laboratory high-fidelity simulation.

The study was conducted in a simulation centre at a university college in Norway, in a laboratory created as an ICU environment and with the use of a high-fidelity human patient simulator.

A convenience sample of 53 registered nurses (RNs) was recruited from seven ICUs in four hospitals within one hospital trust (Table 1).

The RNs worked in two specialties: (i) general intensive care units (G-ICU) (n = 26 RNs) with surgical and medical patients, usually without coronary patients, and (ii) medical intensive care units (M-ICU) (n = 27 RNs) with coronary, medical and in one unit also surgical patients. Participation in the study occurred during the RNs’ scheduled work time and the RNs who wanted to participate signed up on a list. Each team consisted of four to six RNs from same units who knew each other and had worked together. The unit nurse/manager allocated the RNs into teams with regard to their work schedules and the staffing resources to ensure safe care at the unit. A total of 11 teams from the G-ICU (five teams) and M-ICU (six teams) participated.

The cardiac arrest scenario chosen for this study, was a part of a simulation-based team training programme which comprised two half days for each team, and was developed by the research group.

On the first half day of the programme, the RNs received information about the topic of the simulation scenario cases, the CRM learning objectives and recommendations about relevant literature to read. Furthermore, the RNs participated in theory inputs concerning patient safety in the ICU, NTS with a focus on the key CRM points by Rall and Gaba [34] and the use of simulation for training.

The other half day of the programme that referred to conducting two simulation scenarios, was carried out for one team at a time two to four weeks after the first day. In addition to the scenario which involved a patient undergoing cardiac arrest (reported in this study), a scenario of a receipt of a trauma patient was conducted. The two simulation scenarios were carried out in varying order for each team within each specialty. Two trained simulator instructors were responsible for the implementation of the scenario, one for facilitating the simulation and the other for performing the operation of the mannequin software. Three different scenario roles were randomly assigned among the team members by the facilitator: one as a patient charge nurse (leader), one as the charge nurse’s assistant (assistant) and the third with the responsibility for the patient in the neighboring bed (helper). The RNs also had the opportunity to call a physician (acted out by the facilitator), who gave the RNs medical directives for the handling of the emergency over the phone. The rest of the team members were placed in another room and observed the team performance on a screen, with the videotaped scenario lasting approximately 12–15 minutes.

MHPTS [20] provides a brief, reliable and practical measure of NTS that can be used by participants as a self-assessment of CRM training, in which they reflect on and evaluate their team performance. The scale is also recommended for comparisons between self-assessment and expert assessment [20]. Each item is eligible for a 0-, 1- or 2 point assessment: 0 = never or rarely, 1 = inconsistently and 2 = consistently. The first part of MHPTS (items 1–8) was used because the items were judged as being consistent with the simulation scenario.

This study was approved by the Norwegian Social Data Services as well as being approved by the hospital’s administrative heads, and the study was conducted according to the Ethical Guidelines for Nursing Research in the Nordic Countries [36]. Information, and an invitation to participate in the study were given to the RNs in both written and oral form, and referred to the principle of autonomy addressed by voluntariness, informed consent and the right to withdraw from the research project at any time, as well as confidentiality in terms of making data anonymous and assurance that the videotapes would been deleted at the end of the project. The RNs written consent was obtained.

IBM SPSS Statistics 19 was used to analyse the data, with descriptive statistics presenting the frequencies, percentages, means and standard deviations. The units of analyses of team performance were the team score for the expert raters’ assessments and the RNs’ self-assessments, respectively. Chi-square tests were used to compare the two specialties with regard to the RNs’ background in general and with regard to the three different scenario roles. The expert raters’ assessments were calculated by adding each rater’s score for each team, and then dividing by two. Mann–Whitney U tests were conducted to compare the differences between the teams from two types of specialties (G-ICU, M-ICU) with regard to the Ottawa GRS CRM categories and the MHPTS items. The RNs’ self-assessments were calculated by adding each team member’s score for the team, and then dividing the score by the number of team members. Mann–Whitney U tests were conducted to compare the differences between the teams from two types of specialties (G-ICU, M-ICU) concerning the MHPTS items. Additionally, Mann–Whitney U tests were used to compare the differences between the expert raters’ assessments and the RNs’ self-assessments of the MHPTS items. Correlational analyses with Spearman’s rho were used to explore the relationship between the expert raters’ assessments and the RNs’ self-assessments of the MHPTS items. A two-tailed significance level of p < .05 was used for all tests.

Intraclass Correlation Coefficient (ICC) Type III was used to assess inter-rater reliability regarding Ottawa GRS CRM categories. ICC coefficients > .75 indicate a good reliability, with coefficients from .50 -.75 suggesting moderate reliability and values < .50 representing poor reliability [37]. This study showed an ICC ranging from .667-.854 (moderate - good reliability) (Table 2).

Percentage agreements were conducted in relation to MHPTS, and in this study the percentage of agreements between raters was 60%.

Team leadership refers to how to influence team performance by facilitating team problem solving through cognitive processes, coordination processes and the team’s collective motivation and behaviour [41]. In emergencies generally, and in cardiac arrest situations in particular, the behaviour of the first responders is crucial, and the first-responding RN should take the leadership role [42]. RNs from the M-ICU representing units with coronary patients may have experienced cardiac arrest situations in real- and training situations to a greater extent, and were therefore more aware of the leader’s role. Conversely, the RNs from G-ICU had the longest experience as PG-RNs and most years at unit. This may indicate the importance of regular training. A lack of knowledge of procedure or guidelines may result in low leadership performance [43], and research has shown that well-trained, first-reacting nurses have successfully taken leadership roles in CPR [44],[45]. It is important that RNs stay in the leadership role until others with more leadership experience appear. Team effectiveness regarding CPR is found to be most optimal with a leader who demonstrates immediate directive leadership behaviour [46],[47]. Teamwork may be affected by the institutional culture, and RNs from G-ICU may be less comfortable being a leader in a cardiac arrest situation. Still, Andersen et al. [27] found that the RNs’ cultural and professional role inhibited them from participating in decision making in the CPR despite being certified as Advanced Life Support instructors, even though nurses are responsible for decision making in emergencies [48]. According to Künzle et al. [43], an inadequate leadership behaviour in critical care teams could be due to a lack of leadership training; it is therefore important that all RNs in ICUs receive this training to help improve patient safety. RNs often embrace CRM training because it results in a greater respect and an improved quality of work life by flattening hierarchies of authority through team members’ shared responsibility for patient safety [49].

The teams from the M-ICU with the most experience with coronary patients achieved higher scores compared to the G-ICU with regard to problem solving in both the experts’-and the RNs’ self-assessments. However, research demonstrates that a failure to transform theoretical knowledge into effective team activity during CPR appears to be a major problem [26],[50], which is a result that highlights the importance of training team performance. Thomas et al. [51] found that nurses who received team training as part of their neonatal resuscitation training were generally shown to display more teamwork-related behaviours than those who did not receive such training. Skills identified as being the most improved during patient emergencies after followed up simulation-based team training were found to be the ability to respond in a systematic way, handover to the emergency team and airway management [52].

All teams demonstrated positive results regarding the MHPTS’s resource utilization skills as assessed by both the expert raters and the RNs. The RNs from the G-ICU assessed themselves significantly higher than the experts in the items “each team demonstrated a clear understanding of his or her role”. One reason for this more positive assessment may be that the RNs in the scenario were assigned different roles by the facilitator, which helped them to clarify their function in the team. However, patient safety climate studies confirm that RNs in ICUs have positive perceptions of the teamwork within the unit which includes, e.g. supporting others and coordinating activities [53]-[55], which may also influence the team’s resource utilization skills.

A similar picture emerged with the M-ICU teams, with higher scores than the G-ICU teams with regard to communication skills, which may be a result of RN teams from the M-ICU being more familiar with the CPR guidelines. This resonates with “shared mental model” literature, in which teams communicate and coordinate more effectively when members form a shared mental model for goals, tasks and team member roles and responsibilities [56]. As one of the central components of health-care teamwork, communication has been identified for several years as a central root cause of sentinel events [57]. Tools such as SBAR (situation, background, assessment, recommendation), which provide a common and predictable structure to the communication, as well as a “critical language” that gets everyone to stop and listen, have been found to be effective and essential for safe care in complex environments [58]. Furthermore, “closed-loop communication”, with the verbal confirmation of orders and messages for ensuring that sent communications are heard and accurately understood, is recommended [56]. The implementation of standardized communication tools and techniques may improve team performance in an emergency, thereby exerting a positive impact on patient safety, which should be recommended to be adopted for team training.

The present study included both self-assessments of team performance and video-based analyses by expert raters to study RNs’ team performance. Together with direct observation, these are in accordance with the most up-to-date measures of team performance in health care [31]. Video-based analyses offer advantages since they allow for a precise analysis and repeated review by multiple experts [31], and the observational measures may capture the behaviour that teams actually engage in, although observers cannot assess all of the implicit components of teamwork [15]. Situational awareness skills represent implicit components of teamwork, and self-assessments are of great importance. The RNs from G-ICU assessed themselves as generally higher than the expert raters, and significant differences were shown on the two items referring to leadership and resource utilization. On the other hand, the RNs from M-ICU assessed themselves generally lower than the experts, with significant differences on the three items referring to leadership, situational awareness and communication skills. An incongruity between RNs’ self-assessment and the experts’ assessment may indicate that RNs both overestimated and underestimated their NTS competence. Although the correlations between self-assessment and the experts’ assessment had positive directions in 13 of 16 analyses, only one significant relationship was found. Moreover, no correlations (item 5) and a negative correlation (item 8) may also show an incongruity assessment among the RNs. Arora et al. [33] found a strong correlation between surgeons’ self-assessments and expert assessments with regard to technical skills, though no significant correlations were found with regard to NTS. Davis et al.’s [32] systematic review of the accuracy of physicians’ self-assessments compared with observed expert measures of competence documented that in a majority of the relevant studies, physicians did not appear to accurately self-assess. Our results indicate a tendency of poor performers to overestimate their performance while high performers underestimate theirs. Kruger and Dunning [59] studied the calibration of subjects carrying out various tasks such as logical reasoning, grammar and humour. They demonstrated that participants who were in the lowest-scoring quartile greatly overestimated their abilities; those in the middle quartiles were generally accurate in their self-assessment; whereas participants who had the highest scores underestimated their abilities. These tendencies were confirmed by Hodges et al. [60] studying family medicine residents who interviewed a standardized patient in a difficult “telling bad news” scenario.

While self-report measures have their limitations, they offer a means for assessing unobservable components of team performance, which are no less important than observable components pertaining to training [15]. Different approaches in evaluating team performance have both their strengths and weaknesses [15],[31], and a combined approach can help to obtain a more complete picture of the complexity of team performance, while also providing a method to guide learning through systematic feedback [15].

A limitation of this study was the small number of ICU teams; therefore, the generalizability of the results must be interpreted with caution. Additionally, the result of the RNs’ team performance may be influenced by participants who were initially positive to the use of simulation-based training, and on this basis agreed to participate. Even though simulation studies have many opportunities and advantages, there are also limitations, and although behaviour patterns in simulation are similar to those in real-life situations, the transfer of knowledge from the simulator to real-patient situations has not been clearly confirmed [61]. Although rater training was conducted in order to standardize the evaluation of teamwork performance, some moderate ICC scores were demonstrated regarding the Ottawa GRS. Moreover, a 60% agreement was found for the MHPTS rating of the teams’ performance. The complexities of team performance pose a challenge, and measurement and evaluation have weaknesses due to responsible biases of individual raters [15],[30], which may influence the achievement of a high reliability.