The rapid development of information and technology and the spread of COVID-19 have made online learning and assessment more prevalent. In this context, the potential of multimedia-assisted items to provide an alternative method of learning and evaluation is significant. These items can better reflect authentic clinical situations and enhance higher cognitive skills. This study aims to develop and validate multimedia-assisted items that reflect clinical authenticity.
A mixed-method design with qualitative and quantitative approaches was adopted. The study was conducted in two phases: development and validation. Thirty-six multimedia-assisted items were developed using videos, audio files, photos, and pictures. Preliminary testing ensured item comprehensibility and usability. In the validation phase, a mock test was administered to 136 nursing students, with 105 completing the survey. Quantitative analysis included calculating item difficulty, discrimination, and reliability and examining evidence based on relation with another variable through correlations with clinical reasoning scores and differences in test scores based on perceived academic achievement. Qualitative analysis involved thematic analysis of interviews with 10 participants to explore their experiences and validate the response process.
The average total score was 18.1 out of 36, with item difficulty averaging 0.51, item discrimination 0.31, and reliability (Cronbach’s alpha) 0.64. While no significant correlation was found between mock test scores and clinical reasoning scores, a significant difference was observed between high and low-perceived academic achievement groups. Participants expressed that the multimedia-assisted items were more relevant to nursing practice than paper-based items, integrating theoretical knowledge and clinical practicum experience, enhancing understanding, and stimulating active participation in clinical practicum.
Multimedia-assisted items have been verified as an effective strategy for education and for assessing nursing students’ comprehensive competencies.
MAIs are adequate for evaluating nursing students' competencies in theory and practice, suggesting that MAIs can enhance newly graduated nurses' competencies and improve online assessments in nursing education.
Hinweise
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Abkürzungen
IRB
Institutional Review Board
COVID-19
Coronavirus Disease 2019
CVI
Content validity index
NCRS
The Nurse Clinical Reasoning Scale
MAI
Multimedia assisted item
S-CVI
Scale content validity index
Introduction
Nursing education aims to help students acquire the theoretical knowledge and competency needed for patient care in various clinical situations. The rapid development of information and technology and the spread of COVID-19 have made online learning and assessment more common, with learning methods such as e-learning, web-based learning, blended learning, mobile learning, and learning games being attempted [1‐4]. Portfolios, video clips, and digital devices are used in student evaluation [5, 6]. However, text-based tests remain the predominant assessment method in Korea, with limitations in assessing the competencies required for new nurses. Using test items with multimedia elements can be an alternative evaluation method for evaluating clinical practice performance competency in nursing students.
Background
Using digital devices in evaluation and learning enables the utilization of multimedia, such as videos, pictures, and audio. It allows sharing of educational content without regard to time and space through a university’s learning management system [7]. Due to the COVID-19 pandemic, the demand for computer-based evaluations has increased, and non-face-to-face learning has rapidly expanded.
Anzeige
Many studies have been conducted on nursing education since the COVID-19 pandemic. Most of these studies have focused on the transition to, problems surrounding, or necessity of online education [8, 9], as well as the perceptions and experiences of educators and learners in online education [10, 11]. However, only a few studies have evaluated the effectiveness of digital devices in educational evaluation. The evaluation process is critical as it enhances learning and determines a student’s knowledge and skills, which affects the health and safety of others. Since the Korean Nursing Licensing Examination is conducted as a paper–pencil test, it cannot evaluate learners’ problem-solving or practical abilities required for job performance [12]. The National Council License Examination for Registered Nurses defines “critical thinking” and “clinical judgment” as abilities that new nurses should have. It thus evaluates practical skills focusing on clinical judgments and decision-making ability [13].
Since there is a limited ability to evaluate the higher level of cognitive skills required in nursing, such as critical thinking, clinical judgment, clinical reasoning, and problem-solving ability, through paper–pencil items, it is necessary to explore new ways to evaluate performance competency in practice [14]. Authenticity can play a vital role in fostering meaningful learning experiences, facilitating the integration of theory and practicums, enhancing nurses’ understanding of patients, and improving the effectiveness of education [15]. Experiencing authenticity in nursing education is crucial for meaningful learning, enhancing students’ confidence, and developing their ability to make independent decisions. It allows students to integrate theory and practicums, understand patients holistically, and observe the medical process, leading to improved nursing skills and patient-centered care [15, 16]. To evaluate practical performance competency, authentic scenarios can be presented through tools such as multimedia-assisted items (MAIs) via actual data [14].
A computer-based test can compensate for the limitations of a paper-based test regarding practical ability evaluation [12]. In computer-based tests, it is possible to present authentic data using various forms of multimedia, such as videos, photos, and audio sources, to display realistic, practical situations. This clarifies the meaning of the items and allows them to be developed to measure problem-solving ability. Additionally, multimedia materials for education and evaluation effectively improve knowledge and performance [17, 18]. MAIs are used in some Korean healthcare professional licensing examinations to measure learners’ understanding of clinical situations [19]. Online learning methods like virtual reality and simulations can improve nursing students’ clinical reasoning skills and promote active learning [20, 21]. Tests using multimedia items instead of paper–pencil format are more suitable for evaluating critical thinking skills [22]. Therefore, to assess the clinical performance of nurses, including their clinical reasoning and decision-making ability, it is necessary to develop multimedia items that reflect authentic clinical situations. The aim of this study was to develop MAIs that reflect authentic clinical nursing practices and verify their validity through item analysis, content validity, criterion validity, and response process validity. Through the study, we expected to provide evaluation system to access performance competency and clinical reasoning skill in practice and confirm the effectiveness of its application.
Methods
Study design
This study used a mixed-method design with concurrent quantitative and qualitative approaches. There were conducted in two phases: development of Multimedia assisted items and validation of items (Fig. 1). In development phase, it shows MAIs developing process which reflect authentic clinical nursing practices. In validation phase, to ensure the validity of the items, quantitative data were used to provide evidence based on the relation with other variables and item analysis, and qualitative data were used to provided evidence-based on the response process content validity.
Fig. 1
The phases of development and validation of multimedia assisted items
×
Anzeige
Development of MAIs
In the development phase, the examination subjects were identified, and the examination objectives and items were developed. Experts verified the content validity of the developed objectives and items, and usability tests were conducted.
In the first stage of item development, exam contents were identified. The contents were selected based on the learning objectives of adult health nursing. Care for patients with cardiovascular and respiratory disorders was chosen as the examination subject as it is a frequent topic in the Korean Nursing Licensing Examination. The selected subjects included patients with respiratory symptoms (cough/sputum/dyspnea), central venous catheters, chest pain, congestive heart failure, or arrhythmia, and those who had undergone pulmonary lobectomy.
Next, to ensure relevance to actual clinical practice, the description of examination objectives was linked with the activity statement connecting the duties of newly graduated nurses in Korea as developed by Kim et al. [23]. A total of six examination objectives were produced, each of which included the topic, evaluation goals, and specific outcomes. The outcomes were derived from the activity statement, including patient assessment, differentiation between normal and abnormal conditions, interpretation, intervention selection, and education. Seven experts then evaluated the examination objectives using the content validity index (CVI) to verify the appropriateness of assessing clinical practical competency and its suitability for development as multimedia-assisted items.
Draft items were developed using a plan that included examination objectives, relevant activity statements, item types, and case scenarios and were selected through expert consultation. In total, 36 items were developed using video, audio, photo, and illustration. An example of the developed item is presented in Fig. 2. Five nursing professors and five clinical nurses evaluated the CVI score of the designed items to determine whether the items could reflect clinical reality and assess clinical practice competency. In this case, the CVI score was 0.91. Before the mock test, a usability test was conducted to confirm the understanding of the sentences and the appropriateness of the test time. The usability test results revealed no errors in the MAIs and verified their comprehension and readability. The average time it took the ten preliminary test participants to complete the test was 26.2 ± 10.9 min.
Fig. 2
An example of the developed item
×
Validation of items
The validation phase included a mock test, a survey, and focus group interview. For validation item analysis was used included calculating difficulty, discrimination, and reliability. For validation of evidence based on the relation with other variables, a survey was conducted which included items on participants’ sociodemographic characteristics, participants’ perceptions of MAIs and their perceived academic achievement and the Nurses Clinical Reasoning Scale. For the qualitative study to verify the validation of the response process, focus group interviews were performed a total of two 50-min per each focus group, which focused on exploring participants' experience of solving process about MAIs. Through data analysis was complete, all researchers reviewed and discussed qualitative and quantitative components.
Study participants and setting
In total, 136 participants were recruited by convenience sampling for the mock test. The participants included third—and fourth-year nursing students who had completed a cardiovascular and respiratory systems course. Students who had not completed any of the subjects above were excluded. Data were collected from six nursing colleges in five cities in November 2021.
Undergraduate nursing students were invited to use the URL to access and complete the mock test and survey. The MAIs were accessible online and had to be fully completed. The test time was set to 30 min, which had been determined through the preliminary test, and it was automatically terminated when 30 min had passed. Among the 136 students, 31 did not complete the questionnaire and were excluded from the final analysis. A final group of 105 participants were enrolled. Ten participants who completed the mock test and survey were recruited for focus group interviews to collect qualitative data. (Fig. 3).
Fig. 3
Flow diagram of the data collection
×
Measurement
For evidence based on relation with other variables, a Korean version of the Nurses Clinical Reasoning Scale (NCRS) was used. The NCRS was first developed by Liou et al. [24], and the Korean version was translated, validated, and established as reliable by Joung and Han [25]. This assesses clinical reasoning competency through patient data assessments, nursing problem identification, goal-setting, and evaluating problem-solving skills. The Korean version is widely used to assess nurses’ clinical reasoning and provide educational evaluations for nursing students [26, 27]. This scale included 15 items consisting of one factor each. Cronbach’s alpha was 0.90 in this study.
Anzeige
The survey was conducted using a questionnaire on the participants’ perceptions of MAIs and their perceived academic achievement after the mock test. Their perceived academic achievement was reported through a Likert-type scale (0–10; 0 = not at all achieved, 10 = perfectly achieved).
To assess evidence based on the validity of the response process, we conducted semi-structured focus goup interviews to explore participants' experiences with MAIs. The interview questions were: “How was your experience taking the mock test using MAIs?” “How do the MAIs compare to traditional paper-based questions?” “Do you think that the MAIs are effective in enhancing clinical practice competency?” and “What were your thought processes when solving the problems?” The interviews were digitally recorded with the consent of participants, and the audio data were transcribed by the researcher.
Ethical considerations.
This study was approved by the Institutional Review Board (IRB) of Ewha Womans University (IRB No. ewha-202109–0014-01). The need of written informed consent was exempted by IRB of Ewha Womans University. All methods were conducted in accordance with relevant guidelines, ensuring that participation was entirely voluntary. An online informed consent of the study, including its purpose, methods, and procedure, and any potential risks or benefits associated with participation was posted on an online survey. The participants were also informed that they could withdraw from the study at any time and that the data of withdrawn participants would not be included in the final analysis, and that they could appeal to the Institutional Review Board (IRB) of Ewha Womans University for further explanation of the study if they wished. Only those participants who agreed to participate were allowed to complete the study.
Anzeige
Data analysis
The test items were analyzed using quantitative and qualitative data. For the quantitative data analysis, item analysis was conducted, including calculations of item difficulty, discrimination, and validity evidence based on the relation with other variables. The qualitative data analysis involved interviewing participants about their experiences solving multimedia-assisted items (MAIs) to gather evidence based on the response process.
The evaluation of test items is based on test scores, assessing factors such as item difficulty and discriminatory power, which calculate item discrimination according to the difference in the correct answer rate between high and low performers. In this study, we evaluated the appropriateness of the test by calculating the difficulty, discriminatory power, and reliability of the test items using item analysis based on classical test theory (CTT), a method commonly used for test evaluation, such as the Korea Health Personnel Licensing Examination in Korea [19].
The quantitative data were statistically analyzed using IBM SPSS Statistics version 28.0 (IBM Inc, Armonk, NY, USA) for participants’ general characteristics and mock test scores in frequency, percentage, mean, and standard deviation. Microsoft Office 365 Excel 16.0 (Microsoft Inc., Redmond, WA, USA) was used to analyze item difficulty, based on the proportion of participants who answered the item correctly, and item discrimination, calculated by comparing item scores between the higher 27% and lower 27% of participants. The statistical significance of correlations between test scores and NCRS scores and the mean difference of test result scores according to the level of perceived academic achievement in theoretical subjects were analyzed with t-tests.
For the qualitative data analysis, we used the inductive content analysis method suggested by Elo and Kyngäs [28]. In preparation for qualitative data, we selected the unit of analysis, immersed themselves in the data; the interview content was selected the unit. During the organizing phase, we conducted open coding, categorization, and abstraction. Repeated reading of the data allowed us to assign codes to meaningful units, and then grouping similar codes into subcategories and higher-order categories. To enhance the trustworthiness and reliability of our analysis, researchers coded the data, discussed together it, and if there were any discrepancies, the collaborative process of refining the interpretation of the coding and classification was repeated and agreed upon. This iterative process continued until consensus was reached on the coding and categorization. To ensure data saturation and comprehensiveness of the study results, the analysis process continued until no new categories or topics emerged. The entire analysis process was shared in a document.
Anzeige
Results
General characteristics of the participants
The study included 105 participants with a perceived academic achievement level of 6.3 ± 1.4.
Survey results of participants’ experiences taking the MAIs.
The survey results indicate that 91 participants (86.7%) considered MAIs effective in evaluating competency, and 78 (72.4%) expressed that the MAIs were more complex than paper–pencil items.
Item analysis
The average score achieved in the mock test was 18.1 ± 4.35 out of 36; the lowest score was 10, and the highest score was 27. The median score was 18.0, and the mode was 19.0.
The average item difficulty was 0.51, and the average item discrimination calculated by the difference in the correct answer rate between high and lower performers was 0.31, except for two items that indicated harmful discrimination (Fig. 4).
Fig. 4
Distribution of items difficulty and discrimination
×
Cronbach’s alpha for the final 36-item test was 0.59. Cronbach’s alpha for 34 items, excluding two items with inadequate discrimination, was 0.64.
Evidence based on the relation with other variables
The NCRS score was 54.74 ± 6.68. A correlation analysis between the mock test and NCRS scores revealed no statistical significance (r = 0.095, p = 0.336). However, the difference in the mock test's total score between the respective upper and lower groups of perceived academic achievement in theoretical subjects was statistically significant (t = 2.880, p = 0.005), as the upper group had higher scores (Table 1).
Table 1
The difference in the total score of the mock test between the upper and lower groups in the perceived academic achievement (N = 105)
PAA
n (%)
M ± SD
t
p
Upper 50% group
79 (75.2)
17.8 ± 4.5
2.880
.005
Lower than 50% group
26 (24.7)
15.0 ± 3.9
PAA Perceived Academic Achievement
Evidence-based on the response process
Focus group interviews were conducted with ten participants to obtain evidence based on the response process. Six themes were identified: “MAI is difficult to solve with memorized knowledge,” “MAI helps to understand the situation better than when it was expressed in text,” “MAI solves problems by integrating the content learned from practicum and theory,” “MAI reduces the gap between theory and clinical practice,” “The use of various multimedia materials stimulates active participation in clinical practice,” and “MAI can be helpful for future performance in clinical practice by presenting questions that reflect actual clinical situations.”
Theme 1: “MAI is difficult to solve with memorized knowledge”
The participants encountered items reflecting a clinical situation for the first time, and they felt that combining the theoretical content they had learned with the content presented through multimedia was difficult. Additionally, they were not accustomed to integrating the content of practical experience with the learned theory for solving the MAIs.
Previously, I had hardly heard the actual sound, and I had solved the question by memorizing the description of the characteristics of each auscultation sound. It wasn't easy to interpret breathing sounds. (P05)
I was a little embarrassed because questions I could only experience in practice were presented as questions. I learned breathing sounds, such as wheezing, but I could not remember them well, and it was new when I heard them. (P04)
Theme 2: “MAIs help to understand the situation better than when it was expressed in text”
The participants expressed that the MAIs enabled an intuitive understanding of what they had learned from texts and that pictures or videos helped enhance their ability to interpret situations.
We learned that the apical pulse assessment location is on the left side of the chest at the 5th intercostal space at the midclavicular line. Since the MAI shows the location visually, it is easier to remember and identify the area for measuring the apical pulse. (P03)
It was difficult to analyze the cardiac rhythm of the patient monitoring screen during the practicum. I thought my interpreting skills would improve if exposed to many MAIs before the practicum. (P04)
Theme 3: “MAIs solve problems by integrating the content learned from practice and theory”
The participants solved the problems by integrating knowledge with clinical practice or recalling their clinical practicum experiences. They considered the items easy in the cases they experienced in clinical practice. The participants expressed that the MAI was effective, as it connected knowledge with clinical practice experiences, enabling them to apply knowledge to practice and helping them remember what they had learned.
I had experience observing a patient with a chest tube drainage system during my clinical practicum. I remembered from my clinical practice experience that my clinical instructor said it was important to check the color and amount of drainage and identify if there were any changes (e.g., serosanguineous, bright red, serous, creamy). (P02)
Theme 4: “MAIs reduce the gap between theory and clinical practice”
The MAIs included knowledge related to clinical practice. In existing tests, such as the paper–pencil test, no questions focused on observations or learning content experienced in clinical situations; thus, the learners did not focus on learning these aspects of the field and considered clinical and theoretical learning as separate experiences. However, MAIs narrow the gap between clinical practice and theory and require a learning method different from that used for existing tests.
I have been studying based on the theoretical content of the exams. The MAIs were difficult because they evaluated items related to clinical practice. I could recognize the gap between theoretical learning and clinical practice. (P02)
Theme 5: “Using various multimedia content stimulates active participation in the clinical practicum”
The participants said they were motivated to expand their view on clinical situations and actively search for related data in the multimedia item-solving process. They thought that learning using various multimedia materials rather than memorizing knowledge from the textbook was necessary to determine the correct answers to MAIs. Thus, MAIs also induce active observation and participation in the clinical practicum and stimulate additional learning.
I think the experience of taking an exam with MAIs would be helpful for future practice; there will be a lot more to learn and a wider view of the field. (P01)
When MAIs are implemented, I will likely study various devices and media, such as looking for video clips or photos and learning to memorize knowledge and solve previous items. Moreover, I think I will participate more actively in the practice. (P05)
Theme 6: “MAIs can help new nurses improve their practice performance”
The participants thought that the MAIs reflected actual clinical situations and that they could confirm and prepare for them realistically, which would help their future clinical practice or work performance.
Instead of learning through texts, I think learning what to do through multimedia is the most important thing when I become a new nurse. (P02)
If these items are applied in an evaluation, students will likely become accustomed to them more easily and apply the skills when they perform in clinical practice. (P04)
As the interviews indicated, the participants integrated and applied their theory and practice experience in the MAI problem-solving process. The participants anticipated that MAIs could positively affect theory learning and clinical practicum. Furthermore, the authentic clinical situation reflected in MAIs reduced the gap between theory and practicum and helped improve practice performance. Moreover, compared with existing paper–pencil items, participants recognized that different learning strategies were required to solve MAIs.
The participants’ responses to the problem-solving process were as follows: the MAIs more accurately reflected their performance in nursing practice than previously existing items. The MAIs reflected real-world clinical situations; thus, solving MAIs requires inferring the clinical situation by integrating theoretical knowledge and experience in practice. The process of solving MAIs will ultimately be helpful in real-world practice. The evidence-based response process of the MAIs reflects the practical aspects of nursing and can improve the clinical performance of nursing students.
Discussion
The practice-based MAIs were developed to reflect clinical practice in nursing education evaluation, and we confirm the possibility of measuring prospective nurses' practice performance competency through multimedia items. Additionally, MAIs that reflect clinical practice can reduce the gap between theory and practice by improving learning and evaluation. The final score of the participants was lower than the cutoff score for the Korean Nursing Licensing Examination. In their interviews, the participants also expressed that the mock test was not easy; 72.4% of participants indicated that the MAIs were more complex than paper–pencil items. Cronbach's alpha is also a way to ensure comprehensive validity. Cronbach's alpha is one of method of ensuring comprehensive validity, and can be used as an adjunct to the validation. Also, Cronbach's alpha indicates good internal consistency of the knowledge assessment items. The results of this study show that Cronbach's alpha was low (0.64) but it may be seemed influenced by measuring knowledge in two areas: the cardiovascular system and the respiratory system.
Variations in difficulty and discrimination ability according to item type may differ depending on item characteristics and quality [14, 29‐31]. This was because unfamiliar item types were presented, wherein learning content had to be applied in an integrated way by combining the content learned through theory and practicum. Additionally, the ability to interpret and use information in clinical practice instead of simply memorizing or understanding theory was needed. This finding was partially consistent with the results of a previous study, which reported that the difficulty level of MAIs was high due to unfamiliarity with the multimedia format [32]. However, because the participants had different academic and learning achievement levels, determining the difference in difficulty according to the item types was challenging. Therefore, it is necessary to evaluate the difference in the degree of discrimination and difficulty between the MAIs and the paper–pencil items. Most participants indicated that they perceived the MAIs to measure competency more effectively than existing paper–pencil items but that the MAIs were more difficult to answer correctly. The difficulty level of this test is lower than the cutoff score of the Korean nursing licensure examination; however, some Korean nursing licensure examinations that introduce multimedia items do not use these items in all the questions. Therefore, MAIs could be used as part of summative or formative assessments to promote practical and theoretical learning.
Simulation-based clinical education can help students develop problem-solving and effective nontechnical skills and practice uncommon emergencies by allowing them to practice clinical and decision-making skills through various realistic experiences while protecting patient safety. In addition, regardless of fidelity, simulation has strong educational effects [33]. MAIs are a valuable complement to simulation-based assessments. While simulations offer realistic scenarios, multimedia items enhance the experience by providing more immersive content, such as photos, videos, images, and audio. This approach enables a more comprehensive evaluation of students’ clinical competencies, including the integration of nursing knowledge and problem-solving skills. Therefore, MAIs increase student engagement and understanding, leading to better learning outcomes. Combining MAIs with simulations gives students more realistic and comprehensive learning experiences, ultimately improving their clinical skills [34]. Compared with simulations, MAIs are more straightforward and cost-effective and effectively assess problem-solving skills in real-life situations. Multimedia items, especially videos, provide realistic representations of clinical conditions, allowing assessment of skills beyond what can be measured by paper-and-pencil tests and enhancing question comprehension. Compared with paper–pencil items, integrating video, voice, and text in electronic media facilitates content delivery and question comprehension.
Evidence-based on the response process verifies whether the construct was accurately measured through participant responses [35]. The content analysis of the interviews suggested that MAIs can positively impact nursing students’ clinical performance abilities. We confirmed the participants’ thought processes in the problem-solving process. It determined why they found the questions difficult, the perceived advantages of MAIs, and the expected effects of the questions. MAIs require critical thinking ability and clinical reasoning in problem-solving and help interpret observations [32]. This study demonstrated that MAIs necessitate crucial thinking and clinical reasoning in problem-solving, facilitate interpreting observational results, and benefit nursing students' clinical practice. Thus, it can be inferred that MAIs are an effective method for acquiring practical and theoretical knowledge.
The experience of addressing MAIs demonstrates the potential for improving how we learn from assessments in terms of driving active engagement and learning in clinical practicum. Learning clinical reasoning is essential for enhancing knowledge in clinical practicum in nursing education [36]. Tests with MAIs require complex reasoning or critical thinking skills in problem-solving, and they provide opportunities to bridge observation and interpretation [22]. Furthermore, they integrate knowledge and practice through clinical reasoning. This integration allows nursing students to assess patient situations accurately and determine the best course of action, thus bridging academic learning and clinical practice [37, 38]. MAIs scores were not correlated significantly with clinical reasoning score, but interviews regarding the process of solving the MAI revealed that the participants had the opportunity to reflect on their clinical practicum and integrate their in-class learning and practicum experience. It can be expected that this cognitive process in the MAI-solving process can potentially improve clinical reasoning skills among nursing students. Therefore, further research on this topic is necessary.
According to Hao's [39] study, which explored the effects of MAIs on students, most participants had positive attitudes toward multimedia test items but thought the items were more complex. Some participants complained that they needed to make an effort to link test items with a picture/video clip when they looked at the multimedia elements. Despite these barriers, the advantages of multimedia items were confirmed through interviews. The students tried to link their knowledge from the textbook and the multimedia content to see if they could recall something through the multimedia; they also thought that MAIs were effective in evaluating practice ability and helped with their learning through practice. This study revealed no statistically significant correlation between clinical reasoning competency and mock test scores. However, the high academic self-efficacy group had higher mock test scores. Previous research has shown a positive association between clinical reasoning and problem-solving abilities [2, 40], and academic self-efficacy has been linked to clinical reasoning skills in nursing students [41]. Furthermore, previous studies have linked self-efficacy to academic achievement [42, 43]. Therefore, it is possible that students with higher perceived academic success had higher scores because of their relatively higher self-efficacy and clinical reasoning ability. Further investigations into the relationships among self-efficacy, clinical reasoning ability, and academic success are necessary.
The interview and questionnaire results revealed that the MAIs were not to be solved through memorization but through interpretation and judgment. Nursing practice evaluation should evaluate higher levels of cognitive processes; thus, it is helpful to identify whether the item can be used to assess the intended level of mental functioning by testing response process validity when developing an item. After the developed questions were applied, data were collected retrospectively to explore the reaction process for the mock test. However, there was a limitation; the response process of each question could not be confirmed or compared with objective academic achievement.
The participants’ responses were analyzed retrospectively to determine the impact of the developed multimedia questions on improving students’ clinical practice skills and their perceptions and experiences with the MAIs. Focus group interviews were used to identify the potential for future use of MAIs by exploring perspectives on the experience of MAIs and the response process. Focus group interviews can introduce some bias due to interactions between participants. To minimize this bias, we allowed participants to express their views freely and use open-ended questions. Additionally, due to the COVID-19 pandemic, this simulation “involved participants independently accessing websites in a private setting using their own devices. Therefore, the possibility of recall bias cannot be excluded entirely as the student’s academic achievement was self-reported, and the responses were retrospective.
MAIs in mobile-based tests comprehensively evaluate work-related competencies, making learning more engaging and immersive. Integrating MAIs with simulations enhances the assessment of clinical competencies, improving students' nursing knowledge and problem-solving skills.
These findings are consistent with Jang and Suh’s [34] previous research, which showed that MAIs effectively reflect both practicums and theory. They assess authentic situations in the field while providing an immersive learning experience. MAIs can be used to evaluate authentic situations in the field while providing an immersive learning experience. This makes them advantageous in assessing clinical competencies, which can improve nursing students’ knowledge and problem-solving skills. The items developed can serve as valuable educational tools, supporting clinical practice and simulation-based learning.
In addition, the MAIs developed can be effectively used in existing formative assessment courses. The introduction of computer-based examinations that can utilize MAIs in nursing licensure examinations in Korea will underscore the importance of clinical practice assessment in nursing education. In this context, the findings of this study provide valuable evidence for preparing students for the anticipated changes in nursing education.
We have several limitations, including the idea that MAIs may not be directly comparable to traditional testing, which presents a challenge in confirming the validity of MAIs. As convenience sampling was used to select the target nursing students in this study, it is possible that a biased sample was recruited. The representativeness of the study participants may have been limited. It is, therefore, essential to consider the generalisability of the findings. We developed a multimedia-based assessment item and explored its appropriateness for assessing the integration of knowledge and clinical skills. However, its limitations include the application of the item to a single group and the focus on participants' perceptions, which do not provide definitive evidence of the effectiveness of the multimedia item. Future research should include experimental studies with a control group to confirm the efficacy of multimedia-based assessment items. There is a need for further research to address these limitations.
The significance of this study is that it developed assessment objectives derived from nursing activities and then created MAIs based on them, which could be used as a teaching and assessment strategy to assess the comprehensive skills of nursing students in both theory and practicum. The results can be used to suggest a process for developing practice-based multimedia items that nurse educators can use. Furthermore, it can be used as a basis for concurrent education and improve newly graduated nurses' clinical performance.
Conclusions
We developed MAIs based on assessment objectives derived from nursing activities and content validity were verified. Practice-based MAIs have been verified as an effective strategy for education and evaluation in assessing nursing students’ comprehensive competencies in both theory and practicum. Moreover, this revealed the potential for MAIs to bridge the gap between theory and practicum by implementing evaluation techniques that test the understanding required in real clinical scenarios. The results can be used to suggest development processes of practice-based MAIs that nursing educators can utilize when creating multimedia assessments. Furthermore, it can be used as a basis for practice-based education to enhance the practical competencies of newly graduated nurses.
Acknowledgements
Not applicable.
Declarations
Ethics approval and consent to participate
This study was approved by the Institutional Review Board (IRB) of Ewha university (IRB No. ewha-202109–0014-01). The written informed consent was exempted by the confirmation of IRB. All participants expressed their consent to participate in the study by clicking on a statement agreeing to participate during online survey. The research was conducted in accordance with the principles of the Declaration of Helsinki.
Consent for publication
Not applicable.
Competing interests
The authors declare no competing interests.
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
