Instruments for measuring nursing research competence: a COSMIN-based scoping review

Nursing research competence (NRC) refers to the individual nurse’s ability to conduct nursing research activities [1, 2]. Evidence-based nursing has developed rapidly in recent years, and the importance of evidence-based nursing in improving clinical nursing quality has been confirmed by many researchers [3‐5]. However, there is currently a lack of relevant available evidence focusing on clinical problems, so it is necessary for some nurses with nursing research competence to conduct original research on clinial practice in order to generate relevant available evidence and promote evidence-based nursing practices [6]. Specifically, enhancing the NRC of nurses holds significant importance in the advancement of high-quality clinical nursing research. For clinical nurses who are inclined towards research, possessing a strong NRC competence can motivate them to address clinical issues scientifically, apply evidence-based practices, and contribute to bridging the gap between theory and practical application [7]. As future nursing researchers and nurses, improving the NRC of nursing students has a positive promoting effect on the future development of nursing [8, 9]. Using NRC instruments are necessary to evaluate the NRC of nursing staff and the effectiveness of interventions [8, 10].

Measuring the NRC of nursing staff is important for research, education, and management purposes. Research has shown that clinical nurses are the end users and producers of nursing research, and nurses with research competence can promote the development of nursing discipline [11]. The prerequisite for improving nurses' research competence is to clarify the current situation and influencing factors of nurses' research competence, which provides a precise theoretical basis for formulating intervention plans to improve nursing staff's research competence [11]. However, an important way to clarify the current state of NRC and its associated factors was to use precise NRC instruments to measure NRC. They can provide evidence for building effective intervention strategies in research, evaluating teaching quality and promote the development of courses or training programs in education [9]. In addition, using the NRC instruments to measure the NRC of nurses could help nursing managers identify which nurses have good research competence, assist in organizing and conducting research projects, and cultivate research-oriented nurses in a targeted manner [10, 12]. Therefore, it is important to evaluate the measurement properties and the application of existing NRC instruments. This could aid in selecting the most appropriate instrument and in revising or/and developing higher-quality instruments. COSMIN (Consensus-based Standards for the Selection of Health Measurement Instruments) is a consensus-based standard for the selection of health measurement instruments, which can evaluate the methodological quality and measurement properties of measuring instruments and provide recommendations for instrument selection [13]. This study evaluated all measurement properties of the NRC instruments based on COSMIN methodology. For more detailed steps on COSMIN methodology were showed in the 'Methods' section.

This scoping review evaluated eight NRC instruments using the COSMIN checklist, but none of them have assessed all measurement properties. Among the existing eight NRC instruments, NRC instrument ⑦ is the most widely used, and it has only been used by Chinese scholars as of now. This may be because the NRC instrument ⑦ was developed by Chinese scholar Pan et al. In addition, NRC instrument ⑦ was developed in 2011 and was one of the earliest NRC instruments developed in China.

Lack of reference to the target population during development was an important disadvantage in developing NRC instruments. The items in NRC instruments should be both relevant and comprehensive for the "construct" being measured, as well as comprehensible for the study population. These elements are crucial for ensuring content validity, which is crucial for ensuring an instrument's psychometric properties, and? requires cognitive interviews with the target population [13, 20]. However, only two NRC instruments (⑦⑧) conducted cognitive interviews with the target population during development, and these interviews lacked detail. However, details of the cognitive interview process were missing. Additionally, three studies (⑤⑦⑧) asked the target population about the relevance, comprehensiveness, and comprehensibility of the instrument's content validity, while experts were consulted about the relevance and comprehensiveness of the instruments in all three studies. Comprehensive details of the cognitive interview process are necessary to evaluate content validity. However, the published articles lacked such details, which may be due to the COSMIN guideline being published in 2018, while most (75%) NRC instruments in this review were developed prior to 2018 [25].

Confirmatory factor analysis (CFA) and exploratory factor analysis (EFA) were performed on six NRC instruments (③④⑤⑥⑦⑧), with two instruments (③⑦) reporting CFI values of 0.98 and 0.97, respectively. These NRC instruments (③④⑤⑥⑦⑧) are capable of reliably capturing the theoretical structure and idiosyncratic degree [26]. In other words, these NRC instruments (③④⑤⑥⑦⑧) could effectively represent the theoretical concept of nursing research competence.

Most studies focused on internal consistency, which reflects the correlation of items in the NRC instrument or subscales. However, some NRC instruments ①②④⑤⑥ and ⑧ did not meet the criterion for sufficient structural validity [13]. Therefore, reviewers should evaluate the structural validity before assessing internal consistency and provide detailed information in the future. In addition, NRC instrument ③ only reported Cronbach's alpha for the total instrument, whereas in future studies, reliability analysis should be conducted to evaluate Cronbach's alpha for each dimension of NRC instrument ③. It is worth noting that the Cronbach's alpha values of three subscales in NRC instrument ⑦ were below 0.70 (0.68, 0.68, 0.66, respectively). The value of Cronbach's alpha is influenced by factors such as the number of items, item interrelatedness, and dimensionality [27]. The low Cronbach's alpha value suggests that heterogeneity exists between some items of the instrument and that these items should be revised or removed. One straightforward method is to calculate the item-total score correlation and eliminate items with low correlations [27, 28]. Therefore, additional studies are necessary to enhance and assess the internal consistency of NRC instrument ⑦. Moreover, the sample sizes of studies assessing the internal consistency of NRC instruments ① and ② were below 100, resulting in downgrading of the quality of evidence on internal consistency. Consequently, a larger target population is required to further evaluate the internal consistency of these two NRC instruments (①②).

The prerequisite for widespread use of NRC instruments is to ensure reliability, minimal measurement error, and sensitivity to changes. Except 'The Nurse Research Questionnaire among Nurse Clinicians ②', all NRC instruments have been evaluated for reliability. The reported ICC values were not clearly documented in the literature, indicating that their reliability was not satisfactory. Although reliability and measurement error are interrelated measurement properties [25], there is currently no NRC instrument available to evaluate measurement error. Measurement error refers to the systematic and random errors in the target population's rating, which are not attributed to actual changes in the structure to be measured [27]. However, the credibility of the results obtained from the NRC instruments evaluated in this study may be compromised by the lack of measurement error assessment. Therefore, future research should address this issue by evaluating the measurement error of NRC instruments. Moreover, none of the NRC instruments were tested for responsiveness, which may be attributed to the lack of longitudinal validation, including intervention studies [29]. Although NRC instruments have been utilized in some intervention studies to evaluate outcomes, the minimal important change/distribution of scores (MIC/SD) change score for the stable group target was not calculated [13]. Future research should include more longitudinal or intervention studies that employ NRC instruments to assess their reliability and responsiveness [25].

Criterion validity of NRC instruments ④ and ⑧ was reported using the author-defined gold standard in the respective articles. However, we question the appropriateness of using 'The Anxiety Scale Towards Research and the Attention Scale Towards Scientific Research' and 'General Self Efficiency Scale' as the gold standard in the studies, as they may not be ideal measures for NRC instruments ④ and ⑧, respectively [13]. According to the guidance of the COSMIN guideline, the criterion validity reported in the articles for NRC instruments ④ and ⑧ would be more appropriately considered as convergent validity, which pertains to the hypothesis of the targeted instrument's relationship with other relevant measurement instruments [30]. As a result, we opted not to evaluate criterion validity of NRC instruments ④ and ⑧, and instead focused on testing hypotheses for construct validity (specifically, convergent validity). The challenge of identifying a suitable 'gold standard' for NRC instruments may be attributed to the difficulty in establishing an objective index for NRC. To address this issue, we recommend enhancing the development of objective evaluation indicators for NRC, which could lead to the formation of a 'gold standard' instrument. Having a reliable gold standard could aid in the development and validation of more user-friendly and efficient NRC instruments.

Hypotheses testing for construct validity is defined as the relationships of scores on the instrument of interest with the scores on other instruments measuring similar constructs (convergent validity) or dissimilar constructs (discriminant validity), or the difference in the instrument scores between subgroups of people (known-groups validity) [31]. The study on NRC instrument ④ reported hypotheses stating that individuals with high levels of research competency would hold more positive attitudes towards scientific research and experience less anxiety towards research [16]. Although the hypothesis was not explicitly stated in the study on NRC instrument ⑧, the positive correlation observed between NRC and general self-efficiency could be used to draw conclusions about the construct validity of NRC instrument ⑧ [13, 17]. The studies on NRC instruments ④ and ⑧ all formulated hypotheses for testing construct validity, with expected directions of effect. To accurately represent the underlying theoretical structure of nursing research competence, hypotheses should verify both the magnitude of correlations or differences [31].

No studies evaluated the cross-cultural validity of the NRC instruments. Cross-cultural validity refers to the degree to which the performance of the items in the translated or culturally adapted instrument adequately reflects the performance of the items in the original version of the instrument [13, 31]. Cross-cultural validity is important to ensure that a measurement instrument can accurately measure what it is intended to measure among different target populations [32]. The evaluation of cross-cultural validity should be conducted across different groups, cultures, and languages [13, 31, 33]. We recommend that it be conducted for NRC instruments in different groups such as clinical nurses and nursing students, as well as across different cultures and languages to ensure their reliability across different contexts.

The study recommended all NRC instruments as Grade B. However, the lack of specific information regarding the evaluation of content and construct validity may have influenced this rating. Although all NRC instruments could be recommended for use, but further studies are necessary to confirm the reliability of all NRC instruments. It is also important to note that the stringent evaluation method of COSMIN bases the score of each measurement property on the lowest-scoring item across all items. This approach may result in lower evaluations for instruments with insufficient information [25, 34, 35]. In this study, interpretability and feasibility were not evaluated, so future research is suggested to assess these properties.

We have observed that even though the of developers these NRC instruments have limited the target population (nurses/nursing students) and provided clear definitions of NRC, there was no significant difference in their definitions of NRC for nurses and nursing students. Furthermore, minimal discrepancies exist among the NRC instruments developed for distinct populations (with the exception of attitudes towards nursing research, problem finding competence, research design competence, and paper writing competence). Therefore, further research should investigate whether a distinction in NRC between different populations (nurses and nursing students) is necessary. If different populations need to have different NRC, it becomes imperative to delineate the precise implications and extent of NRC for each distinct group. Conversely, if there is no difference, all the NRC instruments can be universally used without limiting the target population.

All the NRC instruments were recommended for use under category B. Considering that the COSMIN guidelines recommend using instruments categorized as (B), and given the current widespread used of NRC instruments, it was not recommended to develop new NRC instruments. Instead, existing NRC instruments that were recommended as category (B) could be optimized to the greatest extent possible. For example, it's worth noting that the evaluation on content validity of NRC instruments ①②③④⑥ did not encompass nurses and/or nursing students (the main reason why these NRC instruments cannot be recommended as category (A)). Therefore, it is possible to consider conducting additional interviews with nurses and/or nursing students, comparing the NRC instruments ①②③④⑥ items with the interview results to estimate which NRC instruments have very good content validity. Furthermore, the evaluation results suggest the Cronbach's alpha values of some dimensions of NRC instruments ⑤⑦⑧ are below 0.70. It's imperative to undertake large-scale studies that validate these dimensions comprehensively. If the Cronbach's alpha values for all dimensions fail to reach the desired threshold even within a large-scale study, a revision of the dimensions and entries within the existing NRC instruments should be deliberated. In a broader context, future researchers are encouraged to develop novel measurement instruments guided by the COSMIN framework. Notably, the development process should incorporate qualitative interviews with the target population, specifically focusing on gauging reliability, comprehensiveness, and understandability of content validity within these instruments. Subsequently, extensive validation of internal consistency within a sizable sample of the target population is pivotal, ensuring that instruments could be categorized as (A) merit recommendation for practical use.

By summarizing the usage of all NRC instruments, we found that nurses and nursing students were currently the main focus of research using NRC instruments, and more than 50% of the research was cross-sectional. This provides a theoretical basis for nursing researchers to understand the current situation of nurses and nursing students' NRC and develop precise intervention plans to improve their NRC. It is worth noting that although RCT and Before-after study in the same patient have been conducted, there were few studies with a large sample size and a lack of longitudinal evaluation of the effectiveness of NRC intervention by nurses and/or nursing students. In addition, almost all research was conducted in China, which may be due to the fact that the majority (87.5%) of NRC instruments were first developed by Chinese researchers. Therefore, in the future, nursing researchers from different countries should improve existing NRC instruments, select appropriate NRC instrument based on specific contexts and cultural backgrounds, and conduct cross-cultural testing to clarify the NRC competence of nursing staff from different countries and provide a theoretical basis for formulating intervention measures.

The study recommended NRC instrument ⑦ as the most suitable among existing instruments, but calls for further research on the measurement properties of NRC instruments, especially cross-cultural validity, measurement error, and criteria validity. Additionally, researchers should evaluate and report on the interpretability and feasibility of NRC instruments, and explore the development of more reliable and feasible instruments for different nursing populations based on a unified concept of nursing research competence.