Evaluation frameworks for digital nursing technologies: analysis, assessment, and guidance. An overview of the literature

This overview [12] was conducted using three different search components (roman numerals). The goal was to identify frameworks of relevance to the nursing context according to the inclusion and exclusion criteria described further below. The search components comprised: (i) a systematized literature search conducted in PubMed; (ii) a narrative review was carried out by searching google scholar, screening already identified literature [4] and reference lists of the systematically identified articles; (iii) information on relevant frameworks collected from four experts in the field of evaluation from the German “Cluster Zukunft der Pflege” (Future of Care Cluster) [13]. In this joint project, regular evaluations of digital nursing technologies have been and are carried out over the period 2017–2022. The search terms of the systematized literature search can be found in Table 1. The complete search strategy, including all three components, is shown in Fig. 1.

Scientific papers included in the search had to have been published between 2005 and 2020 in English. A summary and full text had to be available. All information was gathered in March 2020, which limits the inclusion period from January 2005 to March 2020.

The studies in question had to (i) either apply an evaluation framework to a (nursing) technology or describe the development of an evaluation framework. The selected frameworks had to (ii) be at least based on either a literature study, an empirical evaluation, or an expert survey. They had to (iii) be directly related to technologies in (nursing) care, or comparable fields like the evaluation of digital health applications in general, but they had to be potentially suitable for the evaluation of DNT. (iv) The technology specific focus must lie on Information and Communication Technologies (ICT), telemedicine, telecare, sensor technology or robotics (or their sub-sectors), as these are the most common technology areas in nursing care (definitions by Krick et al. 2019) [2]. (v) The frameworks could refer to the evaluation of different stages of the life cycle of a technology [14] e.g. before, during or after implementation.

Exclusion criteria were: articles (i) focussing on frameworks for the evaluation of specific medical technology fields (e.g. radiology, surgery) with no relation to nursing; (ii) from developing countries or underdeveloped health systems; (iii) that described study protocols; (iv) with overviews that only present categorical systems without creating a framework; (v) focussing on frameworks related to technologies for education or training; (vi) focussing on fitness applications, wellness applications or applications for general disease prevention; (vii) in a psychiatric context; (viii) that are not comprehensive, i.e. only focus on individual areas such as economy or acceptance or satisfaction or usability; (ix) which refer exclusively to the implementation and not the evaluation of the implementation.

All systematic search results were imported into EndNote X8 and reimported into the Excel screening workbook by VonVille [15]. A multi-step screening process was performed. The first step included screening 100 titles and 100 abstracts. The eligibility criteria were then refined. All titles were screened in the second step and the remaining abstracts in the third step. The eligibility criteria were then refined again before screening the full texts. If an identified article only applied a framework, for which the development is described in another article, this was an intermediate step for the identification of the framework. The original article describing the framework was than identified and included in the further steps of the analysis.

The narrative search was performed with the knowledge and eligibility criteria of the first screening process, which enabled a much more precise identification process. Google scholar was searched with the terms “framework” AND “evaluation” AND “nursing”. Articles were screened and reference lists were also examined, snowballing through the reference lists of these articles. Reference lists of the systematically identified studies and literature from a previous search [4] were also included. This method of snowballing is important for such complex search fields. It helped to obtain all relevant information on frameworks as a supplement to those not found in the systematic approach [16].

The expert consultation additionally focussed on the identification of relevant frameworks. The experts were invited to name frameworks known to them and list all frameworks used in their projects to evaluate digital nursing technologies. The whole search and identification process can be found in Fig. 1.

The systematic search in PubMed generated 1957 hits. After removing the duplicates, 1.755 remained for screening the titles. The abstracts of 113 articles were chosen for screening, yielding 69 full-texts eligible for full-text screening. The systematic search generated a total of 22 articles to be analyzed for relevant frameworks. The narrative search and the expert consultations yielded 13 articles with frameworks. There remained 26 frameworks for the final analysis process, after sorting out the duplicates generated by the different search processes. Eight further frameworks were discarded during the data extraction process because detailed analysis revealed that they did not meet the eligibility criteria. This left 18 frameworks filtered out for the final analysis (Fig. 1).

Eighteen comprehensive evaluation frameworks that can be applied to DNT are presented in the results section. According to the technology categories there are n = 7 frameworks related to information and communication technologies (with different sub-sectors), n = 3 frameworks for telemedicine/telecare, and n = 1 framework for sensor technologies. The remaining frameworks were generalistic frameworks from the areas of digital health (n = 3), health (and care) technologies (n = 2), e-health (n = 1) and clinical informatics (n = 1). The classification was based on the technologies derived from the articles with reference to the definitions of technology categories from Krick et al. 2019 [2] (Table 3). The final selection of frameworks can be found in Table 4.

The purposes and the perspectives of the selected frameworks were analyzed to answer research question (ii). Although these frameworks have the common purpose of assessing digital technologies using specific assessment categories, the purposes described beyond this differ. The detailed assessment can be found in Table 5. The qualitative synthesis identified 7 overarching categories of purposes:

A further classification of the purposes could be made by dividing them into two main categories. (A) knowledge-oriented purposes (1–4) that mainly indicate that the frameworks and their use serve to generate a certain form of knowledge. (B) practice-related purposes (5–7). Theses purposes could be summarized as application-oriented knowledge as they indicate that the knowledge will be used for a specific action such as development, implementation or decision making.

For a deeper understanding of the purpose of a framework, it is necessary to also analyze the perspective that the framework takes as it might have an influence on how the framework is intended to achieve that purpose.

Of the 18 frameworks, n = 9 had a universal perspective, which means that they can be applied to different perspectives or remain neutral thus leaving the decision to the evaluator. The remaining nine frameworks indicate or define for themselves, that they were developed from a specific viewpoint, and that the results could be interpreted from this viewpoint. Of these nine, n = 4 take the perspective of a healthcare organization, n = 2 describe the healthcare system as a perspective, n = 2 have the nurse’s perspective as a viewpoint and n = 1 is developed from an investment program perspective.

The frameworks were also scrutinized for definitions or descriptions of what is meant by “success” or “successful technology” in order to gain a better understanding of the differences that might exist (Table 5). Most of the frameworks do not have an explicit definition of success [19, 20, 22, 26, 28‐33]. In such cases, the evaluation criteria described in the frameworks and the resulting interpretation of the evaluation results could be used to make a statement about whether or not the technology in question was successful.

The qualitative synthesis of the success definitions/ descriptions of success identified three categories:

The strengths and weaknesses of the frameworks under consideration were assessed in order to contribute to a better understanding of what constitutes a good (DNT) evaluation framework and answer research question (iii). Previously developed quality criteria for DNT Frameworks were used for the assessment (as described in the methods section). A detailed analysis of the assessment can be found in Table 6. (+) stands for strength and (−) denotes a weakness in a certain assessment area.

The assessment revealed differences and similarities between the frameworks under study. All frameworks included a description of their intended purpose and the question(s) addressed. Of the 18 frameworks, 14 do not explicitly describe an application setting, since these frameworks were developed with generic setting approaches. They therefore received a (+) in the evaluation for the universality of the setting and at the same time a (−) because they are not specifically related to a setting. However, most frameworks describe specific technology areas on which they focus. Only six have universal designs, for potentially different technologies, so that they have been assessed with a (+) for universality and a (−) for being non specific. Due to their elaboration, n = 12 of the frameworks are easily transferable to other contexts. N = 11 frameworks have a very clear visual presentation and n = 9 include a visual representation of connections and relationships of individual aspects within the framework. On the other hand, this means that n = 7 frameworks were not without visual weaknesses and n = 9 frameworks did not show visual connections between the aspects with which they were concerned. Almost all frameworks (n = 15) included transparent definitions of terms and key concepts and are transparent in terms of the development process (n = 14). However, many of the frameworks have weaknesses in applicability, clear guidance, and assistance for the interpretation of the results. Only n = 9 frameworks are strong in the description of an application strategy and instructions for use and only n = 6 include soundadvice on how to interpret the results. Furthermore, many articles on the frameworks do adequately discuss weaknesses and limitations - if at all(n = 11). The frameworks with the highest scores across all assessment categories were:

The detailed analysis of the assessment can be found in Table 6. More detailed descriptions of the strengths and weaknesses can be found in Additional file 3.

A detailed analysis of the areas of evaluation included in the frameworks was carried out to answer research question (iv). The analysis resulted in definitions for nine evaluation areas that are described in Table 3.

Table 7 shows a comparison of the frameworks regarding the evaluation areas they cover. The results of the analysis of these areas indicate where the frameworks have their main areas of focus. The colour coding in the table signals that a framework covers a certain area. The allocation was based on the definitions and the sorting logic described in the Methods section above. The specific perspective described for the frameworks was also included in the table to crosscheck whether it is possible to make generalized statements about the existence of certain evaluation areas in relation to the perspective taken.

Most frameworks (n = 17) contained evaluation aspects of the area of objective value/effect. Also, the evaluation of the specific product/technology aspects (n = 16), aspects of the organization (n = 15) and the relationship of individuals to the technology (n = 14) was largely represented.

Societal (n = 10) and strategic (n = 6) aspects, as well as ethical aspects (n = 4), were not as frequent. A closer look reveals that these aspects are particularly rare when the perspective described is the healthcare organization or the nurses. At the same time, these aspects are jointly represented three times in the assessment if the perspective “universal” was assigned (in the HIS Success Framework [27], MAST Manual [31] and the Khoja–Durrani–Scott Framework for e-Health Evaluation [22]).

Frameworks for which a universal perspective has been described (n = 9) also cover more areas more often in general (coverage of 7,2 areas on average). While frameworks for which a healthcare organization perspective is described (n = 4) only cover 4 areas on average (none of them covers the areas focus, societal or ethics), and frameworks covering the nurse’s perspective (n = 2) only cover 2 and 4 areas respectively (none of them covers the focus, societal, ethics, economic or strategic aspects). The frameworks with a healthcare system perspective (n = 2) cover 4 areas on average (none of them covers the focus, individual, organization, ethics, or strategic areas) and the only framework with an investment program perspective covers 5 areas (does not include the areas focus, societal, ethics or strategic). A mapping of the content of the frameworks to the evaluation areas was carried out and can be found in Table 6.

It is important to understand that although the common purpose of the frameworks dealt with here is to assess digital technologies using specific evaluation categories, the purposes described beyond this differ. Most frameworks included here were developed (i) to help and guide researchers in the design and realisation of an evaluation [14, 22, 25‐27, 32‐35], and/or (ii) to support the identification of success and failure factors (and to help manage them) [14, 19, 21, 23, 27, 34]. (iii) Four frameworks were specifically designed to assess the performance/success of a technology (outcomes, impact, errors, deficiencies) [28‐31], but only one framework (iv) was drawn up to help to make the results comparable [19]. This could be since the other articles simply assume the comparability issue to be implicitly logical and therefore do not name it explicitly as a purpose. These four purpose categories (with the exception of the management of success and failure factors) could be summarized as knowledge-oriented purposes, indicating that the frameworks mainly serve to generate a certain form of knowledge.

Some frameworks indicate a more practice-related purpose such as (v) contributing to the quality and development of the technology [31, 34, 35], (vi) supporting the implementation of a technology [33‐35] or (vii) providing help in decision-making [20, 24, 31]. These purposes can be summarized as application-oriented knowledge generation. As the classification shows, a DNT framework can have several purposes in both categories: knowledge generation and application orientation. The knowledge-oriented categories i, ii, iii, iv and the practice-related category vi are in line with seven out of eight general attributes of evaluation frameworks in healthcare identified by Bradford et al. (2019) These are 1. simplify a complex (evaluation) process, 2. provide structure (for an evaluation), 3. facilitate the evaluation process, 4. promote meaningful evaluation, 5. identify and explain outcomes, 6. generate transferable lessons, 7. identify mechanisms driving or inhibiting change [39]. Bradford and colleagues also consider it an important element that frameworks help to identify relevant stakeholders [39]. This element was not mentioned as a purpose by any of the frameworks in the present study. In general, DNT evaluation and DNT relevant evaluation frameworks can consider knowledge-oriented and application-oriented purposes. It does not make a framework better or worse if it includes only one of the categories, but having a clearly defined purpose is a quality criterion when choosing a reliable framework. Researchers must be aware of the intended purpose of a framework when choosing their evaluation approach.

A closer look at the perspectives of the frameworks discussed here raises the question whether the perspective under which a framework has been developed, may have an impact on the evaluation categories included. The analysis of the frameworks revealed five perspectives: universal, healthcare system, healthcare organization, nurses, and investment program perspective.

Of the 18 frameworks, n = 9 had a universal perspective, which means that they can be applied to different perspectives, leavingthe decision is left to the evaluator. Frameworks for which a universal perspective has been described cover more evaluation areas in general (coverage of 7,2 areas on average), what supports this assumption. Frameworks for which a healthcare organization perspective is described only cover 4 areas on average (none of them covers the areas focus, societal or ethics), and frameworks with the nurse’s perspective only cover 3 areas on average (none of them covers the aspects focus, societal, ethics, economic or strategic). Those frameworks with a healthcare system perspective (n = 2) cover 4 areas on average (with none of them covering the areas focus, individual, organization, ethics, or strategic) and the only framework with an investment program perspective covers 5 areas (does not include the areas focus, societal, ethics or strategic). The perspective with which a framework has been developed is therefore always an important feature to consider when selecting a framework, as this could influence the evaluation aspects included.

In addition, there are several stakeholder perspectives that were not taken as the main perspective in the frameworks. These perspectives are the payors perspective, the perspective of the patient / person in need of care and the perspective of the informal caregiver. All these perspectives are particularly important in the context of DNTs. However, it should be mentioned that these perspectives are often included in the universal frameworks like the patient perspective in the MAST [31] or the 3rd party payment aspect in the comprehensive evaluation framework for telemedicine implementation [24]. In summary, when selecting and using a comprehensive evaluation framework to evaluate a DNT, the perspective of the chosen framework and the intended perspective of the evaluation should always be reflected and contrasted, as there may be a relationship between the perspective and the evaluation areas included in the framework. The frameworks with a narrow perspective, such as the nurses’ perspective, cover fewer evaluation areas overall in this study, while those with a universal perspective cover significantly more. Three examples explaining this in more detail can be found in the discussion section entitled “Discussion of the three most relevant frameworks”. If researchers require a comprehensive framework with as many evaluation areas as possible, they must choose a framework with a universal evaluation perspective.

It is not easy to define a successful digital nursing technology. Nguyen et al. (2014) argue that the success of a technology may be “disputed depending on the interests of the evaluating party.” [40] This is in line with the success description of Lau (2009) who sees it in the context of an “ongoing negotiation and adaptation of interrelationships” of the healthcare professionals involved [41]. A large multi stakeholder Delphi study conducted by McNair and colleagues (2006) [42] concluded that “success cannot be characterized along one single axis” and therefore defined success as the fulfilment or non-fulfilment of five consensus based aspects, namely (1) the wide usage in daily practice, (2) the fulfilment of the role and tasks it was planned for (in a specific environment), (3) the support of good medical practice (benefitting the patient), (4) the benefits to the healthcare organization and working conditions, (5) easy upgradability to adapt to the developments in practice [42].

Aspects 1–4 from McNair and colleagues were also identified as success definitions in the analyzed frameworks in this article (1. Success is when the technology achieves its intended purpose [27]; 2. Success means achieving implementation, dissemination and/or sustainability of a technology [14, 23, 24, 34]; 3. A successful technology must generate a net benefit [21]). This suggests that these aspects could be universal definitions of success for digital nursing technologies. Aspect 5 of the McNair study could be added as equally important.

However, looking at success from a certain perspective in a specific context might lead to additional perceptions or definitions of success – which could influence the evaluation.

In view of these differences in definition, it must be concludedthat the “success” of a DNT is a relative term, made up of various aspects and depths of success definitions, the selection of which depends strongly on the evaluation perspective. This insight is decisive for the evaluation of a DNT because it should always be reflected for whom and from which perspective an evaluation is conducted and what is considered as successful.

The strengths and weaknesses analysis process in this article identified framework components where more guidance would be beneficial, and which are important to consider when selecting a DNT evaluation framework. The assessment included the focus of a framework, the illustration, terminology, instructions for use and scientific quality. The strengths and weaknesses criteria were created especially for DNT evaluation frameworks although there are other quality criteria that could be applied from other healthcare fields. For example, Bradford et al. (2019) [39] provide 6 quality criteria for frameworks, most of which are similar to the criteria in this article. Bradford et al. also suggest assessing whether the frameworks help to identify and include stakeholders as well as mechanisms that drive or inhibit change – which might be also helpful selecting a framework.

The results of the framework assessment presented in this article need to be seen in a wider context. The evaluation was carried out with a view to assessing of perceived strengths and weaknesses in general. Several aspects mighthave been rated differently in other contexts, e.g., assessing the transferability of a framework to only onespecific context, as with the Health Technology Adoption Framework [33] and the CISSM [21] as a strength or a weakness depends on the viewpoint of the planned evaluation. If the context is surgical the Health Technology Adoption Framework might be the perfect choice, although it is not transferable to other contexts.

By showing and applying assessment possibilities for DNT frameworks this article gives guidance for the selection of appropriate DNT evaluation frameworks. An assessment of strengths and weaknesses in combination with the other important information presented in this article is crucial.

This section describes the three highest rated frameworks of the strengths and weakness analysis in more detail in order to give researchers insights into the analytical dimensions described in this article. The complete data for all frameworks are summarized in Tables 4–7. The description is written in an application-oriented way to support researchers. Problematic aspects in the selection of DNT frameworks are also discussed.

The Health Technology Adoption Framework shows validated criteria for assessing new health technologies for adoption at local level with a hospital focus in the surgical context. It is developed generically with respect to the technology to be assessed and can therefore be applied to several DNTs.

The evaluation categories presented focus on the product/technology, its objective value/effect in the adoption process, the evaluation of organizational aspects as well as economic and strategic aspects with respect to the (potential) adoption process. Individual, societal, ethical and aspects of the area “focus” are not covered. This distribution of the aspects considered could be related to the fact that the framework was developed from the perspective of the healthcare organization and is intended to help decision makers (e.g., nurse managers) to evaluate the suitability of new technologies as well as to facilitate smooth adoption from the perspective of the organization. This means that while aspects such as security, costs and strategic fit in the organization are considered, individual aspects like usability or acceptance are not. The Health Technology Adoption Framework covers more evaluation areas (n = 5) than the other three frameworks in this study with a health organization perspective (see Table 7). Overall, however, all frameworks with this perspective cover fewer areas than those with a universal perspective. This limitation regarding the evaluation areas should always be considered, when deciding on a framework.

The Health Technology Adoption Framework performs very well in the framework quality assessment. The descriptions and definitions are accurate and complete. An evaluation tool with appropriate evaluation categories, an application strategy, relevant questions, and a scoring logic is provided. The development process of the framework is also presented transparently, and the corresponding limitations are named in the article. Only the transferability to other settings is limited and there is no visual presentation of the framework. It is displayed in a table and not as a graphic. Overall, the Health Technology Adoption Framework provides a very good evaluation basis with valid assessment criteria.

The CISSM framework is designed to assess the success of clinical information systems (CIS) from the nurses’ perspective. The framework is suitable for all DNTs from the CIS category in the hospital setting. CISSM focusses only on a specific selection of evaluation areas: product/technology, objective value/ effect, individual aspects, and aspects of the healthcare organization. The societal, ethical, economic and strategic aspects are not considered.

Compared to the second framework with the nurses’ perspective from this study (Adapted nursing care performance framework) [30], the CISSM covers two more evaluation areas. The Adapted nursing care performance framework [30] only covers the areas of the objective value/ effect and individual aspects of technology. By comparison, CISSM is the more comprehensive framework - but overall, both cover only a few of the possible evaluation areas. It is certainly critical that none of the two frameworks covers ethical aspects. This should be considered when selecting and applying them, as the ethical aspects are also important in this context. Additional ethical evaluation criteria from other frameworks could be added to fill this gap.

When selecting a framework with such a narrow perspective it should be borne in mind that aspects of the “bigger picture” might not be depicted. The specific success definition of a technology in this framework focusses solely on the nurses’ net benefit. Success in this case is the “degree to which a nurse believes that using a particular system enhances job performance”. This makes the CISSM framework particularly suitable for use when the nurses’ perspective is the object of research. If a broader spectrum of evaluation aspects is to be researched, then this framework would not be the right choice.

At the same time, however, the framework scores very well in the strengths and weaknesses assessment for this specific perspective. It has a very clear description of purpose, target setting, and technology addressed. It is very well illustrated, and the connections are visualized. The key concepts and terms a are completely defined. A procedure and evaluation matrix are provided as well as examples for the interpretation of an evaluation. The development process was fully explained.

The main shortcoming of the article on this framework is that no explanation is given about its limitations. Overall, the CISSM framework is a good basis to assess the success of clinical information systems (CIS) from the nurses’ perspective and the article provides a good basis and example for the methodology.

The Nonadoption, Abandonment, Scale-up, Spread, and Sustainability Framework (NASSS) was designed to help predict and evaluate the success of health and care technologies and to support the design, development, implementation, scale up, spread, and sustainability of technology-supported health or social care programs by identifying key challenges in different domains and the interactions between them. As the description shows, this is the most comprehensive and generic framework of the three described. The framework does not focus on any specific technology or setting which makes it suitable for any DNT. The comprehensiveness of the framework is also reflected in the included evaluation aspects. The framework considers aspects of the focus, product/technology, objective value/ effect, individual aspects, aspects of the organization, societal aspects, and economic aspects – which makes it a compendium of evaluation options for DNTs. Only the ethics and strategic aspects as defined in this review are not covered by the NASSS Framework. Thus, among all the frameworks covered in this study, the NASSS framework is one that covers the most evaluation areas.

A special feature of this framework is that it addresses the issue of complexity. It categorizes the interpretation of the evaluation domains as simple, complicated, or complex. The level of complexity of each evaluation aspect is seen as the greatest challenge to scale-up, spread, and sustainability. No specific evaluation perspective is given, which makes it a universally applicable framework. However, this universality is also accompanied by a limitation. As no specific setting, technology or perspective is given, the researcher using it must always reflect on the individual application strategy. The NASSS Framework is not a directly applicable or formulaic instrument, which is reflected by the authors of the framework themselves. Specific additional tools to solve this problem have been published recently [43].

Apart from this, the NASSS framework scores very well in the strengths and weaknesses assessment.

Looking at all three frameworks in comparison, the question arises whether a framework with a specific perspective is always preferable to a universal framework. This question can only be answered on a case-by-case basis when assessing the fit of a framework to the research situation.

It is important to discuss the limitations of such an extensive procedure as described in this article. A three-component search process was chosen for this article., namely (1) a Systematic search in PubMed, (2) a narrative search in Google Scholar and reference lists, (3) expert consultations. Additional databases could have been systematically searched. Experience from previous systematic search processes in such complex fields has shown that a combination of systematic searches and other methods is a good way to identify relevant articles [4]. However, with such a search procedure there is always a chance that something is left undiscovered. There is also a limitation associated with a single researcher conducting a study that may have affected the search and analysis process. Single studies with frameworks might have been overlooked [44] or specific biases in interpretation could have occurred in the analysis process [12]. These limitations were sought to be minimised through expert workshops and consultations to identify relevant frameworks and discuss the methodology and the analysis process of this article. Also, the exclusion criteria were very strict. Only literature published in the English language was included. The DNT related criteria were based on the knowledge of Krick et al. 2019 [2] and Krick et al. 2020 [4] but still led to a heterogeneity of frameworks due to the broadness of the field.

Only comprehensive, technology related frameworks were included, which ruled out many specific frameworks that could potentially also have been included. Frameworks focussing exclusively on special fields like health economics or acceptance could have been included, because they are certainly relevant for DNT, but were excluded due to their specificity and the large number of different frameworks available. Generalistic evaluation frameworks for (complex) healthcare interventions like the Consolidated Framework for Implementation Research (CFIR) [45] or on HTA Frameworks could have also been included (e.g. Integrate HTA [46]) – but it was necessary to limit the included frameworks to allow statements on this specific field of research.

The framework analysis itself has further limitations. Due to the heterogeneity of the frameworks and the difference in their content the categories and the assignment of the framework content to these categories is subjective. Even though the process of evaluation and classification was carried out with the utmost care, a non-overlapping categorization of these categories was not possible due to the complexity of the frameworks content and the interconnectedness of different categories. There are overlaps between the category of technology and individual as well as ethical and societal aspects. It should also be considered that the generalized statements made in this article can only provide initial indications in this specific field. No statistical analysis was carried out, but a qualitative and hypothesis-generating analysis. Despite these limitations, this article makes an important contribution to further research in the field of DNT evaluation.