British Columbia Healthy Connections Project process evaluation: a mixed methods protocol to describe the implementation and delivery of the Nurse-Family Partnership in Canada

The Nurse-Family Partnership (NFP) is a home visitation program for first-time, socially and economically disadvantaged pregnant women and mothers with young children. The overarching goals of this evidence-based public health intervention are to improve: 1) pregnancy outcomes; 2) child health and development; and 3) parents’ economic self-sufficiency [1]. Over the past three decades in the United States (US), the effectiveness of the NFP has been tested in three randomized controlled trials (RCTs) with diverse populations of American families living in unique geographic locations: Elmira, New York, Memphis, Tennessee, and Denver, Colorado [2]. Across these trials, consistent and enduring effects of the NFP have been established, including demonstrated improvements in prenatal health behaviours, birth outcomes, and child mental health and developmental outcomes [2]. Findings from the Memphis RCT also conclude that participation in NFP reduces preventable death among nurse-home visited mothers and their children [3]. Furthermore, NFP has been established as the intervention with the best evidence for preventing child maltreatment and other associated outcomes, including reductions in child injuries and emergency room visits [4]. Given the level of evidence attributable to this nurse home visitation intervention in improving the lives of disadvantaged children and families, extensive interest has been expressed by societies outside the US in implementing the NFP. Ongoing evaluation and international NFP research is coordinated by Dr. David Olds (the developer of the NFP) and his team at the Prevention Research Center for Family and Child Health, University of Colorado Denver. In countries outside of the US, there is considerable geographic, demographic, socio-economic, public policy, nursing workforce, and healthcare system variation; it therefore cannot be assumed that the impact of the NFP seen in the US trials will be the same in other countries. Dr. Olds has developed a four-phase model for adapting and evaluating the NFP in international contexts: 1) adaptation: exploration of adaptations needed to deliver the program in local contexts while ensuring fidelity to the 18 core elements [5] of the NFP model; 2) feasibility and acceptability pilot studies: initial small-scale assessment and implementation studies to inform what additional program adaptations may be needed to meet the needs in the new country’s context; 3) RCT: expansion and testing of the intervention in a large-scale RCT to determine NFP’s effectiveness in the specified context; and 4) continued refinement and expansion: if the RCT produces outcomes of public health significance, decisions can then be made regarding wide-scale expansion of the NFP program within the new country [6]. The adaptation of the NFP in England (known there as the Family-Nurse Partnership) is a prime example of how this process for adapting existing evidence-based interventions for new societies can be implemented. In England, the feasibility of implementing this home visitation program along with estimating program costs and determining program impact on key stakeholders was evaluated prior to initiation of the RCT to determine overall effectiveness [7].

The first two steps of the process for international NFP replications and evaluation were completed in Hamilton, Ontario, Canada. A concurrent mixed methods pilot study was conducted from 2008 to 2012 to determine overall feasibility and acceptability of the NFP program in Canada with 108 women who met the study eligibility criteria [8, 9]. Based on this work, to successfully deliver NFP in Canada, minor adaptations were made to the US NFP model elements. A list of the model elements to guide subsequent implementations in Canada is provided in Table 1.

Given the findings that the NFP was both feasible to deliver and acceptable to stakeholders in a predominately urban Canadian setting, the McMaster research team sought out provincial partners interested in participating in Phase 3 – the conduct of an RCT to determine overall effectiveness of the NFP to improve maternal and child health outcomes in a Canadian context [10]. As the NFP aligned with key mental health strategic plans and core public health programs, the province of British Columbia (BC) emerged as ready and committed to conducting the RCT. The BC Healthy Connections Project (BCHCP) is the resultant scientific evaluation that includes both an RCT to measure NFP’s effectiveness compared to existing services, and a process evaluation to describe how the NFP is implemented and delivered across the five participating BC Health Authorities (HAs), including describing variances within and between sites [11]. The BCHCP includes one final third project - the Healthy Foundations study that is focused on exploring the biological mechanisms that link intervention and behavioural outcomes in children. This paper outlines the BCHCP process evaluation methods; the BCHCP RCT protocol is the subject of a separate manuscript.

As a complex intervention, the NFP has multiple interacting components, a range of diverse prenatal, child and maternal outcomes to influence, flexibility in intervention delivery, and sensitivity to local organizational and contextual factors [12]. For example, although PHNs have guidelines on when to conduct home visits, the timing and frequency of home visits will be influenced largely by individual client needs, availability, and engagement in the program. At the level of the individual HAs who are responsible for implementing and delivering the NFP as part of the BCHCP RCT intervention arm and process evaluation, organizational factors such as the level and quality of nursing supervision provided, the time allocated for PHNs to deliver the NFP (e.g., full-time or part-time), and administrative supports and resources for the BCHCP in general have the potential to influence fidelity to the program model. Each HA is also unique in its geography, population characteristics, and the pre-existing culture of partnerships and collaborations that are needed to support and sustain NFP at the community level. This level of complexity creates challenges for evaluating the effectiveness of the intervention and for understanding how the causal mechanisms of the intervention may influence intended outcomes [11].

Increasingly, process evaluations are carried out as part of the overall approach to conducting a comprehensive evaluation of a complex intervention [13]. A process evaluation can help to answer how and why an intervention succeeds, or fails, within different contexts. The primary functions of the BCHCP process evaluation will be to describe stakeholders’ perceptions and experiences of the intervention components, to assess if the components are delivered with fidelity, to describe planning, implementation and delivery processes and to provide a detailed description of a range of contextual factors that might explain variances across the five HAs. Process evaluations of public health interventions typically include measurements of: fidelity (quality); dose of the intervention delivered (completeness) and received (exposure, including engagement and satisfaction with intervention materials); reach (participation rate); recruitment (enrolment and retention); implementation; and context (individual, organizational, cultural, social, or environmental influences on implementation) [14, 15]. In evaluating a complex intervention, the process evaluation is ideally planned and implemented alongside an RCT. In BC, the NFP will be delivered by 5 unique HAs –our rationale for conducting the process evaluation was so that the different ways in which this intervention is implemented and delivered can be documented and analyzed. This contextual data will be essential for eventually understanding and explaining any potential inter-site variances that emerge from the trial. Data from this process evaluation will also make a substantial contribution internationally related to the adaptations required to successfully deliver NFP to families living in rural or remote communities. In addition to enhancing our understanding of the process outcomes related to delivery of the NFP, the data collected within the process evaluation will also enrich our understanding of a wide range of professional nursing practice and education issues.

The overarching objectives for the BCHCP process evaluation are:

The framework for conducting process evaluations of public health interventions developed by Linnan and Steckler [14] and operationalized by Saunders, Evans, and Joshi [15] was used to guide the planning, development and conduct of this study. Core components of the framework, and how they have been operationalized for the BCHCP process evaluation, are summarized in Table 2.

Within the overarching BCHCP, the process evaluation has been systematically embedded throughout the full conduct of the RCT. A convergent parallel mixed methods [16] research design will be used to address the process evaluation objectives outlined above. This mixed methods design is characterized by the collection and analysis of different –both quantitative and qualitative – but complementary data. Qualitative data from program documents as well as interviews and focus groups with PHNs, supervisors (data from the NFP Provincial Coordinator will be coded and categorized with data from supervisors to ensure anonymity and confidentiality) and senior public health managers will be analyzed to explore and describe how the NFP is implemented and delivered across five diverse regional HAs. Concurrent with this data collection, NFP fidelity reports with quantitative data on the reach and the dose of the NFP intervention will be collected and analyzed. Summaries of the content and number of NFP site team meetings and supervisory sessions will also be provided to the research team for analysis. The reason for collecting both quantitative and qualitative data is to bring together the strengths of both research traditions to compare, corroborate and explain results and variances across the five HAs.

All sampling, data collection and data analysis decisions within the qualitative component of the process evaluation will be guided by the principles of interpretive description [17], an applied qualitative methodology that provides a structure for answering questions that arise within clinical practice [17]. Use of interpretive descriptive methods will enable us to provide a comprehensive description of how the NFP is implemented in BC, and to discover the associations, relationships and patterns among key concepts that are related to the implementation and delivery of this early intervention program, across the HAs. This qualitative approach is one that is used to address issues that have disciplinary relevance (i.e., public health nursing) and to address very specific practice goals. In this study, this approach will thus allow us to expand and extend what is already known about professional practice issues within the disciplinary field of nurse home visitation. Within the process evaluation, we are specifically seeking to understand the practice issue of how the NFP is implemented and delivered within the Canadian context, and more particularly, how this home visiting intervention may be adapted for delivery in smaller suburban, rural and remote areas compared to urban centres.

In BC, the Ministry of Health collaborates with the regional HAs that are responsible for the provision of all public health and healthcare services to individuals living within their boundaries. Through the BCHCP, each participating HA is also now responsible for the implementation and delivery of the NFP through their public health services. Four of the five HAs are participating in both the process evaluation and the RCT, while one HA (Northern Health Authority) is only participating in the process evaluation due to its remote geography.

Primary data will be collected from all BC NFP PHNs and supervisors (including the NFP Provincial Coordinator). These individuals will be eligible to participate in the process evaluation (basic or expanded) if they: 1) have completed, or are in the process of completing, the NFP education; 2) are delivering the NFP intervention to enrolled participants in the BCHCP (RCT intervention arm or process evaluation); and 3) who speak English. A purposeful sample of senior public health managers from each HA, who have the NFP program within the portfolio of services for which they are responsible, will also be invited to participate. Informed consent will be sought from all NFP service providers eligible to participate in the process evaluation. The total sample size at the commencement of the study is 71 participants including: supervisors + NPF provincial coordinator (n = 11) and PHNs (basic process evaluation n = 13, expanded process evaluation n = 47) (Fig. 1). Through a process of snowball sampling, we estimate inviting 5–15 senior public health managers (1–3 per participating health authority) to also participate. We anticipate that the sample size will increase during the study as new PHNs or supervisors hired to deliver the NFP during the participant recruitment period of the BCHCP will be invited to participate in the process evaluation following the completion of their NFP core education.

The process evaluation will be characterized by the triangulation of multiple data sources and types to address each research objective. A summary of data sources, types and the frequency of data collection for both the qualitative and quantitative components are summarized in Table 3. Using the principles of interpretive description, qualitative data will be collected from each consenting participant. The NFP PHNs, supervisors and managers will be invited to participate in either an in-depth 1:1 interview (basic process evaluation) or focus groups (expanded process evaluation). Individual interviews will be conducted at regular intervals (approximately every 6 months) until 2018, for a total of eight interviews each. At this rate, with a baseline sample of 11 supervisors (+provincial coordinator), a total of 88 interviews will be conducted with this group, and a total of 104 1:1 interviews with the 13 PHNs (basic process evaluation). However, senior public health managers will be invited to participate in only one annual interview (for a maximum of five interviews per individual; a total of 25–75 interviews across the duration of the study). Multiple interviews are required to document the emerging and ongoing procedures and contextual factors influencing both intermediate and long-term program delivery outcomes. Each participant will be asked to complete a short demographic questionnaire at baseline that will be integrated into the interview facilitation guide and then reviewed for changes during subsequent interviews.

The content of each interview has been informed by the study objectives – a list of core topics to be explored across the different phases of the study is summarized in Table 4. In-depth 1:1 interviews with supervisors, managers, the NFP provincial coordinator and those PHNs in the basic process evaluation will allow us to collect more nuanced data about adaptations required to deliver the NFP model in a range of geographic contexts. Through the use of interviews, we will be able to explore PHNs’ individual experiences of delivering the NFP and obtain a rich, detailed description of their overall experiences and perceptions with the BCHCP. Each semi-structured 1:1 interview will last approximately 60 min and will be conducted by the Research Coordinator via telephone at a mutually negotiated time. A copy of the interview guide will be shared with each participant prior to the interview, so that the individual can reflect on the questions and develop rich, detailed responses and case examples to share during the interview. Field notes to document observations noted during the interviews, emerging themes and issues, as well as topics to explore in subsequent interviews will be maintained by the interviewer and the research team.

PHNs enrolled in the expanded process evaluation will similarly be invited to share their experiences and perceptions of the implementation and delivery of the NFP through focus groups conducted within each HA approximately every 6 months. This will provide us with the opportunity to identify unique contextual factors that influence program delivery between and across the four HAs that are also participating in the RCT. Each focus group will include 5–10 participants. Five focus groups will be held in each phase. Over the length of time of the expanded process evaluation, individual PHNs will participate in a total of eight focus groups (overall total number of focus groups for duration of project n = 40). The lead researcher who has over 16 years of experience conducting qualitative research (SMJ), or the Research Coordinator (with 8 years of qualitative experience) will facilitate all focus groups, which will be conducted on-site. The lead researcher will be responsible for the main facilitation of the group interviews and the Research Coordinator will manage the logistics and will maintain observational field notes. To minimize the risks of “group think” where participants shape their answers to conform with more dominant or powerful members of the group a range of strategies for posing the questions in different ways to elicit a range of responses will be used in each group [18]. PHNs who are unable to attend the focus group will be invited to complete an abbreviated 1:1 telephone interview.

Concurrent to the focus groups, field site visits will be completed across all five HAs. These field site visits will provide an opportunity to document observations about implementing agencies (i.e., HAs) [14] and provide the researchers with opportunities to develop rapport with study participants in the process evaluation, particularly the NFP supervisors and PHNs. Observations will also be captured in study field notes. Dr. Jack has successfully used this approach in her process evaluation documenting the implementation of the NFP IPV intervention within the context of an RCT to evaluate the effectiveness of the NFP IPV intervention [19] in the US. Initial and subsequent in-person on-site visits to meet key site leaders (e.g., managers, supervisors) and PHNs is anticipated to promote rapid engagement when subsequent data are collected via distance using teleconference, or webinar technology.

Key documents will be collected and analyzed in this process evaluation to corroborate and augment evidence from other data sources and to clarify terms and concepts that might have been mentioned in an interview. A review of documents might identify new issues for exploration in subsequent interviews or focus groups. The following documents will be collected on a quarterly basis by each NFP supervisor: 1) team meeting and case conference summary forms, and 2) a summary of weekly supervision records.

The BCHCP Scientific Team will be summarizing and analyzing data from NFP program fidelity reports (aggregate data) provided by the BC Ministry of Health. Data contained within these fidelity reports are entered at the local health area level into a health surveillance software application. Data will be reported by implementation location (local health area) and by Health Authority. However, if at any one local unit level, the numbers are small, the Ministry of Health may choose to aggregate the data at the larger HA level only. The following information will be collected and analyzed from the NFP Program Fidelity Reports: 1) stage of pregnancy at time of enrolment; 2) percentage of time spent on each of the six domains during each of the three phases of the NFP program (pregnancy, infancy, toddler); and 3) total number of visits during each of the three program phases. The HAs enter all the data from the NFP nurse assessment forms into the provincial database systems. This information is utilized by the NFP PHNs and supervisors to guide their practice, guide program implementation, inform clinical supervision, enhance program quality, and assess program fidelity. The NFP Program Fidelity Reports are generated automatically from information recorded in the Home Visit/Alternate Encounter form, which is completed after each NFP home visit. The primary purpose of collecting these anonymized and aggregate fidelity data will be to provide a descriptive overview of the fidelity for some of the NFP model elements across local health areas and HAs.

Qualitative data from the interviews will be analyzed using directed content and thematic analysis, and constant comparative approaches [20]. The analytic process in interpretive description includes broad labeling of concepts through open coding, a synthesis of meanings across codes, the theorizing of relationships through the use of constant comparative methods and the contextualization of data into findings [20]. All interview data will be recorded and transcribed verbatim with any identifying information removed. Transcripts will be reviewed and cleaned by the research team administrative assistant and verified by the Research Coordinator. All textual data (field notes, transcripts, documents) will be imported to NVivo 10.0, a qualitative software package used for overall data management, including coding, searching and indexing.

Both directed (deductive) and conventional (inductive) content analytic procedures will be used to develop the codebook [21]. A set of preliminary codes will be derived from the research questions in the interview guides. However, new codes will be developed and defined as concepts are identified within the data. The codebook will contain the following elements: code name, full definition, instructions on when to apply the code, instructions on when not to apply the code and a brief example of data relevant to that code [22]. The initial code list and code definitions will be circulated to the members of the research team for review. Members of the research team will also independently code a sample of interview data and consistency of code application will be assessed. Based on these findings, the codebook will be reviewed and revised. The Research Coordinator will be subsequently responsible for applying the approved codes to the data and completing all first- and second-level coding.

Quantitative data from NFP Program Fidelity forms and information from the Case Conference forms will be collected and analyzed. Depending on sample size, results will be analyzed and reported at both the local health area and HA levels. The main quantitative analysis will consist of descriptive statistics designed to determine various aspects of program fidelity following recommendations by Boller and colleagues [23]. These fidelity indicators will be evaluated under five broad categories: 1) PHN and supervisor caseloads; 2) duration of the program; 3) service dosage of the program; and 4) content of home visits. According to NFP model fidelity requirements (adapted for Canada) a full-time PHN should carry a caseload of no more than 20 active clients and a full-time supervisor should provide supervision to no more than eight individual PHNs. Measures for this category will include mean monthly PHN and supervisor caseloads and the percentage of PHNs and supervisors at or below required caseload for the observation period. Nurse supervisors provide PHNs with clinical supervision through various activities including one-to-one clinical supervision, case conferences team meetings and field supervision. From the Case Conference forms, number of hours spent in 1:1 supervision (in person or teleconference) per month will be averaged and reported. Time spent in team meetings and conferences will also be averaged and reported. Finally, percentage of time spent on various themes during team meetings will be described.

The NFP seeks to enroll clients early in pregnancy and retain them until the child reaches their second birthday (typically 2.5 years duration). The percent of clients enrolled by 12-, 16-, and after 20 weeks of pregnancy will be described. Duration measures will include: percentage of participants with at least one home visit who remain enrolled at 3, 6, 12, 18 and 24 months; the percentage of participants who do not graduate from the program (at the child’s second birthday); and the mean duration of time spent in the program for those who leave (calculated from the discharge date).

Regular contact with families is one of the key facets of the NFP. NFP guidelines articulate expectations for program dosage varying over the stages of program delivery – pregnancy, infancy and toddlerhood. To assess dosage of the program, we will calculate the percent of participants receiving the intended dosage across all the three stages (90 and 80 % of intended dosage levels) and the number of visits provided between enrollment and date of exit. To fulfill our research objectives, qualitative interview and document data and quantitative NFP Program Fidelity data will be collected in parallel, analyzed separately, and then merged at the interpretation stage.

To ensure the overall quality and trustworthiness of the process evaluation findings, a range of strategies will be applied across all phases of the study to achieve overall data credibility (internal validity), dependability (reliability or the consistency of the findings) and confirmability (neutrality) [24]. A summary of these strategies is provided in Table 5.

As the overarching BCHCP is a mixed methods study that includes the RCT and the process evaluation, both studies were submitted as a single application to ten Research Ethics Boards for review. Approval to conduct the BCHCP (inclusive of the process evaluation) was successfully obtained from all ten of these boards – which consisted of five boards from the participating HAs, four boards based in Universities where investigators held their affiliations or faculty appointments, and through the Health Canada and Public Health Agency of Canada Research Ethics Board. The names of the Research Ethics Boards that approved this study are listed in Table 6. Specific to the process evaluation component, informed consent to participate will be sought from all participants. All data will remain confidential and identifying information will be removed from all transcript data. As part of the consent procedure, PHNs and supervisors specifically will be informed about the potential for experiencing distress associated with participation in the process evaluation. The potential to experience distress may occur: 1) when talking to client participants and their families regarding their life circumstances; 2) during reflective supervision; or 3) when discussing their nursing education or role in the BCHCP. Participants will be given the opportunity to take a break or to stop the interview at any point.

The potential benefits of participating in the process evaluation will also be outlined for participants, including: 1) the experience gained through the NFP education and their involvement in the study is expected to facilitate substantial professional growth and development for participating PHNs and supervisors; and 2) information from this study may be used to help improve programs and services for young, first-time BC (and Canadian) mothers and their children in the future. Findings from the Ontario pilot study [8] suggested that almost all participating PHNs and the local supervisor found the NFP experience to be highly enriching and rewarding. By participating, PHNs and supervisors may be making a lasting contribution to these program improvements.