Aims and research questions
The process evaluation was designed to examine the process, challenges, benefits and impacts of the trial to identify the processes and factors associated with degrees of successful and unsuccessful intervention implementation.
The aims of the process evaluation included:
l describing adherence to the required components of the intervention and the quality (or fidelity) of intervention delivery
l understanding staff members’, residents’ and relatives’ perceptions of the impacts of the intervention l understanding the barriers to and facilitators of implementing DCM in practice.
The process evaluation answered research questions aligned to the Medical Research Council guidelines on process evaluations152and included implementation, mechanisms of impact and context.
l What was implemented?
¢ What was the process of setting up the intervention in each care home?
¢ Did this differ, and, if so, how did it differ, from the intended process as outlined in the protocol? ¢ How many cycles of DCM were delivered in each care home? (Dose+ Reach)
¢ To what extent did each cycle in each care home meet the planned delivery as set out in the protocol? (Fidelity+ Reach)
¢ Did care homes deviate from the delivery of the intervention as set out in the protocol and, if so, how? l How did participants react to the intervention?
¢ What were mappers’, managers’, residents’, relatives’ and staff members’ experiences of the intervention and its implementation?
¢ What were mappers’, managers’, residents’, relatives’ and staff members’ perceptions of the impact of the intervention?
¢ Did the intervention have any perceived or unexpected impacts or consequences?
¢ For the perceived impacts, through what mediators/processes did each group perceive the intervention to have operated?
¢ Did the intervention or its mechanisms of impact operate in any unexpected ways? l What contextual factors shaped if, and how, the intervention was implemented or worked?
¢ What were the perceived barriers to and facilitators of intervention implementation, mechanisms of impact and the perceived impact from the perspective of mappers, DCM expert mappers, managers, staff members, residents and relatives?
¢ How did care homes that demonstrated different degrees of intervention implementation manage and address barriers to and facilitators of intervention implementation?
The process evaluation and implementation assessment was intended to support the refinement and improvement of intervention efficacy and the sustainable implementation of the intervention over time, if the intervention was found to be effective.153
Design of the process evaluation
A mixed-methods approach to data collection was used, involving quantitative and qualitative components to embed the process evaluation as part of the main trial data set.
The quantitative data set included an assessment of the levels of adherence and fidelity in each care home, utilising data provided by the mappers from each care home at each cycle. These data included details on the‘dose’ and quality of DCM use in relation to briefing (the number of briefing sessions held and the proportion of care home staff receiving briefing), mapping cycles (the number of mapping sessions, the number of residents observed, the length of the mapping period and the number of mappers taking part), feedback sessions (the number of feedback sessions held and the proportion of care home staff receiving feedback) and DCM and action-planning documentation (successful completion of standard mapping
documents during each cycle using the standard templates provided and the number of action plans developed per resident and at the home level).
The qualitative data were collected from a subset of 18 intervention homes using semistructured interviews with residents, the care home manager, mappers, staff members, relatives and residents. Homes that had achieved varying degrees of success with DCM implementation (no full cycles, at least one full cycle, two or more full cycles) were purposefully selected to explore the factors associated with successful and unsuccessful implementation. Although the selection of care homes took place before the final follow-up data collection point, the process-evaluation interviews took place after all outcome data had been collected in each home (i.e. at the end of the 16-month follow-up data collection). Semistructured interviews were also conducted with the DCM expert mappers to explore their experience of supporting the implementation of DCM within the intervention homes. To enable links between the qualitative and quantitative data, researchers undertaking the qualitative data collection were provided with implementation data by the CTRU from the first two cycles in the home prior to the interviews.
Sampling for the quantitative and qualitative data collection
For the quantitative data analysis, frequency data from the mapping cycles in all intervention homes were used to assess dose, adherence and fidelity, and to understand the variation in the levels of DCM implementation across homes.
For the qualitative data collection, purposive sampling was used to select a subset of 18 homes that had achieved varying degrees of success with DCM implementation in order to explore factors associated with this in greater detail. Owing to the staggered recruitment of care homes and the need to set up the process-evaluation data collection dates with home managers ahead of time, participating homes had to be identified before all three cycles of mapping were due to have been completed. These homes were stratified into three equal groups (six per group) according to if they were considered likely to be ‘successful implementers’ (more than two cycles completed), ‘partial implementers’ (one or two cycles completed) or‘unsuccessful implementers’ (fewer than one cycle completed) of DCM.
Homes that differed according to key characteristics with the potential to affect DCM implementation, including location (six from each hub), size (large≥ 40 beds vs. medium or small < 40 beds) and type of home (nursing, dementia or general residential), were also accounted for in the sampling.
Participant eligibility
Residents from homes taking part in the process evaluation were eligible if they were deemed to have the capacity to consent and were able to take part in a brief interview. Staff were eligible to take part if they were a permanent or contracted member of staff. Relatives/friends were eligible if they had visited the care home at least once a month during the trial.
Identifying staff and relatives/friends to approach was undertaken in conjunction with the home manager and included identification of the staff members who had played a key role in intervention delivery. All potential participants were provided with verbal and written information about the interview, were given time to consider taking part and signed a consent form if they were willing to participate [see www.journalslibrary.nihr.ac.uk/programmes/hta/111513/#/ (accessed July 2019)]. Mappers had already provided consent to take part in the process evaluation as part of their initial consent to become mappers. Data collection, transcription and storage
All researchers were trained in qualitative interviewing ahead of data collection to ensure consistency in the approach. Resident interviews were brief, using a conversational style informed by a flexible interview schedule. Staff and relative/friend interviews were conducted using a semistructured format informed by a topic guide. The interviews focused on experiences of DCM implementation, with prompts to encourage interviewees to discuss the various stages of DCM implementation, the successes, challenges and impacts of implementation, and any changes required to improve DCM implementation or impact in the care
home, as well as future plans for DCM within the care home. Mappers who had left the home during the trial were not interviewed. Relatives/friends of resident participants who had died during the trial were not contacted regarding the process-evaluation interviews. Interviews were conducted within the care homes, in a private room with no other individuals present, and an alternative method of telephone interviews was offered to all relatives/friends [see www.journalslibrary.nihr.ac.uk/programmes/hta/111513/#/ (accessed July 2019) for copies of interview topic guides].
The interviews were audio-recorded using a digital audio-recording device and were professionally transcribed by a researcher independent to the study. Any potentially identifying information about the participants was anonymised or removed during transcription. Audio files were securely transferred in encrypted format and stored securely on computers in university offices.
Data analysis
Data analysis utilised a framework analysis approach.154The initial data analysis by all researchers involved in data collection informed the development of a coding matrix, which guided and created a structure for further data analysis. The focus of the coding matrix (and therefore of the data analysis) was on experiences of utilising and implementing DCM, particularly on identifying patterns and variations in implementation, barriers to and facilitators of implementation, and the impacts of DCM implementation. The coding matrix helped to assimilate the development of coding categories between the team of researchers that undertook the analysis. Each transcript was independently analysed by two researchers to ensure that key themes were identified. Development of the coding categories continued throughout the data analysis, informed by the emerging themes and analytic thoughts of the researchers. Codes and themes were compared and contrasted across homes and between different types of respondents to develop an in-depth, nuanced and contextualised understanding of the implementation and the impacts of DCM.
The quantitative data that informed the process evaluation (measures of adherence and fidelity in each home) were collected and analysed as part of the main trial data set (as described in Screening, baseline, treatment and outcome summaries). Findings from the quantitative data were integrated with the qualitative data to provide an in-depth understanding of DCM implementation and the issues surrounding implementation. Measurement of adherence
Adherence to the prescribed processes for intervention delivery was monitored from randomisation to check that both mappers attended DCM training on time and passed the assessment. At each expected round of mapping, adherence to the processes was monitored to check that mappers delivered all components of the DCM cycle as intended and to the required quality (fidelity) and delivered three full cycles (dose). Anonymised copies of all observation data collection sheets, feedback reports and action plans were collected to assess fidelity. Data were also collected from the DCM expert mapper about cycle 1 completion, following their support of mappers through their first cycle of mapping. For the purposes of the trial, DCM was considered as comprising four required components: (1) briefing, (2) observation, (3) data analysis, reporting and feedback and (4) action-planning.
Care homes were classified according to their compliance with the intervention at each cycle, namely as ‘acceptable’, ‘partial’ or ‘none’.
For a cycle to be classified as:
l acceptable, the care home must have completed all four components l partial, the care home must have completed one to three components l none, the care home must have completed none of the components.
If paperwork was not received for specific components and the researchers had been unable to ascertain verbally from mappers if particular cycle components had been completed, the following rules were used to determine whether a component had been completed:
l If there was paper documentation for observation, it was assumed that briefing had also taken place (at least two components were completed).
l If there was paper documentation for feedback, it was assumed that briefing and observation had taken place (at least three components were completed).
l If there was paper documentation for action-planning, it was assumed that briefing, observation and feedback had taken place (all components were completed).
An assessment of the quality of each component was also conducted when paperwork had been returned, including whether or not all of the required DCM coding frames and accompanying qualitative notes had been used during mapping; if the standard feedback report format had been used and all parts of this had been completed (group data summary and individual data summary for each resident); and whether or not the standard action-planning template had been used, and if there were action plans developed at the care home level and for each resident mapped.