In this section limitations of the study are discussed. They include the sample, data analysis, limited external validity, social desirability and difficulty in inferring causal relationships.
Sample
There are several characteristics of the sample that may limit the generalisability of the results. The three London boroughs that the sample was collected from are urban areas with diverse social and cultural make-up. The sample may differ significantly from rural areas. Differences between houses run by different organisations were not accounted for and there may be systematic differences between organisations and
between sectors (Blumenthal et al., 1998). These differences may be particularly pertinent as the study concerns differences in organisational characteristics
Only 33% of the houses invited agreed to take part in the study. Although many houses did not take part because they were not currently working with challenging behaviour, several houses did not take part because of other reasons (e.g. time pressure, feeling that 'now was not a good time', difficulties in obtaining line manager approval). Organisational characteristics relevant to the results of this study may, in part, determine the likelihood of houses taking part in the study, and this may introduce a systematic bias to the sampling. As residential units were approached through managers it is also possible that personal characteristics and preferences of the managers influenced their decision to take part. Home managers effectively acted as 'gate-keepers' to accessing participants. This is particularly salient because questions were asking about the support they offered their staff, and so if they felt that these were particularly sensitive areas they may have been less likely to take part in the study.
Although the number of residents reported to show challenging behaviour in each house was recorded no attempt was made to take account of the severity of this behaviour. Thus, there may be significantly different ongoing experiences of challenging behaviour in different houses and this may be a factor in determining levels of self-efficacy and emotional reactions to challenging behaviour. Resident characteristics are probably the most significant demand placed on staff (Dyer & Quine, 1998). They may also be a predictor of levels of staff support, as carers working with people with challenging behaviour have been shown to report lower
levels of support (Jenkins et al., 1987). Thus, their exclusion from the analysis may be a significant omission.
Data Analysis
The data analysis in this study did not take account of its clustered nature. Data was collected from 13 houses, and all of these houses have different characteristics. They may vary in terms of the measured variables (for example, different levels of staff support), the relationships between these variables (e.g. the relationship between staff support and self-efficacy might be dependent on the number of residents in each house), and possibly further unmeasured factors (e.g. location, number of vacancies in staff team). Thus, without comparing the data between different houses, it is assumed that the results are representative of all the houses. It may be profitable in future research to take account of the clustered nature of the data in the analysis using higher level modelling. However, this was not possible in this research, as a significantly larger sample would be needed to generate reliable estimates of differences between houses.
The power analysis suggested that 85 participants needed to be recruited in order to find a main effect. 71 total participants were recruited, of which 5 participants’ data was incomplete to a degree that meant that all their data was discarded. Thus the statistical end-point was not reached. This may have limited the power of the study and some of the results that were approaching significance may have reached significance if 85 participants had been recruited. However, the power calculation was based on a correlation between behavioural attributions, self-efficacy and
emotional reactions to challenging behaviours (Mitchell and Brown, 2002). Significant relationships were found between self-efficacy and emotional reaction to challenging behaviour. As previously discussed, attributions did not approach significant correlations with other variables in the study and it appears unlikely that the addition of 19 more subjects would have altered this.
A further weakness of the study is that the analysis did not include consideration of effect sizes. Correlations between measures varied between .29 and .67 (see table 11), and this indicates effect sizes from small/medium through to large (Cohen, 1988). However, the implications of these different effect sizes are not considered in this study, and this may have been useful in giving a more subtle understanding of the implications of the results.
Social Desirability
Much of the data was collected at team meetings. At one team meeting the manager left the room, explaining that staff might find it easier to answer the questionnaire if she was not present. At all other team meetings the manager was present as the staff were filling in the questionnaire. Although staff were assured that all of their responses were confidential the presence of their manager may have influenced their responses. Possible direction of influence may have been to over-report staff support and self-efficacy and under report negative emotions.
Mitchell and Hastings (1998) found that the emotional reaction to challenging behaviour was not affected by social desirability. However, it is unclear under what
circumstances they issued the questionnaire (i.e. by post, at residential homes) and so it is difficult to know the generalisability of this result. The Staff Support Questionnaire (the forerunner to the 3SQ) was administered to a staff team on a unit for people with learning disabilities and challenging behaviour with the Marlowe- Crowne Social Desirability Scale (M-CSDS; Crowne & Marlow, 1964). Carers were found to score higher on the M-CSDS than undergraduates tested in the original paper (Bell & Espie, 2002). There is no record of any of the other measures being tested for social desirability responses.
Limited External Validity
An important limitation of this and much previous research in this area is its limited external validity. Hypothesised relationships between scores on emotional reactions to challenging behaviour, the CHABA and carer behaviour have been made, but there has been no test of these relationships to actual behaviour. There is no clear hypothesised link between scores on the self-efficacy in dealing with difficult behaviour measure and carer behaviour and, as discussed, some of the items on this scale could represent both positive and negative ways of interacting with service- users. The 3SQ has been show to relate to anxiety and depression scores, which have in turn been associated with levels of stress in residential homes, which have been associated with lower levels of positive interaction with service-users (Rose, Jones and Fletcher, 1998). Thus, there is some tenuous evidence for its link to carer behaviour, though these findings are not in the context of challenging behaviour. However, the 3SQ was not independently linked to emotional reactions to
challenging behaviour or self-efficacy scores, further weakening the possible link to actual behaviour.
Difficulty in Inferring Causal Relationships
As a correlation study unequivocal causal inferences cannot be made (Barker et al., 1994). For example, intuitively it would be expected that there would be a causal relationship between staff support and efficacy in dealing with challenging behaviour. However, it is possible that both are caused by personality factors and thus the apparent causal relationship is spurious. Difficulties in concluding that there is a causal link between increased self-efficacy and reduced anxiety when managing challenging behaviour have been discussed previously.