4. Objetivos
6.5. El enfoque en la enseñanza comunicativa del lenguaje
CHAPTER 7 -
STUDY FOUR :
CONSULTATION AND SELECTION OF AN
OUTCOME MEASURE
Two important benefits of consulting practitioners about potential outcome measures are to ( 1 ) obtain feedback about the measure's practical utility and feasibility in everyday practice, and (2) include practitioners in the selection process, as the sense of frontline ownership is likely to facilitate a more successful implementation. This was particularly pertinent for the HCN project as routine outcome measurement was a new process for HCN and its practitioners, the measures being considered were not widely used in New Zealand, and they were not developed for use here.
Before seeking the views of HCN practitioners, the short-listed measures were considered by a group of clinical psychology experts who recommended which were most suited for HCN, and therefore to be presented in Part 11 of the practitioner focus groups. The purpose of taking the measures to the focus groups was to obtain feedback on feasibility and usefulness. The two phases of consultation for Study Four are described next, followed by the final selection of an outcome measure for HCN.
CHAPTER 7 -STUDY FOUR: CONSULTATION AND SELECTION OF AN OUTCOME MEASURE 126
Expert consultation
Members of the HCN Outcome Measurement Project research team (Clinical Psychologists and researchers) independently considered the reviews of each potential measure before discussing as a group those that should be recom mended for presentation to practitioners for feedback. A summary of the discussion follows.
Expert discussion of the short-listed measures
A drawback common to all 1 7 measures reviewed was the lack of data/research about their use or appropriateness for the New Zealand context. A caveat for all six short listed measures was that practitioners would need to be assured of their suitability for use here before they were implemented in routine practice. To recap, the measures were,
• Clinical Assessment Package for Risks & Strengths (CASPARS)
• North Carolina Family Assessment Scale (NCFAS)
• Ohio Youth Problems, Functioning & Satisfaction Scales (Ohio Scales) • Pediatric Symptom Checklist (PSC)
• Strengths & Difficulties Questionnaire (SDQ)
• Youth Outcome Questionnaire (YOQ)
The NCFAS was thought to be the best of the family measures reviewed, a domain of well-being highlighted by the HCN Advisors and practitioners as vital for tracking with HCN you ng people. lt had the advantage of reducing subjectivity with the definitions booklet and offered a broad view of the young person's family situation . Its comprehensive coverage included the parent's functioning in the family, relationships between family members, the material environment, and the well-being of the young person. This meant N CFAS was more likely to capture important information for a wider range of young people than measures focussed specifically on one area of family functioning. Using the measure as designed - at intake and closure - could be problematic for HCN who wished for more frequent, routine adm inistrations. I n addition, because NCFAS was clinician-rated, opportunities to obtain the perspective of the young person and their parent/caregiver were missed. Despite these drawbacks, on balance, NCFAS was seen as the most suitable measure of family functioning .
The PSC was one of the briefest measures reviewed, indicating good practical utility for time-pressured practitioners. lt had a large research base and was psychometrically sound. The PSC is quite mental health focussed, as indicated by the term "symptoms"
CHAPTER 7 -STUDY FOUR: CONSULTATION AND SELECTION OF AN OUTCOME MEASURE 127
in the title, which m ight limit its general applicability for HCN. The multiple informant nature allowed use with a wide age range and subsequent triangu lation could minimise the impact of missing data. Mu ltiple perspectives could also be used therapeutically where differences or similarities might affect the direction of later intervention.
The Ohio Scales was reviewed very favourably by the expert group. Because it was purpose-developed as a global routine outcome measure for 5- 1 8 year olds, it was the most widely applicable measure reviewed. The downside of this was the risk that it might l ack sufficient detail for tracking a very specific problem or area of functioning if that was necessary. Unfortunately, g iven the highly diverse nature of HCN clients, this was true of all measures reviewed. The Ohio Scales was seen as very user-friendly, with an attractive layout and simple instructions. Like the PSC and others, it had parallel forms available for the youth , parent/caregiver, and clinician to complete. An advantage of this measure was the coverage of symptoms, functioning, satisfaction, hopefulness (a quasi quality of life scale), and living environments, in line with literature on outcome measurement (e.g., Koch, et al., 1 998) . lt was the broadest of all measures reviewed, which added to its appeal especially given its relative brevity.
The SOQ also had multiple informant forms and spanned a large age range, although the Ohio Scales was more comprehensive in content. Like the Ohio, the SDQ is not exclu sively focussed on the young person's problems, which was viewed as a strength because of the importance of being alert to both decreases in negative aspects of functioning and increases in positive aspects. The SDQ did this openly by giving scores for both strengths and difficulties, which would be useful for monitoring progress in both areas. In the focus groups, Advisors and practitioners had indicated HCN clients were likely to show change more slowly on an outcome measure than other young people, so the 3-point rating scale might lack the sensitivity to detect small changes, or not be fine-grained enough to use frequently.
The CASPARS was another taking a balanced approach to outcome measurement. The m easures in this battery were explicit in this, as first, a decision is made about whether each item is a risk or strength and then the intensity is rated. As with the SDQ, two scores are obtained and could be used to track the type of change as well as where this had occurred. Like the NCFAS, CASPARS was completed by the clinician, so the view of the client and those close to them were not integrated into the overall picture . I n addition, u nfortunately, there was not the established body of research
CHAPTER 7 -STUDY FOUR: CONSULTATION AND SELECfiON OF AN OUTCOME MEASURE 128
behind it that supported the use of many other short-listed measures. For this reason, it was not recom mended by the expert group.
Finally, the YOQ had a large amount of supporting research, was highly regarded internationally, and widely used. lt was one of the longest short-listed measures, in terms of quantity of items, although they were short statements and the developers indicated administration time was around seven minutes. The YOQ contained items for a large range of issues, but some might be seen infrequently with HCN cl ients. Unfortunately, the YOQ was limited to use with adolescents.
Expert recommendations
All short-listed measures had positive and negative features identified in the expert group discussion. The research team felt that all measures were potential usefu l, however, the CASPARS lacked the research support to be recommended for HCN use at present. Aside from this, the team recommended the remaining five short-listed measures be taken to the practitioner focus groups for feedback.
The team felt that two short-listed measures could be used together to measure outcome across as many of the well-being domains as practical for HCN teams. The NCFAS could be useful where interventions were expected to improve family functioning and relationships. To supplement this with information about change in the well-being of the young person, a general measure including areas of strength and need could be used. The g roup thought the Ohio Scales provided that information as well as giving older youth an opportunity to contribute their perspective on their well being and the success of the intervention plan. HCN teams could then consider multiple perspectives and integrate these to inform future work with the young person . Both measu res were able t o be administered b y any member of the team and were not burdensome in terms of adm inistration, scoring or interpretation time. This would allow the workload of routine outcome measurement to be shared by all and be incorporated
into the everyday work of the practitioners involved.
The Ohio Scales was intentionally quite broad to give a global picture of outcome and therefore had superficial coverage of specific issues. lt may lack detail if HCN teams needed to track a very specific problem, behaviour, or area of functioning, although the diversity of their clients mad e this issue practically impossible to avoid. Therefore, the best result was seen to be to offer a measure that was sou nd and could give a good
CHAPTER 7 -STUDY FOUR: CONSULTATION AND SELECTION OF AN OUTCOME MEASURE 129
general picture of where the you ng person was currently, compared to a short while ago. The research team encouraged teams to supplement this with detailed measures on a case-by-case basis depending on the intervention, client's needs, and resources available.
Although the combination of NCFAS and the Ohio Scales was recommended by the team as most suitable for HCN in theory, consultation with practitioners about the best measure in practice was necessary before the final selection was made.
Practitioner consultation
Focus groups: Part
11The consultation with HCN practitioners was u ndertaken in the focus groups described previously (Study Two) . The second part of each focus group involved presenting the short-listed measures for feedback about suitability and feasibility in practice. Participants were sent sample copies of the measures in advance (see Appendix C), so they could read and consider each before arriving. Based on the discussion of the research team, five measures were presented to the participants:
• North Carolina Family Assessment Scale (NCFAS)
• Ohio Youth Problems, Functioning & Satisfaction Scales (Ohio Scales)
• Pediatric Symptom Checklist (PSC)
• Strengths & Difficulties Questionnaire (SDQ)
• Youth Outcome Questionnaire (YOQ)
For each measure, hard copies of all informant forms and supplemental material (where applicable) were supplied at the focus groups. Recall that there were five g roups: two in Auckland, and one each in Palmerston North, Nelson , and Christchurch.
Practitioner discussion of the short-listed measures
Each focus group found that at least one of the measures presented was to their liking. However, there was no clear unanimity about the most preferred measure. On the other hand, there was universal agreement about which measures were not suitable. The feedback for each measure is summarised below, with direct quotes illustrating the key ideas expressed by participants.
CHAPTER 7 -STUDY FOUR: CONSULTATION AND SELECTION OF AN OUTCOME MEASURE 130
North Carolina Family Assessment Scale
There was general support for this measure, but suggestions for alterations to increase its utility were frequent. One common suggestion related to the potential impact on validity of recording post-i ntervention ratings right next to the pre-intervention ratings. The following comment captured the view also expressed in other groups.
Christchurch
I also think it would be really valuable to have one at intake and one at closure as opposed together so that you are not influenced in any way, shape or form by what's been there before.
However, this concern was not universal, with one participant in Palmerston North noting the practical ease of having the comparison easily visible. Participants highlighted the benefits of using the measure more frequently than just pre- and post intervention. In particular, participants recognised the clinical utility of using the measure to track change over time, and how the interventio n could be enhanced by its regular use. For example,
Christchurch
Yes, because if something's gone wrong in the middle you can look at it and say, "Oops, something's gone wrong here, how can we fix it? What can we put in place ? Do we need to put more support in place in a certain area . . . ?"
Other concerns related to the validity of the measure were about relying solely on clinician ratings. Participants noted disadvantages such as reducing the usefulness of the data and preventing the therapeutic opportunity to collate multiple perspectives. This was linked to a strong desire for the direct involvement of families and young people in com pleting measures. The dialogue that follows illustrates the sentiment in a number of groups.
Palmerston North
X: [Reading one item] "Relationships between parents and caregivers"; they might think they're completely functional and we might think that they've got a really disjointed way in which they deal with and cope with their child.
Y: ... we had Mum coming saying, "Dad's drinking heavily, I'm frightened of him" . . . and we had Dad saying, "Things are great, I'm managing, I'm fine, everything's going smoothly", so you straight away have a few difficulties with how you fill out your form.
X: And yet the idea of getting the parents to do one of these, versus us doing one, you know? That'd be an interesting exercise, to be fair.
CHAPTER 7 -STUDY FOUR: CONSULTATION AND SELECTION OF AN OUTCOME MEASURE 131 Y: lt would be good to have a form you could give to different people
and collate. You 'd have a better idea too about what was happening at the end, because you'd see whether the change was showing across all of them or whether it was in only one environment. I think that'd be good.
A few participants discussed the length and complexity of the measure as potential feasibility problems for busy practitioners. O ne suggested this might lead to less careful ratings being made, which would compromise the reliability of the measure.
Palmerston North
Yip. Either it's too time consuming, or are we going to do a hash job o f it?
While some participants were cautious about the measure, because not all subscales or items were applicable or useful or its design rendered it open to bias, there were more positive comments than negative at all focus groups. From two,
Auckland 1
I can see that working. it's a broad, general sort of measure; that's okay.
Nelson
I think it could have some quite good stuff around the situation for the family and improvements in family functioning . . . it's whether you could use something like this and also have some individual stuff around the specific behavioural intervention . . . yes, this is probably quite useful.
I nterestingly, the bulk of the discussion about the NCFAS across groups was not about the content of the measure. Without prompting from the facilitators, the discussion was dominated by ideas about how the measure could be improved, which implied that in its current form , the NCFAS was not an ideal solution for monitoring outcome .
T h e Ohio Youth Problems, Functioning, and Satisfaction Scales
Of all the measures presented to the focus g roups, the Ohio Scales was the most positively received. A representative sample of comments illustrates what participants liked about this measure,
Auckland 1
I like this a lot, the wellness part at the back. it's the individual answers that is good evidence as far as I'm concerned .. . /'// take this one!
CHAPTER 7 -STUDY FOUR: CONSULTATION AND SELECITON OF AN OUTCOME MEASURE 132
A uckland 2
As a test, I think it's quite concrete. I quite like it. lt gives some very clear information. lt's quite good.
Palmerston North
Yip, and this was the one my client I was sitting down next to today decided she wanted to fill in . . . ! was sitting down reading them outside and she sort of had a look and asked me what I was doing . . . and she decided that she wanted a copy to fill out.
The combination of both problems and functioning aspects were appreciated by participants. A link to the HCN well-being domains was made by a number of participants, who felt it had good coverage of a broad range of areas. In addition , it was seen as relevant to the important issues in HCN cases, for example,
Auckland 2
Facilitator: . . . how closely would those things tie in with . . . what you want to achieve ?
X: "Getting along with friends" definitely. "Getting along with family", that third one, yep. Yes, number 4, definitely. 5; yes. 6 is part of our, definitely part of it . . . Yes, "Controlling emotions ". "Being motivated in finishing projects': not really. "Participating in hobbies", definitely, and "Recreational activities", "Completing chores ", yes. "Attending school"; yes . . . Everything. Everything there.
Y: They're all part of the plan in all those different areas.
There were positive comments about the simple language of the measure, but queries about some of the USA terminology. One participant stated that young people would not complete any rating scale longer than nine items, which appeared related to a wider discussion i n other groups about the difficulty encouraging clients to complete measures. Nevertheless, most participants seemed to hold the contrary view, that the Ohio Scales was very well designed, engaging, and would be readily accepted by young people, as illustrated by these comments :
Palmerston North
lt's worded in a way teenagers can read it and understand, more so than [the measures already discussed].
Auckland 1
. . . often {young people aren't] very interested in participating, but they'd certainly answer a form like that.
CHAPTER 7 -STUDY FOUR: CONSULTATION AND SELECTION OF AN OUTCOME MEASURE 133
The satisfaction scale prompted lengthy discussion about the importance of having input from both the parents and the young person into the intervention plan. This was seen as another strength, as seeking their view on how the intervention was working was highly valued. In addition, the parallel forms were recognised as another strength of the measure. One participant reiterated the view that completing such measures could provide a useful framework for engaging with family members. In particular, that the Ohio Scales had items that could be empowering for parents who would be able to see some objective indicators of positive change and that their son/daughter had some positive attributes.
Nelson
Often in terms of where I sit, a lot of the plan will work or not work on the parents ' participation . . . so I think having an evaluation measure that actually actively includes the parent is more meaningful.
I n addition to offering multiple perspectives, the therapeutic benefits of parallel forms were discussed . Participants saw the identification of differences and similarities in ratings as potentially useful for the ongoing intervention.