The scoring system was developed by the present author and project supervisor. Initially, the pilot data collected during the development o f the test was pooled, and answers were sorted into categories reflecting different types o f solutions. These categories were then classified according to three criteria (see below) as to whether they were judged to show adequate appreciation o f the problem, to be socially
appropriate, and to provide effective practical means o f resolving the problem. Detailed guidelines were written which described each category, and gave sample answers for each. Five alternative solutions were also selected for each problem from the pilot data to represent a range o f solutions for the Judgement o f Alternatives task. Two new raters then attempted to classify each individual’s solutions using the scoring categories, and any difficulties arising were discussed in order to refine the classification system further. Two additional raters then went through the same procedure with the revised scoring system to create the final version.
This final scoring system was used to score the responses o f the participants in the present study. All responses were rated by one rater who was blind as to the identity and group membership of the participants, and by a second rater who was not blind. The two raters agreed for 87% o f ratings. All differences were resolved by reference to an additional blind rater.
7A ,2,2,9,1 Problem Appreciation
The Problem Appreciation measure assessed whether or not the solution demonstrated adequate recognition o f the pertinent interpersonal/practical aspects o f the problem situation that needed to be taken into account in order for a satisfactory outcome to be possible. In the example o f the scenario described above, in which Tony is having difficulties because o f his neighbours’ noisy dogs, the categories could be summarised as follows:
(a) Attempt to negotiate a solution with the neighbours
(b) Make further complaints (e.g. go to landlord/council/police) (c) Alter your own life (e.g earplugs, move house)
(d) Extreme ideas (e.g. kill the dogs)
(e) Irrelevant or incomplete responses (e.g. people shouldn’t keep dogs in flats) Categories a, b and c in this example were assigned a score o f 1, and categories d and e were assigned a score o f 0, to indicate adequate and poor appreciation o f the pertinent issues (whether or not solutions were socially appropriate or effective).
7.1.2.2.9.2 Social Appropriateness
The categories were used to classify solutions according to whether or not the manner o f dealing with the situation was socially appropriate (Social Appropriateness) (scored 1 or 0). In the example o f the “Dogs” predicament shown above, the categories were classified as follows: Category (a) answers scored 1 point, category (c), (d) and (e) answers scored 0 points, and category (b) answers were split according to the degree o f social appropriateness indicated (e.g. aggressive or threatening answers scored 0).
7.1.2.2.9.3 Effectiveness
The categories were also used to classify solutions according to whether or not the manner o f dealing with the situation was likely to provide an effective practical means o f resolving it (Effectiveness) (scored 1 or 0). In the example o f the “Dogs” predicament shown above, the categories were classified as follows: Category (d) and (e) answers scored 0 points, and category (a), (b) and (c) answers were split into those scoring 1 or 0 according to the degree o f practical effectiveness indicated.
7.1.2.2.9.4 Total Solution Quality
Total Solution Quality scores were also calculated by adding the three subscores (Problem Appreciation, Social Appropriateness and Effectiveness). For optimal and personal solutions, this gave two measures scored out o f 3 for each item. Optimal Solution Quality and Personal Solution Quality. The three subscores were not necessarily independent, as failure to identify the pertinent aspects o f the problem (Problem Appreciation) was automatically scored 0 for Social Appropriateness and Effectiveness, since it was associated with generation o f a solution that did not recognise or address the correct problem.
7.1.2.2.9.5 Solution Generation
To assess how efficiently people generated solutions, the number o f ideas generated for each problem situation was added to create a total score (Number o f Solutions). Each o f these ideas was scored for quality on the three subscales as described
above. The Total Solution Quality scores were added together and divided by Number o f Solutions, to provide an Average Solution Quality score, i.e. adjusted for the num ber o f solutions given.
7,L2,2,9.6 Scoring examples
Solutions that scored the maximum 3 points for Total Solution Quality: “Discuss it again with the neighbours and negotiate a compromise” “Take it up with the council/landlord”
Sample solutions given by anterior participants that scored no points for Total Solution Quality:
“Poison the dogs” “Shoot the dogs”
‘Make a loud noise to counteract their noise’ ‘Threaten the neighbours with violence’ ‘Bribe them ’
‘Mention to the neighbours that I was looking after a couple o f rottweilers for a friend’
7.7.2.2. P. 7 Judgement o f Alternatives
In order to create the Judgement o f Alternatives task, five alternative solutions for each problem situation were selected from the pilot data to represent answers from at least three different scoring categories for each problem. Each o f the five alternatives for each problem was scored 1 or 0 according to the system described above for Problem Appreciation, Social Appropriateness and Effectiveness, giving a Total Solution Quality score between 0 and 3. The scoring for these alternative solutions was checked by the additional raters, as described above. The rank ordering o f the alternative solutions was determined by their Total Solution Quality scores. Where two or more alternative solutions shared the same Total Solution Quality score, these were ranked equally. Participants’ rank orderings o f the 5 alternatives were then scored by comparing them with the optimal rank ordering derived from Total Solution Quality scores. For each problem situation, the rank
position o f each answer scored between 0 and 4, according to its distance from the optimal rank position; this gave a maximum score o f 20, where all 5 answers were in the correct position relative to the other 4. The maximum Judgement o f Alternatives score was thus 320.