Average (S.D.)
Condition Trial 2 Trial 3 Trial 4
Base 1.15 (0.21) 1.22 (0.34) 1.32 (0.42) PR 1.02 (0.05) 0.94 (0.43) 0.98 (0.35)
U (* = significant) 21 15* 13*
Table 7.8. Perceived efficiency data for Base and PR conditions.
Self Rated Efficiency
1 .4 0 1 .3 0 1.20 1.00 0 .9 0 0 .8 0 0 .7 0 0 .6 0 T4 T1 T2 T3 -0“ B ase+PR X B ase
Fig. 7.9. Graph showing average relative perceived efficiency, over four trials for the Base and PR conditions.
From the graph, it appears that for the control group, perceived efficiency increases steadily over the four trials. This would appear not to be so for the experimental group (i.e. with the PR manipulation). In the case of trials 3 & 4, the differences between the two groups are statistically significant.
ii) Perceived Difficulty
Average (S.D.)
Condition Trial 2 Trial 3 Trial 4
Base 0.93 (0.22) 0.87 (0.31) 0.74 (0.36) PR 0.99 (0.64) 1.10 (1.01) 1.01 (0.55)
U (* = significant) 20 27 23
Table 7.9. Perceived difficulty data for Base and PR conditions.
Self Rated Difficulty
1 .4 0 1 .3 0 1.20 1.00 0 .9 0 0 .8 0 0 .7 0 0 .6 0 T1 T2 T3 T4 ■O* B ase+PR X- B a se
Fig. 7.10. Graph showing average relative perceived difficulty, over 4 trials for the Base and PR conditions.
The average perceived difficulty for the control group appears to decrease steadily over the four trials, whereas that for the PR (experimental) group remains constant, except for trial 3 where it would seem on average the subjects found it more difficult. This is not supported by the applied statistical test.
iii) Direction of Gaze
T2 T3
Average 12 6
sd 3.1 3.1
Table 7.10. Direction of gaze data for PR condition on trials 2 & 3.
1 4 12 10 8 6 4 2 0 T2 T3
Fig. 7.11. Average percentage of time spent looking in the direction of the PR screen in the PR condition.
On average, subjects spent 12% of the time looking in the direction of the PR screen on trial 2, and only half this on trial 3. All subjects spent some time gazing in this direction. No statistical test is applied here since there would appear to be no requirement. This data is included as confirmation that the PR screen was at least looked at.
iv) Comments, Debriefing and Observations
Comments, both those elicited on the post-trial forms and other voluntary statements made in passing and noted by the experimenter, were approximately equally for and against the PR manipulation. Those in favour included those who clearly appreciated it (to differing degrees):
[the format of the code on the left is SUBJECT/TRIAL, where trial is 1-4, and D for debriefing]
13/T2 “It was easier having the left hand screen ’cos you could see at a glance what needed doing”
15/D “The left hand screen was a slight help, a confidence boost to see things wiped - knowing that there is nothing left to do”
Also, there were those who admitted they they missed it in the final trial: 20/T4 “Missed the information on the left hand screen”
25/D “Was reluctant to use the left hand screen. Had a mental picture, but NOW think it would have been useful”
Comments from subjects who did not find the PR screen helpful included for example: 30/T2 “The left hand screen was not particularly useful. Because of the small number
of cars being dealt with, it was possible to keep in memory the lists of tasks still to be done”
12/D “The left hand screen was no use - was able to use the resource store etc to the same effect”
The (unsupported) observation of the experimenter was that subjects could often perform more efficiently if the PR screen is both present and used. However, some subjects expressed a preference for the single screen trials, finding multiple screens generally unhelpful.
The most common errors made by subjects were concerned with attempting to allocate more resources than were present. This may be a problem which is overcome through learning, but the PR screen could have been enhanced to emphasise the ‘current resource availability’ aspect of the problem. Had this been done, it would be reasonable to expect that more user comments at least would support the notion that the PR screen was useful.
3 . 2 . 2 . The Sol Condition A. Primary Measures i) Strategy Time
Average (S.D.)
Condition Trial 2 Trial 3 Trial 4
Base 0.95 (0.09) 0.92 (0.05) 0.93 (0.09) Sol 0.90 (0.08) 0.87 (0.11) 0.89 (0.10)
U (* = significant) 22 18 24
Table 7.11. Strategy time data for Base and Sol conditions.
Strategy Time 1.00 0 .9 5 0 .9 0 0 .8 5 0 .8 0 T4 T1 T2 T3 ■v- B ase+ S ol X' B ase
Fig. 7.12. Graph showing the average relative strategy time over 4 trials for the Base and Sol conditions.
In a similar way to the equivalent data for the PR condition, there does appear to be a trend in the direction expected if the manipulation were having its effect (i.e. the time, as a proportion of the time taken on the first trial, is lower on trials 2 and 3). However, efficiency measured in this way does not appear to return to a level similar to that of the control group on trial 4, which is not as one might expect. Any differences between these data series are not supported by the statistical tests applied, and so will have to be interpreted in the context of the other measures.
ii) Free Time
Average (S.D.)
Condition Trial 2 Trial 3 Trial 4
Base 1.10 (1.09) 1.10 (1.19) 1.79 (1.89) Sol 3.01 (3.02) 3.75 (3.45) 0.76 (0.64)
U(* = significant) 13* 12* 21
Table 7.12. Free time data for Base and Sol conditions.
Proportion of Free Time
5 .0 0 4 .0 0 3 .0 0 2.00 1.00 0.00 T4 T1 T2 T3
Fig. 7.13. Graph showing average relative free time (as a proportion of strategy time), over four trials for the Base and Sol conditions.
The presence of the Sol manipulation on trials 2 & 3 has an effect in the desired direction, according to this measure. This difference is also statistically supported. In the case of trial 4, the graph indicates that efficiency on this measure drops below that of the base condition, possibly indicating a negative effect of removing the manipulation. Such a difference is not statistically supported however.
B. Secondary Measures i) Perceived Efficiency
Average (S.D.)
Condition Trial 2 Trial 3 Trial 4
Base 1.15 (0.21) 1.22 (0.34) 1.32 (0.42) Sol 1.05 (0.25) 1.07 (0.33) 1.02 (0.25)
U (* = significant) 26 21 13*
Table 7.13. Perceived efficiency data for Base and Sol conditions.
Self Rated Efficiency
1 .4 0 -r 1 .3 0 -- 1.20 - - 1 .0 0 X' 0 .9 0 -- 0 .8 0 -- 0 .7 0 -- 0 .6 0 T1 T2 T3 T4 ■O- B a se+ S o l •X- B ase
Fig. 7.14. Graph showing average relative perceived efficiency, over 4 trials for the Base and Sol conditions.
The experimental group does not show an increase in perceived efficiency over the four trials, even on trial 4, while the control group shows a steady increase. The difference between the two only becomes statistically significant on the final trial.
ii) Perceived Difficulty
Average (S.D.)
Condition Trial 2 Trial 3 Trial 4
Base 0.93 (0.22) 0.87 (0.31) 0.74 (0.36) Sol 0.90 (0.22) 0.70 (0.24) 0.87 (0.38)
U (* = significant) 23 20 27
Table 7.14. Perceived difficulty data for Base and Sol conditions.
Self Rated Difficulty
1 .4 0 1 .3 0 1.20 1.00 0 .9 0 0 .8 0 0 .7 0 0 .6 0 T4 T1 T2 T3 ■©■ B a se+ S ol ■X- B a se
Fig. 7.15. Graph showing average relative perceived difficulty, over 4 trials for the Base and Sol conditions.
There would appear to be a trend in the above such that subjects perceive the job to be less difficult on trials 2 & 3 when the Sol manipulation is present. There is an anomaly in the case of trial 4 where this apparent easing is wiped out when the manipulation is no longer present - to the extent of being worse than the base condition. There differences are not statistically supported and thus the final interpretation of any of these points will have to be in the context of the other measures.
iii) Direction of Gaze
T 2 T3
Average 5.4 5.4
sd 3.0 3.2
Table 7.15. Direction of gaze data for Sol condition.
14 T 12 10 - - 8 - - 6 - - 4 -- 2 -- o -I— I— :--- L + J U T2 T3
Fig. 7.16. Average percentage of time spent looking in the direction of the Sol screen in the Sol condition.
There would appear to be no difference in the average percentage of time spent gazing in the direction of the Sol screen when present.
iv) Comments, Debriefing and Observations
The comments and observations were generally in favour of this manipulation. Many subjects (irrespective of condition) spontaneously tried to issue commands on trial 1 to achieve the effect of queuing. Only a single subject (No. 19), when asked afterwards whether such a facility might have been useful, thought that it would not have been (this subject was in a condition where it was not present). Examples of comments in favour of the Sol manipulation include the following (* indicates the subject was in a condition where they did not experience the Sol screen):
9/T2 “Queuing facility is more natural, i.e. better”
3/T2 “I liked having the queuing facility because I could set things up in advance and so did not have to be vigilant in spotting when a resource became free”
As in the case of the PR manipulation, it usefulness might only have become apparent after it was withdrawn:
6/T4 “Missed the opportunity to queue, despite the fact that I under used the queuing whilst it was available”
27/T4 “Once you get used to using a queue, having it taken away makes mistakes happen more often”
It is also clear from the comments that there is a dissociation between the facility to queue and the screen based reminder of this. The balance of comments probably falls against the need for the screen, given the queuing facility. For example, comments in favour of the the screen included:
*20/D “Queuing would probably be useful, although [unprompted:] would probably need a reminder of what I had queued”
27/D “The right hand screen is useful, but could possibly manage without it” and those against:
*23/D “could probably remember 2-3 items”
7/D “because it was quite slow, didn’t use the screen”
13/D “the right hand screen itself was not useful. Queuing facility was very useful” 35/D “would probably have preferred something on just one screen”
There are no other observations in addition to that of subjects spontaneously trying to queue allocations.
3 . 2 . 3 . The Combined PR + Sol Condition A. Primary Measures
i) Strategy Time
Average (S.D.)
Condition Trial 2 Trial 3 Trial 4
Base 0.95 (0.09) 0.92 (0.05) 0.93 (0.09) PR + Sol 0.88 (0.14) 0.88 (0.09) 0.90 (0.11_
U (* = significant) 21 25 24
Table 7.16. Strategy time data for the combined PR + Sol and Base conditions.
Strategy Time 1.00 0 .9 5 0 .9 0 0 .8 5 0 .8 0 T4 T3 T1 T2
■O- B ase+ P R + Sol X B a se
Fig. 7.17. Graph showing the average relative strategy time over 4 trials for the Base and PR + Sol conditions.
The graph for this measure is very similar to that for the Sol only condition - i.e. there appears to be a trend towards faster performance on both the experimental trials (2 & 3), with a carry over benefit on to the 4th trial. None of these differences is statistically supported.
ii) Free Time
Average (S.D.)
Condition Trial 2 Trial 3 Trial 4
Base 1.10 (1.09) 1.10 (1.19) 1.79 (1.89) PR + Sol 4.30 (4.19) 3.75 (2.62) 1.90 (2.33)
U(* = significant) 17 10* 31
Table 7.17. Free time data for the combined PR + Sol and Base conditions.
Proportion of Free Time
5 .0 0 4 .0 0 3 .0 0 2.00 1.00 0.00 T3 T4 T1 T2
■O- B ase + PR + Sol X B ase
Fig. 7.18. Graph showing average relative free time (as a proportion of strategy time), over four trials for the Base and PR + Sol conditions.
The effect of the experimental manipulation (both extra screens together) displayed in the above graph is as would be expected - i.e. there is a benefit when the manipulation is present (trials 2 & 3), which disappears when removed (trial 4), rendering performance equivalent to that in the control condition. Note that only the trial 3 difference is
statistically significant. Although the difference on trial 2 appears larger, it fails to be supported statistically at the chosen level, using the chosen test. Closer examination reveals one anomalous data point in each group. Also if the equivalent parametric test (i.e. t-test) had been chosen, this would have appeared to be significant.
B. Secondary Measures i) Perceived Efficiency
Average (S.D.)
Condition Trial 2 Trial 3 Trial 4
Base 1.15 (0.21) 1.22 (0.34) 1.32 (0.42) PR + Sol 0.98 (0.31) 1.22 (0.25) 1.08 (0.38)
U (* = significant) 19 30 19
Table 7.18. Perceived efficiency data for the combined PR + Sol and Base conditions.
Self Rated Efficiency
1 .4 0 1 .3 0 1.20 10 1.00 0 .9 0 0 .8 0 0 .7 0 0 .6 0 T1 T2 T3 T4
■O- B ase+P R +Sol X B ase
Fig. 7.19. Graph showing average relative perceived efficiency, over 4 trials for the Base and PR + Sol conditions.
Statistically, there is no significant difference in the 2 conditions on trials 2, 3 & 4. Given the absence of even a suggestion of a trend, it is probably wisest to accept that there is probably no difference in these two series.
ii) Perceived Difficulty
Average (S.D.)
Condition Trial 2 Trial 3 Trial 4
Base 0.93 (0.22) 0.87 (0.31) 0.74 (0.36) PR + Sol 0.97 (0.53) 1.01 (0.66) 0.78 (0.61)
U (* = significant) 25 31 23
Table 7.19. Perceived difficulty data for the combined PR + Sol and Base conditions.
Self Rated Difficulty
1 .4 0 1 .3 0 1.20 1.00 0 .9 0 0 .8 0 0 .7 0 0 .6 0 T4 T1 T2 T3
■O- B ase+PR +Sol X -B a se
Fig. 7.20. Graph showing average relative perceived difficulty, over 4 trials for the Base and PR + Sol conditions.
There is very little difference, either apparent or significant in these 2 data series. The difference on trial 3 might be taken to suggest that subjects do not find the job getting easier whilst the manipulation is present.
iii) Direction of Gaze PI Trial 2 [ screen Trial 3 So Trial 2 1 screen Trial 3 Average sd 6.40 2.40 4.90 1.90 2.60 1.60 2.70 1.90
Table 7.20. Direction of gaze data for the PR and Sol screens in the combined PR + Sol condition. Trials 2 and 3.
14 T 12 - - 10 - - 8 - - 6 - - 4 -- 2 0 2 3
Fig. 7.21. Average percentage of time spent looking in the direction of the PR and Sol screens in the PR + Sol condition.
Subjects would appear to look at the additional screens, although not very much. There is little difference between trials 2 & 3. They look at the PR screen more than the Sol screen.
iv) Comments and Observations
Any comments and observations concerned with the effectiveness of each of the two manipulations per se have been presented above. The purpose of this section is to
present any further points which arise specifically out of the combination. The only issue here is that having three screens at once was thought to be undesirable. For example: 33/D “Preferred to focus on just one screen”
Although for some subjects, this could be weighed against the advantages, e.g.: 13/T2 “Having all three screens going at once involved more time to think, but the
gains were greater”
^ Sol screen E ] PR screen
28/T2 “Easier than before, but only when used to using the other screens”
3 . 3 . The comparative effects of the manipulations
The aim of this section is to reveal the relative effects of the different manipulations. The data presented here have all been presented above, separately, but are shown here again in combination. 3 . 3 . 1 . Primary Measures A. Strategy Time Strategy Time “O" B ase+PR ■d- B a se+ S o l B ase+P R +Sol •X- B ase T1 T2 T3 T4
Fig. 7.22. Graph showing the average relative strategy time over four trials for all four conditions.
The visible effects in the Sol and the PR + Sol conditions are very similar, and slightly greater in apparent extent than the PR conditions. Note also the general learning effect visible in the data for the base condition.
0 .9 5
0 .9 0
0 .8 5
B. Free Time
Proportion of Free Time
5 .0 0 4 .0 0 ■0- B ase + PR 3 .0 0 ■*> B ase + Sol • Base + PR + Sol 2.00 X X- B ase 1.00 0.00 T3 T4 T1 T2
Fig. 7.23. Graph showing average relative free time (as a proportion of strategy time), over four trials for all conditions.
The effects of all three experimental manipulations appear very similar, with the
exception of the two points already discussed above (Trial 2 for the combined PR + Sol condition, and trial 4 for the Sol condition). Note also that any learning effect for the control group only becomes apparent towards the end. How such a curve would progress on subsequent trials is not known.
3 . 3 . 2 . Secondary Measures
A. Perceived Efficiency
Self Rated Efficiency
1 .4 0 1 .3 0 1.20 1.00 0 .9 0 0 .8 0 0 .7 0 0 .6 0 T1 T2 T3 T4 -O- B ase+PR ■n- B ase+ S ol B ase+P R +Sol 'X- B ase
Fig. 7.24. Graph showing average relative perceived efficiency, over four trials for all conditions.
With the exception of the combined PR + Sol group on trial 3, the members of the three experimental groups would appear similar on average in not perceiving themselves becoming increasingly efficient with successive attempts at the job.