Instrumentos

The reasoning questions are split into three types: case study specific, close transfer

scenarios and far transfer scenarios.

Case Study Specific Reasoning

The first question participants answer is related to the A&E case study. In the

case study resources such as cubicles and doctors are shared between all patients

(pooled resources). This pooling of resources helps speed up a process subject to

variation and also increases resource utilisation. A simple explanation of this is that

resources, if available, can go to where demand is waiting. Participants are asked to

predict what would happen if A&E resources were split up and dedicated to specific

emergency streams (i.e. whether performance increases or decreases) and to state

why this change in performance occurs. Answers are analysed qualitatively. Firstly,

on whether the participant predicts a decrease in performance and secondly whether

the reasoning behind the prediction is correct.

Transfer Scenarios

The transfer scenarios are divided into two groupings: close and far. All four close

transfer scenarios are set in a healthcare context. This closeness is therefore the

surface similarity of the scenario to the case study. In contrast the far transfer scenarios have less surface similarity to the case study, but are stillstructurally sim-

ilar. The far transfer scenarios are either set in call centres or a food manufacturing plant. Each scenario details a problem, provides transfer cues, lists multiple choice

answers and provides space for a qualitative answer. A scenario will look for the

transfer of one out of two concepts from the simulation experiment:

1. Recognition of the relationship between resource utilisation and the speed at

which an entity can travel through a process;

2. Recognition that eliminating variation from a process can increase the speed

at which an entity can travel through a process.

The case study contains numerous examples of these concepts in action. For

example, participants run experiments analysing the allocation of nurses to shifts

and the use of resources to learn about resource utilisation and performance. If

a degree of double-loop learning has occurred the participants should be able to

transfer these concepts beyond the case study to analogous scenarios. Table 4.5

lists the context of the scenarios and the concept that is tested within each. Refer

to Appendix A.3 for details of the scenarios.

Table 4.5: Transfer Scenarios

Scenario Context Reasoning required for transfer success S1 GP’s Surgery Process Variation linked to Performance; S2 A&E department Resource Utilisation and Performance; S3 Operating Theatre Resource Utilisation and Performance; S4 NHS walkin Centre Process Variation linked to Performance; S5 Pie Factory Resource Utilisation and Performance; S6 Police Call Centre Process Variation linked to Performance; S7 Pie Factory Process Variation linked to Performance; S8 Call Centre Resource Utilisation and Performance;

There are two ways to analyse the answers participants provide to the transfer

1. Assume that the reasoning behind the multiple choice answers is known be-

forehand.

2. For those correct answers, assume that a significant proportion of the partici-

pant’s reasoning is captured in the qualitative information provided and that

it is accurate. Interpret if the basic description of reasoning provided by the

participant is correct.

The first of these approaches is problematic as it may be that participants have

chosen the correct answer by chance. The second approach requires multiple judges;

differences in interpretation of answers must also be resolved.

Multiple judges are not available for the interpretation of the reasoning answer.

Instead the data are coded multiple times by the same judge with a time lag in

between each coding. Two codings of the reasoning answers will take place with at

least three months in-between. For acceptability these are expected to have high

reliability score - measured using Cronbach’s α and intra-class correlation (Field,

2009). In addition to minimise any judge bias (as the experimental hypothesis

is known) all participant details are hidden from view and the order of answers

randomised in each coding.

Transfer of learning is coded as a binary variable: zero for correct choice and in-

correct reasoning (transfer failure) and one for correct choice and correct reasoning

(transfer success). As an example of the coding procedure consider the informa-

tion/cues provided to a participant by scenario two:

1. A hospital is going to experience an increase in demand (amount unspecified).

2. Current average utilisation of human resources is approximately 75%.

3. Current average utilisation of cubicles is 60%.

The performance target and percentages provide the cues to recall the perfor-

mance of the simulation model. The participants are asked if they agree that more

staff should be introduced. All participants are asked to give the reasons why they

think their choice would improve the performance of the system. An example answer

provided by a participant is

‘As the workload is going to increase, it is important to increase

the number of staff. Depending on increase in workload the number of

cubicles may be necessary to increase, but is unlikely as their utilisation

is currently lower than staff utilisation.’

The second sentence provides the most detail on the participants reasoning.

The participant is thinking in terms of maximum capacity. If the resources are

working at less than 100% then they can cope. If 100% is exceeded then more are

necessary to serve them. Whilst this seems sensible it fails to transfer any learning

from the experiment. An example of the transfer we are looking for would discuss

performance levels and the relationship to utilisation: even if utilisation is increased

to 90% there will be a change in the performance of the system. Hence this answer

is coded as a zero - correct choice, but failure to transfer learning.

Confidence in Reasoning

Measurement of the confidence participants hold in their answers to the transfer

scenarios is, again, based on the scales used by (Petty et al., 2002). At the bottom

of each transfer scenario participants are asked to rate the confidence they have

in their answer on a nine-point scale (1 = not at all confident to 9 = extremely