Validación de los resultados

The previous section reviewed the general OR and simulation literature regarding

learning from the process of building a model. One problem with the empirical

studies in that literature is that it can be difficult to draw out decision maker

learning that occurs primarily from involvement in model building. This section

reviews the literature addressing model use and learning. Perhaps unsurprisingly, it

is easier to identify empirical studies primarily focussed on model use. This would

seem to be due to relative simplicity in the procedure for laboratory based studies

focussed on use compared to building.

This section, firstly, reviews the DES literature on model use. This ranges from

conceptual discussions of the benefits (or problems) associated with visual interac-

tion modelling (Van der Zee and Slomp, 2009; Chwif and Barretto, 2003; Belton

and Elder, 1994; Musselman, 1990) to empirical studies of learning and perceptions

(Tako and Robinson, 2009; Bell and O’Keefe, 1995). Following this is a review of the

SD literature. The SD literature exploring model use is much more closely linked to

the history of simulation gaming (Lane, 1995) and appears to be much more mature

than the field of DES. In fact, there are numerous well known experimental studies

investigating learning from simulation type board games (Sterman, 1989b,a) and simulation models (Paich and Sterman, 1993; Bakken et al., 1994), as well as re-

Rouwette et al., 2004; Gr¨oßler, 2004). The section closes with a summary of the

main points covered in the review.

2.3.1 DES Research

Belton and Elder (1994) argue, based on their combined experience, that use of

Visual Interactive Simulation (VIS) for simulation experimentation is beneficial for

discovery, clarification, change and creation of a client’s views and ideas about sys-

tem management. Indeed these benefits would seem to suggest alternative uses

for DES such as gaming and training (Van der Zee and Slomp, 2009; Chwif and

Barretto, 2003). The reasoning behind this is similar to that expressed by Lane

(1994). Decision makers are able to explore the consequences of their assumptions

and decisions while receiving fast and understandable feedback from the model.

This experience helps makes the tacit views they hold explicit and possibly reveals

inconsistencies or incompatibilities between them (Belton and Elder, 1994).

Bell and O’Keefe (1995) reiterate the argument for visual interactive simulation

and also provide an insight into an opposing view: namely the overemphasis of the

result from a single run and subjective based analysis (Musselman, 1990, quoted in

Bell and O’Keefe, 1995). Bell and O’Keefe point out that most proponents of VIS

will use a combination of batch and interactive experimentation, but also that the

majority of what is known about using VIS is anecdotal.

Hence to explore the proposed benefits of VIS, Bell and O’Keefe (1995) conduct

an experimental study investigating the results users of interactive experimentation

can achieve relative to a statistical analysis. Participants, namely MBA students,

are provided with a model based around a queuing and resource-allocation problem

at a mine. The model is simple and allows for a ‘correct solution’ to be found.

Participants read the case study the night before and provide an initial solution

a revised answer. Statistics on usage are collected throughout.

Findings show that the participant’s solutions were a substantial improvement

on their initial efforts. However, only 39% provided the correct solution. Analysis

of the usage statistics identified that the participants who made most use of the

animated display found the correct results most quickly. Furthermore, participants

that collected multiple data from a scenario (i.e. viewing the results in detail and

drilling into data) before proceeding to the next, performed best.

The most recent study was conducted by Tako and Robinson (2009): an empir-

ical comparison of the use of SD and DES models. Participants in the experiment,

executive MBAs, take the role of a government consulting service and use either

a SD or DES model of the UK prison population to explore and draw conclusions

on how the process could be improved. At the end of the experiment participants

fill in a questionnaire to elicit their perceptions of model understanding, complex-

ity, credibility and interpretation of results. Interestingly, the results showed little

difference between the measures. In fact, only slight differences were found in how

representative of the case study participants found the models (favouring the SD

model) and that the participants found the results of the DES model more difficult

to interpret.

2.3.2 SD Research

The literature exploring and testing learning from using SD models is arguably more

substantial and mature than the DES literature just discussed. The focus is largely

on the use of management flight simulators or SD business games; essentially using

simulation models as games to help the user, who may or may not be a manager,

learn about a specific concept. Several review papers exist to summarise the key

insights that have been discovered (Lane, 1995; Rouwette et al., 2004; Gr¨oßler, 2004).

back to the 1960’s. Business games appear to have had a mixed history with some

early successes, but also failures including an infamous paper by Neuhauser (1976)

declaring that it is the model design process, not use, where the majority of learning

is found. Rouwette et al. (2004) review only the empirical studies of model use found

within the SD literature. They provide an overview of the characteristics of models,

simulators (e.g. a simulation business game) and players that influence decision

making over time. A key finding, already discussed in the section on model building,

was that model transparency has a positive relation to performance. Gr¨oßler (2004)

reviews the methodological issues in using simulation as an educational tool and lists

15 main issues. These include overcomplicated models, accounting for the different

learning styles of participants and the lack of risk for participants’ decision making.

Turning to specific relevant case studies, Bakken et al. (1994) conduct an em-

pirical study concerned with transfer of learning using executive MBA and un-

dergraduate students. In the experiment MBAs, working in groups of two, and

undergraduate students, working individually, attempt to manage an SD simulation

model of either a real estate or oil tanker system. The underlying feedback structure

of these systems is identical. After a certain amount of time the participants use

the other simulation model. To infer transfer of learning Bakken et al. look at the

performance of participants using the second simulation model.

The results show that the students outperform the executive MBAs in the trans-

fer model. This is particularly interesting as a number of the MBAs have substantial

experience in one of the domains used in the experiment. To explain this result

Bakken et al. provide data that shows that undergraduate students are much more

exploratory than the executive MBAs in the first simulation model. In fact, the

undergraduate students make approximately double the amount of decisions com-

pared to the real world managers. This results in an increased bankruptcy rate in

tions. In contrast an underlying assumption of the executive MBAs with real world

experience appeared to be ‘I know a lot about this market; so I just do in the game

what I would do in real life’ (Bakken et al., 1994).

Learning problems are not just limited to managers using simulation games;

students also have difficulties. Paich and Sterman (1993) conduct an empirical

study where postgraduate students, including MBAs, use a SD simulation model to

manage a product from launch through to maturity. The objective of the game is

to minimise the boom and bust dynamics of the market using pricing and capacity

decisions. Participants use the model over five trials and make a total of 200 pricing

and capacity decisions. After each decision stage the simulation model provides

results to the participant on current performance as well as the history of decisions

and performance in the game. Furthermore, participants can take as long as they

need to make decisions.

The results of the study found that on average the participants performed better

over the five trials. However, the participants performed poorly relative to a naive

benchmark: mean performance (profit) was 60% of the benchmark with only 17%

of participants outperforming the benchmark in the final trial. This performance

is explained by the student learning the average demand for replacement of the

product in the market and matching their capacity to this area. However, they

completely ignored other aspects of the market, for example, the growth rate of

demand for new products. In fact their actions worsened the extent of the boom

and bust dynamics.

The results of Paich and Sterman (1993) support the misperception of feedback

hypothesis put forward by Sterman (1989b) that ‘the stronger the feedback process in the environment the worse people do relative to potential’. Paich and Sterman

(1993) conclude that in order for participants to learn when dynamic complexity is

2.3.3 Conclusions on Model Use and Learning

In summary, the literature on learning from the use of simulation models is mixed.

It would seem that there is some benefit from interactively using a simulation model

to improve on decision makers initial solutions. However, the search and results re-

viewing process that a decision maker uses can radically affect this outcome. Indeed,

some studies illustrate that learning can be quite poor relative to potential.

One possible problem with drawing conclusions from this literature review is

the lack of empirical studies using DES models and the prevalence of SD models.

Some confidence can be drawn from the Tako and Robinson (2009) study. This

demonstrated that users’ perceptions of using a specific SD and DES model is largely

the same. One issue may be that participants in the Tako and Robinson study

indicated that the DES results were more difficult to interpret. Hence learning

difficulties may be worse in a complex DES study.

Further to what has been discussed, an important aspect of learning from mod-

elling and model use is the credibility of the study, model and results with the

individuals or group who will actually make the decisions. This is the topic of the

final section of this chapter.

In document Sistema de actividades educativas que desde la biblioteca escolar contribuye a la formación de una actitud ambiental responsable en los alumnos de 6to grado de la escuela Francisco Rosales Benítez (página 65-74)