Pasajes finales de esta experiencia

The evaluation method that was employed in this research was the TA cooperative evaluation method. As discussed in Section 4.5.3, the TA method has a number of variants prescribed on the basis of the designer’s interaction with the user. An evaluation method that is conducive to a design science methodology and action research approach, which requires more than just the functionality of the system, was necessary. A TA process that involves greater interaction between the researcher and the user is cooperative evaluation (Holzinger, 2005; Olmsted-Hawala et al., 2010).

Monk et al. (1993) maintain that cooperative evaluation can be applied to prototypes and simulations, and is envisaged to be used in the early stages of the design process. TA cooperative evaluation combines empirical usability evaluation with a qualitative research design, by integrating interview type questions into the traditional TA method (Dumas & Redish, 1999). This method involves interaction and collaboration, where the user and the designer can both ask questions, but it also involves the evaluator steering the participant in the right direction while using the system (Monk et al., 1993). This is specifically suitable for this research, as some of the participants using the system will not have carried out LCC calculations to any great extent and thus will need to be guided in relation to what to do. It is also applicable based on qualitative data that is generated from engaging with the 5D BIM process and the views of the participants in relation to adding on LCC functionality to this work-flow. The level of interaction between the participant and the evaluator depends on some extent to the goals of the evaluation (Dumas & Redish, 1999). The objectives regarding the evaluation do not just specifically relate to the product but also include general questions on the overarching process of LCC and BIM, which are primarily addressed in the post-evaluation (debriefing) questions. Thus the method is both a usability evaluation and a research interview. This also complements an action research strategy because of the social interactions and the qualitative nature of the data.

146

Monk et al. (1993) claim that TA cooperative evaluation requires very little training and can quite simply identify and rectify problems with model development. In this research, two TA evaluations were carried out, one on the prototype (where feedback is included in a more developed version) and a second evaluation on a more developed version. Monk et al. (1993) outline the procedure to cooperative evaluation as follows:

 Recruiting users – Define the target population and recruit users.

 Preparing & establishing the tasks – Tasks should be specific. Tasks should explicitly outline what is required by the user.

 Interacting with the user and the system and recording what happens – It is important to record what the users say and do. Recording can be carried out by video, tape recorder or note taking. The session also includes a post- evaluation interview (debriefing session).

 Summarising the observation – When analysing the data from the evaluation, two broad categories of information should be examined: behaviour and user comments.

Faulkner (2000) maintains that participants enjoy TA cooperative evaluation more than the other formal techniques, as they have a sense of helping to find and fix the problems rather than feeling they are under investigation. Scheiderman & Plaisant (2005) and Monk et al (1993) agree the informal nature of TA puts the participants at ease and often leads to spontaneous interaction and comments that were not envisaged. These methods also have advantages in cases where the user population is small, specialised and the participants may not have the time nor interest in participating in lengthy performance evaluation (Nielsen, 1993). Faulkner (2000) suggests this method is as close a usability evaluation to simulating the users actual environment other than observing them in the field.

Henderson et al. (1995, cited in van Velsen, van der Geest, & Klaassen, 2007) state that TA identifies more usability issues than interviews, questionnaires and field observation. The TA process includes interview type questions in the debriefing stage of the TA exercise (post-evaluation questions) to generate subjective feedback on the topic addressed in the system rather than just the usability of the system itself (van Velsen et al., 2007). In the debriefing stage more general questions on the topic

147

of LCC are asked to gather opinions and attitudes on the issue of LCC and BIM, rather than just the usability of the LCC tool. This enables qualitative data to also be generated on the process the participants are engaged in and the relationship between the actions being taken and the results being obtained. The debriefing process is important in this context because it focuses on the human interactions at play which is necessary in action research. This broadens the data being generated and enables links and observations between the system and the theoretical grounding from which it emanates.

As illustrated in Figure 4.6, there were two TA cooperative evaluations carried out - firstly with six participants on model development (prototype) and subsequently with ten participants to evaluate the developed version/process. The procedures adhered to in this research followed the formulated structure evident in Nielsen (1993), Faulkner (2000) and Wahl (2000) for TA cooperative evaluation but are particularly applicable to the steps outlined by Monk et al. (1993) which are presented above. The procedures outlined by Monk et al. (1993) were the same for both evaluations. Further detail related to this process is discussed in Chapter 6.