Componentes de la Conducta Asertiva

The evaluation approach adopted in year 2 followed the model applied in year 1, with the exception that all participants were asked to complete questionnaires electronically. A total of 14 participants were involved in the second evaluation session, with all volunteering their time to participate.

During the session, at one point there were some minor technical problems with the server hosting LAVA. Groups were notified of these issues during the session, and they were very quickly resolved. These issues were commented upon by two participants in their questionnaire responses, with both showing frustration at their investigation work being interrupted:

“It kept logging us out, so we had to keep starting over, which was highly frustrating” “On first attempt to open a hotspot, had to log in again and received another error message (can’t remember) and took a couple of tries to load”

From the participant responses there was frustration at the problem encountered. Whilst only 2 participants out of a group of 14 noted this issue in their questionnaire responses, a number mentioned the issue during the session. When considering usability and educational value, the following results were returned.

Usability

In year 2, all of the 14 session participants filled in the SUS and Educational Values sections of the post session questionnaire. All respondents answered all questions, with many providing additional comments on their impression of the evaluation session in the open ended section of the questionnaire. A summary of the results returned for the SUS and Educational Value multiple choice questions is shown in Figure 74, with Table 15 providing details of the raw scores returned. For this evaluation session, the average SUS score recorded for LAVA was 59. As the scale goes from 0 to 100, with 50 being neutral, this gave LAVA a positive usability score. It is of note that this result is lower than the average usability score of 65 returned during the first evaluation session. The lowest absolute usability score awarded was 30, with the maximum being 82.5. As with the average score, these results show a slight shift to the left when compared with the results of the first evaluation session. However, as 11 of the 14 participants returned scores in excess of 50, the second year results still show a significant number of users rating the usability of the system positively. It is also important to note the reduced sample size in the second evaluation session which amplifies the effect of any outlying results to the

average score calculation. Furthermore, given the technical difficulties experienced at the start of the session, it is likely that participants’ opinions with respect to usability would have been negatively affected.

When considering individual questions, responses to the question; “I felt the system very cumbersome to use”

solicited the highest number of responses indicating disagreement with the statement, therefore indicating that participants felt that they were able to use the system with reasonable ease. In terms of the lowest scores returned, the question;

“I found the various functions in this system were well integrated”

faired less well for the second time running, with many participants again indicating that whilst they were able to use the system with relative ease, they were aware that is was not as cohesive and polished as more mature systems.

Throughout the session a number of usability issues were reported, with many being related to the login issue caused by the technical problem at the start of the session. Of the remaining issues, most related to the mechanisms designed to allow groups to investigate separate contexts within the main body of the excavation. Of the three groups surveyed when dealing with contexts, all reported having initial problems, with one group showing complete confusion which led to one participant commenting in the post session questionnaire:

“What were we supposed to do with the contexts?”

However, the remaining two groups did seem to be able to grasp the concept after two or three attempts, with one group offering the following response:

“Initially, found contexts a little difficult to fully understand, couldn't tell at a glance how to further explore them”

This seemed to indicate that perhaps the initial instructions were not necessarily helpful in determining how to use this system. This was further confirmed by feedback from another group who commented that:

“The directions of the program after you start the excavation are not as dire[c]t and not as engaging as one would hope”

Participant SUS Score Educational Value Score 1 55 53.3 2 65 60 3 30 41.6 4 45 61.6 5 82.5 65 6 32.5 35 7 62.5 55 8 60 38.3 9 57.5 61.6 10 67.5 53.3 11 77.5 63.3 12 65 81.6 13 72.5 63.3 14 60 75

Table 15 - Year 2 Participant SUS and Educational Value Scores

2007/8 SUS Results Returned

0 1 2 3 4 5 0 - 10 _{11 - 20 21 - 30 31 - 40 41 - 50 51 - 60 61 - 70 71 - 80 81 - 90} 91 - 100

Score (in range)

Fre

q

u

enc

y

Figure 74 – Year 2 System Usability Scale Scores

2007/8 Educational Value Results Returned

0 1 2 3 4 5 6 0 - 10 11 - 20 21 - 30 31 - 40 41 - 50 51 - 60 61 - 70 71 - 80 81 - 90 91 - 1 00

Score (in range)

F

req

uency

Once clarified, however, the concept of separate contexts within the excavation did appear to be desired amongst the participant groups, with one participant making a note in their post evaluation questionnaire responses that it was a desirable feature which should be present in the system.

“The ability to pick what tools you use to excavate a certain area was good”

Given this, it seems as though many of the problems associated with the contexts system are related to the level of instruction and the interfaces provided to manage the functionality, and not fundamental to the approach being adopted per se. This is reassuring and emphasises the importance of redesigning the contexts interface to help develop a more positive user experience.

On the whole, 5 of the 7 groups stated that they found that the visualisations offered by the 2D interface of benefit, with many suggesting minor changes to help further clarify things: for example, using different coloured hotspots to distinguish newly found artefacts from those already discovered and recorded. In addition, one group suggested that replacing the site maps with more detailed site photographs would be beneficial as this would make it possible to show the different stages of the excavation more clearly, i.e. before and after clearing the top soil etc. One participant commented that this type of feedback would be useful when determining whether to finish or continue a line of investigation:

“It was unclear when to progress as basic pictures do not show the entire site [soil, etc.], which is necessary when you need to know when to continue”

Given the design of the system, this adaptation could easily be accommodated if there were existing photographs showing the details required.

Educational Value

All participants completed the educational value section of the post session questionnaire, with all respondents answering all questions. A summary of the results is shown in Figure 75, with raw values provided in Table 15. For the second evaluation session, the educational value score returned was approximately 58, which is slightly lower than that returned in the first evaluation session. However, as the scale ranges from 0 to 100, with 50 being neutral, 58 is still broadly encouraging and provides a positive indication of the level of engagement and value that participants apply to their use of LAVA. Despite a slightly lower average, the range of scores returned in the second session is similar to that of the first evaluation session, with a low of 35 and a high of 82 returned.

The spread of scores is skewed slightly to the left as shown in Figure 75. However, with most scores falling in the 51-70 range (9 of 14) the effect of the skew is limited. As with the results from the first evaluation session, the overall distribution of scores reflects a broadly positive outlook; with

significantly more results returned in the 51-100 range than in the 0-50 range (11 participants in the 50- 100 range versus 3 participants in the 0-50 range respectively), participants associate value with their use of LAVA. Given that the effect of the technical problems experienced early in the session cannot be quantified, it is encouraging to see that participants still return broadly positive results which show a promising reaction to their interactions with LAVA.

Of the feedback obtained through the open questions included in the post session questionnaire, participants made the following encouraging comments which again seem to backup the conclusion that, in general, the user response to LAVA is positive:

“The pictures and artefacts were good”

“The simplicity of the program is appreciated as it makes the [excavation] process simple” “It sounds complicated, but very simple to pick up once you start, and the prompts (with one day penalty) for missing skills [are] very useful to remind/teach you what you need. Also enjoyed the witty remarks on the staff! Based on real life examples?”

“Being able to investigate the finds was useful to understand what was there [on the excavation site]”

“A great tool to help people who might not have the opportunity to dig to actually understand the processes and demands of an excavation - as organiser or grunt [student?]. Relatively easy to grasp, teaches a lot, and interesting”