ARANCELES PARA LAS TAREAS DE VERIFICACIÓN DE ESTRUCTURAS CONSTRUIDAS

The Participant reaction to the usability of the AMS was mostly positive theme was

coded 57 times overall, and it pertains to the observation that the participants managed to work through the AMS without encountering issues most of the time. There were

3 codes associated with this theme, and these are outlined in Table 5.7. below.

Code Number of times

coded In general, the participants found the AMS to be easy

to use (C9)

27

There were cases where the participants encountered issues with the AMS interface (C10)

12

Participants were mostly against the idea of adding a time limit to the AMS (C11)

18

Table 5.7: Codes associated with the Participant reaction to the usability of the AMS was mostly positive theme.

Referring to code C9, all of the participants reacted similarly to the AMS, regardless

of what their overall performance on it was. All of the participants read through the instructions at the beginning of the AMS, and all of them answered all of the questions.

When faced with a free-response question for the first time, participants P1, P3, P7 and P8 read through the question first, before clicking into the text-entry box. On the

other hand, participants P2, P4, P5 and P6 immediately clicked into the text-entry box when they saw it, and then read the question. These points demonstrate that in general, participants were able to use the interface and work through the AMS without

difficulty.

In the cases of code C10, participants occasionally had problems with some of the finer details of the AMS. For example, participant P1 did not realize that they needed

to scroll down the page containing Q3 to reach Q4; participant P2 felt diagrams would be useful in some of the questions that did not have them; and participant P3 had

issues with answering Q3 because the dictionary did not recognize one of the words that they used in their answer. The AMS had a technical failure at one point for

participant P6, and one of the observers had to enter the testing room and reset it manually.

Without a time limit in place, the participants worked through the AMS at their own pace, and took different amounts of time to answer the questions. P1, P2 and

P5 took the longest time, with P2 taking 1 hour 16 minutes to complete the AMS, P1 taking 59 minutes, and P5 taking 52 minutes. P3, P4, P6 and P8 took less time, with

P3 taking 39 minutes to complete the AMS, P6 and P8 each taking 36 minutes and P4 taking 33 minutes. P7 was particularly fast, completing the AMS in 22 minutes.

Thus most participants took between half an hour and an hour to complete the AMS, and this information is given graphically in Figure 5.7 below. When asked to suggest

a possible time limit for the AMS, participants P1, P4, P6, and P7 suggested various amounts of time, which are shown in Table 5.8 below.

Participant P1 suggested their time limit based on how long they thought the AMS

would take to do on average. On the other hand, participants P4 and P6 estimated that since there were roughly 30 questions on the AMS, and if it took approximately 2

minutes to answer each question, then 1 hour would provide enough time to complete the AMS. Participant P7 instead suggested their time limit based on how long they took to complete the AMS, explaining why it is shorter than the other suggested times.

Figure 5.7: Graph showing the time taken to complete the AMS by each of the par- ticipants.

Participant Suggested time limit

P1 45 minutes

P4 1 hour

P6 1 hour

P7 30 minutes

Table 5.8: Table showing the possible time limits for the AMS suggested by the us- ability laboratory study participants.

Since there was no time limit in place, no participant seemed rushed, and no other possible effects of adding time pressure emerged in the study. Instead, the

participants highlighted some of the possible problems that the introduction of a time limit could bring, and these were gathered together in code C11. Participant P2

noted that adding a time limit would encourage students to answer the questions quickly; however, they also identified the drawbacks of this approach, since it would

stop students from thinking carefully about the questions. Participant P6 felt that adding a time limit would put extra pressure on the test-taker, which could cause

them to rush the questions and answer incorrectly. Participant P5 identified the key point that it is important to consider what the AMS was supposed to achieve before deciding whether to put a time limit on it:

“I think that depends on what you’re planning to get out of it - Because it would

then give somebody pressure, which would in some ways, stop you from thinking about what you are answering” [P5].

Implications for the development of the AMS

Occurrences of code C9 highlighted that participants were able to work through the AMS interface without issues most of the time. This was also found in the large-scale

administration of Version 1 of the AMS, as students managed to enter responses to the questions, and there were no complaints about the usability of the AMS in the

responses to Q34. When issues did arise, these were picked up by code C10; the issues highlighted by this code were mostly minor and question specific, meaning that they

were easily resolved. Taking the findings from these codes together implied that the AMS was presented in a format that could easily be worked through by test-takers.

This was a positive outcome from the usability testing, as it meant that the overall AMS did not need to be fundamentally re-designed for later versions.

Code C11 captured participants’ ideas about adding a time limit to the AMS. Participants were not placed under any time limit when they worked through the

AMS, and they each took different amounts of time to complete it. When asked, some participants suggested possible time limits for the AMS, and these were chosen from

their own experience, or based on an estimate of the mean length of time taken by all participants. In general, the participants were not strongly in favour of adding a

time limit, since it was not perceived to add anything useful to the AMS. However, some participants were against the idea of adding a time limit. These participants

reasoned that if the AMS was designed to learn about student understanding, then adding a time limit would add unnecessary pressure to test-takers, which could lead

to their levels of understanding being misrepresented. These reflections highlight the importance of considering what the purpose of the AMS is before trying to develop it

further, since making unnecessary additions to the AMS could cause it to diverge from its original goals. This point is considered further in Subsection 5.3.5.