LCS in the Ria de Vigo

The third and final issue relates to Task 4 exclusively. Although all 15 participants managed to complete the task as expected, two required guidance in the form of having to be reminded to press the ‘Save’ soft key before returning to the Home screen (see Figure 6.23).

Further examination of the results revealed that both instances occurred with participants who performed the study using the feature phone, the design guidelines of which specifically require users to save changes before they are applied [158].

6.8. ANALYSIS AND DISCUSSION 134

Figure 6.23: Update profile, screen ‘Save’ soft key

Coineidently, both of these participants owned the identical handset to that of which they used in the study (a Nokia Asha 201), suggesting that their mistakes were likely not to have been caused by unfamiliaritv with the device. Both participants however specified isiXhosa as their preferred language, which may have been a contributing factor. Interestingly, the issue was not experienced by any participants who performed the study using the smartphone, even though the same ‘Save’ option was mandatory. As a solution, the client application was modified so as to prompt users to save changes before navigating back to the Home, screen.

The median number of button presses made by participants when attem pting Task 4 was identical to the minimum number required (3), highlighting the ease with which the majority of parti­

cipants were able to complete it as expected. This favourable result may have been due to the fact that the task was always performed last by participants. In addition, the task was simpler than other tasks, merely requiring participants to change the client application’s language from within the Update profile, screen.

Additional observations

It was decided to examine the number of log in attem pts a participant made when performing their first assigned task, compared to their second task. Eight participants (1, 2, 5, 10, 12, 22, 26, 27) made fewer log in attem pts when performing their second task, compared to four participants (14, 19, 20, 24) who made a greater number. This suggests that the client application possesses a certain degree of learnabilitv. Similarly, 12 (1, 5, 6, 8, 19, 21, 24, 25, 26, 28, 29, 30) out of the 14 participants who did not manage to complete their first task were able to complete their second. While only a single participant (Participant_3), who managed to complete their first task, did not manage to complete their second. Again, highlighting the learnabilitv of the client application.

6.8. ANALYSIS AND DISCUSSION 135 6.8.4 Visualisation

Another aim of the usability evaluation was to determine how effectively different chart visual­

isations conveyed poll results to participants. Particular emphasis was placed on determining which charts participants found easiest to understand, as well as what role interactivity within the charts played in helping participants gain further insight into the data. Results are presen­

ted in three sections: chart preference (Section 6.8.4), interactivity and understanding (Section 6.8.4) as well as usability concerns (Section 6.8.4).

Chart preference

Two questions were posed to participants which aimed to determine which chart type they preferred. The first question (“Which chart did you find easiest to understand?”) was presented to participants after having completed Task 3. Responses indicated that slightly under half (46.67%) of participants found the pie chart the easiest to understand; 20% indicated preference for the bar chart; while 33.33% rated them as both equal. The second question (“Which of the visual representations did you find most useful?”) formed part of the post-intervention questionnaire, to which 46.67% of participants again indicated their preference for the pie chart.

Although these results cannot be considered unusual due to the self-explanatory nature of a pie chart, Cleveland, in his book titled The Elements of Graphing Data, argues that human per­

ception is less accurately able to judge differences in area (the technique used by pie charts) compared to differences in length (the technique used by bar charts), possibly leading to inac­

curate readings (31]. Conversely, pie charts are very effective at conveying the “part-to-whole relationship” (44, p.6], an important characteristic when comparing what proportion of residents within a given suburb are experiencing a particular service delivery issue.

Written responses from Participant_ 10 and Participant_25 highlighted confusion with the choice of unit used on the dependent variable (y-axis) of the bar chart. This may possibly have been a contributing factor as to why the pie chart took preference. Interestingly, many participants were observed using a combination of the charts, along with the numerical table when interpreting results, especially those who did not realise the smartphone charts were interactive. Ultimately, it was decided to keep both the pie chart and numerical table, and remove the bar chart, the technique rated least favourably by participants.

Interactivity and understanding

Study participants using the smartphone (n = 15) were presented with unlabelled, interactive charts. Participants were expected to ‘tap’ areas of interest within the charts, thereby revealing

6.8. ANALYSIS AND DISCUSSION 136 hidden information (such as the size of the selected sector). Part of the study aimed to determine whether this interactivity encouraged participants to explore the results, helping them gain further insight into the presented data.

As described in Section 6.5.1, three-quarters (75%) of participants who performed Task 1 using the smartphone indicated that they did not make use of the chart interactivity. Although this seemed surprising at first, observations revealed that many participants were unaware that the charts were in fact interactive, even though the screen title clearly stated: ‘Tap chart for more details’. Three participants (2, 22, 24) indicated that instead of having to interact with the chart, they would prefer the necessary information to appear directly on the chart itself, thereby allowing them to glean the necessary information more easily.

Interestingly, although only 25% of the participants who performed Task 1 using the smartphone indicated that they had used chart interactivity, the majority of responses to the post-intervention questionnaire statement “The interactive charts made information more easily understandable”, were positive. This resulted in a usability rating of 76.67%. A possible reason for this favourable rating is that participants who used the smartphone when performing the study (n = 15) were comparing this additional information provided to the blank chart initially presented (see Figure 5.8a).

Observations and concerns

After having performed Tasks 1 and 3 as expected, participants were presented with the Results screen, similar to that shown in Figure 6.24. The screen contained a numerical table, as well as the option to view two charts summarising responses to the given poll question. By examining these charts, participants were asked to note the number and percentage of people experiencing the exact issue they were experiencing.

Charts presented to participants who completed Task 1 displayed the number and percentage of people without water (answer: 131 people/83% of all votes). While charts presented to participants who completed Task 3 illustrated the number and percentage of people who were experiencing a water quality issue. Participants were asked to note those experiencing brown water (answer: 83 people/57% of all votes).

6.8. ANALYSIS AND DISCUSSION 137

Figure 6.24: Numerical table summarising poll results

Consulting the written answers from participants who performed Task 1, it can be seen that only four participants (9, 10, 16, 17) correctly noted these figures. Furthermore, two of these participants (9, 10) were observed incorrectly consulting the numerical table, instead of the charts (as instructed by the questionnaire) when determining their answers.. Although this might seem like a low number, it should be noted that only eight out of the 15 participants performed Task 1 as expected, and as such only these participants were presented with the R esults screen.

Interestingly, although only four participants correctly read the values from the charts, the majority of participants responded favourably when presented with the statement: “ftlhc charts were easy to understand”. This resulted in a usability rating of 88.57%. Two possible reasons for this result have been identified. Firstly, as participants were not told the correct answer after having performed the task, they may have been unaware that they interpreted the values incorrectly. Secondly, as with previous statements, participants may not have wished to convey a negative reflection of themselves, and therefore answered more positively than they otherwise would have.

W ritten answers for Task 3 were noticeably more accurate when compared to Task 1, with 66.67%.

(n = 10) participants correctly interpreting chart values. As previously mentioned, this improved result is likely to have been caused by participants’ exposure to other tasks before performing Task 3, thereby improving their familiarity with the client application. This may potentially highlight the application’s learnabilitv. As a ease in point, 10 out of the 15 participants who performed Task 3 had performed a prior task, while none of the 15 participants who performed Task 1 had done so. An eciual number of English and isiXhosa participants performed each task, therefore language preference is unlikely to have affected the results. Observational notes made during Task 3 again highlighted that two participants (18, 25) read values from the numerical table instead of from the charts, as instructed. Participants responded slightly more favourably to Task 3’s statement: “ftlhc charts were easy to understand”, resulting in a usability rating of 90.67%..

6.9. SUMMARY' 138 A minor usability concern highlighted by five participants (6, 7, 11, 20, 23) was the use of the term

‘votes’ on the Results screen. This issue is likely as a result of participants misunderstanding that when asked to inform Makana Municipality about a service delivery problem, they were in fact answering a service delivery poll. Alternative suggestions from participants included renaming votes to either respondents, people or comments.

6.9 Summary

This chapter presented the first of two usability evaluations, performed using the cross-platform client application. The study highlighted problems experienced by participants while navigat­

ing the client application, particularly misunderstandings surrounding the use of certain words (‘poll’, ‘votes’) and phrases (‘Take a poll’) as well as ambiguities within the menu options (‘Water monitoring’, ‘Talk to your municipality’). The study also indicated which chart type participants found most understandable, as well as whether interactivity within the charts helped better con­

vey meaning. The task performance measures from several different sample groups within the study were compared and discussed, drawing out interesting trends and findings. The follow­

ing chapter details the comparative usability evaluation performed using the MobiSAM and MobiSAM Report cross-platform client applications.

Chapter 7

Evaluation: client application