A short lab-based pilot study was conducted on June 12-14 2007 to explore user reactions to the SnapAndGrab concept. The experiment assessed whether new users were able to access and use the core features of SnG without any prior training. We sought to answer the following questions:
1. Could the users complete an interaction based on the visual instruction set shown on the display? (See Figure 5-2)
2. Did the users know where and how to find and access the returned multimedia on their handsets?
3. Did they find the interaction too time consuming?
Figure 5-2: Visual instructions on the SnapAndGrab display
5.5.1 Experimental subjects
The author recruited eight University of Cape Town (UCT) students, all between 18 and 24, who owned Bluetooth-enabled camera phones. According to Nielsen’s [118] research, a usability study with eight subjects can theoretically uncover ~90% of the underlying usability
problems – a value deemed acceptable by the author for a pilot study. All eight students had prior PC and Internet experience and had completed UCT’s computer training programme in their first year of study.
5.5.2 Experimental plan
The laboratory experiment aimed to capture data showing whether the users (subjects) were able to complete a basic interaction with SnapAndGrab (SnG) without prior training; that is, based solely on their interpretation of the visual instructions shown on the SnG screen. An SnG interaction relied on the user being able to effectively use the Bluetooth and camera features of their cellular phone. It was therefore necessary to gain an understanding of each user’s Bluetooth and camera-phone experience levels to assist the analysis of the task-based data gathered in the experiment.
The subjects completed a pre-experiment questionnaire (see Appendix A) to gauge these experience levels. The pre-questionnaire included a consent form, gathered basic demographic information (age, occupation, gender and home language) and probed the following topics:
1. Had the subject ever used their mobile phone’s camera and Bluetooth features before the study?
2. How often had they used their camera?
3. For what purposes had they used their camera?
4. Had they ever shared pictures or video via Bluetooth?
5. Had they ever been sent a file via Bluetooth?
6. Did they know where to find the files after receiving them?
Each subject was then asked to complete a simple task, which required them to download three media packages from the SnG system. The users were not briefed or trained regarding the use of SnG. Instead, they had to rely on a simple three-step instruction displayed along the bottom of the screen (see Figure 5-2). The subjects were given a maximum of fifteen minutes to complete the task. If the subjects were unsuccessful, the facilitator (the author) would provide the necessary clarification and assistance before asking them to attempt the task again until success was achieved.
The subjects were then required to complete a post experiment questionnaire (see Appendix
A).
The post-questionnaire probed the following topics:
1. Did the user understand the SnG concept and have an idea how they might use it in real life?
2. Was there anything about SnG the user would change?
3. Did they know where to find the files that SnG sent to them?
4. Did the users feel that a SnG transaction took too long for what it was worth?
The resultant data was analysed to determine whether:
1. The users were able to successfully complete a SnG interaction.
2. An SnG interaction was efficient enough, from the users’ perspective, to be of practical use.
3. Users were able to find and consume the content they downloaded from SnG.
5.5.3 Results
See Appendix A for pre and post questionnaires.
Table 5-1: Lab test: Pre-questionnaire results Numerical scale used (1 = rarely…5 = a lot)
Questions Subjects
1 2 3 4 5 6 7 8 Avg.
Gender F F M F M -* M F -
Age 18 22 20 19 18 19 21 24 20,3
Home language Zulu Sesotho English Zulu English Zulu English Sesotho - Has used
camera feature Yes Yes Yes Yes Yes Yes Yes Yes -
Has taken pictures at an event
3 5 3 4 5 5 2 5 3,67
Has taken pictures of friends and family
4 3 4 4 5 5 4 5 4,11
Has taken pictures of interesting or amusing things
2 4 5 3 5 5 3 5 4,11
Questions Subjects
1 2 3 4 5 6 7 8 Avg.
Has recorded videos of friends or family
2 1 1 3 5 3 2 4 2,44
Has recorded videos of an event
3 5 1 3 5 1 1 4 2,67
Has recorded videos of interesting or amusing things
3 5 3 3 5 3 1 4 3,56
Has used Bluetooth feature
Yes Yes Yes Yes Yes Yes Yes Yes -
Has shared
pictures or video Yes No Yes Yes Yes Yes Yes Yes -
How? MMS or
Bluetooth? Both - MMS Bluetooth Bluetooth MMS Bluetooth Both -
Types of files sent by someone else
Photo,
video Photo
Photo, video, audio
Photo, audio
Photo, video, audio, contact details
Photo, video, audio, contact details, themes
Photo, video, audio
Photo, video, audio, games
-
Did you know where to find the files after receiving them?
Yes Yes Yes Yes Yes Yes Yes Yes -
* The subject did not wish to disclose their gender on the questionnaire
User performance during the test was assessed on the following scale : 1. The user managed to use SnG without the facilitator’s (author’s) help.
2. The facilitator provided minimal assistance to the user.
3. The user was unable to use the board despite the facilitator’s help.
4. The user’s phone was incompatible with SnG.
Table 5-2: Laboratory test results
Subjects
1 2 3 4 5 6 7 8 Avg.
Scale
value 2 2 2 4 2 2 2 1 2.33
Users completed another questionnaire after the study, with the following results:
Table 5-3: Post questionnaire results
Question Subjects
1 2 3 4 5 6 7 8
Did you feel that the total time taken to complete an interaction was too long?
No No No - No No No No
Did you know where to find all the
files that were sent by SnG? No No No - Yes Yes Yes No
Which file types were you unable to
find? All Text
Found all eventu ally
- - -
VCal took a while
VCal
Could you imagine using SnG in an
everyday public setting? Yes Yes Yes Yes* Yes Yes Yes Yes
* Result based on a demonstration of SnG by the author.
Users were also asked for suggestions and to say what their favourite SnapAndGrab features were, with the following results:
Table 5-4: User suggestions for SnapAndGrab Subject
1 Put it in secure places in big malls and ensure that data sent and received doesn’t contain any viruses that would damage phones.
2 I would not change it.
3 Perhaps mention the space requirements below the files and mention where the files get saved
4 Make it compatible with a wider range of cell phones Allow auto updates
5 Add more information. Things such as urgent announcements or breaking news.
6 It would help if it had more instructions about Bluetooth settings.
7 Allow for multiple pictures to be taken at once.
8 Include a dictionary or at least definitions that can be sent back to me if I send a word through.
Table 5-5: Users' favourite features of SnapAndGrab Subject
1 Get updates about Beyonce’s shows and other information.
Promotions on campuses, targeting students who use phones regularly.
2 The ability to provide more than one item that comes with a picture.
3 The option to choose the content you download, and then save it and take it with you.
Once you have the content, you can share and spread it.
4 The speed and different media packages’. (This user’s phone was not compatible with SnG; her comments were based on a demo by the facilitator.)
5 The high definition picture quality, which makes it even brighter then usual billboards.
The simplicity and speed.
6 It’s paperless but just as effective as the information brochures that people end up throwing away.
7 Besides Bluetooth being free, it was very fast.
8 Fast, don’t have to log in to get it.
Don’t wait long for it to be sent back.
5.5.4 Findings
The data revealed that SnG was not ideally suited for deployment as a walk-up-and-use public information system for new users.
1. Only one of the eight users was able to work out how to use SnG based on the visual instructions alone.
2. Five of the eight users were able to complete an interaction with minimal assistance.
3. One of the eight users possessed an incompatible phone.
4. One user was unable to find the returned multi-media on their phone.
Analysis of the pre-questionnaire data ruled out the subject’s handset as a usability barrier and a cause of the subjects being unable to complete the task. All of the subjects used their camera phone to take photos on a regular basis (the average value for the photo activities was 3.96 out of 5). All had used Bluetooth to share photos and videos on a previous occasion. We concluded that the subjects were all experienced camera and Bluetooth users.
The author did, however, notice that subjects assumed that the absence of a keyboard and mouse meant that they should interact with the display via a physical touch. SnG’s unique style of interaction -- taking a photo of the screen and sending the photo to SnG -- was not immediately obvious to new users despite the visual instructions. The data showed that once the facilitator (the author) entered the room and explained how SnG worked, five of the six users, who initially failed to complete an interaction, were able to do so without any further assistance.
Further analysis of the data showed that SnG’s reliance on OBEX services and the user’s familiarity with the storage of OBEX files did not pose a usability problem for five of the eight users. Four of those users found it difficult to locate less common OBEX file types such
as text and calendar entries. This finding is consistent with the data supplied in the pre-questionnaire: Before the study, users had only been sent photos, videos, audio or game files via Bluetooth. Calendar entries and text files were not mentioned, indicating that the users would not be familiar with these file types.
None of the users felt that the interaction took too long for the value of the media returned.
The post questionnaire feedback (see Tables 5-4 and 5-5) revealed that the subjects were satisfied (often excited) with their experience of SnG.
We concluded that users who were experienced at using the Bluetooth and camera features of their phone were able to learn how to use SnG effectively with minimal training. SnG’s unique style of interaction was a potential usability, and therefore access barrier; but its learnability, efficiency in use and provision of a satisfactory experience compensated for this.
The study did, however, show that the provision of local training and support would be essential during future field studies, particularly as SnG was intended to be deployed within resource-constrained communities in which technology skills and knowledge were not guaranteed.