Una Universidad comprometida con la cultura

Interestingly, though there were no questions about data collection, several participants expressed thoughts about the impact of data collection on search results and privacy while discussing their intelligent assistant use. With regard to search results, two participants reported that they felt that intelligent assistants learn to answer the most frequently asked questions more effectively.

Regarding personalization, three other participants related that they want the data collected about them to be used to enhance their intelligent assistants. In particular, they want their assistants to be able to tune or personalize answers based on their previous successful intelligent assistant uses.

Finally, regarding privacy, two participants felt that the fact that intelligent assistants are now often always listening for voice activation is creepy. One participant

felt that this creepiness factor was so impactful, that it has led them to have a declining opinion of voice activated technology in general.

Discussion

The interviews and diaries collected in this study reveal that experienced users of intelligent assistants have established frequent and common behaviors. These users have come to turn to intelligent assistants for repetitive, factual information needs. In

particular, users make frequent use of their assistants to find weather information and travel directions. Interestingly, both of these behaviors are typically related to the users preparing to do something else. As designs move to predict user needs, this may be a valuable insight for designers.

Additionally, the five participants in this study described awareness of the limitations of intelligent assistants with regard to complex or nuanced questions. The diaries also suggest that users are not likely to turn to their assistants to ask complex questions as 76% of diary entries focused on readily available factual information such as weather, traffic, local business information and factual trivia. While not all users reported thinking explicitly about which questions to use intelligent assistants for, there is clearly decision-making going on. For many participants, it seems possible that their intelligent assistants have become related but distinct information seeking tools from search engines.

Several aspects of participants’ comments suggest that they are developing relatively complex system models of their intelligent assistants. First, five participants reported that their thoughts about their assistants’ system models factors into their routing of information needs to their assistant or elsewhere. Second, three participants reported

thinking about how well the system might have learned the answer to their need based on how frequently other users had asked the same question.

Some of the findings from this research also reflect on previous findings in other work, such as that of Luger and Sellen (2016). For example, Luger and Sellen (2016) reported that preprogrammed jokes established unrealistic expectations for users which ultimately led them to feel like their assistants weren’t capable. However, in this research, many users reported that humorous and frivolous responses were their entry point into using their intelligent assistant. Additionally, while Luger and Sellen found that users’ expectations were rarely met by system performance, the participants in this study reported extremely high satisfaction in their diaries. Furthermore, in their interviews, all 11 participants reported that, while they understood there were limitations, their

impression of their intelligent assistants was positive. One possible explanation for this difference is that the users in Luger and Sellen’s study were not the same as the

population in this study. Specifically, Luger and Sellen’s participants were required to have used intelligent assistants for only one month while the participants in this study had been using their assistants frequently for much longer. The participants in this study had clearly found useful tasks for their intelligent assistants. Additionally, it is also possible that the passage of time may have affected perceptions about intelligent assistants though it is difficult to know if that is because intelligent assistants have improved rapidly or if people have developed more nuanced opinions of their assistants over time.

Porcheron, Fischer, and Sharples’ (2016) work is also relevant to the qualitative behaviors found in this research. In their work, Porcheron, Fischer, and Sharples found that voice interaction allowed groups of users to interact with the same intelligent

personal assistant to solve a collaborative information need. Two participants in this study reflected that they thought of their intelligent assistant as useful for getting answers that would be relevant to everyone they were around. However, two participants felt differently, stating that they felt that using their assistant around others would be rude because it would force others to stop speaking or, as Porcheron, Fischer and Sharples write, to participate in the “mutual production of silence” (2016, p. 212). More research is

required to better understand the nuances of this difference of opinion.

Finally, this research also compares and contrasts interestingly to the finding of Church, Cousin, and Oliver (2012). In their work on the social context of mobile information needs, the researchers found that their participants typically used mobile search in the presence of people who were close to them. This finding was reflected in the present study, where 75% of uses that occurred around others occurred around friends, family, and significant others. Furthermore, Church, Cousin, and Oliver (2012) reported very few mobile searches in the presence of strangers while the present research found none. In contrast, Church, Cousin, and Oliver found a greater diversity in the types of information needs addressed. They also found that pop culture trivia was the most commonly searched for topic, accounting for around one third of all queries. While trivia queries were present in the present study, they only accounted for 12% of queries Instead, weather information was the most common topic, though weather only accounted for 1.6% of queries in Church, Cousin, and Oliver’s work. It is difficult to be certain why this difference is so striking, but it may be related to intelligent assistant users’

understandings of what their assistants can and cannot do. In other words, these users may have had the same kinds of mobile information needs described by Church, Cousin,

and Oliver (2012), but they are only choosing to employ their assistants for tasks that the assistants can handle well. Future research should look at intelligent assistant usage in the context of peoples’ other contemporaneous information needs.

Conclusion

While the concept of intelligent personal assistants is not new, they have never been so widely available to the public thanks to their inclusion in mobile operating systems. Despite this fact, there has been little research into their use, particularly with regard to information seeking. This study adopted methods used in previous work in mobile, social, and voice search and applied them to the study of intelligent personal assistants. In particular, this study used diaries to collect data about specific uses of intelligent assistants and interviews to collect general perceptions of intelligent assistants among active and persistent users of these systems. The results show that these users have indeed already established clear behavioral trends. The users in this study typically search for factual information to meet repeated needs such as searching for weather and directional information. Participants in this study also had developed thoughts about when it is appropriate or inappropriate to use intelligent assistants. Interestingly, several participants felt that social interaction often spurred information needs that intelligent assistants are suited to handle. Furthermore, users understand that their assistants have limitations but still tend to be quite happy with them for the purposes that they are good at. Nonetheless, while this research has identified behaviors and trends, further research is clearly needed about intelligent assistants and information seeking.

References

Amin, A., Townsend, S., Van Ossenbruggen, J., & Hardman, L. (2009, August). Fancy a drink in canary wharf?: A user study on location-based mobile search. In IFIP Conference on Human-Computer Interaction (pp. 736-749). Springer, New York. Anastasiou, D., Jokinen, K., & Wilcock, G. (2013, July). Evaluation of WikiTalk–user

studies of human-robot interaction. In International Conference on Human- Computer Interaction (pp. 32-42). Springer, New York.

Barkhuus, L., & Polichar, V. E. (2011). Empowerment through seamfulness: smart phones in everyday life. Personal and Ubiquitous Computing, 15(6), 629-639. Callejas, Z., & López-Cózar, R. (2009). Designing smart home interfaces for the

elderly. ACM SIGACCESS Accessibility and Computing, (95), 10-16. Church, K., Cousin, A., & Oliver, N. (2012, September). I wanted to settle a bet!:

understanding why and how people use mobile search in social settings. In Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services (pp. 393-402). ACM, New York. Church, K., Smyth, B., Cotter, P., & Bradley, K. (2007). Mobile information access: A

study of emerging search behavior on the mobile Internet. ACM Transactions on the Web (TWEB), 1(1), 4.

Church, K., & Oliver, N. (2011, August). Understanding mobile web and mobile search use in today's dynamic mobile landscape. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and

Services (pp. 67-76). ACM, New York.

Google Official Blog. Retrieved from http://googleblog.blogspot.co.il/2014/10/omg-

mobile-voice-survey-reveals-teens.html [Accessed 2016-09-20].

Guy, I. (2016, July). Searching by talking: Analysis of voice queries on mobile web search. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval (pp. 35-44). ACM, New York.

Jiang, J., Hassan Awadallah, A., Jones, R., Ozertem, U., Zitouni, I., Gurunath Kulkarni, R., & Khan, O. Z. (2015, May). Automatic online evaluation of intelligent assistants. In Proceedings of the 24th International Conference on World Wide Web (pp. 506-516). ACM, New York.

Jokinen, K., & Hurtig, T. (2006, September). User expectations and real experience on a multimodal interactive system. In Proceedings of the International Conference of Spoken Language Processing (pp.1049-1055). International Speech

Communication Associations, Baixas, France.

Kamvar, M., & Baluja, S. (2006, April). A large scale study of wireless search behavior: Google mobile search. In Proceedings of the SIGCHI conference on Human

Kamvar, M., & Beeferman, D. (2010). Say What? Why users choose to speak their web queries. Proceedings of the International Conference of Spoken Language

Processing (pp. 1966 – 1969). International Speech Communication Associations, Baixas, France.

Kane, S. K., Jayant, C., Wobbrock, J. O., & Ladner, R. E. (2009, October). Freedom to roam: a study of mobile device adoption and accessibility for people with visual and motor disabilities. In Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility (pp. 115-122). ACM, New York. Kelly, D. (2009). Methods for evaluating interactive information retrieval systems with

users. Foundations and Trends in Information Retrieval, 3(1—2). 1-224. Kiseleva, J., Williams, K., Jiang, J., Hassan Awadallah, A., Crook, A. C., Zitouni, I., &

Anastasakos, T. (2016, March). Understanding User Satisfaction with Intelligent Assistants. In Proceedings of the 2016 ACM on Conference on Human

Information Interaction and Retrieval (pp. 121-130). ACM, New York. Komaki, D., Hara, T., & Nishio, S. (2012, March). How does mobile context affect

people's web search behavior?: A diary study of mobile information needs and search behaviors. In 2012 IEEE 26th International Conference on Advanced Information Networking and Applications (pp. 245-252). IEEE, New York. Luger, E., & Sellen, A. (2016, May). Like Having a Really Bad PA: The Gulf between

User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 5286-5297). ACM, New York.

Naftali, M., & Findlater, L. (2014, October). Accessibility in context: understanding the truly mobile experience of smartphone users with motor impairments.

In Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility (pp. 209-216). ACM, New York.

Nylander, S., Lundquist, T., & Brännström, A. (2009, April). At home and with computer access: why and where people use cell phones to access the internet.

In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1639-1642). ACM, New York.

Pew Research Center. (2011, July 11). Smartphone Adoption and Usage. Retrieved from http://www.pewinternet.org/2011/07/11/smartphone-adoption-and-usage/.

Pew Research Center. (2016, February 22). Smartphone Ownership and Internet Usage Continues to Climb in Emerging Economies. Retrieved from

http://www.pewglobal.org/2016/02/22/smartphone-ownership-and-internet-usage- continues-to-climb-in-emerging-economies/.

Porcheron, M., Fischer, J. E., & Sharples, S. (2016). “Do animals have accents?”: talking with agents in multi-party conversation. In Proceedings of the 2017 ACM

Conference on Computer Supported Cooperative Work and Social Computing (pp. 207-219). ACM, New York.

Portet, F., Vacher, M., Golanski, C., Roux, C., & Meillon, B. (2013). Design and evaluation of a smart home voice interface for the elderly: acceptability and objection aspects. Personal and Ubiquitous Computing, 17(1), 127-144.

Rieh, S. Y. (2004). On the Web at home: Information seeking and Web searching in the home environment. Journal of the American Society for Information Science and Technology, 55(8), 743-753.

Schalkwyk, J., Beeferman, D., Beaufays, F., Byrne, B., Chelba, C., Cohen, M., Kamvar, M., & Strope, B. (2010). “Your Word is my Command”: Google Search by Voice: A Case Study. In Advances in Speech Recognition (pp. 61-90). Springer, New York.

Sohn, T., Li, K. A., Griswold, W. G., & Hollan, J. D. (2008, April). A diary study of mobile information needs. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 433-442). ACM, New York.

Tamine-Lechani, L., Boughanem, M., & Daoud, M. (2010). Evaluation of contextual information retrieval effectiveness: overview of issues and research. Knowledge and Information Systems, 24(1), 1-34.

Appendix A: Interview Guide

Questions for Participants

1. How often do you use your intelligent assistant? 2. Where do you typically use your intelligent assistant? 3. How is your usage affected if other people are present?

4. Are there types of tasks that you like to use the intelligent assistant for? 5. What kinds of things do you think the intelligent assistant can do well?

a. Do you do anything to help the intelligent assistant with your commands? 6. What kinds of things do you think the intelligent assistant does poorly?

a. How could these problems be improved?

b. Would you use the assistant more often if it offered some other functionality? 7. How do you feel about the accuracy of the information that the intelligent assistant

provides you?

8. Has your intelligent assistant used changed the way you use your phone? a. If so, how?

b. If not, why do you think it hasn’t changed?

c. What other ways do you find information on your phone? 9. Do you notice any trends in your own intelligent assistant usage? 10. Has your intelligent assistant usage changed over time?

a. If yes, why?

Appendix B: Diary Questionnaire

1. Have you used your personal assistant since your last diary entry? If NO: the questionnaire will end

If YES: the following questions will be asked.

For the following questions, think about one specific use of your intelligent assistant since your last diary entry.

2. What was the approximate date and time of your intelligent assistant use? Open ended

3. What was your location at the time intelligent assistant use (e.g., home, work, grocery store)? Open ended

4. Who were you with (e.g., friends, co-workers, spouse)? Open ended 5. What were you doing? Open ended

6. What information did you need? Open ended

7. Were you able to complete the task with the intelligent assistant? Yes or No 8. (if the answer to Q6 is no) If you were unable to accomplish your task, why?

Open ended

9. (if the answer to Q6 is no) If you found the information another way, what was that? Open ended

10. For the task you described above task, rate your satisfaction in the intelligent assistant from 1(very unsatisfied) to 5(very satisfied) 1 to 5 rating scale

Appendix C: Informed consent form

How long will your part in this study last?

There are three parts to this study: 1) a brief introductory session (20 minutes), 2) five online questionnaires that you will completed during a two week period (about 5 minutes each), and 3) an interview in which we will ask you questions about your use of

intelligent assistants (30 minutes). In total, we expect that your part in this study will take 1 and a half to two hours.

What will happen if you take part in the study?  Overall design:

o You will participate in a meeting where you will learn how to complete diary entries.

o Diary entries will be presented in survey form using a Qualtrics form. They will not take more than 5 minutes to complete.

o Every 2 days, you will receive text message reminders containing links to the Qualtrics survey.

o After 2 weeks, the investigator will contact you to schedule an in-person interview.

o The interview will be performed in person in Manning Hall at UNC and will focus on questions about your use of intelligent personal assistants. The interview will not last more than half an hour. Your interview will be audio recorded.

o You will receive $25 upon completion of the entire study, $15 if the diaries are completed but the interview is not, or $5 if the diaries and interview are not completed.

What are the possible benefits from being in this study?

Research is designed to benefit society by gaining new knowledge. You will not benefit personally from being in this research study.

What are the possible risks or discomforts involved from being in this study? Because this study involves asking participants about their information behavior, it is possible that your responses may include personally identifying information. Data will

not be named or labeled with any personal identifiers and personally identifying information will obscured or omitted in reporting.

How will information about you be protected?

 Research data will be stored securely according to university guidelines.

 Only the principal investigator will have access to personally identifying information.

 During reporting, personally identifying information will be obscured to protect participant privacy.

 All data will be deleted upon completion of the study.

We will make every effort to prevent participants from being identified by name in any report or publication about this study. More specifically, personally identifying

information will be obscured as much as possible and participants will never be never identified by name. Although every effort will be made to keep research records private, there may be times when federal or state law requires the disclosure of such records, including personal information. This is very unlikely, but if disclosure is ever required, UNC-Chapel Hill will take steps allowable by law to protect the privacy of personal information. In some cases, your information in this research study could be reviewed by representatives of the University, research sponsors, or government agencies (for

example, the FDA) for purposes such as quality control or safety.

What if you want to stop before your part in the study is complete?

You can withdraw from this study at any time, without penalty. The investigators also have the right to stop your participation at any time. This could be because you have had an unexpected reaction, or have failed to follow instructions, or because the entire study