ANÁLISIS E INTERPRETACIÓN DE RESULTADOS

decision-related perceptions. The open-ended questions asked participants to describe their recommendation and the information they considered during the decision making process. The closed questions asked participants to indicate their agreement with a series of statements about constructs about time (time pressure and affect, impact to behavior, time adequacy), their decision (confidence, difficulty), the information they found (adequacy), and their search (difficulty). Each construct was measured using 3-5 questions, and the statements for all constructs except search difficulty were placed on four separate pages and ordered so that only one question from each construct appeared on each page. The search difficulty questions only appeared for participants if they searched for information, and these statements appeared together on the last page of the post-task questionnaire. Tables 4.8-4.10 show the text of the questionnaires and their source and/or from where they were adapted. Each of these are described below.

Recommendation decision. The post-task questionnaire started with four open-ended questions about the participant’s recommendation to their friend as shown in Table 4.8. Immediately after pressing the “Make Recommendation” button, participants were asked 1) which option they recommended as best and why, 2)

Table 4.8: Post-task questionnaire: Questions about recommendation (open-ended).

Name Question text

Decision(open-ended)

which option Which option do you recommend as the best? Why?

other options Did you consider any other options? Which ones? Use an asterisk to indicate options you knew

about before starting this task.

which information What information was most important to you to decide which option to recommend?

other information Did any other information help you decide which option to recommend? What information?

which, if any, other options they considered, 3) what information was most important to them as they decided which option to recommend, and 4) what, if any, other information helped them decide which option to recommend. The first question was intended to capture the recommendation made. The second question was intended to determine which and how many other options were considered, and the third and fourth questions were intended to determine which and how many attributes of the options were considered.

Time-related perceptions. The post-task questionnaire included items for three time-related constructs: 1) time pressure and affect, 2) time inadequacy, and 3) task pace. Although some studies have measured time pressure using one question from each of these constructs (e.g., Bowrin & King, 2010), this study uses multiple items from each construct to attempt to differentiate between three factors. Time pressure and negative affectindicate an affective response to time constraints. Time inadequacyprovides insight into whether participants thought there was enough time to complete the task; this may or may not be accompanied by a feeling of time pressure and negative affect. A fasttask paceenables comparison of the pace of work across time conditions to identify an accelerated work pace; working faster may or may not accompanied by a feeling of time pressure and negative affect or a perception that time was inadequate. There is some precedent for this decomposition: Ballard and Seibold (2004) identified these three dimensions as separate factors in a set of 10 dimensions of temporal experience in work environments. In a previous study, higher time pressure was reported if participants experienced significant system delay but there were no significant differences in perceiving working fast or feeling rushed or hurried (Crescenzi et al., 2016).

Table 4.9 shows the time-related items which asked about the time pressure, stress, and anxiety they felt; their perceptions of time adequacy (not enough, prefer more, consider more information if more time), and their perceptions of how time influenced their behaviors during the task including the pace of interactions (hurried, work fast) and how much they thought about time during the task The first time-related construct,

Table 4.9: Post-task questionnaire: Time-related items. Participant responses to statements were on a 1-7 scale where 1 = Strongly disagree and 7 = Strongly agree.

name Question text Source

Time pressure and affect(H1a)

time pressure I felt time pressure while completing this task. e, f, g

stressed I felt stressed while completing this task. f, g

anxious I felt anxious while completing this task. h

Time inadequacy(H1b)

time think I would have preferred to think longer about my decision. f, g

time info If I had more time, I would have considered more information. e, f

time not enough I did not have enough time. f, i

Time monitoring(H1c)

meta time While I was working on this task, I thought about how much time I had left. e, k

Task pace(H3b)

hurried I felt hurried or rushed during this task. e, j

work fast I needed to work fast to complete this task. e, j

Source of questionnaire items: (e) Crescenzi et al. (2016), (f) adapted from Svenson et al. (1990), (g) adapted from Rieskamp and Hoffrage (2008), (h) adapted from Maule et al. (2000), (i) adapted from Cheng (2003), (j) adapted from Hart and Staveland (1988), (k) inspired by Claessens (2004)

(2000), Rieskamp and Hoffrage (2008). These questions asked whether participants felt time pressure, stress, or anxiety during the task.

The second time-related construct,time inadequacy, measured the extent to which participants felt they had enough time for their tasks. Two statements, adapted from Rieskamp and Hoffrage (2008), asked whether participant would have preferred longer to think about their decision, or if they would have considered more information if they had more time. A final statement was whether participants thought they did not have enough time, and it was adapted from Cheng (2003).

The third time-related construct,task pace, asked about participants’ thoughts about their task pace and was measured by three items (Crescenzi et al., 2016; Hart & Staveland, 1988). Two items asked participants to express agreement with statements that they needed to work fast, or they felt hurried or rushed (adapted from the temporal demand item from the NASA-Task Load Index (NASA-TLX) (Hart & Staveland, 1988). The third statement related to participant’s attention to time (i.e., how much they thought about the time remaining while doing the task).

Table 4.10: Post-task questionnaire: Decision and search-related items. Participant responses to statements were on a 1-7 scale where 1 = Strongly disagree and 7 = Strongly agree.

Name Question text Source

Information adequacy(H5)

adeq info I had adequate information to make a good decision. a

adeq options I had adequate decision options to choose from to make a good decision. a, d

adeq info about options I had adequate information about the decision options to choose from to make a

good decision.

a

Search difficulty(H4c) (asked if participant searched)

diff overall Overall, it was difficult to search for information to make the decision. b, c

diff query It was difficult to determine search terms to use to find relevant information. b, c

diff on SERP It was difficult to choose which items to view from the search results. b, c

diff decide continue It was difficult to decide whether to continue inspecting the search results or to

search again.

b, c

diff decide stop It was difficult to determine when to stop looking for information. b, c

Decision confidence(H6b)

dec confidence I am confident in my decision. l

dec comfort I am comfortable with my decision. k

dec satisfaction I am satisfied with my decision. k, m

Decision difficulty(H6c)

diff make choice It was difficult to make a choice. k, l

diff dec option It was difficult to decide which option to recommend. n

diff decide search It was difficult to decide whether to search for information to make this

recommendation.

n

Source of questionnaire items: (a) ad-hoc, (b) inspired by D. Kelly et al. (2015), (c) inspired by Brennan et al. (2016), (d) inspired by Fitzsimons et al. (1997), (k) inspired by Claessens (2004), (l) adapted from Kerstholt (1992), (m) ad-hoc, inspired by Knijnenburg, Willemsen, Gantner, Soncu, and Newell (2012), (n) adapted from Sainfort and Booske (2000)

Search and decision-related constructs. As shown in Table 4.10, the post-task questionnaire also included items on four constructs relating to search, information adequacy, and decision confidence and difficulty.

Theinformation adequacyitems were created for this study to ascertain whether participants thought they had adequate information overall, adequate options from which to choose, and adequate information about the options. The search difficultyitems were created for this study or adapted from existing task difficulty questions used in IIR studies (Capra et al., 2015; Capra et al., 2017; Crescenzi et al., 2015, 2016; D. Kelly et al., 2015). These statements focus on the difficulty of aspects of the search process that are measured by search behaviors: creating queries, choosing items from the search results page, and deciding when to re-query. Participants who did not search for information for a given task were not presented with these statements.

Table 4.11: Stopping questionnaire administered after the last task completed. Quotes and original item numbers (e.g., MT1) are from Gerhart (2018, p. 2). Participants indicated agreement with statements on a 1-7 scale where 1 = Strongly disagree and 7 = Strongly agree.

Decision rules Name Orig.

Judgment rule: “based on the information that is available to the decision maker”

Magnitude Threshold: stop when “they have ‘enough’ information based on their initial goals”

enough info I felt I had enough information. MT1

adeq. info I felt I had adequate information to make a decision. MT3

collected info I collected enough information to make a decision. MT6

Difference Threshold: stop when “no new value in the information gained”

same info I kept finding the same information in every search. DT1

no new info I stopped searching because I was finding no new information. DT2

continuing waste I felt like continuing to search was a waste of time, because the same information

was showing up.

DT8

Reasoning-based rules: “when the decision maker reaches the limit of their cognitive capacities”

Representational Stability: stop when “the situation no longer chang[es] when new information is acquired”

understanding stable My understanding of the topic was no longer changing. RS2

view of topic stable My view of the topic was no longer changing. RS8

no longer learning I was no longer learning about the topic. ad hoc

Single Rule: “based on information for only one single search criteria”

one thing I was focused on getting information on one thing. SC1

more imp. info I only considered looking for the piece of information most important to me. SC6

one aspect I was most concerned with finding information on one specific aspect. SC10

Mental List: stop when “a predetermined set of criteria” met

list of things I had a list of certain things I was interested in. ML1

option satis. I stopped searching when I found an option that satisfied the things that were

important to me.

ML3

certain things I have certain things I look for for a decision like this. ML6

Participants were also asked two sets of questions about their recommendation: their confidence in their recommendation and their difficulty making a choice. There were threedecision confidencestatements; the decision confidence item was selected from Jacoby, Speller, and Kohn (1974), and the decision comfort and decision satisfaction statements were selected from a set of five items from Sainfort and Booske (2000). Three questions asked about difficulty making the choice. One item was adapted from Sainfort and Booske (2000) and two items were created for this study inspired by Knijnenburg et al. (2012).

4.7.3 Decision questionnaire. After their final task, participants were asked to complete an additional