Definición, codificación y escala de medida de las variables

Two questionnaires have been designed for the study: the demographic questionnaire, which asks for some information about the participant’s background, and the task review questionnaire, which is filled after each research task.

Demographic

This is a reproduction of the demographic questionnaire. The titles between the questions have been inserted for convenience and were not part of the original form. Processing instructions are placed within square brackets.

1. What is your gender? a) female

b) male c) other

d) I prefer not to answer 2. What is your age?

a) 17 or younger b) 18 to 24 c) 25 to 34 d) 35 to 44 e) 45 to 54 f) 55 to 64 g) 65 or older

h) I prefer not to answer

3. How would you rate your English skills? (according to the Common European Framework of Reference for Languages, CEFR)

a) A1: Beginner (Can understand and use familiar everyday ex- pressions)

b) A2: Elementary (Can under- stand sentences and frequently used expressions related to areas of most immediate relevance) c) B1: Intermediate (Can under-

stand the main points of clear standard input on familiar mat- ters)

A.2. Survey 97

d) B2: Upper Intermediate (Can understand the main ideas of complex text on both concrete and abstract topics)

e) C1: Advanced (Can understand a wide range of demanding, longer clauses)

f) C2: Mastery (Can understand with ease virtually everything heard or read.)

Education and field of study

4. What is the highest level of education you have completed so far?

a) High school diploma b) Bachelor’s degree

c) Master’s degree d) Doctor

e) I prefer not to answer

5. Are you a student or did you study at a university?

a) yes b) no

6. Please specify your field of study (multiple answers possible) [only if

5 a) selected, multiple choice]

a) Physics b) Mathematics c) Computer Science d) Mechanical engineering e) Electrical engineering f) Other:

7. How many semesters did you study?

[only if 5 a) selected] a) 1-2 b) 3-5 c) 6-8 d) 9-12 e) 13+

f) I prefer not to answer

Working experience

8. Please select all fields from the list, in which you have working experience

[multiple choice] a) Physics b) Mathematics c) Computer Science d) Mechanical engineering e) Electrical engineering f) Other engineering

g) No working experience in one of these fields

9. How many years of experience do you have in the selected fields? (please use combined experience in case of multi- ple selection) [only if one or more of

8 a-f selected] a) 0-1 b) 2-3 c) 4-5 d) 6-10 e) 11+

Usage of search engines

10. How often do you typically use search engines?

a) several times a day b) daily

c) several times a week d) once a week

e) less frequently

11. Some search engines provide key- words or filters to help you refine the search results. Which of these fil- ters have you used before? [multiple

choice]

a) by language or region b) by file type

98 A. Appendix

d) by domain name e) boolean expressions f) none of the above

Experience with scientific work

12. How would you rate your experience with scientific work? (e.g. research- ing and writing papers)

a) 7 point Likert scale: Beginner ... Expert

13. Which of these activities related to scientific work have you already done?

[multiple choice]

a) wrote a term paper

b) wrote a thesis to gain a scientific degree

c) published a scientific paper d) held a talk on a conference or

workshop

e) published a doctoral thesis

Usage of academic search engines

14. How often do you use academic search engines (such as ACM Digital Library or Google Scholar) during your re- search?

a) 7 point Likert scale: never ... ex- clusively

15. Which of these academic search en- gines are you using for literature re- search? [multiple choice]

a) Google Scholar b) Microsoft Academic c) IEEE Xplore

d) ACM Digital Library e) arXiv

f) CiteSeerX g) Other:

16. Some academic search engines offer advanced features that generic web search engines typically do not offer. Which of these features do you use?

[multiple choice]

a) find other papers of an author b) filter by release date

c) find papers that quote a publi- cation

d) find papers with similar content e) export Bibtex quotation

f) others

Task Review

This is a reproduction of the task review questionnaire. The titles between the ques- tions have been inserted for convenience and were not part of the original form. Processing instructions are placed within square brackets.

System Usability Scale (SUS)

[All SUS statements use a 5 point Likert scale from strongly disagree to strongly agree]

Please rate your experience with the search engine you’ve been using on a scale from 1 (strongly disagree) to 5 (strongly agree)

1. I think that I would like to use this system frequently. 2. I found the system unnecessarily complex.

A.2. Survey 99

3. I thought the system was easy to use.

4. I think that I would need the support of a technical person to be able to use this system.

5. I found the various functions in this system were well integrated. 6. I thought there was too much inconsistency in this system.

7. I would imagine that most people would learn to use this system very quickly.

8. I found the system very cumbersome to use. 9. I felt very confident using the system.

10. I needed to learn a lot of things before I could get going with this system.

Task Rating

[For the rest of the questionnaire, a 7 point Likert scale from strongly disagree to strongly agree is used for all statements]

Please rate your experience with the task you’ve been assigned on a scale from 1 (strongly disagree) to 7 (strongly agree)

11. I had enough time to finish the task 12. I found the task was very simple

13. I was able to draw on a lot of previous knowledge, which helped me with the task

14. I was overwhelmed with the task

15. I was able to gain a very good overview of the topic at hand 16. I am very pleased with the result of my research

Search Engine Features

Please rate your experience with the search engine you’ve been using on a scale from 1 (strongly disagree) to 7 (strongly agree)

17. The search results fit very well with my search terms 18. The presentation of the search results was very clear 19. The filter functions (e.g. year, author) were very useful

20. I was able to judge the relevance of a search result just by looking at its preview

21. Among the search results were mostly unfitting documents

22. Among the search results were often relevant documents which I would not have expected based on my search terms

100 A. Appendix

Features of the Prototype

[only displayed, if the prototype was used. For questions 23-25, explanatory screenshots were provided]

Please rate the features of the prototype on a scale from 1 (strongly disagree) to 7 (strongly agree)

23. The additional tools in the right column (sidebar) were very helpful to me 24. The terms in the "topic centroid" (sidebar) represented the topic of my

research very well

25. The suggested search results (sidebar) were relevant to my search

26. The possibility to save search results by clicking _{F was very useful to me} 27. The possibility to navigate freely in the search history was very useful to

me

A.2.4. Participant’s Answers