Data were collected from multiple methods: pre-questionnaires, screen recording, transaction logs, think-aloud verbal protocols, and post-questionnaires.
First, pre-questionnaires were used to collect demographic information about participants, such as gender, age, race, major, computer skills, and others.
Second, users were asked to conduct three search tasks using Library of Congress Digital Libraries (LOC-DL). All of the users’ search activities were recorded using Morae usability testing software (http://techsmith.com/morae.html) to collect users’ behavioral data, such as pages viewed, input device operations (e.g., clicks, keystrokes, etc.), facial expressions, and voices.
Third, subjects were asked to verbalize their intentions, thoughts and feelings in relation to their search activities during the search process. All think-aloud utterances along with facial expression and voice tone were recorded using Morae software.
Fourth, post-search questionnaires were used to measure subjects' perceptions of system support, difficulty, and satisfaction about search tactic application after the search. Figure 3-1 summarizes the data collection procedures of this study.
3.2.2. Selected Digital Library System
For this user study, Library of Congress Digital Libraries (LOC-DL) was selected as a test digital library system. LOC-DL is one of the representative national-level digital libraries in the United States, which covers a wide variety of topics. By selecting a currently operating digital library system, instead of an experimental system, the author aimed to observe users' interactions with real resources of a real system that reflect more real situations. In addition, digital libraries create a new searching environment in which many of the searchers are novice users (Xie 2009), so this study intends to uncover users' unique search behaviors in digital libraries, which have not been widely studied in IR research. To be more specific, LOC-DL was chosen as an IR system to be examined based on the following reasons:
Coverage of topics — LOC-DL covers a wide variety of topics such as history, maps & geography, biography, arts & culture, religion, and philosophy, amongst others.
Resource formats — LOC-DL offers multiple formats of sources such as text, images, audio files, video clips, and maps.
Search strategies — LOC-DL provide a variety of search features in support of different types of search tactics.
Help features — LOC-DL offers different types of explicit help features, including help pages, FAQ, search finding aids, instructional pages, etc. Credibility of contents — resources of LOC-DL are originated from reliable,
Representation of digital libraries in academia — LOCDL is one of representative national-level digital libraries run by Library of Congress.
This study involves digital library systems, so the results of this study cannot be
generalized to other IR system settings. External validity indicates how well the results of a study can be generalized across different populations and settings. In this study, all the subjects were recruited from different disciplines including humanities and arts, social sciences, sciences/ engineering, and they represents both undergraduate and graduate student groups. Therefore, the results of this study can be generalized into the setting of digital library uses in a research university.
3.2.3. Search Tasks
In this study, three types of search tasks were designed to explore users' interactions with LOC-DL, including known-item search task, specific information search task, and exploratory search task. Search task types can be classified by search results that a user intends to obtain. Known-item searching refers to finding an item when a user knows particular information about that item, such as author, title and so forth. Specific
information searching represents looking for exact data or a fact. Exploratory searching
indicates looking for items with common characteristics (Xie, 2008a). Using multiple types of search tasks, the author planned to observe more diverse user engagement and corresponding system support during the search process.
First, as a known-item search task, subjects were asked to find a video clip of "Coca-Cola advertisements in 1964." LOC-DL has special collections about "Fifty years of Coca-
Cola." In the collections, there are several video clips of Coca-Cola advertisements, and subjects were asked to locate one of them broadcasted in 1964.
Second, as a specific information search task, subjects were asked to find who were the four US presidents assassinated and when they were assassinated. This task requested users to find very specific factual pieces of information. In this task, they were asked to find the names of four president who were assassinated while in office (Lincoln, Garfield, McKinley, Kennedy) and the dates of the assassinations (1865, 1881, 1901, 1963).
Therefore, in total, eight pieces of specific information were supposed to be searched to successfully complete this search task. LOC-DL has a special collection on American presidents, which includes information about president assassinations in the United States.
Third, as an exploratory search task, subjects were asked to collect as many aspects as possible on a certain topic within eight minutes. The selected topic was "Jackie
Robinson’s life and his career as a major league baseball player." LOC-DL has a special collection on Jackie Robinson with various aspects of information ranging from overview, timeline, essays, photos, achievement, and to his family. Subjects were allowed to apply any search strategies they wanted to solve the search task within the boundary of LOC- DL. To objectively calculate aspectual recall rates, subjects were instructed either to copy and paste the findings to the MS-Word file or to speak out whether to use the information they accessed. Table 3-3 summarizes three types of search tasks designed in this study. For simplicity sake, task ID numbers are used to indicate each type of search task throughout this dissertation.
Table 3-3. Three search task types: known-item search (Task 1), specific information search (Task 2), and exploratory search (Task 3)
Task ID Search task type Task Time limit
Task 1 Known-item
search
1. Find a Coca-Cola advertisement video clip
in 1964 5 minutes
Task 2
Specific information
search
2. Who are the four US presidents assassinated during their presidency? In which year was each of the president assassinated?
5 minutes
Task 3 Exploratory search
3. Assume that you are supposed to write a final report on Jackie Robinson’s life and his career as a major league player. Please collect as many aspects as possible that could be useful for your report (e.g., biography, achievement, images, teams, records, etc.)
8 minutes