As indicated by the epigraph, many challenges are associated with conducting lon- gitudinal research. They include challenges associated with your sample and attrition from your sample, the difficulty of measuring the same variables on each occasion, and the effects of extraneous events (including your study procedures) on the phenomenon being observed. In each case, you will need to plan for the study that you want to conduct and, in addition, plan for the study you will need to conduct when your original plans don’t work. Try to anticipate the challenges you might face, and plan for how to address them if they do occur (Bauer, 2004).
Because longitudinal studies occur over extended periods of time, selecting or recruit- ing a sample that is willing to participate can be more challenging than for a study that will occur at only one point in time. Because those who will volunteer to participate in a lon- gitudinal study are likely to be different than those who won’t, even your original sample may be biased (Bauer, 2004). In addition, attrition (i.e., people dropping out of the study after it’s begun) could introduce even more bias because those who drop out are likely to be different from those who continue in the study (Bergman & Magnusson, 1990). There are a number of things you can do to minimize attrition (Bauer, 2004; Murphy, 1990). First, you will want to carefully track all your participants so that, when you need to follow up with them for the next data collection, you will be able to find them. At the first data collection session, be sure you record accurate contact information for each participant (Goldstein, 1979). Second, you should consider ways in which you can develop rapport with the study participants (e.g., having the same interviewer contact each participant on each occasion) to encourage participants to continue. A personal relationship with the researcher will lessen attrition. Third, be sure that the frequency of data collection or the amount of time required for each data collection session is not too much of a burden on the participants. Finally, consider offering an incentive (e.g., cash or a gift of some kind) to participants. If necessary, the value of the incentive can be increased for later data collection sessions. For instance, if you were conducting a study of college students’ use of the library over their undergraduate careers, you might provide them with gift certificates to the campus bookstore, with the amount of the gift certificate increasing each year. With appropriate planning and study procedures, a valid sample can be recruited and maintained over the course of the study. You might also consider using multiple panels simultaneously to be better able to evaluate the effects of attrition within each of them.
76 APPLICATIONS OF SOCIAL RESEARCH METHODS
There are also challenges associated with the measurements to be taken during a lon- gitudinal study, particularly with measures of psychological constructs (i.e., individual characteristics that cannot be directly observed such as attitudes or satisfaction). Since your goal is to observe changes over time, you need to be able to compare the measures over time. Thus, as Singer and Willett (1996) point out, they need to be equitable, if not exactly the same. Once you have valid measures to use for data collection, you will want to consider how to analyze those changes: as simple change scores (i.e., the difference between scores at one time and scores at another time) or some type of residual change score, taking into account the expected trajectory of change and looking for divergence from that trajectory (Bauer, 2004; Bergman & Magnusson, 1990). You may need to seek statistical consulting advice to resolve these issues.
Finally, several challenges are associated with the context of any given study. The most obvious is the possibility of history effects (Bauer, 2004; Stanley & Campbell, 1963). The danger here is that some extraneous event occurs during the course of the study that has an effect on the phenomenon of interest, influencing it in unusual ways. The longer the study, the more likely that something unexpected will occur. Even for relatively short studies, researchers should be continually monitoring the environment to be able to take into account such unexpected and unwanted influences. A second problem is called panel conditioning (Ruspini, 1999) and is equivalent to the testing effects described by Stanley and Campbell (1963). Because the same measurements will be taken on multiple occasions, the process of data collection may influence the behavior of the participants. For example, if you were studying the development of information literacy skills among middle school students, the skills test you administer to them every two months may have an effect on their skill development, in addition to the training they are receiving. Thus the data collection procedures themselves influence the process being studied. A third issue is that while some change is linear, for many phenomena of interest in information and library science (ILS), linear change cannot be assumed (Plewis, 1985). Instead, there are “fluctuations over time” (p. 2). None of these problems can be easily resolved, but careful consideration of them while planning your study procedures will allow you to minimize their negative effects on the validity of your results.
EXAMPLES
Given the narrow definition of longitudinal research that we are using here, there are not many examples of this research design in the ILS literature. Both the examples to be discussed here have relatively small sample sizes, making them more feasible for an individual researcher or a small research team to conduct. They vary in their length; the longer the study, the more planning and resources it requires. In spite of the challenges of conducting longitudinal research, these two examples illustrate that longitudinal research is possible and that it can yield rich rewards. The first is a study of Web users’ information behaviors and the contexts in which they occur, and the second is a study of scholars’ information use.
Example 1: The Effects of Context on Information Behaviors
Kelly’s (2004,12006a, 2006b) dissertation research investigated the role of context as doctoral students sought information on the Web over the course of one semester.
Longitudinal Studies 77
Seven doctoral students were provided with laptops instrumented with logging software. They were interviewed and completed a questionnaire at the beginning of a semester and indicated a set of tasks and topics of interest at the time. Each week during the semester, they returned to the researcher’s office to review some of the documents they had retrieved during the past week. They classified each document in relation to the task and topic for which it was retrieved and rated its usefulness. During these weekly sessions, they could also add tasks and topics to their active lists. Each week, they also rated the endurance of each task, the frequency with which they had worked on each task, the stage of their progress in completing the task, and the persistence of each topic and their familiarity with that topic. At the end of the 14-week semester, they were interviewed about the tasks and topics, the process of classifying the documents by task and topic, the history of their work on each task and topic, and their reactions to the study procedures.
Our discussion here will focus on the aspects of this study design that define it as a longitudinal study. It clearly meets the first criterion of collecting data on each variable over multiple time periods—14, in this case. It also meets the second criterion, that the study participants were the same people on each occasion. Finally, it also meets the third criterion because the analysis of the results includes a focus on changes in endurance, frequency, and stage ratings of the tasks, and changes in the persistence and familiarity ratings of each topic.
While Kelly (2004, 2006a, 2006b) began the study with a small convenience sample, she was able to retain all the members of that sample throughout the 14 weeks of the study. Several of her procedures promoted continuing participation. By meeting personally with the participants each week, she established a relationship with them, increasing their motivation to continue in the study. The weekly meetings were scheduled often enough to develop rapport and to become a routine part of each participant’s schedule, but were not so frequent that they would be burdensome. She restricted the number of documents to be reviewed each week to control the level of the burden on the participants. She provided the laptop and printer to the participants at the end of the study—a significant incentive for them to participate in all 14 data collection sessions. This combination of procedures was successful in avoiding attrition in the study sample.
The study procedures also demonstrate the benefits of careful advance planning for the study. In advance of the beginning of the study, the data collection instruments were developed and pilot tested. While the topic familiarity ratings were still not as reliable as desired, the pilot testing did allow Kelly (2004, 2006a, 2006b) to minimize any problems with the data collection procedures. In particular, they allowed her to use the same data collection procedures at all 14 occasions, making the longitudinal data comparable across occasions. The use of Web logs was also supported by careful planning. The PCs were instrumented to collect data, plus a copy of each Web site visited was captured by a proxy server, through which all the participants’ Web sessions were routed. Finally, at the end of the study, Kelly implemented clear procedures for “cleaning” the laptops and legally transferring them to the participants. This combination of procedures contributed to the quality of the data collected during the study.
The data analysis for this study focused on two types of questions typical of lon- gitudinal studies: the history of change in the outcome variables over the course of the study, and the relationships among variables over the course of the study. The first type of analysis is illustrated in the longitudinal analysis of participants’ perceptions of their stage of task completion. While there were a small number of participants, there
78 APPLICATIONS OF SOCIAL RESEARCH METHODS
Week
Note: Week 6 was semester break. Figure 9.1-a. Participant 2
Week
Note: Week 6 was semester break. Figure 9.1-b. Participant 5
Task Number Task Number
Figure 9.1. Ratings of task stage by participants 2 and 5 (from 1, started, to 7, finished; dash
indicates not applicable). Adapted from Kelly (2006b, Fig. 1). Reprinted with permission of John Wiley & Sons, Inc.
were a large number of cases (i.e., 127 tasks) included in this analysis. The results for participants 2 and 5 are shown in Figure 9.1. The goal of this analysis was to try to detect patterns that occurred over time. From the results for all seven participants, Kelly was able to conclude that very few tasks were completed during the semester: participant 2 (Figure 9.1a) completed only 2 of 11, and participant 5 (Figure 9.1b) completed only 4 of 12. In addition, while it would be expected that each task would progress toward completion as the semester proceeded (as illustrated by participant 2), this was not always the case. Par- ticipant 5 rated half of the tasks as progressing forward but then moving further away from completion later in the semester. During the exit interviews, this phenomenon was ex- plored and explained by the participants as times when “you get thrown back . . . because you hit a little snafu” (Kelly, 2006b, p. 1866). While Kelly needed to develop these anal- ysis procedures specifically for this study, they were quite effective in helping her to understand changes in her participants’ stages of task completion over time.
The second type of question addressed with these data focuses on relationships among the variables. Specifically, Kelly investigated the relationships between the context vari- ables (i.e., attributes of the tasks and topics) and the participants’ ratings of Web site usefulness. She used two different statistical analysis techniques to examine these rela- tionships. First, she checked the correlations among the context variables and usefulness ratings (see Kelly, 2006b, table 6). While there were a number of relationships between context variables that seemed to warrant further study (e.g., for six of the participants, the endurance of a task was related to its stage of completion), this analysis provided relatively little evidence that the context variables were related to ratings of Web site use- fulness. A second analysis examined the question of whether usefulness ratings varied based on the particular task or topic. For this analysis, chi-square was used (even with some concerns that the sample size was insufficient to draw valid conclusions). Never- theless, the analyses did indicate that usefulness ratings did vary by task and topic for all seven participants. As in many longitudinal studies, this type of question is somewhat less concerned with the longitudinal nature of the data and might also be investigated with a cross-sectional study design.
In summary, Kelly’s use of a longitudinal study design demonstrated many of the ways in which a researcher can address the challenges associated with this approach. Her careful planning of the study procedures was exemplary, in particular in relation to keeping the data collection procedures comparable across occasions and minimizing
Longitudinal Studies 79
sample attrition. It serves as a good example of how to apply this research design to questions related to change over time.
Example 2: A Longitudinal Study of Scholars’ Use of Information
Over the course of several years, Wang and her colleagues2studied scholars’ selection,
reading, and citing of documents originally identified in an online search. Twenty-five agricultural economists (11 faculty and 14 graduate students) were recruited to partici- pate in the study, which began in 1992. At that time, an online search was conducted to support a project of each participant; the participants then reviewed the retrieved items and identified those that would be relevant to their projects. Three years later, 15 of the same participants were interviewed concerning which of the documents they had read and which they had cited. There are a number of aspects of this study design that are of interest to others planning a longitudinal study, including the timing of the data collection waves, the ways in which retrospective data collection was combined with concurrent data collection (including the use of external stimuli to improve recall), and finally, the timing of the publication of results from the study.
The information behavior of interest for this study was the selection, reading, and citing behavior of the participants as they worked on intellectual projects (e.g., articles, book chapters, and dissertations). Each of these three types of behaviors happens inter- mittently: intensely during some periods, and rarely during other periods. In addition, each of these types of information behavior can continue over an extended period of time. Therefore the study began in 1992 (Wang & Soergel, 1993, 1998), when each participant was beginning a particular project. At that time, the participants provided think-aloud protocols while they identified relevant papers on which they expected to follow up during the project. Wang and White (1995, 1996, 1999; White & Wang, 1997a, 1997b) contacted each participant a second time, in 1995, three years later. At that point, 10 of the original 25 participants had dropped their projects or modified them so sig- nificantly that they could not provide data appropriate to the study. Of the remaining 15, 3 were not able to provide data about citing decisions: “two participants still had research in progress and the third was a joint author whose colleague had made the citing decisions” (White & Wang, 1997b, p. 124). Thus the three-year interval between the two waves of data collection was appropriate for most of the participants, but still too short for some.
Another point of interest is that this study was concerned with three information behaviors (selection, reading, and citing of documents) that occur at different times, but data were collected on only two occasions. Data about selection of relevant documents were collected through concurrent think-aloud protocols while the participants reviewed their search results. Data about reading and citing of documents were collected retrospectively, that is, after the behaviors had occurred. In some cases, this would have been years after the behavior had occurred. For example, it is easy to imagine a participant who identified a few relevant items from the search in 1992 and, within months, had read several of those items. It was not until 1995 that the participant was asked about that reading behavior and, at the same time, was asked about whether the item had been cited in project publications. It’s very possible that the results of this study were not as complete or unbiased as would be desirable because participants had forgotten how they reacted to the document when they read it or decided to cite it. Wang and her colleagues tried to overcome this problem, primarily, by encouraging the
80 APPLICATIONS OF SOCIAL RESEARCH METHODS
participants to use external stimuli to aid their memories. For example, one participant referred to referee comments on an article to see if the referee had recommended a particular citation. Others referred to their notes, drafts of publications, and other documentary evidence. An ideal study design would have the researcher present with each participant as each reading or citing decision was made. However, in the real (rather than ideal) world, the use of retrospective recall of reading and citing decisions, aided by external stimuli, was an appropriate approach to increasing the validity of the study data.
Finally, it is worthwhile to take note of the publications that resulted from this study because the timing of longitudinal study results can be an issue. The results of the 1992 interviews were published in Wang’s (1994) dissertation. In addition, preliminary results were reported in an American Society for Information Science and Technology paper (Wang & Soergel, 1993), and the full results were reported in a Journal of the
American Society for Information Science and Technology (JASIST) article (Wang &
Soergel, 1998). There is no evidence that when the original study was undertaken, it was planned as the first phase of a longitudinal study; thus the second wave of data collection, in 1995, appears to have been opportunistic. It resulted in several more publications: a report of preliminary results in two ASIST papers (Wang & White, 1995, 1996), a
JASIST article reporting the results (Wang & White, 1999), a technical report (White &
Wang, 1997a), and a Library Quarterly article examining scholars’ citing behaviors in more detail (White & Wang, 1997b). This is a typical pattern for publishing the results of longitudinal studies that occur over a period of years. Usually, the first wave of data collection is of interest, even without the additional comparisons that will be possible after the follow-up data are collected. Most often, the second and later waves of results are published in combination with the earlier waves of data, comparing the results over time.
Wang’s work, in collaboration with Soergel and White, resulted in a fairly unique view of scholars’ selection, reading, and citing behaviors as they occurred over time. The interviews regarding selection decisions allowed the later data on reading and citing decisions to be explicitly anchored in a set of documents that could be used to stimulate the study participants’ memories. While all the original study participants were available and willing to participate in the second wave of data collection, the