Within the broad classification of quasi-experimental designs provided by Campbell and Stanley (1963), they identified a number of specific single- and multigroup designs. Three are summarized here: time series designs, nonequivalent control group designs, and counterbalanced designs. These three distinct designs were selected because they demonstrate useful designs that may be particularly appropriate for application in ILS practice settings. You will want to consult Campbell and Stanley (1963) or other re- search design texts concerning additional quasi-experimental designs that may suit your purposes.
Quasi-experimental Studies 95
In borrowing from the graphical treatments of Campbell and Stanley (1963), the following descriptions include a formulaic representation of these distinct quasi- experimental designs—O represents a measurement or observation of the dependent variable(s) and X represents subjects’ exposure to a treatment, the independent variable (e.g., a particular instructional method or a particular information system). Each row of a diagram represents one group of subjects and the treatment (Xs) and observations (Os) applied to that group (in the order shown). For multigroup designs, each row of the diagram depicts application of the treatment and observations for a particular group.
Time Series Design
The time series design is based on intermittent measurements taken before and after
exposure to a treatment (Campbell & Stanley, 1963; Powell & Connaway, 2004). Simply put, this design tests for changes over time due to a treatment that occurs at a particular point in time. Time series designs are referred to as within-subjects designs because the comparison of interest is within each subject’s performance before and after the treatment.
Fife-Schaw (2000) characterizes this design as having a minimum of two data col- lection points in total, pre- and posttreatment, but encourages use of more than this minimum to allow for sufficient opportunities to assess the effects of the treatment. A time series design with six observation points is represented as follows:
O1 O2 O3 X O4 O5 O6
Data would be collected at three points prior to the treatment (X) and at three points after the treatment. One hypothetical case in which such a design might be used is to investigate the effectiveness of a new tool developed for catalogers. Productivity resulting from this new tool, or treatment, could be ascertained by observing productivity each month for several months before the tool is introduced. Following introduction of the tool, or treatment, productivity would continue to be measured monthly, for several months, to assess the impact of the tool. It is important to take measurements on several occasions before and after the tool is introduced to make sure that the evaluation takes into account normal variability in productivity.
Nonequivalent Control Group Design
The nonequivalent control group design is likely the most frequently applied type of quasi-experimental design (Fife-Schaw, 2000; Trochim, 2006). This design is also referred to as the nonequivalent groups design (Campbell & Stanley, 1963; Trochim, 2006); the pretest-posttest nonequivalent control group design (Powell & Connaway, 2004); and the controlled comparison study (Lorenzetti, 2007). The nonequivalent con- trol group design can be diagramed as follows:
Experimental group: O X O
Control group: O O
There are two groups of study participants/subjects; often, they are naturally occurring groups such as two classes. One group receives the treatment and the other (the control
96 APPLICATIONS OF SOCIAL RESEARCH METHODS
group) does not. This design can be expanded to more groups if there are multiple forms of the treatment.
Although individual subjects are not randomly assigned to one group or the other, the groups should be formed so that they are as equivalent to each other as possible under existing conditions. For example, if evaluating a new method of library instruction, you would want to select two groups of students with similar characteristics such as the same class, age, college, and so on (Lorenzetti, 2007).
Each group is administered a pretest, and following application of the treatment to the experimental group, a posttest is administered. The pretest helps you to understand the ways in which the experimental group differs from the control group, even before the treatment is implemented. If there is no difference between the two groups on the pretest, then the comparison of their posttest scores is relatively straightforward. In addition, the pretest serves as a comparison with the posttest in determining the effects of the treatment.
This research design is referred to as a between-subjects design. Each subject partic- ipates in only one group, so comparisons between groups are comparisons between two independent sets of subjects.
Counterbalanced Design
In the multigroup counterbalanced design, multiple treatments, or interventions, are applied to each of the subjects. Since the comparison of interest is within each subject’s performance in the multiple treatment conditions, this design is referred to as a within- subjects design. The counterbalanced design is particularly useful for situations when pretesting is unavailable or inappropriate. This design can be implemented in a variety of ways, depending on the number of treatments to be compared. The following example is depicted as a Latin square, to compare four treatments:
Group 1: X1 O X2 O X3 O X4 O
Group 2: X2 O X4 O X1 O X3 O
Group 3: X3 O X1 O X4 O X2 O
Group 4: X4 O X3 O X2 O X1 O
A typical study using this design might be trying to compare four different Web search engines and users’ effectiveness in conducting searches with them. Group 1 would conduct searches on search engine 1, then search engine 2, then search engine 3, then search engine 4. The observation of user performance with each might be the amount of time taken to search or the number of relevant items retrieved with each. The second group of subjects would interact with search engine 2 first, then 4, then 1, then 3. By counterbalancing the order in which the subjects are exposed to the search engines, you will eliminate concerns about the learning that might occur as users get more and more practice with searching during the course of the study.
Like the nonequivalent control group design, counterbalanced designs are not consid- ered true experiments because the researcher is not able to randomly assign individual subjects to the four groups. Instead, it is most common that intact groups are used. For example, four sections of an introductory English course might be used as the four experimental groups. While not a true experimental design, this is the most rigorous of the three designs discussed in this chapter and is more similar to true experimental designs than nonequivalent control group designs and time series designs.
Quasi-experimental Studies 97