Experimental designs differ from other research designs in the degree of control exerted by the researcher over the conditions under which data are collected. When observational or survey designs are employed, the re- searcher measures the variables of interest but does not attempt to manipu- late or control the variables. However, in experimental (causal) designs, re- searchers are exercising control over the variables of interest.
For example, a retailer trying to isolate the influence of price changes on the sales volume of a product, must identify the nonprice variables such as location of the product in the store, time period of the day/week, promotion,
etc., and control these variables because they also influence sales volume. Simply altering price and measuring sales is not an adequate design because of the impact of nonprice variables on sales.
Essentially, the key differences in experimentation and other designs are: 1. In an experiment, one or more of the independent variables are delib-
erately manipulated while others are controlled.
2. Combinations of conditions (particular values of the independent variables, e.g., different prices) are assigned to sample elements (e.g., different stores) on a random basis. This reduces the likelihood of preexisting conditions affecting the results.
The Terminology of Experimentation
There are several terms that are used to describe the concepts involved in an experiment. Knowledge of these terms is essential to understanding ex- perimental designs.
Experimental Treatments
The term “treatment” is used to describe a specific manner of manipulat- ing an independent variable. For example, if an experiment was designed to measure the influence of three different levels of advertising on sales, then each level of advertising would be a different treatment. If the effects of the three levels of advertising were combined with two levels of price, then there would be six experimental treatments as shown in the following:
Price level one with
Advertising level one Advertising level two Advertising level three
Price level two with
Advertising level one Advertising level two Advertising level three
Since price level one with advertising level one represents a specified combination of independent variables, it is referred to as an experimental treatment. Each combination of price and advertising levels creates another experimental treatment. Therefore, this example consists of six different treatments.
Experimental Units
Experimental units are the geographic areas, stores, or people whose re- sponses are measured in determining the effect of the different treatments. If the price and advertising levels described earlier are used in different geo- graphical areas, then the areas represent experimental units. Each group of experimental units is a sample of all possible units and is used for compar- ing the impact of the treatments. Continuing the example, each of the six treatments may be assigned to 40 different geographic locations for a total of 240 units. If we took a measure of sales before and after each treatment in each location we would have a total of 480 (i.e., 240 × 2) “observations.”
Experimental Designs
The experimental design is the specific process used to arrange inde- pendent variables into treatments and then assign treatments to units. There are many possible designs and great care should be exercised in choosing and/or modifying the design. Choosing the best design is the most important aspect of experimentation. Several designs will be discussed in a later sec- tion, but it should be noted that researchers unfamiliar with experimentation should seek advice in setting up an experiment.
Control Group
In many experiments the use of a control group is a necessary element of the design. A control group is randomly selected just like other groups of units in the experiment. However, the control group does not receive a treat- ment and thus serves as a benchmark for comparing the effects of treat- ments. If an experiment was used to determine the impact of two new levels of price on sales, prices held at the previous level in some areas would en- able these areas to serve as controls in analyzing the results.
Validity and Experimentation
While there are a number of different types of validity, the two major va- rieties are considered here: internal validity and external validity. Internal validity has to do with whether the independent variables that were manipu- lated caused the changes in the dependent variable or whether other factors involved also influenced the dependent variable. There are many threats to internal validity. Some of the major ones are:
1. History. During the time that an experiment is taking place, some events may occur that affect the relationship being studied. For exam-
ple, economic conditions may change, new products may be intro- duced, or other factors may change that alter the results.
2. Maturation. Changes may also take place within the subject that are a function of the passage of time and are not specific to any particular event. These are of special concern when the study covers a long pe- riod of time, but may also be a factor in tests that are as short as an hour or two. For example, a subject can become hungry, bored, or tired in a short time and this can affect response results.
3. Testing. When pretreatment and posttreatment measures are used, the process of taking a test can affect the scores of a second measurement. The experience of participation in the first measurement can have a learning effect that influences the results on the second measurement. 4. Instrumentation. This threat to internal validity comes from changes
in measuring instruments or observers. Using different questions or different observers or interviewers is a validity threat.
5. Selection. One of the more important threats to internal validity is the differential selection of subjects to be included in experimental and control groups. The concern is over initial differences that exist be- tween subjects. If subjects are randomly assigned to experimental and control groups, this problem can be overcome.
6. Statistical regression. This potential loss of validity is of special con- cern when subjects have been selected on the basis of their extreme scores. For example, if the most productive and least productive sales- people are included in an experiment, there is a tendency for the aver- age of the high scores to decline and the low scores to increase. 7. Mortality. Mortality occurs when the composition of the study groups
changes during the experiment. Subjects dropping out of an experi- ment, for example, causes the makeup of the group to change. While internal validity concerns whether experimental or nonexperi- mental factors caused the observed differences, external validity is con- cerned with whether the results are generalizable to other subjects, stores, or areas. The major threats to external validity are:
1. Subject selection. The process by which test subjects are selected for an experiment may be a threat to external validity. The population from which subjects are selected may not be the same as the target market. For example, if college students were used as subjects, gener- alizing to other types of consumers may not be appropriate.
2. Other factors. The experimental settings themselves may have an ef- fect on a subject’s response. Artificial settings, for example, can give results that are not representative of actual market situations. If the
subjects know they are participating in a price experiment, they may be more sensitive to price than normal.
Field versus Laboratory Experiments
There are two types of settings in which experiments are conducted— field and laboratory. Field experiments result in an experimental setting which is more realistic in terms of modeling actual conditions and is there- fore higher on external validity. Field experiments are carried out in a natu- ral setting with a minimum of artificial elements in the experiment. The downside of field experiments is that they usually cost more, have lower in- ternal validity due to lack of control over variables that influence the de- pendent variable, and also may alert competition to changes a company is contemplating in marketing their products.
Laboratory experiments are experiments conducted under artificial con- ditions, such as testing television ads in a movie theater rather than in buyers’ homes. Such experiments usually are lower in costs, have higher internal validity due to more control of the experimental environment, and provide greater secrecy of potential marketing actions. It is also possible to use more elaborate measurement techniques in a laboratory setting than in field ex- periments.
Disadvantages of laboratory experiments include loss of realism and lower external validity. These are due to the artificial conditions used in the experiment. For example, an experiment where people are given play money and asked to shop in a laboratory setting may reveal a great deal about some aspects of consumer behavior but there would be great diffi- culty in trying to generalize the findings to the purchase environment en- countered in the marketplace.
The type of information to be generated from the experiment and its in- tended use dictate which type of experimental setting is more appropriate in a given situation. It is even possible to use both—a laboratory experiment followed by a field experiment—to validate the findings of the laboratory study under actual market conditions.
Experimental Design Symbols
Certain symbols have been developed to help describe experimental de- signs. These symbols have been used because they help in understanding the designs. These symbols include:
• X = exposure of a group of subjects to an experimental treatment or a level of an independent variable (e.g., to a particular ad or product price, etc.). If different levels are used, then such as X1, X2, X3, . . ., Xnare used.
• O = observation or measurement of the dependent variable in which the researcher is interested.
• R = random assignment of people to groups or groups to treatments. Other notions are also helpful. The left-to-right notation indicates the time sequence of occurrence of events. Thus the notation
RO1XO2
would mean that subjects were randomly assigned to this group (R); a be- fore measure (O1) was taken; the subjects were then exposed to the treat- ment (X); and then an after measure (O2) was taken.
All the notations on a single line refer to a single group of respondents, and notations that appear together vertically identify events that occurred at the same point in time. Thus,
O1X1O2 X1O3
refers to two groups of subjects, one of which received a pretest measure and a posttest measure (O1 and O2), whereas the other received only a posttest measure (O3). Both groups were exposed to the same treatment (X1) and that the treatment and posttest measures occurred at the same time for both groups.
Ethics and Experimentation
Over the past few years, there has been increasing concern for protecting the rights of subjects used in research projects. This is a potential problem in all studies involving human subjects. The researcher should give careful consideration to the potential negative effects on those participating in an experiment to avoid violating the subjects’ rights and deflect potential law- suits. Luckily, most marketing research experiments are not likely to in- volve negative effects, but the possibility of such effects should be carefully evaluated.5
Experimental Research Designs
There are many possible experimental designs to choose from and they vary widely in terms of both complexity and effectiveness. The most widely accepted classifications of designs are: (1) preexperiments, (2) true experi- ments, and (3) quasi experiments. While complete coverage of experimen- tal designs is beyond the scope of this book, a brief explanation and exam- ples of these follows:
Preexperimental Designs
Preexperimental designs are designs that are weak in terms of their abil- ity to control the various threats to internal validity. This is especially true with the one-shot case study.
One-Shot Case Study. This design may be noted as:
X1O1
An example of such a study would be to conduct a sales training program without a measure of the salespeople’s knowledge before participation in the training program. Results would reveal only how much they know after the program but not how effective the program was in increasing knowl- edge.
One-Group Pretest-Posttest. This design can be represented as:
O1X1O2
It is an improvement on the one-shot case study because of the addition of the pretest measurement, but it is still a weak design in that it fails to con- trol for history, maturation, and other internal validity problems.
Static-Group Comparison. This design provides for two study groups,
one of which receives the experimental treatment while the other serves as a control. The design is:
X1O1 O2
The addition of a control group makes this design better than the previ- ous two designs. However, there is no way to ensure that the two groups were not different before the introduction of the treatment.
True Experimental Designs
The major deficiency of the previous designs is that they fail to provide groups that are comparable. The way to achieve comparability is through the random assignment of subjects to groups and treatments to groups. This deficit is overcome in the following true experimental designs.
Pretest-Posttest Control Group. This design is represented as:
RO1X1O2 RO3 O4
In this design, most internal validity problems are minimized. However, there are still some difficulties. For example, history may occur in one group and not the other. Also, if communication exists between the groups, the effect of the treatment can be altered.
Posttest-Only Control Group. Pretest measurements are not used in this
design. Pretest measurements are well-established in experimental research design but are not really necessary when random assignment to groups is possible. The design is represented as:
RX1O1 R O2
The simplicity of this design makes it more attractive than the pretest- posttest control group design. Internal validity threats from history, matura- tion, selection, and statistical regression are adequately controlled by ran- dom assignment. This design also eliminates some external validity prob- lems as well. How does this design accomplish such notable achievements? We can think of the O1observation as a combination of the treatment effect (e.g., a change in price, or the advertising campaign we ran, or a change in package design, etc.), plus the effect of all the extraneous variables (i.e., his- tory, maturation, testing effect, instrumentation, selection, statistical regres- sion, and mortality). However, since O2will theoretically be subject to all the same extraneous variables, the only difference between what we ob- serve between O1and O2(e.g., a measure of sales at two stores) will be the fact that O1(e.g., sales at store #1) was measured after the treatment X1 (e.g., where prices were lowered), and O2(e.g., sales at store #2) did not re- ceive the treatment. Therefore, all the other extraneous variable effects “wash” or cancel out between the two observations, so any difference in the observations must be due to the treatment. These results are dependent, however, on the random assignment of multiple units (stores in this exam- ple) to either the test or control conditions. Sample sizes for each of the test and control conditions must be large enough to approach a “normal distribu- tion” of stores so that we can assume that the only difference that we see be- tween O1and O2is the result of the treatment, and not due to some extrane- ous variable differences between the test and control stores.
Quasi Experiments
Under actual field conditions, one often cannot control enough of the variables to use a true experiment design. Under such conditions, quasi ex- periments can be used. In a quasi experiment, comparable experimental and control groups cannot be established through random assignment. Often the researcher cannot even determine when or to whom to expose the experi-
mental variable. Usually, however, it is possible to determine when and whom to measure. The loss of control over treatment manipulation (the “when” of the experimental variable exposure) and the test unit assignment (the “who” of the experimental variable exposure) greatly increases the chance of obtaining confounded results. Therefore, we have to build into our design the ability to account for the possible effects of variables outside our control in the field, so that we can more safely conclude whether the treatment was, in fact, the thing that caused the results we observed.
A quasi experiment is not as effective as a true experiment design, but is usually superior to available nonexperimental approaches. Only two quasi- experimental designs will be discussed here.
Nonequivalent Control Group. This is one of the most widely used quasi-
experimental designs. It differs from true experimental design because the groups are not randomly assigned. This design can be represented as follows:
O1X1O2 O3 O4
Pretest results are one indicator of the degree of comparability between test and control groups. If the pretest results are significantly different, there is reason to doubt the groups’ comparability. Obviously, the more alike the O1and O3measures are, the more useful the control group is in indicating the difference the treatment has made (i.e., comparing the difference in the O2to O1with the difference between O4to O3). Close similarity allows for control over the extraneous variables of history, maturation, testing, instru- mentation, selection, and mortality. However, statistical regression can be a problem if either the test or control group has been selected on the basis of extreme scores. In such a case, the O2or O4measures could be more the re- sult of simply regressing back from the extreme to the average score, rather than the result of anything that intervened between the first and second ob- servation of either group.
Separate Sample Pretest-Posttest. This design is most applicable in
those situations where we cannot control when and to whom to introduce the treatment but can control when and whom to measure. The design is:
R O1 R X1O2
This design is more appropriate when the population is large and there is no way to restrict who receives the treatment. For example, suppose a com- pany launches an internal marketing campaign to change its employees’ at- titudes toward customers. Two random samples of employees may be se- lected, one of which is interviewed regarding attitudes prior to the campaign. After the campaign the other group is interviewed.
Limitations of Causal Research
It is natural for managers to be attracted to the results that can be gained from doing experimentation. After all, are we not often looking to know what really causes the effects we see such as satisfied customers, rising (or falling) sales, motivated salespersons, etc.? However attractive causal re- search may be, managers should recognize the following limitations:6
1. Field experiments can involve many variables outside the control of the experimenters, resulting in unanticipated differences in conditions surrounding treatment groups.
2. It may be difficult or expensive to gain the cooperation of retailers and wholesalers when setting up the experiment.
3. Marketing personnel may lack knowledge of experimental proce- dures, reducing the chance of results which demonstrate causality. 4. Experiments are notoriously expensive and time consuming.
5. The experimenter must be careful not to introduce bias into the experi- ment by saying or doing something that may consciously or uncon- sciously affect the behavior of the test participants.