One way of ensuring that the participants in the experi-mental and control group are similar on variables which might be expected to affect the outcome of the study is to use matching. Participants in an experiment will vary in many ways so there may be occasions when you want to ensure that there is a degree of consistency. For instance, some participants may be older than others unless we ensure that they are all the same age. However, it is difficult, if not impossible, to control for all possible individual differences. For example, some participants may have had less sleep than others the night before or gone without breakfast. Some might have more familiarity with the type of task to be carried out than others and so on.
We could try to hold all these factors constant by making sure, for example, that all participants were female, aged 18, weighed 12 stone (76 kilograms), had slept
7 hours the night before and so on. But this is far from easy. It is generally much more practicable to use random assignment. To simplify this illustration, we will think of all these variables as being dichotomous or only having two categories such as female/male, older/younger, heavier/lighter and so on. If you look at Table 9.1, you will see that we have arranged our participants in order and that they fall into sets of individuals who have the same pattern of characteristics on these three variables.
For example, the first three individuals are all female, older and heavier. This is a matched set of individuals. We could choose one of these three at random to be in the experimental condition and another at random to be in the control condition. The third individual would not be matched with anyone else so they cannot be used in our matched study in this case. We could then move on to the
next set of matched participants and select one of them at random for the experimental condition and a second for the control condition.
You might like to try this with the rest of the cases in Table 9.1 which consists of information about the gender, the age and the weight of 24 people who we are going to randomly assign to two groups.
Matching is a useful tool in some circumstances. There are a few things that have to be remembered if you use matching as part of your research design:
z The appropriate statistical tests are those for related data. So a test like the related t-test or the Wilcoxon matched pairs test would be appropriate.
z Variables which correlate with both the independent and dependent variables are needed for the matching variables. If a variable is unrelated to either one or both of the independent or dependent variables then there is no point in using it as a matching variable. It could make no difference to the outcome of the study.
z The most appropriate variable to match on is most probably the dependent variable measured at the start of the study. This is not unrelated to the idea of pre-testing though in pre-testing participants have already been allocated to the experimental and control conditions. But pre-testing, you’ve guessed it, also has its problems.
Table 9.1 Gender, age and weight details for 24 participants
Number Gender Age Weight
9.3 More advanced research designs
We have stressed that there is no such thing as a perfect research design that can be used irrespective of the research question and circumstances. If there were such a thing then not only would this book be rather short but research would probably rank in the top three most boring jobs in the world. Research is intellectually challenging because it is problematic. The best research that any of us can do is probably a balance between a wide range of different considerations. In this chapter we are essentially looking at the simplest laboratory experiment in which we have a single independent variable. But even this basic experimental design gathers levels of complexity as we try to plug the holes in the simple design. The simplest design, as we are beginning to see, has problems.
One of these problems is that if a single study is to be relied on, then the more that we can be certain that the experimental and control conditions are similar prior to the experimental manipulation the better. The answer is obvious: assess the two groups prior to the experimental manipulation to see whether they are similar on the dependent variable. This is a good move but, as we will see, it brings with it further problems to solve.
It should be stressed that none of what you are about to read reduces the importance of using random allocation procedures for participants in experimental studies.
■ Pre-test and post-test sensitisation effects
The pre-test is a way of checking whether random assignment has, in fact, equated the experimental and control groups prior to the experimental manipulation. It is crucial that the two groups are similar on the dependent variable prior to the experimental manipulation. Otherwise it is not possible to know whether the differences following the experimental manipulation are due to the experimental manipulation or to pre-existing differences between the groups on the dependent variable.
The number of mistakes is the dependent variable in our alcohol-effects example. If members of one group make more mistakes than do members of the other group before drinking alcohol, then they are likely to make more mistakes after drinking alcohol. For example, if the participants in the 8 ml alcohol condition have a tendency to make more errors regardless of whether or not they have had any alcohol, then they may make more mistakes after drinking 8 ml of alcohol than the participants who have drunk 16 ml.
This situation is illustrated in Figure 9.4. In this graph the vertical axis represents the number of mistakes made. On the horizontal axis are two marks which indicate participants’
performance before drinking alcohol and after drinking alcohol. The measurement of the participants’ performance before receiving the manipulation is usually called the pre-test
FIGURE 9.4 Performance differences before the manipulation
and the measurement after receiving the manipulation the post-test. The results of the post-test are usually placed after those of the pre-test in graphs and tables as time is usually depicted as travelling from left to right.
Without the pre-test measure, there is only the measure of performance after drinking alcohol. Just looking at these post-test measures, people who drank 8 ml of alcohol made more mistakes than those who drank 16 ml. In other words drinking more alcohol seems to have resulted in making fewer mistakes (and not more mistakes as we might have anticipated). This interpretation is incorrect since, by chance, random assignment to conditions resulted in the participants in the 8 ml condition being those who tend to make more mistakes. Without the pre-test we cannot know this, however.
It is clearer to see what is going on if we calculate the difference between the number of mistakes made at pre-test and at post-test (simply by subtracting one from the other). Now it can be seen that the increase in the number of mistakes was greater for the 16 ml condition (12 − 4 = 8) than for the 8 ml condition (14 − 10 = 4). In other words, the increase in the number of mistakes made was greater for those drinking more alcohol.
We can illustrate the situation summarised in Figure 9.4 with the fictitious raw data in Table 9.2 where there are three participants in each of the two conditions. Each participant is represented by the letter P with a subscript from 1 to 6 to indicate the six different participants. There are two scores for each participant – the first for the pre-test and the second for the post-test. These data could be analysed in a number of different ways. Among the better of these would be the mixed-design analysis of variance. This statistical test is described in some introductory statistics texts such as the companion book Introduction to Statistics in Psychology (Howitt and Cramer, 2011a).
However, this requires more than a basic level of statistical sophistication. Essentially, though, you would be looking for an interaction effect. A simpler way of analysing the same data would be to compare the differences between the pre-test and post-test measures for the two conditions. An unrelated t-test would be suitable for this.
Experimental designs which include a pre-test are referred to as a pre-test–post-test design while those without a pre-test are called a post-test-only design. There are two main advantages of having a pre-test:
z As we have already seen, it enables us to determine whether randomisation has worked.
Table 9.2 Fictitious data for a pre-test–post-test two-group design
Pre-test Post-test
Condition 1 P1 9 13
P2 10 15
P3 11 14
Sum 30 42
Mean 30/3 == 10 42/3 == 14
Condition 2 P4 3 12
P5 4 11
P6 5 13
Sum 12 36
Mean 12/3 == 4 36/3 == 12
z It allows us to determine whether or not there has been a change in performance between pre-test and post-test. If we just have the post-test scores, we cannot tell whether there has been a change in those scores and what that change is. For example, the post-test scores may show a decline from the pre-test. Without the pre-test, we may suggest incorrectly that the independent variable is increasing the scores on the dependent variable.
Look at the data shown in the graph in Figure 9.5. Concentrate on the post-test scores and ignore the pre-test. That is, pretend that we have a post-test-only design for the moment. Participants who had drunk 16 ml of alcohol made more errors than those who had drunk 8 ml. From these results we may conclude that drinking more alcohol increases the number of mistakes made. If the pre-test number of errors made were as shown in Figure 9.5, this interpretation would be incorrect. If we know the pre-test scores we can see that drinking 16 ml of alcohol decreased the number of errors made (10 − 14 = −4) while drinking 8 ml of alcohol had no effect on the number of errors (6 − 6 = 0). Having a pre-test enables us to determine whether or not randomisation has been successful and what, if any, was the change in the scores. (Indeed, we are not being precise if we talk of the conditions in a post-test-only study as increasing or decreasing scores on the dependent variable. All that we can legitimately say is that there is a difference between the conditions.)
Whatever their advantages, pre-tests have disadvantages. One common criticism of pre-test designs is that they may alert participants as to the purpose of the experiment and consequently influence their behaviour. That is, the pre-test affects or sensitises participants in terms of their behaviour on the post-test (Lana, 1969; Solomon, 1949;
Wilson and Putnam, 1982). Again we might extend our basic research design to take this into account. We need to add to our basic design groups which undergo the pre-test and other groups which do not. Solomon (1949) called this a four-group design since at a minimum there will be two groups (an experimental and control group) that include a pre-test and two further groups that do not have a pre-test as shown in Figure 9.6.
One way of analysing the results of this more sophisticated design is to tabulate the data as illustrated in Table 9.3. This contains fictitious post-test scores for three participants in each of the four conditions. The pre-test scores are not given in Table 9.3.
Each participant is represented by the letter P with a subscript consisting of a number ranging from 1 to 12 to denote there are 12 participants.
The analysis of these data involves combining the data over the two conditions. That is, we have a group of six cases which had a pre-test and another group of six cases which did not have the pre-test. The mean score of the group which had a pre-test is 8 whereas the mean score of the group which had no pre-test is 2. In other words we are ignoring the effect of the two conditions at this stage. We have a pre-test sensitisation effect
FIGURE 9.5 Change in performance between pre-test and post-test
if the means for these two (combined) conditions differ significantly. In our example, there may be a pre-test sensitisation effect since the mean score of the combined two conditions with the pre-test is 8 which is higher than the mean score of 2 for the two conditions without the pre-test combined. If this difference is statistically significant we have a pre-test sensitisation effect. (The difference in the two means could be tested using an unrelated t-test. Alternatively, one could use a two-way analysis of variance. In this case we would look for a pre-test–no pre-test main effect.)
Of course, it is possible that the pre-test sensitisation effect is different for the experimental and control conditions (conditions 1 and 2):
z For condition 1 we can see in Table 9.3 that the difference in the mean score for the group with the pre-test and the group without the pre-test is 5 − 1 = 4.
FIGURE 9.6 Solomon’s (1949) four-group design
Table 9.3 Fictitious post-test scores for a Solomon four-group design
Had pre-test Had no pre-test Row means
Condition 1 P1= 4 P7 = 0
P2= 5 P8 = 1
P3= 6 P9 = 2
Sum 15 3 18
Mean 15/3= 5 3/3 = 1 18/6= 3
Condition 2 P4= 10 P10= 2
P5= 11 P11= 3
P6= 12 P12= 4
Sum 33 9 42
Mean 33/3 = 11 9/3 = 3 42/6 = 7
Column sums 48 12
Column means 48/6 = 8 12/6 = 2