So far in this chapter we have looked at the logic behind tests of statistical significance. As explained earlier, these are used to test the null hypotheses that there is no association between populations or that there is no difference in the central tendency (or dispersion) of populations. What these tests do not tell us is the size or extent of any such associations or differences. So if we are interested in whether training programme A is more effective than training programme B, a statistical test of significance can be helpful, but if we want to know how much more effective one training programme is than the other, the test will not provide any help. If we want to know the size of differences in central tendency, or the strength of the associations, we need information about effect sizes. An effect size is simply a measure of how big an association or a difference is.
You can refer to effect sizes in relation to:
■ samples (for example, how strong is the association between age and salary in a sample of 250 Polish employees?);
■ populations (for example, how strong is the association between age and salary in Polish employees?).
Table 2.11 Interpreting p values for the differences between means
p value Interpretation Statistically
significant?
0.023 The probability of obtaining a difference between the Yes sample means as big as this if the null hypothesis is true,
is 23 in 1,000
0.049 The probability of obtaining a difference between the Yes sample means as big as this if the null hypothesis is true,
is 49 in 1,000
0.1 The probability of obtaining a difference between the No sample means as big as this if the null hypothesis is true, is 1 in 10
0.0001 The probability of obtaining a difference between the Yes sample means as big as this if the null hypothesis is true,
is 1 in 10,000
0.05 The probability of obtaining a difference between the No sample means as big as this if the null hypothesis is true, is 5 in 100 (one time in 20)
Effect sizes are very important. For example, when you carry out research into the effec-tiveness of a training programme, you are not necessarily interested in whether one training programme is marginally better than another, but whether one training programme is substantially better than another. And when you are interested in whether there is an asso-ciation between the extroversion of sales people and their sales figures, you are not necessarily interested in whether there is a very small association between them, but a substantial one.
But don’t you already have an answer to the question of whether an association or differ-ence is important and significant – the idea of statistical significance? Is it not the case that a statistically significant difference is also a practically important difference? The answer is, no. Statistical significance does not mean that you have a substantial association, or a substan-tial difference – an association or difference which is big, practically important, and always worth taking notice of and doing something about. To understand why this is so, recall what you mean when you say that something is statistically significant.
Statistical significance at the .05 probability level means that an association (or differ-ence) you have found in your sample data would occur less than 1 time in 20 by chance, if the null hypothesis is true.
Can you find that your association or difference is statistically significant if there is only a small association or a small difference? That is, can you obtain statistical significance if the effect size you have found is small and, from a practical point of view, insubstantial and uninteresting? The answer is, yes. This is because the chance of obtaining a statistically signifi-cant result depends not just on the population effect size, but also on the sample size you have used in your research (and, as mentioned earlier, on the significance level adopted). With two very small samples of the salaries of men and women, you are unlikely to find that the difference between the means of the two samples is statistically significant even if there is actually a very big difference in the mean salaries of men and women (i.e. a big population effect size). And conversely, if you take two very large samples of men and women, you are likely to find that even if the difference in the salaries of men and women is very small indeed, you will still obtain a statistically significant result.
Some examples
■ A correlation is a measure of the strength of association between two variables. The strength of correlation varies from 0 (no association at all between the variables) to 1 (for a perfect positive association) or ⫺1 (for a perfect negative correlation).
So a correlation of 0.1 is very weak, and one of 0.9 is very strong. If the correlation between gender and salary is actually 0.1, this would mean that there is a very weak association between how much people earn and whether they are male and female.
However, if you sampled 1,000 men and women you would have a very good chance of finding that the association between salary and gender is statistically significant even though, at 0.1, it is very weak.
■ Someone might try to sell your organization a psychometric test to use in selection.
They might have data to show that the correlation between the score that people 1111
THE PRINCIPLES OF INFERENTIAL STATISTICS
obtain on the test and their job performance is statistically significant. This would imply that the test is worth buying since it will helpfully predict job applicants who will perform well or perform badly at work. However, if the sample size used to examine the association between the test scores and job performance was very large (say 1,000 or more) this would be likely to be statistically significant even if the actual correlation between the test scores and job performance is very small (say about 0.1).
Small effect sizes can also be important
Sometimes a small effect size is very important. Imagine that a researcher found a way of improving the performance of staff by just 1 per cent with no disadvantage to the staff and at no cost to the organization. The effect size here is small, but because a 1 per cent increase in performance would be very useful to the organization, and very attractive if it entailed no concomitant costs, the research finding would be important. What this shows is that it is not simply whether a finding is statistically significant that matters, nor even whether the effect size is large or small. What matters is statistical significance, and the size of the effect, in the general context of the research.
You should consider statistical significance in the context of sample size, and you should consider effect size in the context of the theoretical implications and practical applica-tions of the research findings.
What all this means is that:
1 you should treat the notion of statistical significance with caution;
2 when interpreting the results of statistical analysis, you should be aware of the sample sizes used, and the effects these are likely to have on the chance of obtaining
statistically significant results;
3 the larger the sample size, the more likely that you will obtain a statistically significant association or difference, if one exists;
4 when examining research findings you should consider effect sizes as well as statistical significance;
5 you should consider effect sizes in the context of the possible implications and practical applications of research findings – although large effects are generally more desirable than small ones, in some contexts a small effect can be important.
Reporting the size of an effect
Finally, if you are to report the size of an effect, how can this be done most effectively? A variety of effect size measures have been suggested, covering all the major forms of asso-ciation and difference. However, only some are used with any regularity, and it is these that we will focus on here.
The effect size for the association between two continuous variables For the association between two continuous variables, the correlation coefficient is an excel-lent measure of effect size. The correlation coefficient will be dealt with in some detail in Chapter 6 but, as we have seen, the correlation coefficient may range between 0 if there is no association between two variables, to 1 if there is a perfect positive relation between them, and ⫺1 if there is a perfect negative relation between them. In organizational research, and social science research generally, a small effect is viewed as a correlation of about 0.1 (or⫺0.1 for a negative correlation), a medium effect is a correlation of about 0.3 (⫺0.3), and a large effect would be a correlation of 0.5 (⫺0.5) or more (Cohen and Cohen 1975).
Effect size for the difference between two means
For the difference between two means, the effect size used is referred to as ‘d’. It is obtained by taking the difference between two means and dividing by the standard deviation of the scores on which they are based. So, if the mean performance for women is 22.7, the mean performance for men is 13.4, and the standard deviation of the performance of all of the men and women is 14.5, the effect size of this difference between the means is:
(22.7⫺13.4)/14.5⫽0.64
For the social sciences and organizational research, a small effect is viewed as a d of about 0.2, a medium effect as about 0.5, and a large effect as 0.8 or more (Cohen 1988). So, in this case, the difference between the performance of men and women has a medium to large effect size.