Overview
CHAPTER 4
z Psychological research is based on a complex decision-making process which is irreducible to a simplistic formula or rules-of-thumb.
z A number of factors are especially influential on setting the limits to generalisation from any data. The sampling procedures used and the statistical significance of the findings are very important. At the same time, psychologists generalise because psy-chology has a tendency towards universalism which assumes that what is true of one group of people is true of all people.
z Psychological research (as opposed to psychological practice) is usually concerned with samples of people rather than specific individuals. This allows general trends to be considered at the expense of neglecting the idiosyncratic aspects of individuals.
z Much psychological research depends on samples being selected primarily because they are convenient for the researcher to obtain. The alternative would be random sampling from a clearly specified population which is much more expensive of time and other resources.
z Characteristically, a great deal of psychological research is concerned with study-ing principles of human behaviour that are assumed to apply generally. As the
Î
generalisations being tested are assumed to be true of people in general, the neces-sity to ensure that the sample is representative is minimised.
z Statistical analysis is often concerned with answering the simple question of how safe it is to generalise from a particular study or sample of data. The usual model of statistical testing in psychology is based on postulating what would happen if the null hypothesis were true and then comparing this with what was actually obtained in the research.
z The probability of accepting that the results are not due to chance sampling if the null hypothesis were true is usually set at the .05 or 5 per cent level. This means that the probability of the finding being due to chance when the null hypothesis is in fact true is 5 times out of 100, or less. Results that meet this criterion are called statistically significant, otherwise they are statistically non-significant.
z The bigger the sample the more likely it is that the results will be statistically significant – all other things being equal. Consequently, it is necessary to look at the size of the result as well as its statistical significance when evaluating its importance.
4.1 Introduction
This chapter discusses in some detail the process of generalisation of research findings.
Are we justified in making more general statements about the findings of our research beyond the research itself? This is a crucial step in any research. There are three import-ant themes that need consideration:
z The lack of limitations placed on generalisation by the universalism of psychological theory.
z The limitations placed on generalisation by the sampling methods used.
z The limitations placed on generalisation by the strictures of statistical significance testing.
We will deal with each of these in turn. They are equally important but there is more to be said about qualitative analysis and generalisation in this context and so this will receive a disproportional amount of space. Each of these has a different but important influence on the question of the extent to which a researcher is wise or correct to generalise beyond the immediate setting and findings of their research study. There may be a temptation to regard statistical considerations as technical matters in research, but this is not altogether the case. Many statistical considerations are better regarded as having a bearing on important conceptual matters. For example, one might be less likely to generalise in circumstances in which your measures of concepts or variables are relatively weak or ineffective. This will tend to yield poor or low correlations between such variables and others – hence the disinclination to generalise from this finding with confidence. However, statistics can help show you such things as what the correlation would be if the measures were good and reliable. This may revise your opinion of what can be said on the basis of your data. See Figure 4.1 for a summary of some of the issues to do with generalisation.
It is important to realise that issues such as the generalisability of data are really aspects of the process of decision-making that a researcher makes throughout their research. The
FIGURE 4.1 The main issues in the generalisation of psychological research findings
task of the researcher is to reach a balanced judgement at every stage of their research based on the information that they have in front of them and reasoned evaluations of the available choices of action available at that point in time. It is impossible to reduce this decision-making process to a few rules of thumb. It might be appealing to students to have such rules of thumb but it distorts the reality of research too much to try to reduce it to any simplistic formula. So even things such as significance testing which are often reduced to a formula in statistics textbooks turn out to be much more of a matter of judgement than that implies. Research is not simply a matter of deciding whether a hypothesis is supported by one’s data or not. There are important issues such as whether it is desirable to develop further questions for further research in the area, whether an important next step is to establish whether one’s findings apply in very different circumstances or with very different groups of participants or using very different methods, and the degree of confidence one should have in one’s findings. There are other questions, of course, such as the desirability of abandoning this particular line of research. Again this is not simply a matter of failing to find support for a hypothesis in a particular study but a decision-making process based not simply on basic statistical outcomes but on a finer judgement as to whether the hypothesis had been given a fair chance in the research study.
4.2 Universalism
One of the characteristics of psychology is its tendency towards universalism. This is the fundamental assumption that the principles of psychology will not vary. Psychological findings will apply anywhere and are the same for all people irrespective of their society
and their culture. So when psychologists propose a hypothesis there is an implicit assumption that it is true of all people – unless it is one of those rare cases where it is stated or implied that the principle applies only to restricted groups of people. In other words, psychologists in practice appear to be interested in making generalisations about behaviour that apply unrestrained by context and circumstances. Psychological principles are assumed to be laws of human behaviour anywhere. Increasingly psychologists ques-tion this idea of universalism and argue that a culturally specific approach to psychology is more realistic and productive (Owusu-Bempah and Howitt, 2000). Historically, many of the principles put forward by psychologists are assumed to apply not only to people but also to other animals. So it was only natural that studies of basic processes were car-ried out on animals and the findings applied to human beings. Examples of this include classical conditioning theory (Pavlov, 1927) and operant conditioning theory (Skinner, 1938).
While universalism is characteristic of a great deal of psychological thinking, it is rarely, if ever, stated as such in modern psychology. Nowadays psychologists are likely to be aware of the problem but, nevertheless, this awareness is not built into their practices for designing their research studies. Universalism operates covertly but reveals itself in a number of different ways – such as when university students are used unquestioningly as participants in a great deal of academic research as if what were true for university students will be true for every other grouping and sector in society. Seldom do psycho-logists build into their research a variety of groups of participants specifically to assess whether their findings apply throughout.
Universalism defines quantitative research in psychology much more than it does qualitative research, of course. Qualitative researchers invariably adopt a relativist perspective which rejects the idea of a single reality which can be discovered through research. Instead, qualitative researchers assume that there is a multiplicity of viewpoints on reality. This is clearly incompatible with universalism and is discussed in more detail in Part 4 of this book on qualitative research methods.
4.3 Sampling and generalisation
Many criticisms have been made of psychology for its restricted approach to sampling.
As already mentioned, psychological research has sometimes been described as the psychology of psychology students or sophomores (Rosenthal and Rosnow, 1969, p. 59) (a sophomore is a second-year student in the USA). This criticism only means something if the idea of universalism in psychological research is being questioned, otherwise it would not matter since the laws of human behaviour might just as well be determined from studies using psychology students as any other group of participants. Whatever the group used, it would reveal the same universal laws. The emphasis of psychology on the processes involved in human behaviour and interaction is a strength of the discipline and not something to which sampling has anything particular to contribute from one perspective. So although sampling methods in psychology may to some extent be found to be lacking, this is not the entire story by any means.
Not all research has or needs a sample of participants. The earliest psychological research tended to use the researcher themselves as the principal or only research participant.
Consequently, experimental psychologists would explore phenomena on themselves. This was extremely common in introspectionism (or structuralism) which was the dominant school of psychology at the start of modern psychology and was eventually replaced by behaviourism early in the twentieth century. Similarly, and famously, Ebbinghaus (1913) studied memory or forgetting. There are circumstances in which a single case may be an
appropriate unit for study. Some psychologists still advocate using single cases or rela-tively few cases in order to investigate changing a particular individual’s behaviour (Barlow and Hersen, 1984) and this is common in qualitative research too. A single-case experimental study in quantitative research involves applying the independent variable at different random points in time. If the independent variable is having an effect then the participant should respond differently at the points in time that the independent vari-able is applied than when it is not. Its obvious major advantage is that it can be helpful when the particular sort of case is very rare. For example, if a particular patient has a very unusual brain condition then such a procedure provides a way of studying the effect of that condition. Clinical researchers working with a particular patient are an obvious set of circumstances in which this style of research might be helpful.
The problems with the approach are largely to do with the high demands on the participant’s time. It also has the usual problems associated with participants being aware of the nature of the design – it is somewhat apparent and obvious what is happening – which may result in the person being studied cooperating with what they see as the purpose of the study. Although this sort of ‘single-case’ method has never been very common in mainstream psychology and appears to be becoming less so (Forsyth et al., 1999), its use questions the extent to which researchers always require substantial samples of cases in order for the research to be worthwhile or effective.