Though the nomenclature differs between authors, three main criteria for assessing the quality of social research can be identified: reliability, replication and validity (Bryman 2001). Reliability is the extent to which a measurement procedure yields the same answer, however and wherever it is carried out. Validity is the extent to
which the research gives the correct answer, and replicability the extent to which the study can be repeated (Kirk and Miller, 1986).
However, there is debate about these criteria. Two camps exist16. One camp argues that qualitative methods can be objective and should adopt the criteria applied to quantitative methods (namely reliability, replication and validity as defined by quantitative texts). They outline strategies to ensure such criteria are met (LeCompte and Goetz, 1982; Kirk and Miller, 1986; Silverman, 2000; Bryman, 2001; Silverman, 2001).
For Kirk and Miller (1986) reliability can be achieved in qualitative methods by making detailed field notes that eliminate the idiosyncrasies of the researcher.
Validity can be achieved by adopting a variety of different data collection methods.
If each method points to similar results, it is likely that the results are valid. If one method involves face-to-face interaction over a period of time, validity claims are strengthened.
LeCompte and Goetz (1982) suggest that external reliability can be achieved by identifying the role and status of the researcher within the group studied (acknowledging the impact they have on the research findings); giving reasons why the informants were selected and how atypical informants were avoided; and acknowledging the social setting and its effect on research findings. External validity can also be achieved via multi-site research and theoretical sampling.
Internal reliability can be achieved by good field notes; using multiple researchers if possible; checking the results with participants; and the examination of the study by peers. Qualitative methods are considered to have high internal validity because the researcher is constantly checking with field (LeCompte and Goetz, 1982).
16 Hammersley (1992) identifies a third camp that rejects the need to judge the quality of social research. Proponents of this position argue that the existence of multiple realities denies that any one account is better than another. I, however, accept the need to prove the quality of my research in order to have the results viewed as credible (Hammersley, 1992).
Others argue that qualitative methods stem from a paradigm quite different to quantitative ones. Qualitative research, therefore, needs to be judged by standards that are appropriate to what it is attempting to achieve. Alternative criteria are presented in the literature, indeed Altheide and Johnson (1994) identify 14 definitions for a validity criteria.
The most referenced set of criteria are by Guba and Lincoln (1994). Lincoln and Guba pose trustworthiness to replace reliability and validity and transferability to replace replicability. Trustworthiness is defined as:
• Credibility (a replacement for validity): the extent to which the findings are believable and accepted by the people studied;
• Dependability (a replacement for reliability): the extent to which the findings are likely to apply at other times; and
• Confirmability (a replacement for the condition of objectivity implicit in quantitative methods): the extent to which the inferences are based on the data and are logical and of high utility.
I acknowledge the paradigmatic differences inherent in qualitative and quantitative research and the need to judge them by different standards. In conducting and writing up my research I aim to fulfil the criteria outlined by Guba and Lincoln (1994) in the following ways:
• Credibility. Two common techniques for meeting the credibility criterion are data triangulation and respondent validation (Guba and Lincoln, 1994). Both are employed in this research. Data triangulation has been mentioned previously.
Respondent validation is a process whereby the researcher provides the people who were interviewed in the research with an account of the findings. I drafted a preliminary research findings report on completion of the case studies and invited comments from key respondents. Several took the time to provide comment and confirmed that the findings were appropriate.
• Dependability. Guba and Lincoln propose that researchers seek an audit of their work to ensure they are making sensible coding decisions. Complete records should be kept of all phases of the research process and peers audit the work.
Given the large data sets involved in qualitative research, this is not a widely adopted practice. Instead, in the presentation of the research findings direct quotes are used as outlined by Miles and Huberman (1994), to allow the reader to confirm my categorisations and conclusions.
• Confirmability is the criterion that shows that the conclusions drawn are based on the data gathered. Important in this is recognition of the personal values the researcher brings to the research and the potential affect of these on the research.
As discussed previously, it is important to be self-reflexive during the research process (Bryman 2001). Through making detailed field notes describing not only the actions taken but also the responses to them and my critical thoughts about my impact on the research process, I attempted to be as reflexive as possible.
• Transferability is possible when there is a ‘thick description’ of both the sending and receiving contexts so a reasoned judgement about the degree of transferability is possible (Hammersely, 1992; Bryman, 2001). Undertaking in-depth case studies using multiple methods of data collection allows for thick description.
4.5 Summary
A multi-case study research strategy has been used. Four cases were undertaken of voluntary organisations that provide social services. The specific cases were selected using Yin’s theoretical replication. Three data collection methods were employed:
in-depth interviews, document analysis and direct observation. Qualitative data was sought. A number of steps were also taken to ensure the results are reliable, including data triangulation, seeking comment on the findings by respondents and taking detailed field notes.
94