This section addresses two important characteristics of measurement, namely reliability and validity. Reliability refers to the accuracy and precision of the measurement process (Cooper and Emory 1995) and tests of data reliability are concerned with ensuring the replicability of results. Validity on the other hand, relates to whether a test actually measures what it purports to measure. If it does, inferences drawn from the results are said to be valid. Reliability and validity are both key
methodological concerns for research such as surveys and content analysis where subjectivity may be an issue (Weber 1990). The following sections provide details of the efforts taken in both the survey and the content analysis to ensure that results are reliable and valid.
4.5.1 Reliability
Survey: The survey was administered through the independent survey facility,
SurveyMonkey76. Respondents entered their answers directly into the online questionnaire and these were then downloaded twice from the website, once as a pdf file to ensure a permanent record of the detail of all responses and the second time as an Excel spreadsheet. The use of the survey facility removed the need for manual data entry of results and therefore removed the potential for data entry errors. The online analysis tool allowed both detailed responses and an overall summary to be downloaded. Summarised data for subsets of responses were generated by applying various filters (such as stakeholder type, industry, etc.). All summaries were downloaded in pdf format. The analysis was then repeated using the data in the Excel file to verify the accuracy of the calculations.
Content analysis: Rating the quality of disclosure rather than merely its presence or
absence arguably introduces additional subjectivity to the scoring process (Judd et al. 1991). However, although the use of judgement cannot be completely avoided in the coding process, subjectivity and bias was minimised in this study through a combination of detailed descriptions of categories, coding rules, uniform scoring guidelines and the use of rating scales that involved a series of simple dichotomous rankings, counting the presence (1) or absence (0) of five attributes of each element in the disclosure index. Such ‘counting’-oriented scales require less degree of judgement and are more reliable than the use of classification schemes in which information is categorised (Milne and Adler 1999). Despite this, the potential always remains for some subjectivity in the analysts’ interpretation of text (Deegan and Rankin 1996; Milne and Adler 1999). Steps to ensure and evaluate the reliability of the results were conducted in addition to the training and pre-testing outlined in Section 4.4.2.
The reliability of the results was examined by testing for both inter-rater and intra- rater reliability, as follows. Inter-rater reliability tests were performed by training a second coder in the coding process and having them analyse the sampled reports. The use of multiple, trained coders is recognised to increase reliability (Kolbe and Burnett 1991), as results are deemed to be reliable if they can be replicated by an
76
independent researcher (Marston and Shrives 1991). The results obtained by each coder were compared against an analysis of the same reports undertaken by the primary coder (the author). Eleven differences were identified in the results of the two coders (an error of 0.03%). All discrepancies were then individually reviewed and resolved by the primary coder.
The primary coder also repeated the analysis of reports numbered 11-20 to test for intra-rater reliability. Given there was a familiarity with those first reports that may potentially affect the results of an intra-rater reliability test, the 11th to 20th reports were chosen for analysis as these were the earliest reports coded that had not been subject to review during the initial stages as the coding process was refined. A time period of approximately five months had elapsed between the first and the repeat analysis. There were no significant differences found between the first and second analysis with two minor errors in coding detected (one relating to the number of prior years’ data provided in a sustainability report, the other to the detection of contingent liability information in an annual report). Consequently, the author is confident that, although the coding process involves inherent subjectivity, given the detailed and structured nature of the evaluation process, replication of final results would be possible.
4.5.2 Validity
The term ‘validity’ is used in various ways within the literature. Weber (1990) distinguishes between two key forms of validity: external validity or generalisability, and internal validity or correspondence, which is often described as the extent to which an instrument measures what its user claims it to measure (Cooper and Emory 1995; Holsti 1969; Thorndike and Hagen 1969). Internal validity may be further classified as including face validity, social validity and empirical validity (the latter comprising content validity, construct validity and criterion validity) (Krippendorff 2004). A discussion of the various forms of validity relating to the survey and content analysis is provided below.
Survey: This section discusses efforts to promote or evaluate each of the forms of
validity relevant to this survey: external, face, social validity and content validity. External validity refers to the extent to which research findings may “be generalised across persons, settings and times” (Cooper and Emory 1995). The purposive sampling technique was used to target a broad range of respondents from across the spectrum of relevant stakeholder groups, seeking results that are likely to be indicative of the
wider population. Nevertheless, the non-random nature of this sampling process may reduce the external validity of the results.
Face validity has its roots in common sense and refers to the appearance of validity
(Hodson 1999) in that “a category has face validity to the extent that it appears to measure the construct it is intended to measure” (Weber 1990, p18). This is the weakest form of validity and is necessary, but not sufficient, for establishing validity
(Hodson 1999). Face validity was established by pre-testing the survey instrument with expert judges from academia, the accounting profession and the safety profession. Social validity refers to “that quality of research findings that leads us to them on account of their contribution to the public discussion of important social concerns” and socially valid research often has “relevance and meaning beyond an academic audience” (Krippendorff 2004, p314). The high response rate achieved in this survey suggests respondents saw the study as sufficiently important to both contribute to and to distribute more widely. Where the survey was distributed, people with no direct connection to the researcher were motivated to take the time to complete the survey. This demonstrates the strong social validity of the questionnaire.
Content validity requires an evaluation of the extent to which a measuring instrument provides adequate coverage of the various facets and domains of the intended research issue. It is primarily concerned with the construction of a research instrument rather than its findings (Cooper and Emory 1995; Hodson 1999). Holsti (1969)
suggests that content validity is established through the informed judgement of the researcher or other expert and is normally sufficient to establish validity if the purpose of the research is purely descriptive (p143). The content validity of this survey was established by surveying the literature to identify relevant topics that need to be addressed in the survey (McMurray et al. 2004) and, as mentioned above, subjecting the draft instrument to the review of experts.
Content analysis: Holsti (1969) suggests that in cases such as the present study, where content analysis is used for problems where the question is answered from a description of the attributes of the content, “the investigator is in large part freed from problems of validity, except to the extent that validity is related to sampling and reliability” (Holsti 1969, p43). Issues of reliability have been addressed above. In terms of sampling, content (and face) validity of the disclosure index instrument was promoted by selecting categories and elements for inclusion based on a thorough review of the literature (McMurray et al. 2004) as shown in Section 4.4.2. These categories were again reviewed following receipt of the stakeholder survey results to
confirm that stakeholders had not raised any issues that had not already been captured in the disclosure index. Furthermore, the scoring process used to collect data was based on a previously used and validated rating system.
4.6 Summary
This chapter has presented the methodological foundations for the research, revealing a study grounded in the scientific method but incorporating some use of qualitative analysis. An outline of the research design is presented along with the structure of the research process, and the logical progression from research aim to research objectives to research questions is highlighted. In doing so, this chapter has shown how the research questions justify the selection of a mixed method approach as the appropriate process for data collection and analysis.
This mixed model approach involved the use of survey questionnaires and content analysis to explore a field of enquiry as yet under represented within this literature, namely, the construction of corporate accounts of OHS accountability. Consequently, the study produced a combination of quantitative and qualitative data for analysis. The final section of this chapter has addressed methodological issues of reliability and validity that are generally associated with the use of techniques such as surveys and content analysis and those specifically relating to the way in which this study has been conducted. Chapters 5 to 7 now present an analysis of these results and a discussion relating the results to the research objectives.