6.2.1 Introduction
This discussion includes the pilot-survey results, response rates, problems encountered with the data collection process, reliability and non-response bias.
6.2.2 Pilot Survey
Prior to the mail survey being sent out, the survey was pilot-tested in two stages1. From the first stage, three questions were identified as requiring amendment to provide for clearer expression. These questions were amended to make the mail survey more user- friendly. The second stage, being the formal pre-test stage, achieved a response rate of 20% (six responses). The pilot-test results were reviewed and any necessary changes made to the survey document. There were no major changes resulting from the pilot survey.
Subsequent to the pilot survey, the formal survey was sent out to potential respondents. The response rate to the mail survey is now discussed.
6.2.3 Response Rates
The mail survey was sent to a total of 536 potential respondents in March 2009. One
128 survey was returned due to an incorrect address, providing a net sample of 535. A total of 191 responses were returned which provided an overall response rate of 35.70%. One response was eliminated as two responses were received from the same council. As this was the only council that had completed two responses, if left in, it could have introduced bias into the survey results; thus, it was removed. Of the two responses received, one was completed by the finance manager and the other by the corporate planning manager. As the survey document was targeted at the CFO, it was thought that the finance manager’s response would be more closely aligned to the CFO’s response. Thus, the corporate planning manager’s response was removed from the sample for the purposes of analysis. However, whilst being removed from the sample, any differences in the two responses may be useful in discussion later to explain differences in results between survey respondents.
The final sample for this study was, therefore, 190 usable responses equating to a usable rate of 35.51% (Table 6.1). This response rate was consistent with previous Australian studies conducted in this area of research, ranging from 25% for the National Local Sustainability Survey conducted in 1996 by Whittaker (1997) to 52% in conducting an LA21 study focusing on local government in South Australia by Kupke (1996).
Table 6.1
Mail Survey - Response Rates
Survey Document Number %
Local Government Bodies 536
Incorrectly Addressed 1
Potential Respondents 535
Returned Responses 191 35.70
Eliminated 1
129 6.2.4 Receipt of Surveys
As completed surveys were received, each was opened, date-stamped and consecutively numbered. Each survey was then examined to ensure that it was acceptable for processing. The completed surveys were subsequently sorted and then coded manually. A preliminary code structure had been developed during the pilot stage of the survey design in an attempt to highlight and eliminate any possible coding issues early on in the process. This code structure was used to code the surveys manually. Manually coding each survey allowed for the inputting of the data into a computer spread-sheet to be straightforward and less time consuming.
6.2.5 Problems of Data Collection
Some respondents did not complete all of the questions in the mail survey. No respondent omitted answers to more than four questions. There were two main sections of the survey that were not fully answered, being Section 2 and Section 5. Section 2 examined the level and type of voluntary sustainability reporting and Section 5 investigated what guidelines are utilized in the preparation of the sustainability report. Where a question was not answered by respondents, to enhance the validity of the study (Cavana et al. 2001), the blank responses were recorded as a nil (‘0’) response.
6.2.6 Reliability
To test the internal consistency and reliability of the multi-item measurement scale in the survey, Cronbach alpha testing was conducted. This method is one of the most accepted methods of reliability testing today (Thomas et al. 2005). Table 6.2 presents the Cronbach alpha coefficient for the major multi-item scaled questions used in the statistical analysis. Nunnaly (1978) indicated that a 0.70 coefficient is generally deemed to be an acceptable reliability coefficient. The results of these calculations indicate overall reliability and consistency as the values exceed conventional levels of acceptability.
130 Table 6.2
Mail Survey - Cronbach Alpha Reliability Analysis Question Cronbach’s
Alpha
Cronbach’s Alpha Based on Standardized Items Number of Items 7 .899 .894 12 8 .916 .917 13 10 .830 .831 4 13 .914 .915 11 17 .839 .837 6 21 .764 .810 7 22 .794 .789 8 24 .795 .796 5 28 .951 .947 9 29 .971 .971 7 30 .875 .865 7 31 .969 .969 42 32 .883 .882 5 35 .922 .923 5 6.2.7 Non-Response Bias
The non-response rate for this survey was 64.49% (346 respondents). That is, of the 536 surveys that were mailed out, 64.49% of all potential respondents did not respond. Possible explanations for this were lack of resources and time to complete the survey (three potential respondents did make contact and advise accordingly) or that they may have had no interest in the subject matter. This latter explanation was consistent with a study conducted by Oppenheim (1966) who considered that similar levels of non- response rates are typical in samples for respondents who may have no special interest in the subject.
131 Non-response in surveys may introduce bias as the actual returns may not be representative of the population. To determine if bias had been introduced into this study, a method developed by Oppenheim (1966) was utilized. Using this method, the survey results of respondents who sent their surveys in on time were compared with the survey results of late respondents. Oppenheim argued that respondents who sent in their surveys late are similar to non-respondents. Where there was no bias and, thus, the results were representative of the population, it would be expected that the survey results would be similar between the respondents who sent their surveys in on time and the late respondents.
To compare survey results between the two types of respondents, the mean response scores for respondents that sent their surveys in on time were compared with respondents who sent their surveys in late. It was assumed that where no bias existed, the mean response scores would be similar. Respondents were deemed to have sent their surveys in on time if the mail survey was received prior to the first reminder letter being sent out. There were sixty survey responses received by the due date and 130 responses received late. Mean responses were calculated for questions 1 to 5 being the main descriptive data of the survey document. The mean responses were then compared by conducting a Levene test for equality of variances and t-test for equality of means. The differences were not significant at the 0.05 level for all questions (Table 6.3). Therefore, it can be assumed that there are no significant differences between the on-time respondents and late respondents.
132 Table 6.3
Mail Survey - Response Bias Results
Levene’s Test and t-test for Equality of Variances for Non-Response Bias
Equal Variances Levene’s Test for Equality of Variances – F Value
Sig. level t-test for Equality of Means df Sig. (2-tailed) 1 are assumed not assumed 1.226 .270 -.685 -.731 188 135.242 .494 .466 2 are assumed not assumed .123 .726 1.415 1.428 188 117.444 .159 .156 3 are assumed not assumed .056 .813 .434 .435 188 115.886 .665 .664 4a are assumed not assumed .922 .338 .816 .798 188 108.830 .415 .426 4b are assumed not assumed .007 .932 .883 .881 188 114.208 .379 .380 5 are assumed not assumed .061 .806 .311 .310 188 114.414 .756 .757