7. TRABAJO DE CAMPO
7.2. REVISIÓN DE DOCUMENTACIÓN DE TENDENCIAS
It is important to assess the validity (i.e. does the method measure what it intends to measure) and reliability (i.e. how consistent are these measures) of the CV
survey instrument (Cameron and Englin, 1997). It is difficult to assess the validity and reliability of CV estimates because, among other things, “there is no benchmark against which to compare the results” (Imber et al., 1993: 91). Besides, by the nature of the technique, every CV survey is different and subject to different constraints and opportunities. These issues are considered in the context of Bangladesh.
4.8.1 Validity
An instrument is considered valid to the extent it measures what it is supposed to measure, that is, validity measures how accurately the CV of a public good estimates the good’s true value to respondents. A fundamental problem arises at this point as the
CV instrument aims to measure a value which cannot be identified in the real world because of the hypothetical nature of the scenario and valuation question. Conditions are being defined which are either necessary preconditions for an instrument’s validity or appropriate tests of that validity (Hoevenagel, 1994). The guidelines set by the Panel appointed by the National Oceanographic and Atmospheric Administration (NOAA) of the United States Department of Commerce are well-known for achieving valid responses in CV surveys (Arrow et al., 1993). Careful consideration was given to the
NOAA Panel’s recommendations to achieve valid responses from respondents (which is elaborated later on in Section 4.8.3). Furthermore, to assess the validity of WTC
answers, a comparison of the stated WTCM with the stated income and a set of
demographic characteristics is conducted in Chapter Five. In addition, a similar comparison of the WTCT responses with a set of demographic and socio-economic
characteristics is done.
4.8.2 Reliability of Responses
Reliability measures the variability among responses; valuations with relatively low variation among responses are considered more reliable estimates of value (Breedlove, 1999). As Hoehn (1990) pointed out “[t]he reliability of the CVM has proven difficult to assess” (p: 105) and only a few researchers have conducted tests of measurement replicability (Georgiou et al., 1997).
In order to test the reliability of the ECV results of the survey, a post survey check approach, a variant of test-retest, was undertaken at a minimal level21 due to time
and budget constraints, following Vaus (1995). As mentioned in Section 4.5, 40 respondents out of total 71 telephone-equipped households were selected randomly and then at a later time (after about four weeks) reinterviewed22. After several attempts, thirty-one agreed to talk23. They were asked a few questions again including the
valuation question (see Appendix VIII for full version of the re-interview schedule). This re-interviewing found a high correlation between the respondents’ WTCM in the
survey and the post-survey check experiment, indicating that the survey appeared to give a reliable measurement of the ECVM. This is also in line with other studies (see, for example,Georgiou et al., 1997).
In the process of verification, only in three cases out of the total 31 selected re- interviews, were some discrepancies found. For one, I could not find anyone with the given name at the telephone number provided24. For two interviews, the answers given
over the phone were inconsistent with the answers given in the interview regarding the bid values. In the telephone interview, they opted for higher values. All these discrepancies were adjusted. As a whole, the discrepancies and item non-responses are considered minimal. This was achieved mainly because of the intensive training and close collaboration during the data collection process by checking item non-response regularly.
4.8.3 The National Oceanic and Atmospheric Administration Panel Guidelines
Although the CVM has been used for more than three decades to estimate economic values of non-marketed goods and services, no standard rule exists for examining its validity and reliability. These issues were the subject of heated debate following the Exxon Valdez oil spill in the Gulf of Alaska in 1989. Against the
21Minimal in the sense that only 40 telephone-equipped respondents were interviewed with few questions repeating
from the IS and after only four-weeks.
22 There is a debate whether (i) the same or different respondents should be interviewed; and (ii) after how long, this reinterview should take place (Teisl et al., 1995). Following Vaus (1995), I reinterviewed the same respondents after four weeks using only eight questions including the valuation question.
23 As mentioned in Section 4.2.1, only the richer class of population has telephones in Bangladesh. Therefore, these 31 respondents may represent neither the survey respondents nor the overall population.
24 This happened as one (a young girl) of the relatives of that household was visiting them and she was selected for the interview as she mentioned herself as a member of that household. She was also the contact person for that household and she used a false name out of fun. This was corrected later by interviewing another member of that
backdrop of the controversy of CV estimates, the National Oceanographic and Atmospheric Administration of the United States Department of Commerceappointed a blue-ribbon panel (hereafter the NOAA Panel) of multidisciplinary experts with two Nobel laureate economists, Kenneth Arrow and Robert Solow as co-chairs in 1992 to assess the reliability and validity in the context of a liability suit25. After weighing evidence by both proponents and critics of the method, the NOAA Panel endorsed the use of CVM, with suggestions for the design and implementation of the technique (Arrow et al., 1993). The Panel’s recommendations are a set of guidelines for CV
survey design, administration and data analysis (Carson et al., 1995a, 1996). The Panel addressed the sources of potential bias inherent in CVM studies and then made recommendations that help to eliminate these sources. Over the years, many CV studies have been conducted following the guidelines of the NOAA Panel26 (Giraud et al., 1999
and White and Lovett, 1999). The Panel’s guidelines are now considered as standard for a CV survey (Smith, 1996 and Navrud and Pruckner, 1997). In the Panel’s view, “…the more closely the guidelines are followed, the more reliable the result will be” (Arrow et al., 1993: 4609).
Although the NOAA Panel guidelines are considered as a standard for deriving reliable CV estimates, they have also been subject to criticism, even from the proponents of CV, for their loose recommendations (Randall, 1997). As Navrud and Pruckner (1997) state:
…the guidelines are not sufficiently strict. On the one hand, the [NOAA Panel] report says that the guidelines should be followed as close as possible. On the other hand, however, it does not provide reference to which deviations would be accepted with the empirical results still remaining valid (p: 12).
There is no standard rule on how far to relax these recommendations, although the Panel suggests that “[i]t is not necessary, however, that every single injunction be completely obeyed; inferences accepted in other contexts are not perfect either” (Arrow
et al., 1993: 4610). Lindsey et al. (1995) state that “[s]trict adherence to the NOAA
guidelines would make use of CV prohibitively expensive for most routine applications” (p: 256). Considering cost and other local conditions, several studies were undertaken relaxing some guidelines of the NOAA Panel27. Navrud and Pruckner (1997) also state
25 For a background of the NOAA Panel, see Arrow et al. (1993) and Portney (1994).
26 The NOAA Panel guidelines are also being used in ex post evaluation of studies (Lindsey et al., 1995).
27 Relaxation can take many forms, such as, survey administering in a class-room (group) setting rather than in-
person (Berrens et al., 1997) and using relatively small sample size, that is, 252 interviews for before scheme survey and 260 interviews for after scheme survey (Tunstall et al., 1999).
that “CV [contingent valuation] surveys for purposes demanding a lower level of accuracy… need not follow all of these guidelines” (p: 23, footnote 6).
The survey instruments and procedures used in this study followed the guidelines recommended by the NOAA Panel, with only a few exceptions which were required due to specific local conditions. A comparison between the NOAA Panel’s major recommendations and the survey procedure followed for this study is presented in Table 4.3.
Table 4.3 shows that most of the recommendations made by the NOAA Panel were incorporated in the design of the survey instrument. A few were left out because of local conditions and time and budget constraints. Most importantly, careful consideration was given to keeping the IS, as much as possible, simple, understandable and plausible to the respondents many of whom are not only illiterate, but also economically extremely poor.
Table 4.3: Comparison between the NOAA Panel guidelines and adopted survey design procedures
NOAA Panel guidelines Adopted survey measures
General Guidelines: a. Sample size and type
b. Minimize and explain non-response c. Personal interview
d. Pre-testing of interviewer effects & survey instrument
e. Reporting results
f. Careful pre-testing of CV questionnaire
a. Reduced size was used*. The NOAA Panel
suggests a split sample, but time restrictions did not allow for this.
b. An 85.10 % response rate was achieved which is considered reasonably good.
c. NOAA Panel recommendation was followed (hereafter followed)
d. Followed e. Followed f. Followed Value Elicitation Surveys:
a. Conservative design b. Elicitation format c. Referendum format d. Accurate description of programme/policy e. Pre-testing of photographs f. Reminder of undamaged substitute commodities
g. Adequate time lapse from the accident h. Temporal averaging
i. “No answer” option j. Yes/no follow ups k. Cross tabulations
l. Checks on understanding and acceptance
a. Followed b. Followed
c. Payment card was used**
d. Followed e. Followed f. Followed g. Not applicable h. Followed i. Followed j. Followed k. Followed l. Followed
Goals for Value Elicitation Surveys: a. Alternative expenditure possibilities b. Deflection of transaction value c. Steady state or interim losses d. Present value calculations of interim losses
e. Advance approval f. Burden of proof28
g. Reliable reference surveys
a. Followed
b. “Warm-glow” effect was not tested*** c. Followed
d. No sensitivity test was conducted e. Not applicable
f. Followed g. Not applicable
Notes: * Four hundred samples were used instead of 1000 which is recommended by the
NOAA Panel (see Chapter Five for more information on sample size
requirement for normal approximation).
**
Although the NOAA Panel recommends dichotomous choice referendum format, payment card was used. The reasons are explained in Section 4.7.1.
*** Reasons are explained in the section to follow.
Source: For NOAA Panel guidelines see Arrow et al. (1993).