6. CASO DE ESTUDIO: PATENTE DEL PROCESO PARA LA DESTRUCCIÓN DE
6.9. PARÁMETROS DE ENTRADA DEL CASO DE ESTUDIO
In order to assess whether the NPS scheme was able to reduce survey bias due to noncontact, its effect on unweighted survey estimates was tested in two ways. The first involved a comparison on frame data. The second compared survey estimates to data from census or election returns.
Table 34 presents the results of the frame comparison, averaged across the three simulated surveys. As with the response disposition data above, average results are presented because the trends were consistent across all of the simulations. Specifically, in all but a few cases the NPS scheme led to a valid group that was more representative of the total frame than an SRS. This was achieved irrespective of whether the variables were part of the propensity models used in the NPS scheme (e.g., occupation and individual gender were not included in any of the propensity models underlying the NPS scenarios tested).
Table 34: Effect of the NPS scheme on estimates for frame variables†
Entire SRS NPS Scenario (Valids)
Frame Variable Frame (Valids) No Adj. Stepped Adj.
Age (Mean) 48.4 50.9 50.5 50.0 Maori Descent (%) 13.5 11.0 11.4 12.0 Gender: Male (%) 48.0 45.4 45.4 45.6 Roll: General (%) 92.6 94.3 94.0 93.6 Occupation: Benefit (%) 1.8 1.1 1.1 1.1 Occupation: Employed (%) 58.8 61.2 61.1 61.1 Occupation: Homemaker (%) 13.4 14.6 14.5 14.2
Occupation: Not Stated (%) 4.7 3.1 3.2 3.4
Occupation: Retired (%) 11.1 12.4 12.1 11.7
Occupation: Student (%) 7.9 6.0 6.3 6.7
Occupation: Unemployed (%) 2.3 1.6 1.7 1.8
Household: Electors (Mean) 3.0 2.8 2.8 2.8
Household: Avg. Age (Mean) 48.3 50.2 49.8 49.4
Household: Surnames (Mean) 2.0 1.7 1.7 1.8
Household: Males (%) 47.3 46.3 46.4 46.5
Household: General Roll (%) 92.6 94.4 94.2 93.8
Household: Maori Descent (%) 13.4 10.8 11.2 11.7
†
Figures represent averages over the three simulated surveys for 2003, 2004 and 2005
At least on these frame variables, then, it appears the NPS scheme consistently reduces noncontact nonresponse bias. Overall, across the variables and survey scenarios examined, it led to an average 28% reduction in absolute error between the returned valid group estimates and the known frame parameters.
An analysis of the standard deviations of the estimates from each simulation scenario, for each frame variable, suggests that the NPS procedure generally increases the variability of results under a ‘constant sample size’18 application. Although it does not do so for all variables, on average the stepped adjustment NPS procedure increased estimate variability by 4% compared to that for the comparative
18
That is, where the NPS sample size is the same as would have been taken under a SRS scheme, rather than increasing the sample size to accommodate the requirements of oversampling likely noncontacts. See page 111 for a discussion of these possible approaches.
SRS scheme (e.g., an average standard deviation of 10 for the SRS scheme would increase to 10.4 under the NPS scheme). Overall, then, the increase is not large. That said, in the 2003 simulation two variables did increase in variation by 20%, while in the other simulations the maximum increase in variability for a variable was 11%. Thus, the scheme may have a substantive influence on variability in a small number of cases.
Turning to survey-only variables, Table 35 presents the results of an analysis to determine which scheme gave estimates closest to known census or election parameters.
The results are not as clear as for the frame variables. However, the NPS scheme did generally produce better point estimates across the variables and scenarios examined. Where the NPS1 (no adjustment) procedure outperformed the NPS2 (stepped adjustment) procedure, the estimates were typically very close between the two, so either would have produced a better estimate. Indeed, in many cases the estimates from all of the sampling scenarios were close; on average they differed by under a percentage point.
Furthermore, where the NPS procedure improved estimates, it led to an average 17% reduction in absolute error between the survey estimates and the census parameters across the simulation scenarios. Thus, although the NPS scheme was generally effective in reducing bias in many survey estimates, it cannot be said to have reduced it substantially in this analysis. In part, this result may be due to other biases inevitably present in a census-based comparison, such as measurement and coverage. Moreover, future development to improve the propensity modelling and underreporting adjustment processes may see greater levels of bias reduction achieved.
Table 35: ‘Best scheme’ results for survey estimates compared to census
Scheme Resulting in Best Estimate
Survey Variable ISSP03 ISSP04 ISSP05
Gender (Male) SRS NPS2 NPS2 Age 20-34 NPS2 NPS2 NPS2 Age 35-49 SRS SRS NPS2 Age 50-64 NPS2 NPS2 NPS2 Age 65+ NPS2 NPS2 SRS Not Religious NPS2 NPS2 SRS Christian NPS2 NPS1 SRS
Employed Full Time SRS NPS2 SRS
Employed Part Time SRS SRS NPS2
One Person Household NPS2 NPS2 SRS
Two Person Household NPS2 NPS2 NPS2
Three Person Household NPS2 NPS1 SRS
Four Person Household SRS SRS SRS
Five+ Person Household SRS NPS1 NPS1
Qualification Bachelor Degree+ NPS1 NPS1 SRS
Own Income <$20k NPS2 NPS1 SRS
Own Income >$50k NPS2 NPS1 SRS
Marital Status: Married NPS2 NPS2 NPS2
Marital Status: Single NPS2 NPS2 NPS2
Marital Status: Widowed NPS1 NPS2 SRS
Ethnicity: NZ European SRS NPS2 NPS2
Ethnicity: NZ Maori SRS SRS NPS2
Election '02 Vote: Labour NPS1 SRS N/A
Election '02 Vote: National SRS SRS N/A
NPS best estimate in x of y cases 15 of 24 18 of 24 11 of 22
Note: SRS represents the Simple Random Sample scenario, NPS1 represents the ‘No Adjustment’ NPS scenario, and NPS2 represents the ‘Stepped Adjustment’ NPS scenario.
As for the frame variables, an analysis of the standard deviations of the estimates from each simulation scenario, for each survey variable, suggests that the NPS procedure generally increases the variability of results. Specifically, on average the
scaled adjustment NPS procedure increased variability by 2% compared to that for the comparative SRS scheme (e.g., an average standard deviation of 10 for the SRS scheme would increase to 10.2 under the NPS scheme).
Again, then, the increase is not large. Moreover, the maximum increase for any one variable across all of the scenarios was 17%. Thus, the NPS scheme appears to have had even less of an effect on point estimate variation, as measured by the standard deviations of simulated estimates, for the survey variables than for the frame variables.
Together, these results lend moderate support to hypothesis 2, that the NPS scheme would consistently improve estimates without a substantive increase in estimate variability. Additionally, the mixed performance of the NPS procedure in the 2005 simulations is consistent with expectations given the relatively poor performance of the underlying propensity model for that scenario (as per hypothesis 4).