Online surveys
Stage 1 of the project consisted of two large scale online surveys, using an existing commercial internet panel, and this may raise a number of concerns. Thefirst is the nature of the sample. Commercial online panels enable the researcher to specify the sampling frame of the respondents out of a long list of criteria (e.g. age, gender, education level, employment status, marital status, disability status, ethnicity, housing status, residential area, number of children and their age, etc.), and thus achieve‘representativeness’in the attributes of their choice. Nevertheless, it can only achieve representativeness in terms of observed characteristics (assuming they are correctly self-reported), and the issue of self-selection into the panel will
always remain a concern (the average 75-year-old panel member may not be an average 75-year-old in the wider population). Furthermore, by definition, internet panels cannot have people without access to online computer facilities. However, it should also be noted that a similar argument could be made of other modes of recruitment. An interview survey may suffer from selection into accepting an interviewer into one’s home–something not everybody has an equal propensity to accept.
The second concern is of legitimacy. Most commercial online panels offer some form of reward to
participating members. In other words, panel members are people who have put themselves forward to be surveyed for various (mostly marketing research) purposes in exchange for afinancial payback. There may be a concern over using the same facility for the purpose of academic research which may subsequently inform public policy. One may argue that democratic legitimacy would require that all citizens have an equal chance of being invited to take part in research that may form the basis of policy. There are two things to note. First, it is indeed the case that some panels appear to offer significant rewards, with no restrictions on the number of surveys a given member can complete; however, other panels offer‘points’ to be converted into donations to charities of the members’choice, and/or have restrictions on the number of surveys a given member can complete in a month. The researcher needs to be aware of such aspects of the panel and select them wisely. Second, it is of interest to note that the stage 3 CAPI survey found the majority (51%) of face-to-face interview respondents replying positive to the question‘Do you think it is okay to base policy on the views of people who volunteer to answer internet surveys for a small reward?’ The third concern is with respect to the quality of the actual responses. Unlike interview-based surveys, but similar to postal surveys, there is no information about the environment in which the survey was taken, and whether the respondent was sufficiently focused on the tasks. Unlike postal surveys, information on time taken between individual clicks or data on the movement of cursors could be collected and analysed, but even they cannot distinguish between a respondent who is in deep contemplation and another who is distracted by other activities. This impact of poor respondent engagement is likely to affect different methods differently, depending on how the analyses treat error and noise. Methods such as iterative TTO, which elicit exact values from each individual for each state, are more likely to be vulnerable to poor data quality than methods such as DCETTO, which collect only ordinal preferences from individuals and models them, taking into account that data contain error and noise.
When the researcher makes a decision on the mode of administration, he/she needs to weigh up the pros and cons. The major advantages of online surveys are that a sample of thousands can be achieved within weeks, at relatively low cost. PRET (including PRET-AS) was an 18-month project, and the stage 1 survey of 6000 respondents was feasible only through using online technology.
Binary choice methods
With the exception of the iterative TTO and LT-TTO used in stage 4, all of the experimental questions used in PRET were of the binary choice kind. This means most of the data collected within this project do not allow the identification of a particular health-statevaluefor a givenindividualrespondent. However, the distribution of respondents across each binary choice allows the analysis to infer the violation or otherwise of different assumptions involved in health-state valuation. Arguably, it is particularly suited to studies such as stage 1 of PRET, in which a large number of methodological topics were tested out on a large number of respondents in a relatively short time frame.
Of interest, however, is whether binary choice methods can be used as the main method of a health-state valuation study, which aims to produce a value set of a health-state classification system. In this project, type VIII data using binary choice LT-TTO was found to be capable of generating health-state values. Unlike iterative LT-TTO, as it does not aim to identify the point of indifference for all states from all individuals, it has the major advantage of not being affected by exhaustion of lead time.19At the same time, further significant developments regarding the selection of the health scenarios and the health-state modelling are required before it can be used for actual health-state valuation studies. On the other hand, DCETTOis much better developed. This project has furthered the work carried out in the development study
of DCETTOwith the three-level EQ-5D,22and shown that it is capable of valuing large descriptive systems such as the EQ-5D-5L.
One fundamental challenge for the use of binary choice methods for health-state valuation is the assumption from random utility theory that the difference in value across the paired scenarios will
determine the distribution of responses between the two scenarios. It has no scope to distinguish between values and strength of preference; in other words, it does not allow a strong preference over a slight difference in value. Although random utility theory may not recognise such a preference, it is clear that they exist: indeed, the DCETTOvalue set estimated from PRET-AS implies that the difference in value between EQ-5D-5L 11111 and 21111 is negligible, although it is quite likely that most people in a binary choice would select 11111 for afixed duration over 21111 for the same duration (if such a choice were given).
It is sometimes suggested that binary choice methods are easier for respondents than iterative methods because although in iterative methods respondents need to give an exact value for an answer, binary methods require respondents to give only an ordinal preference. However, it is rare that respondents of an iterative exercise are asked to come up with an exact value on their own. Most iterative health-state valuation methods are designed as a series of binary choice tasks, in which the parameter of the second task onwards is determined by the response to the preceding task. In this respect, answering, for example, eight independent binary choice questions where all the parameters of the scenarios change from question to question may well be more challenging than answering eight binary choice tasks from within a single iterative question. In the latter, only one parameter is likely to change from task to task. Results reported in stage 4 of this project support this view: respondents typically found iterative TTO and LT-TTO easier than binary choice DCE or DCETTO. However, earlier work comparing DCE and TTO found that both techniques could be understood equally well and had high completion rates.62On the other hand, the very factor that makes iterative tasks easier may introduce bias of its own. There is a literature on contingent valuation and willingness to pay in health economics, where it has been shown that iterative tasks may be susceptible to biases because respondents do not interpret the series of binary choices as independent.73–75In effect, the binary choice versions of TTO and LT-TTO can be interpreted as iterative (LT-)TTO surveys, but ones in which the individual tasks are presented at random order.
The use of DCETTO to value EQ-5D-5L
The move from the three-level EQ-5D (with 243 distinct health states) to thefive-level EQ-5D-5L (with 3125 distinct health states) was to allow for improved sensitivity. An increased number of health states that can bedescribeddifferently do not necessarily lead to improved sensitivity unless (1) patients recognise them as distinct states for self-reporting their own healthand(2) the valuation studies result in distinct coefficients for the added levels within each dimension. In this respect, it is of interest that during stages 3 and 4 of the project at least some respondents expressed uncertainty (both unprompted during the think-aloud process, and prompted as part of a follow-up question) about the relative ordering between level 4,‘severe’, and level 5,‘extreme’, used in two dimensions of EQ-5D-5L (Anxiety/depression and Pain/discomfort).
However, the DCETTOdata in PRET-AS found that at the aggregate, with one exception, all 20 anchored level dummies were ordered within each dimension, and statistically significantly different from level 1. The only exception–mobility level 2–had the‘incorrect’sign, but this was not significant. It is highly unlikely that respondents had any conceptual uncertainty regarding the ordering of level 1‘no problems’ and level 2‘slight problems’; and therefore, this non-significant coefficient for mobility level 2 is likely to be caused by actual preferences rather than a cognitive challenge. In fact, compared with value sets based on TTO data, one attraction of a value set based on DCETTOis its apparent ability to model very small decrements from full health.
Furthermore, when a series of chi-squared tests were conducted to test the null hypothesis that adjacent level dummies (i.e. levels 2 vs. 3; 3 vs. 4; 4 vs. 5) within each dimension were no different from each other, all 15 resulted in rejecting the null (13 hadp< 0.001;p= 0.029 for level 4 vs. 5 in usual activities;
p= 0.009 for level 2 vs. 3 in self-care: details available from authors). The overall implication thus is that although there may be individual respondents who are uncertain about the relative ordering of levels, at the aggregate, a clear pattern exists across the levels within each dimension, so that EQ-5D-5L can be valued with an online survey using DCETTO.
Respondent characteristics
For stages 1–3 of PRET and PRET-AS, the characteristics of the samples used are generally representative of the UK population in terms of age and gender. Throughout the questions, there do not seem to be any covariates that are constantly significant or never significant. In stage 1, age, gender, own health and satisfaction (with own health or life) were often found to be significant, but not always. Furthermore, marital, employment and education status tend to be found significant for the severe states (as opposed to mild states or pooled models), but again, not all of the time. The results from stage 2 indicate that different covariates affect different question types differently.
Although we have the age and gender characteristics of those who did not respond to the online surveys, we do not know how this sample differs in other ways, and how this may have had an impact on results. This may be important as we have demonstrated at stages 3 and 4 that respondent characteristics (in terms of personal factors such as having children) or subjective factors (such as experience of coping with illness) have an impact on the way in which both iterative and binary choice tasks are perceived. We have outlined some of the personal and subjective considerations that are important in the valuation process. It is possible that these could be considered in the design of valuation studies, either by directly including questions about the factors, or including them in the valuation process.