The research team at the National Centre and Centre for Research in Social Policy developed the screening questionnaire with the help of researchers from the DSS, and with input from the Voluntary Sector Resource Group appointed by the DSS and DfEE. In developing the questionnaire we drew on other surveys, most notably the Evaluation of DWA conducted by Karen Rowlingson and Richard Berthoud11.
It was felt to be particularly important that the document was short and inviting, so the questionnaire was kept to four sides in length, and only the most vital questions were included. The aim of the questionnaire was to identify those sample members who were relatively ‘close to’ or ‘far from’ paid work, and who might take part in the New Deal for Disabled People Personal Adviser scheme should it be offered in their area.
Expert Panel
At the first stage the draft questionnaire was subjected to scrutiny by an ‘expert panel’ of experienced researchers from within the National Centre (not connected with the study) who made various recommendations about question wording, overall structure, routing and content.
Cognitive testing
The second stage of testing was a series of cognitive interviews carried out by a National Centre researcher, with respondents recruited using purposive sampling. A total of 7 interviews were carried out on 22nd to 23rd February 2000.
The cognitive interviews were designed to check the length and acceptability of the questionnaire, and respondents’ understanding of key phrases and concepts. Cognitive interviewing is qualitative in nature and makes use of techniques drawn from cognitive psychology in order to uncover aspects of the response process, which are usually hidden. Cognitive interviews reveal where respondents misunderstand survey questions or key concepts, do not know or cannot recall the requested information, use an inappropriate strategy for making a judgement, or prefer to hide certain information or provide a ‘socially desirable’ answer.
Interviews were carried out on a one-to-one basis. Each respondent was asked to
complete the questionnaire, and was then asked a series of retrospective questions about how they had gone about answering the questions, how they had followed the routing and what they understood particular terms to mean. Results from these interviews led to several improvements in the questionnaire. The interviews provided positive feedback on the acceptability of the questions and ease of questionnaire completion.
Interviews were taped with the permission of respondents, and a gift voucher was given for their time.
Dress-rehearsal pilot
A small dress-rehearsal pilot was carried out by the researchers concurrently with the cognitive pilot. The aim was to test the survey procedures and quality of address
information. Ninety pilot sample members were sent the questionnaire and accompanying letter, together with a prepaid envelope. Thirty replies were received within a relatively short period of time. The pilot indicated that the response rate was likely to be acceptable (note that reminder letters were not sent as part of the pilot), and that the mail-out system was likely to be effective. In addition, replies were examined for non-response and any interesting comments, and this was fed into probes used in the cognitive interviews.
The final postal questionnaire
The final document was professionally laid out. The design work was subcontracted by the National Centre to Davenport Associates. The questionnaire was colour washed in yellow, with white boxes for respondents to enter their answers. Particular effort was made to ensure that the questionnaire would be as reader-friendly as possible, in particular considering the needs of respondents with visual impairments. Questions captured data about current economic activity; intention to work, enter study or training or engage in some productive activity now, or in the future; and whether the respondent was interested in receiving help to find work, study or training opportunities. A copy of the final
questionnaire wording is presented at Appendix A.
Mailout details
The postal questionnaire was sent to respondents with an accompanying letter on behalf of the DSS (see Appendix B), and a pre-paid envelope for its return. Two further reminders were sent, consisting of a reminder letter (the first reminder was designed to be informal in style, the second formal, in order to maximise effectiveness) along with another copy of the questionnaire and pre-paid envelope on each occasion.
The mailout dates were as follows: 14th April 2000 Initial mailout
5th May 2000 1st reminder mailout 19th May 2000 2nd reminder mailout
The cut-off date for productive questionnaires was 5th June 2000. Respondents were provided with a telephone number they could contact for help completing the questionnaire and some were completed over the phone (for example where someone had a visual impairment or had difficulties with reading or writing skills). It was also made clear that help could be provided by a friend or carer in completing the questionnaire. In this way we sought to minimise non-response bias among those with specific disabilities of difficulties.
Response rates
The overall response rate to the postal survey was 66 per cent.