It is stated by De Vos (2001: 158, 182), Oppenheim (1992: 47), Veal (1997: 194), and, Converse and Presser (1986: 51) that in order to improve the success and effectiveness of an investigation, a piloting strategy of the questionnaire needs to be undertaken before being administrated to the research population. This testing has to prove that the instrument can be used with confidence in the rest of the empirical investigation. Usually, this includes subjecting the instrument to the scrutiny and criticism of a representative panel of the entire research population. The whole process, according to Oppenheim (1992: 47), of designing and trying out questions and procedures is usually referred to as “pilot work”. Maas (1996: 106), underscores that the questionnaire must be tested in its totality. The following pilot work strategy was undertaken:
5.7.6.1 Testing 1: Concept questionnaire tested by a panel of SMME academic experts
A concept (preliminary) questionnaire was compiled and tested with a panel of SMME academic experts. Converse and Presser (1986: 65) underscore that this stage represents the qualitative stage of testing the research instrument and alternatively term this phase of testing as the developmental or pre-testing phase. This phase is focussed on addressing two broad purposes, state Converse and Presser (1986: 54-59): firstly, to assess the specific questions for: meaning, difficulty, and, the respondents’ interest and attention; secondly, to assess the research instrument as a whole in regard to the flow and “naturalness” of the sections, the order
of the questions, skip patterns, timing, overall interest and attention of the respondents, and the respondents’ well-being. Furthermore, if the respondents in the pretest phase are told that this is a practice run and are asked to explain their reactions and answers, this testing phase is called a “participating” pretest. In other cases where the participants in the pretest are not told that this research instrument is under construction, this would have been called an “undeclared” pretest (Converse and Presser, 1986: 52-53).
A specialist panel of seven academic personnel from Walter Sisulu University, Human Sciences Research Council and Cape Peninsula University of Technology, who were deemed to be specialists on SMMEs and/or strategy, and/or questionnaire design, were invited to participate in the testing of the preliminary instrument. Four completed preliminary questionnaires were returned by this panel.
A cover letter with accompanying full explanations was sent to the test panel inviting them to participate in completing the preliminary questionnaire. The panel was required to validate that the pool items actually measured the dimensions of the research problem. To this end, the panel were asked to respond to three broad categories of questions: what they thought of the research instrument’s questions; they were asked to assess whether the formulation of the questions could be clearly understood, convey one concept each, whether they are appropriate in terms of language or readability for the intended target market (so-called face validity); and, the appropriateness of the response method (for example, is the use of Likert scales, binary response and rankings appropriate for the research design?). The cover letter and checklist used by the specialist panel of academic experts, is included in Appendix 3.
The recommendations from the pretesting specialist panel variously referred to questionnaire layout, question response option reformulation, question reformulation and language improvements. Overall, however few recommendations were forthcoming from the pretesting specialist panel; as far as possible these were used to improve the questionnaire.
A concept (preliminary) questionnaire was compiled and administered to a panel of SMMTE business owners.
The pilot-testing panel should, ultimately, not form part of the final research sample. This second stage can alternatively be called the pilot testing stage. Converse and Presser (1986: 74) alternatively call this phase the “polishing” or “dress rehearsal” phase. Furthermore, this phase is not seen as an opportunity to repair gross errors or to make new explorations, but, rather it is an opportunity for cutting, trimming, re-arranging or reformatting for clarity (Converse and Presser, 1986: 75).
5.7.6.2 Testing 2: Pilot testing by SMMTE business owners
A panel of 14 SMMTE business owners was selected using convenience sampling. Cover letters with accompanying full explanations were sent to the test panel inviting them to participate in completing the preliminary questionnaire. Nine completed preliminary questionnaires were returned by the SMMTE panel.
This phase is also focussed on addressing two broad purposes, according to Converse and Presser (1986: 54-59): firstly, to assess the specific questions for: meaning, difficulty, and, the respondents’ interest and attention; secondly, to assess the research instrument as a whole in regard to the flow and “naturalness” of the sections, the order of the questions, skip patterns, timing, overall interest and attention of the respondents, and, the respondents well-being. The pilot-testing panel were asked to respond to three broad categories of questions: what they thought of the research instrument’s questions; they were asked to assess whether the formulation of the questions could be clearly understood, convey one concept each, whether they are appropriate in terms of language or readability for the intended target market (so-called face validity); and, the appropriateness of the response method (for example, is the use of Likert scales, binary repose and rankings appropriate for the research design?). This cover letter and checklist used during the pilot-testing phase by the panel of experts, are included in Appendix 2.
Recommendations from the panel of 14 SMMTE business owners, based on the nine completed questionnaires, were used to improve the preliminary questionnaire. Comments from the SMMTE panel were collated, analysed and ultimately used to improve the validity of the preliminary questionnaire.
Overall, very few recommendations were forthcoming from the pilot SMMTE panel phase with most panellists variously recommending some language improvement. As far as possible, these recommendations were used to improve the preliminary questionnaire. Overall, the panels recommended very few changes to the research instrument. The duration for completing the preliminary questionnaire was between 15 and 30 minutes with most of the test panel members completing the preliminary questionnaire within about 20 minutes. The cover letter and checklist used by the SMMTE business owners are included in Appendix 4.
5.7.6.3 Testing 3: Final specialist evaluation
The preliminary questionnaire was subjected to a further specialist evaluation by Professor Tobie De Coning of the University of Stellenbosch. The preliminary questionnaire was further assessed for grammatical correctness and cases of ambiguity. In such cases, the items were either corrected or removed. The final draft of the questionnaire was compiled after comments had been received from this phase; these mostly referred to refinements of the instrument (ref. Appendix 5 for the final draft of the questionnaire).