Despite several favorable characteristics that are ascribed to online administration of questionnaires, several concerns were voiced which largely address the quality and generalizability of data that are obtained through Web-based procedures.
Sampling bias. The perhaps most apparent problem pertains to a potential bias in samples which have been recruited online. To date, no explicit requirements exist regarding the drawing of representative samples. In fact, many studies rely on self-selection of participants (e.g., who decide to follow a link to a survey that is placed on a website). Doubts about the generalizability of findings obtained with convenience samples are particularly relevant in the context of online studies because characteristics of individuals who use the Internet may not be representative of the targeted population. Eventually, this fact brings up the question whether results obtained with samples that were recruited online can be generalized to the general population.
Internet users may differ from non-users with regard to specific characteristics such as sex, age, income, education and psychosocial adjustment (Gosling et al., 2004). As mentioned before, Gosling et al. (2004) found that online recruited, self-selected samples tend to be more diverse with regard to age, sex, ethnicity and socioeconomic status than samples consisting of college students. Furthermore, research has shown that self-selected samples provide clearer and more complete data compared to not self-selected participants, such as undergraduate students (Pettit, 2002). However, none of these authors attempted to answer the question whether online recruited self-selected samples are more representative of the general population than, for example, clinical samples that are often studied in investigations of childhood abuse.
Even though online data collection enables researchers to access groups of individuals who would remain covert to offline-methods (Lieberman, 2008), there continue to be populations that are hard to access online, such as older persons, homeless people, or people with
outdated hardware or software (Gosling et al., 2004). Finally, regarding the preconception that Internet samples are unusually maladjusted (for a summary of preconceptions about Internet questionnaires, see Gosling et al., 2004), empirical data indicate that this concern is unsubstantiated (Kraut et al., 2002).
Drop-outs. For online-surveys, higher drop-out rates are expected, as there is usually no direct contact between participants and researchers (Fricker, Galesic, Tourangeau, & Yan, 2005). In order to control for adverse effects of drop-outs, relevant demographic variables should be assessed at the beginning of the survey. Furthermore, the questionnaire should make it possible to retrieve how many participants left the questionnaire at which item.
Psychometric properties of online questionnaires. Many self-report questionnaires that were initially developed for paper-pencil administration are nowadays being administered online. However, it should not be taken for granted that psychometric properties such as reliability and validity of an online administered questionnaire are equal to the ones of its offline counterpart. Hence, several investigations analyzed the quality criteria of Web-based instruments for data collection and compared them to conventional paper-pencil methods (e.g., Fortson, Scotti, Del Ben, & Chen, 2006; Ritter, Lorig, Laurent, & Matthews, 2004). Read, Farrow, Jaanimägi and Ouimette (2009) analyzed the Web-based version of the Traumatic Life Events Questionnaire (TLEQ; Kubany et al., 2000) and of the PTSD
Checklist – Civilian Version (PCL-C, Weathers, Litz, Huska, & Keane, 1991) and compared them to paper-pencil versions of the same measures. They found significant correlations between related constructs in both administration modes as well as between each of the two measures and the Clinician-Administered PTSD Scale (CAPS-1; Blake et al., 1995). Fouladi, McCarthy, and Moller (2002) found only small and unsystematic effects of administration mode on outcomes of self-report questionnaires regarding parental attachment in adulthood and emotion regulation. These effects were further reduced when sex and ethnicity were controlled for. Furthermore, internal consistency and construct validity have shown to be sufficient for both administration types. Several other investigations failed to detect mode- based differences in reliabilities (Fortson et al., 2006; Ritter et al., 2004) and validities
(Buchanan & Smith, 1999a) of health-related self-report questionnaires. Finally, Gosling et al. (2004) made a very important remark about the comparison of online and offline methods. The authors pointed out that, in case of inconsistent findings, it is not justified to conclude automatically that the online method is the inaccurate one.
Intentional distortion of information. Researchers usually have less control over the context in which Web-based questionnaires are completed (e.g., at home, at work, in Internet-cafés, etc.) and they can rarely assess whether participants invested adequate time and effort to complete the questionnaire or whether they intentionally distorted their answers. These factors that are difficult to control may contribute to a reduction in the validity of the obtained data (Buchanan, 2000). On the other hand, Gosling et al. (2004) assumed that paper-pencil instruments are probably just as prone to deliberate falsification as Web-based methods. For detecting dishonest answers, Johnson (2001) suggests scanning data for long sequences of uniform answers. Additionally, an analysis of the time required to complete the questionnaire may point to participants who answered extremely quickly and thus most likely in a random way, perhaps even not having read the questions. Another method to detect willfully distorted answers is to analyze scale reliabilities and discriminant validities (Gosling et al., 2004). Random or dishonest answers would lower scale reliabilities, while they would cause discriminant validities to increase. This scenario would occur, for example, if individuals wanted to draw a particularly favorable picture of themselves. Gosling et al. (2004) compared online and offline-studies with regard to their reliabilities and discriminant validities and concluded that Web-based surveys are not affected to a greater degree by random or otherwise intentionally distorted answers.
Technical aspects. In order to enhance the comparability of data which were collected over the Internet, simple layouts and designs should be used which will be displayed accurately and in a similar manner on most computers (Whitehead, 2007).