In this section, both Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) are discussed because, as will become clear, both types were used in the course of study three.
Origins of Factor Analysis
Spearman (1904) established the use of factor analysis within psychometric testing, having concluded that there was an underlying ‘general mental ability’ that seemed to tie together apparently unrelated abilities within the test scores of school children. As such, factor analysis is a term given to a group of statistical tests whose primary aim is to reduce complex sets of data. It is characterised by the assumption that there are a number of latent (unobserved) variables within a data set which, if drawn out, would be able to explain the variance in the observed variables. There are two sub-types – EFA and CFA; the former is used when the researcher wishes to explore the data set, without any pre-conceptions as to what might exist there. CFA is characterised by the testing of a theoretical model, such that there is an attempt to ‘confirm’ pre-existing ideas.
Nature of Factor Analysis
Exploratory Factor Analysis (EFA)
Cattell and Kline (1977) argued that certain criteria must be met for the successful application of an EFA to be achieved, relating to the sample size, the variables to subject ratio, the number of factors to be extracted, and the choice of rotation.
There has been extensive research on the topic of how many participants are required in EFA – with arguments based either on the total number of participants, or
158 the ratio of subjects to variable items. For the total number of participants, Kline (2005) advocated a minimum of 100 participants whilst Comrey and Lee (1992) argue that 100 is a poor sample, 200 fair, 300 good and 500 very good, with 1000+ being excellent.
Concerning the subjects-to-variable (STV) ratio, Bryant and Yarnold (1997) argue for a rule of 5, with a subject-to-variable ratio of no less than 5:1. Kline (2000) argues for a ratio of 2:1, although he also mentions the work of Arrindell and van der Ende (1985) who claim that the STV ratio is less important than the Subject-to-Factor (STF) ratio which they argue that it should be more than 20:1.
There are a number of methods of deciding how many factor to extract from an EFA. Typically eigenvalues have been used to determine the number of factors retained, with Guttman (1954) arguing that all factors with an eigenvalue over 1 should be
maintained (Kaiser’s criterion). However, Hayton, Allen, and Scarpello (2004) argued that there are problems associated with it – first, that the rule is intended to be the upper boundary for the number of factors to be retained. However, often it is taken as the exact number to be kept. Second, the cut-off point is relatively arbitrary (Fabrigar, Wegener, MacCallum, & Strahan, 1999). Another method is to use Cattell’s (1978) Scree Plot, where a few major factors account for the majority of the variance producing the
‘cliff’ effect, with a multitude of smaller factors as the scree. However, the decision as to where the scree starts is both subjective and ambiguous, and a scree plot should only be in conjunction with other criteria (Hayton et al., 2004).
As a product of the criticisms of Kaiser’s criterion and the Scree Plot, Parallel Analysis (Horn, 1965) was primarily used in this thesis. Parallel analysis argued that the real data should have an underlying factor structure which will be revealed when compared to random data using the same sample parameters, such that “nontrivial components from real data with a valid underlying factor structure should have larger eigenvalues than parallel components derived from random data having the same sample size and number of variables” (Hayton et al., 2004, p.194). In practice, the eigenvalues of both data sets are compared, to test out the value of each eigenvalue against its paired partner (i.e. the first eigenvalue of the real data set is compared to the first eigenvalue of the random data set). At the point there the random eigenvalue is higher than the real data set eigenvalue, the factor is rejected, since it indicates that the model value is effectively random and could be as much sampling error as true variance.
159 The process of identifying factors requires the rotation of factors to such a position as best explain the data (Kline, 2005). With orthogonal rotation factors remain uncorrelated with each other, whilst in oblique rotation the correlation of factors is allowed.
With EFA, there is a best fit for explaining the given correlations in a rotation, known as the ‘simple structure’. Simple structure is a state where variables load either near 1 (perfect loading) or 0 (no loading) onto a factor, so clearly indicating which variables are important in the interpretation of a factor. There are a number of
advantages to having simple structure – first, simple structure factors usually have a few high loadings and are therefore being easier to interpret (Kline, 2005). Secondly, the factors in a simple structure are replicable, an important factor in establishing rigour (Cattell, 1978).
Confirmatory Factor Analysis
CFA seeks to test an a priori model, with the expectation of a number of factors and particular structure being represented in the final model. In CFA a number of statistical tests are applied to the model produced, not with the intention of declaring a model as being ‘correct’ but indicating that the model is a plausible means of explaining the variation shown in the data. Kelloway (1998) writes that there are two traditions in terms of assessment of model fit – Absolute Fit and Comparative Fit, with the latter being broken down into Assessments of Comparative Fit, and of Parsimonious Fit.
Absolute fit reflects the ability of the model to reproduce the actual covariance matrix, whilst an assessment of Comparative Fit compares two or more models to see which fits the data better. Finally an assessment of Parsimonious Fit argues that the better fit can always be achieved through the addition or elimination of pathways within the computed model Kelloway (1998). It has been argued that it is prudent to report goodness-of-fit criteria across these assessment types (Kelloway, 1998). Following the recommendations of Bryne (2012) the following indices were used, with their
membership of the fit measures being indicated:
160
Chi square ratio (X2/df; Absolute Fit)
Root Mean Square Error of Approximation (RMSEA; Absolute Fit)
Standardized Root Mean Square Residual (SRMR; Absolute Fit)
Comparative Fit Index (CFI; Comparative Fit)
Tucker-Lewis Index / Non-normed Fit Index (TLI / NNFI; Comparative Fit)
The Chi square (X2) indices is affected by sample size, so making it increasingly unlikely for the Chi square value to be accepted as the sample size increases (Joreskog, 1969). As such, the chi square ratio (X2/df) was used in this study. Kelloway (1998) warns that there is little consensus on the appropriate levels of fit using the Chi square ratio and advocates caution, though later work by Brown (2011) reports a ratio of less than 2 as being good, with a ratio of less than 3 as being acceptable, and greater than 5 as non-acceptable. The Root Mean Square Error of Approximation (RMSEA - J.
Steiger, 1990) was also reported, with Hu and Bentler (1999) reporting that a figure close to .05 or less as being an useful cut-off, though 0.08 is also seen as acceptable (Brown, 2011). The Comparative Fit Index (CFI: Bentler, 1990a) was also used, with a cut-off value of above .90. Hu and Bentler (1999) recommend the use of the
Standardized Root Mean Square Residual (SRMR: Joreskog & Sorbom, 1981), with a cut-off value close to .8 (Hu & Bentler, 1999). Finally, the Tucker-Lewis index (TFI or NNFI - Tucker & Lewis, 1973) was used, with a cut-off of 0.90 (Hu & Bentler, 1999).
Certain common measures such the Goodness-of-Fit index (GFI) and Adjusted Goodness-of-Fit index (AFGI) were not included because, as indicated by Marsh, Balla and McDonald (1988) and Anderson and Gerbing (1984), they are both affected by sample size. As the sample size increases, larger values of GFI and AFGI are required to indicate an acceptable model fit resulting in a situation where, with a suitably large sample size, any model might be seen as having a poor goodness-of-fit. Table 5.3 provides a summary of the appropriate cut-off levels.
161 Table 5.0.3: Cut-off measures for Goodness of Fit Indices
Measure Target Score Reference
Chi square ratio Good < 2 Acceptable < 3 Not acceptable > 5
Joreskog (1969).
RMSEA Good < 0.05
Acceptable < 0.08 Marginal < 0.10 Bad > 0.10
Steiger (1990); Hu and Bentler (1999).
CFI Good > 0.90 Bentler (1990a).
Standardized RMR Good < 0.05 Acceptable < 0.08
Joreskog (1981).
NNFI (TLI) Acceptable > 0.90 Good > 0.95
Tucker (1973).
RMSEA, Root Mean Square of Approximation; CFI, Comparative Fit Index; SRMR, Standardized Root Mean Square residual; TLI (NNFI), Tucker-Lewis Index (Non-normed Fit Index);
Reliability
Reliability relates to the estimation of the accuracy of a questionnaire (Rust &
Golombok, 2009). Cronbach (1951) developed the test statistic Cronbach’s Alpha, where he proposed that a test be broken into a series of sub-tests, with each sub-test being one test item in length – so resulting in X number of subtests. Cronbach
correlated each sub-test with the others and created an average correlation which acts as an estimate of reliability. Values over 0.7 are seen as an indication of having a test with acceptable levels of internal consistency, though Cronbach’s alpha values over 0.9 should be treated with caution (Kline, 2005). This suggests that inter-item correlations are too high, with item redundancy.
Rationale for Choosing Factor Analysis
Studies One and Two used qualitative analysis to create a theoretical model for the presentation of self in MMOs; however, to establish whether the model was representative of the underlying data within the general population, it was necessary to
162 analyse the data from a quantitative version of the model. Factor analysis was an
appropriate way of achieving this, since it could reduce a large amount of data to find any underlying factors that might account for the variance in the model – with EFA and CFA being the two variations available.
With respect to EFA, there has been extensive debate on whether principal components analysis or maximum likelihood factor analysis should be used as the method of extraction; Bentler and Kano (1990b) and Floyd and Widaman (1995), amongst others, insisted that principle components analysis is not a true factor analytic method, but only a data reduction method. Others, such as Steiger (1990), argued that there is no discernible difference between PCA and EFA. A significant criticism by Ford, MacCallum and Tait (1986) was that principle components analysis is that components are calculated using all the variance available in the variables – including error and unique variance. As a consequence of these criticisms, maximum likelihood EFA was used.
For CFA, following the argument by Fabrigar, Wegener, MacCallum and Strahan (1999) who said that maximum likelihood factor analysis is best used because
“it allows for the computation of a wide range of indexes of the goodness of fit of the model” (p.277), maximum likelihood was used as the means of analysis.
Factor analysis is a powerful analytic technique, which can clear away irrelevant information and inform theorising (Eysenck, 1991), but it is not without its faults.
Given that the underlying mechanism looks at correlations between variables, no causal inference can be established; in itself this is not problematic, but the technique does require a degree of subjectivity in the choice of factor analysis, type of rotation, and number of factors, etc. which might allow experimenter bias to creep in (Eysenck, 1991). This possibility can be diminished by remembering that the results of a factor analysis must be placed within the context of existing research, so being cautious in extrapolating too far from the study results (Widaman, 1993). More generally though, there has been research questioning whether the identification of general factors (particularly at population levels) can reveal anything useful about psychological
structures on an individual level (Cervone & Pervin, 2007). As Borsboom, Mellenbergh, and van Heerden (2003) argued, populations and individuals are very different things, with the only reliable way of verifying a model being to test it on each individual.
Obviously this would be unfeasible and, in some respects, antithetical to the concept of
163 factor analysis as a data reduction technique however, it should be acknowledged that in creating increasingly generalised accounts of human interaction, some detail is lost in the process.
5.2.2 Research Procedure
As with Studies One and Two, to ensure transparency in the research process this section provides details the sampling strategy used to acquiring participants, how they were recruited, and ethical information concerning the anonymity of the
participant, and the security of the data.
Sampling Strategy
As with Studies One and Two, Study Three employed a purposive expert sampling methods. This was due to the necessity of having only gamers complete the online questionnaire, since they needed to possess knowledge specific to the playing of games in order to answer the questions accurately. It is possible that the participants were not active gamers, but this would not have been problematic, since the research was not looking exclusively at the presentation of the current self in MMOs. As with existing research (Osborne, 2012), gamers were simply required to have had gaming experience.
Recruiting Participants
Given the impetus to gather as many participants as possible, the questionnaire was advertised across a number of game forums and Facebook groups (with the permission of forum and group administrators) to order to gather participants. The full list of forums and groups can be found in appendix XI – it should be noted that
particular effort was made to include smaller MMOs since not all gamers play the major MMOs (e.g. World of Warcraft, Rift, Star Wars: The Old Republic, etc.). The
questionnaire advertisements were active for seven weeks.
164 In total 42 forums and three Facebook groups were canvassed, with the result that 679 participants contributed to the study. To check for the possibility of individuals doing the questionnaire multiple times, the ISP number of each participant (provided automatically by the online company) were checked for duplicates. Aside for duplicates for responses from the student cohort, as might be expected if University computers were being used, no suspicious activity was detected.
Of this number, 44 were removed from the dataset as a product of their demographic response indicating they were below the minimum age of 18 years; one participant provided a spurious age (99 years) so creating sufficient uncertainty as to remove his / her data. An additional 223 participants were removed from the study, either for having insufficiently completed data sets (i.e. 50% completion) or for having provided no age at all. Finally, three participants were removed as part of the internal checks on the data, since their answers indicated that the questionnaire had been completed randomly. Overall, the participant’s cohort was made up of 408 participants, after partially completed data sets were imputed to fill the missing data.
In appendix XII the statement provided in the initial thread of the game forums can be found. There are three versions of the initial statement, depending on whether the game required payment to play, was free to play, or if the statement was being sent to a student population. All participants who took part in the research were entered into a prize draw, if they so wished, with a main prize of £100 of Amazon vouchers, with four runners-up prizes of £25 worth of Amazon vouchers. These were allocated at the end of the study by giving each ‘willing’ participant a number, and then using a random number generator to ‘pull out’ the winning numbers.
Anonymity, Confidentiality, and Informed Consent
The ethics for the third study followed the code of ethics laid down by the BPS and NTU ethics committee. Since Study Three involved no direct contact with the participants all statements pertaining to ethical matters required a separate page in the questionnaire. As can be seen from the structure of Study 3 in appendix XIII, the initial pages of the questionnaire provided the participants with relevant information required to make an informed decision. This included information concerning the anonymising of personal data, the procedure for the withdrawal of data, and concerning the security
165 and destruction of the study data. Having been told this, and been provided with
additional ethics material, participants were required to tick a box should they agree to the ethical conditions and wish to carry on. They were also required to provide a Unique Identifier so that, in the event of wishing to withdraw their data, they could be identified.
At the end of the questionnaire, participants were again reminded what the topic had been and were provided with contact details for assistance should they wish to talk to a third party. In addition they were provided with my email address should they wish to make contact for any reason.
5.2.3 Data Collection
Development of the Online Questionnaire
The questionnaire for the study was created directly from the concepts created in the first two studies. Initially there were 44 concepts in the model but it was decided that to include all 44 would have been excessive since each concept required multiple items to establish reliability (Bollen, 1989). Some concepts also had multiple
dimensions, which could have resulted in some concepts having upwards of 6 questions associated with them. Such a structure would have produced upward of 150 test items, which would have produced an unwieldy test.
Theoretically there could have been a number of tests constructed for different sub-sections of the model, but as Streiner and Norman (2008) argued, that level of specificity might have produced a series of tests that suffered from a lack of
generalizability. In addition, well designed general questionnaires often yield comparable results to specific questionnaires (Parkerson et al., 1993).
The original 44 concepts were considered for their Uniqueness and Supporting evidence, such that inclusion in the study three questionnaire would both enrich the quantitative data set by representing all facets of the model, but with sufficient evidence as to justify their inclusion. On this basis the concepts in the theoretical model were sorted into one of four categories, as can be found in appendix XIV.
In total there were 19 concepts, 31 associated properties of those concepts, with 66 questions arising from those properties, with an addition 12 demographic questions.
166 For the creation of the questions, it was important to stay close to the qualitative work, so the concept statements were drawn directly from the Grounded Theory concepts.
Generally there were two statements for each concept, but more complex concepts had three.
Participants were given a five point Likert scale with which to provide their answer, ranging from Strongly Disagree, through Neutral, to Strongly Agree. Following good research protocols, some questionnaire statements were phrased to be opposite of the concept meaning.
When the raw scores were compared against the normally coded statement for that code, it was easier to check if the respondent was exhibiting response bias by randomly filling out the questionnaire (Nardi, 2006). If this was found the data was removed from the data set. The questionnaire was created using a commercial online survey website – http://www.kwiksurveys.com/. As outlined earlier on, the use of the online
environment allowed for the distribution of a questionnaire across a wide spectrum of potential participants (Wood, Griffiths, & Eatough, 2004). With respect to the questions themselves, a randomised pattern was used to distribute the questions – this was
achieved by allocating the 1st, 8th, 15th, etc. question to the first page, and the 2nd, 9th, and 16th to the second, etc. This was done to ensure that no two questions relating to the same concept would be on the same page, so reducing any potential carry-over
achieved by allocating the 1st, 8th, 15th, etc. question to the first page, and the 2nd, 9th, and 16th to the second, etc. This was done to ensure that no two questions relating to the same concept would be on the same page, so reducing any potential carry-over