The independence model (C3 in Table 5-1) corresponds to the position that response styles do not generalize across different sets of items. The other two models in column C of Table 5-1 correspond to the position that response styles are unstable (no
common factor) and only have a local effect (the autoregressive coefficient), which can be time variant (C1) or time invariant (C2). Model B3 corresponds to the stance that all sets of items are affected only by a common response style factor and this with equal strength for all sets. This model is assumed when constraining response style factor loadings to one for different (sets of) items (Billiet and McClendon 2000; Mirowsky and Ross 1991). The other B models hold the latter assumption too, but allow for an additional autoregressive component of response styles. Model A3 assumes a single underlying response style that may have a different impact on different sets of items (Greenleaf 1992a, 1992b; Watson 1992; Baumgartner and
Steenkamp 2001). The other models in column A again present hybrid extensions of this model that may be important because the autoregressive model and the common factor model are not mutually exclusive, but seem to have been treated as such in the literature nonetheless.
Models that are in the same row or column are nested within one another, that is, the set of freely estimated parameters of each model is a subset of those estimated in the model(s) preceding it in the same row as well as the model(s) preceding it in the same column. Note that A1 is not nested in any other model. This model is overly liberal, in that for small numbers of sets (like in the current study, where k=4 or k=5), the degrees of freedom are limited. This model will mainly serve as a reference model. Each model is estimated for each response style and evaluated in three major ways. As pointed out by Marsh, Hau and Wen (2004), meeting common goodness-of-fit cutoff criteria is not a sufficient criterion for having a valid model. Goodness-of-fit criteria usually perform better in comparing alternative models based on the same data (Marsh, Hau and Wen 2004). Therefore the different models are also evaluated with respect to one another. Additionally, the theoretical viability, statistical significance and substantial size of the parameter estimates are assessed.
To sum up, first, model fit of the stand-alone models will be evaluated. Second, model fit will be evaluated relative to the other models (taking into account nesting). Third, the substantive meanings of the model estimates are appraised. Each of the three steps is now discussed in more detail.
Absolute model fit
The chi square statistic allows for a formal test of model fit. However, since some sample sizes are large enough to expect some oversensitivity of the chi square test statistic (Marsh, Balla and McDonald 1988), alternative fit indices are also taken into
account (Hu and Bentler 1999). The RMSEA (Root Mean Square Error of
Approximation, Steiger 1990; Browne and Cudeck 1993) takes into account model complexity by dividing the minimum discrepancy by the number of degrees of freedom for testing the model. This is important since the number of parameters relative to the number of distinct sample moments varies widely over the models and parsimony is considered a plus. Additionally, the confidence intervals around the RMSEA estimates are helpful in comparing models. The CFI (Comparative Fit Index; Bentler 1990) is particularly relevant in this context since it evaluates the decrease in misfit (captured by the noncentrality parameter) relative to the independence model, i.e. model C3. This means that the CFI of model C3 will be zero by definition, while a saturated model will have a CFI of 1. The range and meaning of the CFI precludes its use in assessing model C3, but if the latter model is rejected based on other criteria, the CFI becomes useful in assessing how well the other models account for the covariances between the indicators that are constrained to zero in model C3. Values close to 1 indicate very good fit, .95 is commonly used as a cut-off value (Hu and Bentler 1999). The CFI and RMSEA are two alternative fit indices often referred to by experts (e.g. Flora and Curran 2004).
Relative model fit
Since models in the same column or row are nested, nested chi square difference tests are performed. Here again, chi square may be oversensitive due to the sample size (in the primary data). Therefore, a decrease in CFI equal to or higher than .01 is evaluated as indicative of a relevant deterioration in fit (Grouzet, Otis and Pelletier 2005), a decrease of .05 or more as a substantial non-acceptable deterioration in fit (Little 1997; note however, that this recommendation was based on multi-group invariance tests; generalization to the current setting is therefore somewhat tentative). Another
marker of a substantial deterioration of fit is the extent of separation/overlap between RMSEA confidence intervals.
Estimates
In addition to the above evaluations of model fit, model estimates were evaluated by checking whether the relevant estimates were significantly different from zero and were signed in the expected direction. In particular, in the congeneric and tau- equivalent models (all models A and B), factor loadings were expected to be significantly positive. If the loading of a specific response style indicator was not significantly different from zero, this would imply that the indicator in question is not significantly related to a common response style factor. If its loading is negative, this would indicate that higher levels of response styles in other sets of items are
predictive of lower levels of response styles in the set in question. In the
autoregressive models (all models 1 and 2), the autoregressive weights were expected to be significantly positive. A similar reasoning applied here. If the autoregressive coefficient of a specific response style indicator was not significantly different from zero, this would imply that the indicator in question was not significantly related to the previous indicator. If its coefficient is negative, this would indicate that higher levels of response styles in the previous item set is predictive of lower levels of response styles in the set in question. In addition to the evaluation of significance, size and direction of the loadings and autoregressive coefficients separately, the relative size of the estimates related to autoregression were compared with those related to a common factor.
RESULTS
The correlation matrices provided by Hui and Triandis (1985) were analyzed using a ML estimator (MPlus version 4; Muthén and Muthén 2006). The primary data were
analyzed using a FIML estimator which takes into account missing values (Amos 5.0.1; Arbuckle 1994-2003). All proposed models were fit to four correlation matrices (NARS and ERS in H&T1; NARS and ERS in H&T2) and four covariance matrices (ARS, DRS, ERS and MRS for the primary data). It was chosen to estimate a separate model for each response style to get results that could be directly compared to the results obtained from the H&T data and because this allowed being very specific about what causes misfit in the models. Also, the scenario where data on different response styles fit different models is considered a possibility. Note that model A1 (the time variant autoregressive congeneric model) cannot be estimated with four indicators because this would result in negative degrees of freedom. Hence, model A1 was not estimated for NARS and ERS in H&T1. All other models were identified and the estimations converged without any problems. There were no instances of
inadmissible solutions.