The effort subscale consisted of seven items that were intended to measure the extent to which the work unit devotes constant attention towards work, uses resources like time and care in order to be effective on the job, shows willingness to keep working
under detrimental conditions and spends the extra effort required for the task.
5.6.6.1 Item analysis
The results for the item analysis are depicted in Table 5.16. A satisfactory (>.80) Cronbach’s alpha of .853 was obtained. This indicates that approximately 85.3% of the variance in the items was systematic or true score variance and only 14.7% was random error variance.
The item means ranged from 3.847 to 4.213 on the five-point Likert scale and the item standard deviations ranged from .779 to .938. This indicates that most participants rated their work unit on this competency by choosing the second-highest response option. None of the item distributions were truncated due to extreme means. The items of the subscale were able to detect reasonably small differences in the level of competence that the participants’ work units achieved on the effort competency. The inter-item correlations in the correlation matrix shown in Table 5.16 ranged between .213 and .639, and the mean was .462. Item Q14 consistently correlated lower than the mean inter-item correlation with the remaining items in the subscale. Item Q14 therefore reflected to a different source of systematic variance than the remaining items of the subscale which caused it to respond out of step with the remaining items. The corrected item-total correlations in the item-total statistics section of Table 5.16 ranged from .374 to .727 and were above the cut off (<.3). The squared multiple correlation ranged from .177 to .565 and are considered satisfactory. Item Q14 showed itself as an outlier in the corrected item-total correlation distribution and, especially so, in the squared multiple correlation distribution. Item Q14 therefore was a bit of a closed book to its colleagues in the sense that they could between them only explain circa 18% of the variance in item Q14.
Furthermore, the results revealed that item Q14 from the subscale would increase the current Cronbach alpha if deleted. The deletion of item Q14 improved the internal consistency of the subscale because item Q14 and the remaining items of the subscale responded to different sources of systematic variance. However, based on the aforementioned findings, it was decided that the item statistic evidence against the
item was not severe enough to warrant the immediate deletion of the item Q14 and that the item would be kept for further analyses. Therefore, based on the basket of evidence, none of the items were deleted from the scale. Item Q14 was, however, flagged for specific critical scrutiny in subsequent studies
Table 5.16
Item analyses output for the effort subscale
Reliability Statistics Cronbach's
Alpha Cronbach's Alpha Based on Standardized Items N of Items
.853 .858 7 Item Statistics Mean Std. Deviation N Q14 4.01485 .938283 202 Q15 3.84653 .914892 202 Q16 4.19802 .858320 202 Q17 4.21287 .874875 202 Q18 4.09406 .878751 202 Q19 3.88119 .789069 202 Q20 4.15347 .779854 202
Inter-Item Correlation Matrix
Q14 Q15 Q16 Q17 Q18 Q19 Q20 Q14 1.000 .252 .213 .263 .288 .365 .384 Q15 .252 1.000 .558 .513 .476 .381 .479 Q16 .213 .558 1.000 .580 .503 .395 .512 Q17 .263 .513 .580 1.000 .621 .556 .543 Q18 .288 .476 .503 .621 1.000 .561 .639 Q19 .365 .381 .395 .556 .561 1.000 .628 Q20 .384 .479 .512 .543 .639 .628 1.000 Item-Total Statistics Scale Mean if Item
Deleted Scale Variance if Item Deleted Corrected Item-Total Correlation Squared Multiple Correlation Cronbach's Alpha if Item Deleted Q14 24.38614 15.781 .374 .177 .870 Q15 24.55446 14.477 .593 .398 .836 Q16 24.20297 14.640 .619 .457 .832 Q17 24.18812 14.094 .698 .535 .821 Q18 24.30693 14.065 .699 .536 .820 Q19 24.51980 14.878 .648 .486 .829 Q20 24.24752 14.516 .727 .565 .819
Table 5.16
Item analyses output for the effort subscale (continued)
Summary Item Statistics
Mean Minimum Maximum Range Maximum /
Minimum Variance Items N of Item Means 4.057 3.847 4.213 .366 1.095 .022 7 Item Variances .746 .608 .880 .272 1.448 .010 7 Inter-Item
Correlations .462 .213 .639 .427 3.009 .017 7
5.6.6.2 Dimensionality analysis
All of the eight items in the effort subscale were factor analysed as they all produced satisfactory results in the item analysis but for item Q14.
The correlation matrix for the effort subscale indicated that most of the correlations were larger than .3, except for item Q14 that correlated poorly with items Q15 to Q18 (but reasonably with items Q19 and Q20), and that all the correlations were statistically significant (p<.05). Furthermore, a KMO of .876 (>.6) was obtained and the Bartlett's Test returned a statistically significant chi-square statistic (p<.05) that allowed for the identity matrix null hypothesis to be rejected. This presented strong evidence that the correlation matrix was factor analysable.
Only one factor obtained an eigenvalue greater than one (3.841), however the results hinted towards two factors as the second factor obtained an eigenvalue of .906. The scree plot further suggested that two factors should be extracted. The single-factor factor matrix revealed that all of the items, except item Q14, loaded onto one factor satisfactorily since all factor loadings were larger than .50 (λi1>.50) but for λ71=.405.
The greatest evidence pointing towards two factors was that nine (42%) of the nonredundant residual correlations obtained absolute values greater larger than .05. The unidimensionality assumption was thus not corroborated for the effort subscale. The extraction of two factors was subsequently requested and oblique rotation (direct oblimin) was utilised in an attempt to rotate the factor matrix to simple structure. The pattern matrix reflects the partial regression slope coefficients when regressing each
item on the two extracted factors. The pattern matrix contains the partial slope regression coefficients for the weighted linear combination of the latent variables, were partial regression coefficients reflect the effect of one factor on an item when statistically controlling the effect of the other factors that were extracted in both the item and the focal factor.
The pattern matrix therefore formally recognises that due to the oblique rotation correlations are likely to exist between the extracted factors and therefore they to some degree share variance. As shown in Table 5.17 items Q14, Q18, Q19 and Q20 all grouped together to load positively on factor 1. These items refer to time, commitment, energy investment and dedication. Therefore, based on common themes in these items, the first factor was interpreted as a giving/investing/applying the unit Factor. Items Q15, Q16 and Q17 grouped together to load negatively on factor 2. These items refer to care, perseverance and effort. Therefore, based on common themes in these items, the second factor was interpreted as a continuous focus factor. Item Q18 showed itself as a complex item. The two extracted factors correlated -.691 in the factor correlation matrix. The factor fission was regarded as subtle but nonetheless meaningful.
Furthermore, zero (0%) of the nonredundant residual correlations obtained absolute values greater larger than .05. This suggests that the two-factor model provided a valid and credible explanation for the observed inter-item correlation matrix.
Table 5.17
Pattern matrix for the effort subscale
Factor 1 2 Q19 .810 .025 Q20 .679 -.183 Q18 .495 -.339 Q14 .478 .040 Q16 -.079 -.855 Q15 .065 -.645 Q17 .301 -.532
Two courses of action were possible in response to the factor fission. The first course of action was to divide the effort subscale into two separate subscales designed to measure the two extracted effort factors (and to write additional items for each). The second possible course of action was to acknowledge the (unanticipated) multidimensional nature of the effort subscale and to evaluate the ability of the effort subscale items to validly reflect effort as a second-order factor. Forcing a single factor in the EFA was one possible option that was considered to evaluate the ability of the subscale items to validly reflect effort as a second-order factor. The current study, however, would want to question the methodological rigour of this procedure. Firstly, it is not clear in terms of the underlying logic of this procedure whether the single extracted factor should be interpreted as a second-order factor or multidimensional latent variable. Secondly, in as far as the percentage of large residual correlations represent an evaluation of the fit of the factor structure, and given that the forced single-factor factor structure typically fits poorly, the validity and credibility of the factor loadings come into question. The inference that all the items satisfactorily reflected a higher-order factor thus becomes unconvincing because of the inability of the single- factor factor structure to accurately reproduce the observed inter-item correlation matrix (Wessels, 2018).
A methodologically more rigorous approach seemed to fit a second-order measurement model and to evaluate the statistical significance of the indirect effects of the second-order effort factor on the individual effort subscale items. This seemed justifiable only if the first-order measurement model fitted. The first-order measurement model in which items Q14, Q18, Q19 and Q20 loaded only on factor 1, and Q15, Q16 and Q17 loaded only on factor 2 fitted the subscale data reasonably closely (RMSEA=.054, p>.05). All factor loadings were statistically significant (p<.05). The second-order measurement model in which items Q14, Q18, Q19 and Q20 loaded only on first-order factor 1, items Q15, Q16 and Q17 loaded only on first-order factor 2 and the two first-order factors loaded on a single second-order factor fitted the subscale data closely (RMSEA=.061; p>.05). The solution was, however, inadmissible due to a negative structural error variance estimate for factor 1 and a R² estimate larger than unity.
Setting starting values for 1j did not solve the problem. The modification indices
suggested, both in the first-order and the second-order model, that a path be added from factor 2 to Q18. This dovetailed with the EFA findings as shown in the pattern matrix (Table 4.17). Adding a path from factor 2 to item Q18 in the second-order effort measurement model produced an admissible solution and a close-fitting model (RMSEA=.049; p>.05).
The factor loadings and gamma estimates for the revised second-order effort measurement model are shown in Table 5.18 and in Table 5.19. The completely standardised solution for the second-order effort measurement model is shown in Figure 5.1.
Table 5.18
Unstandardised factor matrix for the second-order effort measurement model Factor 1 Factor 2 Q14 0.39 Q15 0.62 Q16 0.62 (0.57) 1.10 Q17 0.72 (0.59) 1.22 Q18 0.71 (4.68) 0.15 Q19 0.58 (3.59) 0.16 Q20 0.64 (4.58) 0.14
Table 5.18 indicates that all the factor loadings in the second-order effort measurement model were statistically insignificant (p>.05)23.
23 This finding stands in sharp contrast with the finding that the factor loadings in the first-order effort measurement
model were all statistically significant (p<.05). This raises the question exactly how the factor loadings should be interpreted in the second-order measurement model. More specifically it raises the question whether the factor loading estimates should be interpreted as estimates of the slope of the regression of the items on the first-order factors when controlling for the second-order factor?
Table 5.19
Unstandardised gamma matrix for the second-order effort measurement model EFFORT --- FAC1 .995794 (1.099794) .905437 FAC2 .797801 (.124738) 6.395800* * p<.05
Note: EFFORT refers to the effort latent variable
Table 5.19 indicates that 11 was statistically insignificant (p>.05) but that 21 was
statistically significant (p<.05).
Figure 5.1. The second-order effort measurement model (completely standardised solution)
The eight indirect effects were calculated by translating the second-order measurement model SIMPLIS syntax to LISREL syntax, requesting the calculation of eight additional parameters via the AP command on the model (MO) command line, calculating the eight products λijji via the CO command and testing the statistical
significance of these indirect effects. The unstandardized indirect effects, their standard errors and the corresponding z-scores are shown in Table 5.20.
Table 5.20
Unstandardised indirect effects for the second-order effort measurement model
PA(1) PA(2) PA(3) PA(4) PA(5) PA(6) PA(7)
0.42 0.75 0.62 0.69 0.49 0.50 0.57
(0.07) (0.07) (0.07) (0.07) (0.07) (0.07) (0.07)
5.94 10.70 8.72 9.72 7.01 7.03 8.08
Table 5.20 indicates that all the indirect effects were statistically significant (p<.05). This means that respondents standing on effort as a second-order factor statistically significantly (p<.05) affected the scores obtained on each of the eight items. This justified the use of all eight items of the effort subscale as indicators of effort interpreted as a second-order latent competency and in the calculation of two composite indicators for the effort latent variable in the model24.
5.6.7 Psychometric Evaluation of the Counterproductive