A pool of 59 items was generated by two researchers with expertise in regulatory focus theory and scale development. Items were constructed deductively i.e., based on a
comprehensive review of the regulatory focus theoretical literature (Higgins, 1997, 1998; Higgins et al., 1985) and were designed to measure all aspects of regulatory focus theory. The nurturance aspect of RFT was reflected in items like: “Whenever possible, I try to improve my current situation”. The security aspect of RFT was reflected in items like: “I prefer to maintain the status-quo”. The actual-ideal aspect of RFT was reflected in items like: “I strive to be my ideal self”. The actual-ought aspect of RFT was reflected in items like: “I view goals as duties I must complete”. The initial item pool (see Appendix B for full list) included 16 items designed to measure nurturance, 14 items designed to measure security, 14 items written to measure actual- ideal orientation, and 15 items designed to measure actual-ought orientation. Prior research (Haws, 2010; Summerville & Roese, 2008) has questioned the psychometric properties and item quality of existing measures of regulatory focus. Thus, for the present study, no items from existing measures of RFT were used; all new items were written. The item pool contained only forward-scored items as prior research suggests that reverse-scored items may lower scale validity (Schriesheim & Hill, 1981), introduce systematic error to a scale (Jackson et al., 1993) and may even result in an artificial response factor consisting of negatively worded items (Schmitt & Stults, 1985). All items were written to be comprehensible to individuals with no prior experience or exposure to regulatory focus theory.
theory, one academic psychologist familiar with regulatory focus theory and with expertise in testing and assessment, and one doctoral candidate in I/O Psychology who is familiar with regulatory focus theory and with good item writing techniques. The SME’s read a brief
description of regulatory focus theory and then rated each item based on whether they thought it represented a promotion item, a prevention item, or other. SME’s also provided feedback on the quality and clarity of the wording of each item. Items that exceeded 80 percent agreement among raters and that contained no wording ambiguity were retained for the final item pool. The final item pool consisted of 39 items and is shown in Table 1 (shaded items were retained, non-shaded items were eliminated). Each reviewer’s rating of the items is presented (pro = promotion item; pre = prevention item; oth = other) along with an aggregate agreement rating (a rating of 5 indicates 100% consensus on the item), notes, and whether the item was included in the final scale (indicated by a Y or N in the Final Scale column).
Table 1. Subject matter expert ratings of initial regulatory focus items. Item # Item Final Scale? Rev 1 Rev 2 Rev 3 Rev 4 Rev 5 Agree Notes
38 When I succeed I feel a sense of joy Y pro pro pro pro pro 5
1
I think about how I will achieve my hopes and
aspirations Y pro pro pro pro pro 5
3 I strive towards my ideals Y pro pro pro pro pro 5
4
I feel a strong sense of duty to those around
me Y pre pre pre pre pre 5
5
I am more focused on achieving success than
avoiding mistakes Y pro pro pro pro pro 5
6 I hate making mistakes Y pre pre pre pre pre 5
8 I strive to meet my obligations Y pre pre pre pre pre 5
10 I play not to lose Y pre pre pre pre pre 5
11 Success to me is achieving my aspirations Y pro pro pro pro pro 5
13 I am comfortable with change Y pro pro pro pro pro 5
14 I prefer to maintain the status-quo Y pre pre pre pre pre 5
15 I think about how to achieve an optimal life Y pro pro pro pro pro 5
17 I like winning more than I dislike losing Y pro pro pro pro pro 5
23 I pursue goals eagerly Y pro pro pro pro pro 5
24 I pursue goals cautiously Y pre pre pre pre pre 5
30 I pursue ideals in life Y pro pro pro pro pro 5
33 I dislike losing more than I like winning Y pre pre pre pre pre 5
34 I view goals as opportunities not to be missed Y pro pro pro pro pro 5
35 I view goals as duties I must complete Y pre pre pre pre pre 5
36 I like new opportunities Y pro pro pro pro pro 5
39 When I succeed I feel a sense of calm Y pre pre pre pre pre 5
40 I play to win Y pro pro pro pro pro 5
42 When I fail I feel dejected Y pro pro pre pro pro 5
43 When I fail I feel anxiety Y pre pre pre pre pre 5
46 I strive to be my ideal self Y pro pro pro pro pro 5
47
I am concerned that I will fall short of my
duties Y pre pre pre pre pre 5
49
I think about how I can prevent failures in my
life Y pre pre pre pre pre 5
50 Advancing in life is important to me Y pro pro pro pro pro 5
52
Whenever possible, I try to improve my
current situation Y pro pro pro pro pro 5
53
I am focused on preventing negative events in
my life Y pre pre pre pre pre 5
54 I savor winning Y pro pro pro pro pro 5
57 I am a cautious person Y pre pre pre pre pre 5
59
I am really hard on myself when I make
mistakes Y pre pre pre pre pre 5
60 I have lofty aspirations for life Y pro pro pro pro pro 5
9 When I make a mistake I feel sad Y pro pro pro pro pro 5
7 I tend to be a spontaneous person N oth pro pro pro oth 3
16 I tend to be a responsible person N oth pre pre pre oth 3
19 I live a fast-paced life N oth pro oth pro pro 3
27 I pay attention to the big picture N oth pro oth pro oth 3
37 I pay attention to details N oth pre oth pre pre 3
56 I am a carefree person N oth pro pro pro pro 3
58
I like to strike out on my own by setting my
own goals N oth pro oth pro pro 3
21 I like changes in life N pro pro pro pro oth 4
22 I prefer not to take risks in life N oth pre pre pre pre 4
25 I see mistakes as opportunities to grow N oth pro pro pro pro 4
26 I live a slower-paced life N oth pre pre pre pre 4
28 I take risks in life N oth pro pro pro pro 4
32 I do not like missing opportunities N pro pro pro pro pre 4
41 I usually follow the rules N oth pre pre pre pre 4
44 I am achievement oriented N oth pro pro pro pro 4
2
Whenever possible, I try to maintain my
current situation N pre pre pre pre pre 5 vague
12 I see mistakes as betraying my duty N pre pre pre pre pre 5 awkward
29 I like the routines of life N pre pre pre pre pre 5 vague
31 I pursue obligations in life N pre pre pre pre pre 5 vague
45
I am more focused on avoiding mistakes than
achieving success N pre pre pre pre pre 5
51 I strive to be the person I am obligated to be N pre pre pre pre pre 5 awkward
Participants
An online (Qualtrics) sample of 619 individuals responded to the initial set of scale items during the week of April 24th 2017. Researchers (Hinkin, 1995; Tabachnick & Fidell, 2007) recommend a subject to item ratio of at least 10:1 for scale development studies. This is especially important when using multivariate techniques like factor analysis because results have been shown to be sample specific with smaller sample sizes (Schwab, 1980). Utilizing a large sample size may ameliorate some of these issues. Further, a large sample is necessary so that final CFA results can be validated against an adequately sized hold out sample. Scholars have also raised concerns about online samples (e.g., Couper, 2000), arguing mainly that they suffer from self-selection bias and thus limit generalizability. However, Highhouse & Gillespie (2009), cite findings from five meta-analyses and make an empirical argument that the specific nature of the sample does not impact the (theoretical) generalizability of research findings. These findings, coupled with the ease of data collection associated with online samples, and the fact that online samples are likely more diverse (demographically) than a typical introductory psychology course sample, justifies the use of an online sample for the current research.
On average, participants were female (70.3%), employed either part or full time or retired (76.5%), 48 years old, making between $50,000 and $60,000 annually. A hold-out sample of 151
There were no significant demographic differences between the main sample and the holdout sample. The hold-out sample was used to validate the final CFA models described below.
Procedure
Survey respondents were asked to complete all questions and were given the following prompt “The following section contains questions pertaining to your general approach to life. When answering each question think about what is typically true for you. For example if a statement is very typical of you, answer ‘strongly agree’; if a statement is not very typical of you, answer ‘strongly disagree’. Participants were asked to rate their responses on a Likert scale with the following options: 1 (Strongly Disagree), 2 (Disagree), 3 (Neutral), 4 (Agree) or 5 (Strongly Agree). Administration of the online survey constituted the data collection procedures approved by the Kansas State University Institutional Review Board and participants indicated informed consent before participating. The order of the items presented to the participants was randomized to prevent any potential problems with order effects.
Chapter 3 - Results
Item reduction and preliminary model testing
A reliability analysis of the original pool of 39 items was examined to guide the first stage of item reduction. Ten items that were lowering the internal consistency of the promotion and prevention subscales (‘alpha if removed’ > .80) were eliminated. Two survey items were redundant (i.e., the items were identical) to other items and were also eliminated. Thus, a
preliminary item pool of 27 items was retained. Data screening procedures following Tabachnick & Fidell (2007), were conducted using SPSS programs. There were no missing data. Linearity, normality, and homoscedasticty assumptions were checked and confirmed using a normal
probability plot of standardized residuals (to test linearity), a regression of standardized residuals onto a dummy dependent variable (to test homoscedasticity), and a visual check of the normal curve superimposed over histograms for each item to “test” normality. Multicolinearity was evaluated using a conservative variance inflation factor (VIF) threshold of three (larger VIF values indicate higher standard error). Using an iterative process, 38 of the 39 independent variables (IV) were regressed onto one IV which was used as a dependent variable (DV). All VIF values were below 3 indicating no issues with multicolinearity in the data set. To check for univariate outliers, item scores were converted to zscores. Outliers were those cases that
exceeded the z +/- 3.29 criterion. There were 43 univariate outlier cases which were deleted from the data set. Multivariate outliers were checked using Mahalonobis Distance (MD), which were computed by regressing all 27 scale items against a dummy dependent variable (ID#). MD values were then compared against a chi square distribution table. Any values that fell below a p < .001 were identified as outliers. There were 25 multivariate outliers which were deleted from
the data set. Thus, 551 cases were retained for model development and confirmatory factor analysis (CFA).
As mentioned, 551 cases were retained for model development and CFAs. Factor models were tested with the structural equation modeling software program AMOS 18 (Arbuckle, 2009) using maximum likelihood estimation (MLE). MLE is the most commonly used estimation technique and made sense in the current research because data was normally distributed. A primary concern in CFA is assessing the fit of the proposed model(s) to the data (more
specifically, the fit between the sample and estimated population covariance matrices). “Fit”, as noted by Tabachnick & Fidell (2007), is commonly tested against a non-significant χ2 as the criterion. However, with large sample sizes (such as with the current research) trivial differences between the sample and estimated covariance matrices are often significant because the
calculation of χ2 involves multiplying the minimum of the function by N – 1 (in the current research N – 1 = 550). Thus, researchers have proposed a variety of other descriptive model fit indices; and which indices are used is a matter of personal preference for the researcher
(Tabachnick & Fidell, 2007). Fortunately, good-fitting models tend to produce consistent result on the various indices, so if several indices suggest good model fit to the data, chances are the other indices will be good as well. For the current research, the following test statistic and
indices were used to assess goodness of fit: χ2, χ2 / df comparative fit index (CFI), goodness-of-fit index (GFI), root mean squared error of approximation (RMSEA), and the standardized root mean square residual (SRMR). The CFI is important because it is a relative index i.e., it compares the estimated population covariance matrix against both an independent model
(comprised of unrelated variables) and a saturated (perfect fitting) model. The resulting fit index lies in the 0 to 1 range with values approaching 1 considered optimal. Hu & Bentler (1999),
recommend a CFI value of .95 (or higher) as indicative of a good fitting model. RMSEA is also important because it provides a slightly different fit assessment. RMSEA estimates the lack of model fit compared to a saturated (perfect) model. When model fit is perfect the RMSEA yields a value of zero; as model misspecification increases the RMSEA value increases indicating a poor fitting model. Hu & Bentler (1999), recommend a RMSEA value of .06 (or less) as
indicative of a well-fitting model. SRMR is also commonly used in assessing model fit because it uses the residual values from the comparison of the sample and estimated population covariance matrices. Good fitting models will have small residual values because the sample covariance matrix and the estimated population covariance matrix approximate one another. The SRMR values range from 0 to 1, with values of .08 or less indicating good model fit (Hu & Bentler, 1999). The final fit index used in the current research (GFI) is analogous to R2 in regression in that it calculates the proportion of variance in the sample covariance matrix that is explained by the estimated population covariance matrix (Tabachnick & Fidell, 2007). Values range from 0 to 1 with values of .95 or higher indicating greater variance explained, and thus a better fitting model. In sum, the current research uses a variety of model fit indices as criteria for testing model fit: χ2, χ2/df F <= 2 (Tabachnick & Fidell, 2007); for CFI, GFI, RMSEA, and SRMR the current research follows the cut-off recommendations of Hu & Bentler (1999) mentioned above.
The twenty seven items that were retained from the initial item reduction were divided into their theoretically-derived categories to test both a four factor and a two factor model of regulatory focus. The first CFA models were tested on the main sample (N=400) with two first- order latent factors (promotion regulatory focus and prevention regulatory focus) each indicated by the measured items that reflect promotion and prevention regulatory focus. Both factors were tested simultaneously. The 27-item model demonstrated moderate fit.
After exploring the patterns of item loadings in the two-factor model, items were
eliminated in a series of iterative factor models. Items were eliminated if they demonstrated poor loadings on their hypothesized factor. Per Comrey & Lee (1992), factor loadings exceeding .71 are considered excellent, .63 very good, and .55 good. Consequently, a factor loading cut-off of .50 was used so that the shared item-factor variance was greater than 25 percent. All tested models were congeneric, which means that items were allowed to load on only one of the two first-order factors and measurement error of the items was not permitted to be correlated. Through this iterative process, a final model emerged that was a good fit to the data:
[χ2(76)=162.00, p=.000; χ2/DF = 1.97; RMSEA=.053; SRMR=.04; CFI=.96; GFI=.95]. This model appears in Figure 2. The RMSEA, SRMR, CFI, and GFI were within the cut-off range recommended by Hu and Bentler (1999). The chi-square test was significant (an indication of model non-fit); however, it should be noted that the chi-square statistic has been criticized as an overly-sensitive test that frequently suggests rejecting potentially useful models, particularly as N becomes large (Bollen, 1989).
Figure 2. Standardized parameter estimates for a first-order confirmatory factor analysis (AMOS) for the Promotion Prevention Scale (PPS). Numbers in boxes correspond to PPS questionnaire items presented in Table 2.
Consistent with the initially proposed two-factor model, this model consisted of two first-order latent factor (promotion regulatory focus and prevention regulatory focus). The final 14 retained items appear in Table 2 along with their descriptive statistics across samples. In the final model, the promotion subscale had eight items, and the prevention subscale had six thus satisfying the recommended minimum number of observed variables per latent variable (Kline, 2004; Wegener & Fabrigar, 2000). All retained items had standardized loadings on their first-order factors above .50 (range = .53 to .72).
Table 2. Descriptive statistics for the final items of the Promotion Prevention Scale (PPS)
Sample 1 (N = 400) Sample 2 (N=151)
M SD Skew K M SD Skew K
1. When I succeed I feel a sense of joy 4.34 0.56 -0.12 -0.72 4.34 0.65 -0.47 -0.69
2. I strive towards my ideals 3.98 0.66 -0.3 0.28 3.99 0.69 -0.37 0.24
3. I think about how to achieve an optimal life 3.83 0.77 -0.38 0.07 3.79 0.76 -0.56 0.79
7. I strive to be my ideal self 4.06 0.66 -0.27 0.07 3.99 0.67 -0.26 0.09
8. I think about how I will achieve my hopes and
aspirations 3.94 0.71 -0.49 0.31 3.9 0.75 -0.52 0.33
10. Success to me is achieving my aspirations 4.01 0.7 -0.31 -0.02 3.89 0.7 -0.31 0.11
15. I view goals as opportunities not to be missed 3.9 0.75 -0.3 -0.16 3.76 0.72 -0.16 -0.17
19. Whenever possible, I try to improve my current
situation 4.18 0.64 -0.46 0.59 4.11 0.63 -0.25 0.19
23. I strive to meet my obligations 4.24 0.67 -0.48 -0.13 4.19 0.64 -0.5 0.75
24. Success to me is meeting my responsibilities 4.13 0.65 -0.3 -0.04 4.05 0.66 -0.48 0.76
25. I view goals as duties I must complete 3.72 0.89 -0.44 -0.28 3.64 0.91 -0.76 0.33
26. When I succeed I feel a sense of calm 4.03 0.71 -0.46 0.21 3.98 0.67 -0.52 0.88
36. I think about how I can prevent failures in my life 3.74 0.83 -0.51 0.12 3.79 0.85 -0.62 0.26
37. I am focused on preventing negative events in my life 3.78 0.87 -0.45 -0.26 3.87 0.89 -0.96 1.46
A primary purpose of the current research was to compare the fit between a two-factor model of regulatory focus (with promotion and prevention regulatory focus as latent factors) and a four- factor model of regulatory focus. As discussed in the introduction, a four-factor solution may offer superior fit because regulatory focus might be better conceptualized as consisting of four latent factors: nurturance, actual-ideal, security, and actual-ought orientations. Thus, using the 14 remaining items from the initial CFA, a second CFA model was tested on the main sample with
the measured items that reflect the four potential dimensions of regulatory focus. All factors were tested simultaneously. The 14-item four-factor model demonstrated nearly identical fit to the two-factor model with all fit indices within the cutoff ranges suggested by Hu & Bentler (1999), [χ2(71)=140.32, p<.000; χ2/DF = 1.97; RMSEA=.05; SRMR=.04; CFI=.97; GFI=.95]. This model is shown in Figure 2. The chi-square test was also significant with this model, but, reiterating from above, the chi-square statistic has been criticized as an overly-sensitive test that frequently suggests rejecting potentially useful models, particularly as N becomes large (Bollen, 1989). A comparison of model fit statistics and indices for the two and four factor models is presented in Table 3, below.
Table 3. Chi-square statistics and model fit indices for two and four factor models of regulatory fit.
Sample 1 (N = 400)
Χ2 Χ2/DF RMSEA SRMR CFI GFI
Two Factor Model 162 1.97 0.05 0.04 0.96 0.95 Four Factor Model 140.3 1.97 0.05 0.04 0.97 0.95
In sum, both model fit statistics and indices were very good. However, given that the four-factor model had only two items reflecting one of the sub-dimensions (security) making any internal consistency estimates for this factor suspect, and for reasons of parsimony (all other things being
equal, simple models are preferred over more complex), the two-factor model is preferred.
Cross-validation
Because the model pruning approach can result in a model that achieves acceptable fit by capitalizing on chance error, both the two-factor and four-factor CFAs were performed on data drawn from an independent hold-out sample (N=151) to test the replicability of the models derived from the preliminary analyses. MLE was again used. The fit indices suggest that the correspondence between the proposed models and the data was acceptable. The test statistic and fit indices for the two-factor model were similar to those from the first sample: [χ2(76)=120.93, p<.001, χ2/DF = 1.59; RMSEA = .063; SRMR = .05; CFI = .95; GFI = .90]. Also, the test statistic and fit indices for the four-factor model were similar to those from the first sample: [χ2(71)=112.54, p<.001, χ2/DF = 1.59; RMSEA = .062; SRMR = .05; CFI = .96; GFI = .90]. A summary of all model fit statistics and indices are presented in Table 3. Given the similarity of model fit between the first and second samples, it is reasonable to be confident that the proposed