Ponderación de criterios de evaluación para el sistema de

The purpose of this study is to examine the relationship between AP STEM course-taking in high school and selection of college STEM major and determine whether the relationship differs across racial/ethnic groups and male and female students. The study develops a model of STEM major selection with AP STEM course-taking as the key factor, controlling for other factors that the literature documents as being significant predictors of STEM major selection. With the study findings, we seek to provide insight to educators and policymakers in shaping college preparation programs and policies as well as counsel students during their course selection process in high school. This study is guided by the following research questions: (1) After controlling for student background, high school experiences, and college experiences, how is AP STEM course-taking related to the likelihood of selecting a STEM major? (2) Does the relationship between AP STEM course-taking and STEM major selection differ by gender and race/ethnicity?

Research Model

The conceptual model for this study is based on a combination of Lent, Brown, and Hackett’s (2000) social cognitive career theory and St. John, Asker, and Hu’s (2001) social construct theory as presented in Chapter Two. This conceptual model incorporates both student- level and institution-level factors, and is the framework for the two-level logistic regression model with fixed effects used in this study.

The major constructs in the proposed STEM major selection model include: ● Student background characteristics (gender, race/ethnicity, socioeconomic status)

● High school student experiences (total number of high school STEM courses, exposure to AP STEM courses, math self-efficacy, science self-efficacy, academic achievement, education aspirations)

● College student experiences (receipt of need-based financial aid)

● High school context (high school ID). There are likely unobserved school-level

characteristics affecting the likelihood of both AP STEM course-taking, the key factor, and STEM major choice. Therefore, I will be utilizing high school ID, rather than the individual school-level characteristics discussed in the literature review, to account for all school-level factors in my model. The use of high school ID as a fixed effect will be discussed in more detail later in the chapter.

Data Source and Sample

The High School Longitudinal Study of 2009 (HSLS:09) is utilized as the data source for this study. HSLS:09 is the fifth survey in a series of educational longitudinal studies that include the Educational Longitudinal Study of 2002 (ELS:2002), the National Educational Longitudinal Study of 1988 (NELS:88), the High School & Beyond Longitudinal Study of 1980 (HS&B), and the National Longitudinal Study of the High School Class of 1972 (NLS-72). HSLS:09 data collection is ongoing, with base year, first follow-up, and second follow-up data currently available on the NCES website (Duprey, Pratt, Jewell, Cominole, Fritch, Ritchie, Rogers, Wescott, Wilson, 2018). These five studies capture data on the secondary and postsecondary experiences of cohorts of students representing each of the past five decades. The overall purpose of the longitudinal studies program is to examine the relationship of personal, family, social, institutional, and cultural factors with the personal, educational and career development of students (Duprey et al., 2018).

The HSLS:09 baseline survey is representative of high school freshmen in fall 2009 who were followed up with two years later during the spring of 11th grade (first follow-up), the summer after the majority graduated from high school (2013 update), and in the spring of 2016, three years after the majority finished high school (second follow-up). Unlike earlier NCES longitudinal studies, HSLS:09 has a particular focus on STEM learning experiences and outcomes with the intention of helping researchers and policymakers investigate the nature of paths into and out of the STEM pipeline and what personal and educational factors influence those decisions. Thus, HSLS:09 presents a unique opportunity for this study to examine factors related to selection of a STEM major.

The base year survey in fall 2009 included a random sample of 25,206 high school freshman from 944 public and private high schools across the United States. Student participants completed a survey and a mathematics assessment. The student survey collected information on a variety of topics, including student background, math and science course-taking, math and science self-efficacy, and educational and career aspirations. Each student’s parent, science and mathematics teachers, and school counselor all completed questionnaires. An administrator from each school included in the survey also completed a questionnaire (Duprey et al., 2018).

The first follow-up data collection once again included student, parent, counselor, and administrator questionnaires, which included many of the same topics as the base year surveys. The 2013 update was utilized to collect high school transcripts and survey students and parents regarding high school completion status. The second follow-up survey administered

questionnaires to students only, inquiring into students’ postsecondary, employment, and personal experiences. Postsecondary transcripts and financial aid records from institutions that students in the sample attended were also collected as part of the second follow-up. It is

important to note that the base year, first follow-up, and second follow-up surveys all collected information regarding decision-making on education and careers related to STEM fields (Duprey et al., 2018). HSLS:09 was designed with the intent to study student access to and participation in STEM courses as well as their decisions to pursue and persist in STEM majors and careers.

HSLS:09 was designed utilizing two earlier NCES longitudinal studies, namely NELS:88 and ELS:2002, as a model; however, HSLS:09 also included design updates to improve upon the earlier studies. Similarities to earlier NCES studies can be found in the development of scales for composite variables in HSLS:09, such as socioeconomic status, math self-efficacy, and science self-efficacy. To further ensure validity and reliability, their development was also based on advice received from HSLS:09 Technical Review Panel (TRP) members and TRP meeting participants (Ingels, Pratt, Herget, Dever, Fritch, Ottem, Rogers, Kitmitto, & Leinwand, 2014).

However, there are also key differences between HSLS:09 and earlier NCES longitudinal studies. HSLS:09 adjusted timeframes of data collection to improve the quality of the data collected. For example, the second follow-up data collection occurred three years after expected high school graduation rather than two as in prior studies. Doing so allowed for more complete and accurate collection of data on postsecondary education experiences (persistence, majors, etc.) as students had more of their college experience under their belts (Duprey et al., 2018). It should be noted that cross-cohort comparisons cannot be made with earlier NCES secondary longitudinal studies due to new measurement points. However, the improvement in data quality due to the improved design of HSLS:09 and the focus on STEM education and careers is a worthwhile trade-off.

Research Variables for STEM Major Selection Model Outcome variable

The outcome variable in this study is a dichotomous variable indicating whether a student chose a STEM or non-STEM major after up to two years of college enrollment at a four-year institution. The college major variable is recoded so that all STEM majors are recoded as 1 and all other majors, including undecided, are recoded as 0.

Independent variables

● Student background characteristics

● Gender (this categorical variable indicates a student’s gender. This variable is recoded into a dichotomous variable with Female as the reference group.)

● Race/ethnicity (this categorical variable indicates a student’s race/ethnicity. White students are the reference group.)

● Socioeconomic status (this continuous variable is a composite of five questions from the parent questionnaire — father’s education, mother’s education, family income, father’s occupation, and mother’s occupation.)

High school student experiences

● High school STEM courses (this continuous variable represents the total number of Carnegie units of STEM courses a student took in high school. This includes all courses a student took in math, science, computer science, and engineering. A Carnegie unit represents 120 hours of class or contact time with an instructor over a one-year period.)

● High school exposure to AP STEM courses (two variables will be utilized to measure this. Sensitivity testing will be discussed later in this chapter as the method to

determine to what extent the results are sensitive to the two different measures of AP STEM course-taking.)

○ Number of AP STEM courses (this continuous variable indicates the total number of Carnegie units of AP/IB STEM courses a student took in high school. This includes all AP/IB courses in math and science. While this variable includes IB (International Baccalaureate) courses as well as AP courses, there are only 900 participating high schools in the IB program in the United States compared with the more than 22,000 U.S. schools participating in AP (College Board, 2017; International Baccalaureate Organization, 2018). Thus, the number of IB courses only represents a small portion of the data collected for this variable, whereas the number of AP courses represents the majority of the data collected for this variable.)

○ Has taken any AP STEM courses (this dichotomous variable represents whether or not a student has taken any AP/IB STEM courses in high school. This includes any AP/IB courses in math or science.)

● Math self-efficacy (this composite continuous variable represents a student’s math self-efficacy, with higher values representing higher math self-efficacy. This variable is a composite of four questions from the student questionnaire — confidence in taking math tests, understanding the math textbook, mastering math skills, doing well on math assignments.)

● Science self-efficacy (this composite continuous variable represents a student’s science self-efficacy, with higher values representing higher science self-efficacy. This variable is a composite of four questions from the student questionnaire — confidence in taking science tests, understanding the science textbook, mastering science skills, doing well on science assignments. This variable has not been previously measured and is a new variable that has been included in HSLS:09 that was not measured in the previous NCES longitudinal studies.)

● High school math achievement (this continuous variable represents a student’s college entrance exam (i.e., SAT, ACT) math section score standardized in terms of SAT.)

● Education aspirations (this categorical variable indicates whether high school students aspire to a graduate degree or higher. It is recoded as 1 for “yes” and 0 for “no”.)

College student experiences

● Receipt of need-based financial aid (this categorical variable indicates whether a student was offered a Pell Grant during their first year of college. This variable will be recoded into a dichotomous variable with 1 for “yes” and 0 for “no.”)

High school context

● School ID (this is a continuous variable representing the school identifier assigned for the base year sample high school. The use of fixed effects, discussed later in this chapter, will create a dummy variable for each school. However, the coefficient for each school will not be reported.)

Data Analysis

This study utilizes a two-level logistic regression model with fixed effects. In order to proceed with the inferential analysis methods, the variables have been recoded as described in the prior section. Additionally, I utilized multiple imputation to deal with missing cases that existed for some of the variables used in this study. Multiple imputation essentially predicts what the missing data values would be, filling them in by randomly drawing observations from the distribution (Allison, 2001; Schafer, 1999). Then, the researcher can perform analyses on the imputed dataset as if all of the data had been empirically observed. Multiple imputation can be applied to virtually any kind of data or model using conventional software (Allison, 2001). Multiple imputation has been widely accepted as an effective method for dealing with missing data in large data files from sample surveys, which makes it appropriate to use to in this study (Schafer, 1999).

Logistic Regression

A two-level logistic regression model with fixed effects was run to determine the relationship between AP STEM course-taking and STEM major selection, controlling for all relevant student-level and high school-level variables. Logistic regression is the appropriate form of regression analysis when the outcome variable is dichotomous, as is the case in my study (Peng, So, Stage, & St. John, 2002).

Since Stata software does not allow for including a weight variable in a logistic

regression model with fixed effects, I also ran a linear probability model with fixed effects after the logistic regression model to determine whether incorporating a weight variable had any impact on the significance of the predictors to the model. A weight variable is necessary to

include in a model to adjust for unequal probabilities of selection in the sample design and help ensure that the results of the analysis are representative the population (Thomas & Heck, 2001). While any oversampling at the school level in my sample was accounted for through the use of fixed effects, oversampling at the student-level could not be accounted for in this way. As Asian 9th grade students were oversampled in HSLS:09, it was important to determine whether

inclusion of a weight variable would impact my results (Ingels et al., 2014).

Linear probability modeling, like logistic regression, is also appropriate to use when the dependent variable is binary, as it the case in my study (Caudill, 1988). The coefficients

generated by a linear probability model represent the change in probability of the student selecting a STEM major for a one-unit change in the predictor variable of interest, holding all other predictors constant (Caudill, 1988). The main drawback of using linear probability modeling is that the model can produce probabilities outside of the acceptable range of 0-1 (Caudill, 1988).

My analysis also included a series of interaction effects tests, examining the variation of gender and racial/ethnic differences in STEM major selection as a function of AP STEM course- taking. The two sets of interaction terms are gender and AP STEM course-taking and

race/ethnicity and AP STEM course-taking. Each set of interactions was incorporated into the baseline model independently. Each model with a set of interaction terms was then compared with the baseline model using a post-estimation test to determine whether either of these models represented a significant improvement over the model without the interaction effects.

Fixed effects. The option of conducting a randomized controlled trial whereby students

are assigned to the treatment group or the control group was not feasible in this study. However, I needed to account for the fact that school-level factors are likely correlated with the probability

of a student being in the treatment group (i.e., enrolling in an AP STEM course) as well as with the outcome (i.e., selecting a college STEM major). Therefore, in addition to controlling for student-level characteristics, I controlled for the school-level characteristics, which necessitated a two-level model to examine the data (Clark et al., 2010). Doing so accounted for the fact that students are nested within different educational institutions and may behave differently based on their different contexts (Hox, 2002; Clarke et al., 2010; Clarke et al., 2015; Huang, 2016).

In determining whether to treat the school-level factors as random or fixed effects, I first had to consider whether or not the regression assumption held in my study (Clarke et al., 2015). Random effects models assume that unobserved school-level characteristics are uncorrelated with other covariates. This is referred to as the regression assumption (Clarke et al., 2015). However, prior research indicates that clustering often occurs in educational studies looking at student outcomes across different schools because of the influence of unmeasured school characteristics such as teacher quality and school culture (Clarke et al., 2015). The students in my sample come from thousands of different high schools with different characteristics, some of which are not measurable. Thus, the regression assumption does not hold for my study, which indicates that the school-level factors should be treated as fixed effects. Fixed-effects models account for all effects of higher level variables, both observed and unobserved (Clark et al., 2010; Huang, 2016). By accounting for all variability associated with any school-level variables, the omitted variable bias is significantly reduced (Huang, 2016). Additionally, while it is difficult to draw causal inferences in observational studies, a fixed-effects model allows the researcher to draw inferences that are closer to causal than other methods, because the fixed-effect model accounts for the possible correlation of all higher level factors, observed and unobserved, with both the outcome and the treatment (Clark et al., 2015).

In order to determine whether a fixed effects approach is appropriate for my analysis, I need to decide whether I am interested in the effects of both level 1 (student) and level 2 (school) variables. A fixed-effects model does not allow for analysis of the influence of school-level factors. However, education methodologists support the use of fixed effects in a study when the researcher is only interested in the effect of level 1 variables, while controlling for second level observable and unobservable factors (Hahs-Vaughn, 2005; Huang, 2016; McCoach & Adelson, 2010; Thomas & Heck, 2001; Thomas, Heck, & Bauer, 2005). As the focus of my study is the influence of student-level (level 1) factors — specifically AP STEM course-taking, gender, and race — on the outcome, using a fixed-effects model is an appropriate method for my study. Therefore, I included the variables representing high school IDs as covariates in my regression model (Huang, 2016).

Sensitivity testing. Sensitivity testing is necessary in my analysis in order to determine

which subset of variables accounts for more of the output variance, if any (Hussain, 2008). HSLS:09 includes more detailed data regarding high school student STEM experiences than earlier educational longitudinal studies. Therefore, I needed to run multiple models that

incorporated different measures for the variable in my study measuring exposure to AP STEM courses in high school in order to determine which, if any, of the variables, have the most significant correlation to STEM major selection. Exposure to AP STEM courses is measured in HSLS:09 by whether or not a student took any AP STEM courses in high school was well as by the actual number of AP STEM courses a student took. Thus, sensitivity testing allows me to examine whether exposure versus number of courses matters in the model explaining STEM major selection. Thus, this study builds on earlier research, testing a two-level logistic regression

model with fixed effects, using sensitivity testing to determine which high-school STEM exposure variables have the most significant relationship to STEM major selection.

Interaction effects. In order to examine the variation of gender and racial/ethnic

differences in STEM major selection as a function of AP STEM course-taking, I also needed to run a series of interaction effects tests. Prior to running the interaction effect models, I generated interaction terms for the two variables, measuring exposure to AP STEM course-taking with gender and race/ethnicity. I ran the interaction effects tests using both a logistic regression and a linear probability model. After determining whether any of the interaction effects were

significant to the model, I utilized Jaccard’s (2001) method of generating predicted probabilities to more closely examine the interaction effect on STEM major selection.

Limitations

There are several limitations to this study that warrant discussion. First, the sample only includes students who declared a major by the time of the second follow-up survey. Therefore, students who declared majors later in their college career are not included in the sample.

Second, the study is constrained by data included in HSLS:09. Other factors that the literature has found to be related to STEM major selection and were measured in earlier NCES longitudinal studies, such as interaction with faculty and math and science readiness (Pascarella & Terenzini, 2005; Rosenbaum, 2001; Wang, 2013), are not included in the study as they are not measured in HSLS:09. Additionally, while HSLS:09 includes more STEM-specific data than