Caracterización de la oferta de productos.

As shown in Figure 3.5 below, the post-PD survey began by asking educators to rate the PD opportunity as high-, medium-, or low-quality based on a Likert scale from 1 (low) to 5 (high). The participants were asked to explain their rankings and to record the total number of hours they had devoted to the PD. They were also asked to evaluate their interactions with the PD provider. (To view the post-PD survey in its entirety, see Appendix G.)

1) Which PD opportunity did you attend? (Drop-down with the three opportunities listed)

2) How would you rate the overall quality of this PD opportunity? (Scale 1 – 5: Low = 1; Medium = 3; High = 5)

3) Please explain why you rated the PD as such from the previous question. 4) How much time did you devote to this specific PD opportunity thus far?

a. 0-8 hours (1 day) b. 9-16 hours (2 days) c. 17-24 hours (3 days) d. 25-48 hours (4-6 days) e. 49-72 hours (6-9 days)

Please give your opinions about working with your PD provider. To what extent... (Not at all, Some of the time, Most of the time, All of the time) (Questions 5 – 8 were presented in a gridded format.)

5) Did you have adequate contact with the service provider? 6) Did you receive materials and resources in a timely manner? 7) Was the service provider responsive to your questions and needs?

8) Did your partnership with the service provider meet your overall expectations? Figure 3.5. Post-PD STEM Educator Questions Focusing on Quality of the PD and

Interactions with PD Provider

For these questions, descriptive statistics, including mean and standard deviation, were calculated. In addition, Likert scale rankings allowed generalizations to be made regarding the overall quality of the opportunity (question 2) and the educators’ interactions with the service provider (questions 5 – 8).

In relation to question 2 associated with ranking the quality of the PD, after an extensive search, no clear cut-offs or criteria appeared to exist in the research related to a ‘high-quality’ PD experience. Consequently, this prompted the questions: How many

participants need to agree that the opportunity is of ‘high quality’ to classify it as such? Does one participant defining an opportunity as low-quality equate to the entire opportunity being eliminated as a high-quality option? Prior to collecting the data, the working assumptions were that an opportunity would be deemed to be of high quality if it met the following criteria: 1) No participants ranked the PD as low-quality (equivalent to ranking it as a 1 on the Likert scale) and 2) At least 75% of the participants ranked the PD as high-quality (ranking it as a 4 or 5 on the Likert scale).

Skills and Knowledge Gained from the PD: Questions 9 – 24

The next three sections of the post-PD survey represented questions that were derived from three sources: Darling-Hammond and Richardson (2009); Smith and Gillspie (2007); and the Iowa Governor’s STEM Advisory Council’s 2015-2016 Evaluation Report.

Questions from this post-PD survey section were rated on a Likert scale of 1 – 6 using the terms ranging from strongly agree (coded as 1) to strongly disagree (coded as 6).

The Darling-Hammond and Richardson (2009) questions (shown in Figure 3.6 below) focused on increasing educator knowledge of the content, how students learn the content, the delivery format of the PD to the educator, and the time provided for educator collaboration.

Please answer the following questions as strongly agree, agree, somewhat agree, somewhat disagree, disagree, strongly disagree. (Questions were presented in a gridded format.)

9) The PD deepened my knowledge of content and how to teach it to students. 10) The PD helped me understand how students learn specific content.

11) The PD provided opportunities for active, hands-on learning.

12) The PD enabled me to acquire new knowledge, apply it to practice, and reflect on the results with colleagues.

Figure 3.6. Post-PD STEM Educator Survey Questions Related to Darling- Hammond & Richardson (2009)

The Smith and Gillespie (2007) questions illustrated in Figure 3.7 focused on

relevance to current teaching assignment, overcoming barriers to implementation, increasing content knowledge, and delivery format of the PD.

Please answer the following questions as strongly agree, agree, somewhat agree,

somewhat disagree, disagree, strongly disagree. (Questions were presented in a gridded format.)

13) The PD made a strong connection between what I learned and my current work assignment.

14) The PD helped me plan for application and to identify and strategize barriers to application.

15) The PD focused on subject-matter knowledge.

16) The PD included a strong emphasis on analysis and reflection, rather than just demonstrating techniques.

17) The PD included a variety of activities.

Figure 3.7. Post PD STEM Educator Survey Questions Related to Smith & Gillespie (2007)

The Iowa Governor’s STEM Advisory Council also used the CTEq STEMworks model to select PD and curriculum for six regions throughout Iowa. Each of these six regions is linked to a regional university or community college. Because Iowa was in the third year of implementation using the STEMworks model, there were a number of evaluation reports available online (iowastem.gov/sites/default/files/evaluation/2015-16-Iowa-STEM-

Evaluation-Report.pdf). Based on the similarities between Idaho and Iowa methodologies, a number of questions from the Iowa PD survey were used for Idaho educators, focusing specifically on educator confidence, knowledge, effective teaching, student questioning, diverse learners, integration, and administrative support (as shown in Figure 3.8 below).

Please answer the following questions as strongly agree, agree, somewhat agree,

somewhat disagree, disagree, strongly disagree. (Questions were presented in a gridded format.)

18) The PD helped me to better understand how to engage diverse learners in STEM (i.e. females, low SES, students from rural communities, students with diverse races/ethnicities).

19) The PD showed me how to utilize more than one STEM subject in my classroom (i.e. science and engineering, math and technology, etc). 20) I gained new knowledge and/or skills as a result of this PD.

21) I believe that I will receive the administrative support to implement most aspects of the PD.

22) I have more confidence to teach STEM topics. 23) I have increased my knowledge of STEM topics.

24) I am better prepared to answer students' questions about STEM topics. 25) I have learned effective methods for teaching STEM topics.

Figure 3.8. Post PD STEM Educator Survey Questions Related the Iowa STEM Evaluation Report (2015 – 2016)

For questions 5 – 25, all participant responses were recorded using a Likert scale, and descriptive statistics were run to analyze differences. Tables and figures in Chapter Four will be shown to illustrate the percentages of teachers in each of the Likert scale categories. No other tests were performed on this data because there were 20 variables and a final sample

size of 101 participants in the post-PD data files. The loss of significance with 20 variables and a reduced sample size simply do not permit extensive testing (Zar, 2009). Strong conclusions were still drawn on the questions, and trends are visible and will be analyzed in Chapter Four and within the Implications section of Chapter Five.

Qualitative, Extended Response Items: Questions 26 – 31

The remaining questions were qualitative, extended response questions related to challenges and successes, needed and expected supports, knowledge gained, and expected student interactions. The survey concluded by asking if other educators would benefit from this opportunity and to explain if the PD should be offered in subsequent years (see Figure 3.9 below).

Extended Response:

26) Describe challenges or barriers, if any, you faced in working with your PD provider.

27) What did you find helpful during the PD and would recommend to others? This might include helpful partners, administrative support, training, or unique local circumstances.

28) How will you implement what you learned from this PD into your teaching practices?

29) What additional supports do you need to be successful?

30) Would other educators benefit from this opportunity? If so, why? If not, why not?

31) Should this PD be offered in future years? (i.e. Would you recommend this PD to others?). Please explain why or why not.

Figure 3.9. Post-PD STEM Educator Survey Qualitative Questions

Post-PD responses for Likert scale ranking questions 5 – 25 were aggregated from the three opportunities and exported into Excel for data coding. Once exported from the

Salesforce Community Grant Portal, duplicates were removed, and then responses were stripped of any personally identifiable information prior to file manipulation. Consistent with other qualitative coding methods used in this research and throughout the literature, the

works of Saldaña (2009) and DeCuir-Gunby et al. (2011) were used to guide the primary descriptive and secondary focused coding techniques from the post-PD extended response questions. As a result, main themes from each of the extended response answers were identified. Pertinent quotes and insights are used in Chapter Four as evidence of outcomes. For example, the primary coding methodology involved descriptive coding which is

illustrated in Figure 3.12 below, and shows how the primary codes were produced using the participant responses. These primary codes were then used to generate secondary themes for each of the six extended response questions.

Figure 3.10. Primary Coding Technique Used for Post-PD Survey Questions 26 – 31 This methodology was systematically repeated for questions 26 – 31. Data was drawn from Excel and imported into Word to be coded. Codes were then secondarily aggregated into themes and placed back into Excel for calculation. Data collected from this process is analyzed in detail in Chapter Four. In addition, to ensure accuracy of the primary code and subsequent secondary themes, additional STEM AC staff reviewed the codes; moreover, discussion followed to verify the accuracy of the coding and the associated themes, which served to validate the results and associated outcomes.

CHAPTER FOUR: RESEARCH FINDINGS – RESULTS AND DISCUSSION

This chapter analyzes the research findings of this study and discusses the results of these findings. It begins by focusing on Stage 1 of the study regarding the pre-PD aspects. Stage 1 included the large group survey, the focus group, and an additional set of adult learning theory questions that participants were asked prior to participating in one of the three research-based, rubric-selected PD opportunities. The chapter then shifts into Stage 2, the post-PD survey data. For both stages, qualitative and quantitative data were

simultaneously studied, measured, and analyzed with equal weight given to both types of data in order to more accurately evaluate educator feedback and results (Morse, 2010; Johnson & Onwuegbuzie, 2004). The purpose of this work was to determine if the research- based, rubric-selected PD opportunities were truly viewed as ‘high quality’ (with the

potential of being offered and funded in subsequent years), and/or if these rubrics should be modified for future use in the selection of high-quality PD opportunities for Idaho STEM educators.

In total, 116 unique individuals, meeting the criteria of currently practicing K-12 teachers, participated in the research-based, rubric-selected PD opportunities and provided pre-PD survey information. Only 101 of these individuals participated in the post-PD survey as six teachers dropped the PD prior to participating and nine failed to complete the post-PD by the required date of August 18, 2017.

For the post-PD research, opportunities were analyzed collectively, though, in principle, each individual opportunity could be analyzed individually to determine the most

important characteristics associated with that particular PD. However, comparability between opportunities was not the focus of this research. As noted previously, these opportunities spanned various grade levels and used slightly different approaches to STEM education (i.e. PBL, inquiry-based, and train-the-trainer models) and, as a result, targeted slightly different groups of educators, although there was minor overlap with four educators participating in two of the three opportunities. The comparisons between the individual opportunities were not the focus of this study (assuming that all three opportunities are deemed to be of high quality) because it was irrelevant if one high-quality opportunity ranked higher than another high-quality opportunity. Rather, the goal was to determine characteristics of opportunities that, when taken together, identify the most important factors associated with high-quality STEM PD and to determine if the rubrics used to select PD opportunities for Idaho STEM educators are viable.

This study was not designed as a repeated measures study nor was it attempting to measure changes over time in educator attitudes and/or student achievement measures, as these variables were beyond the scope of this work. Pre- and post-PD surveys served three distinct purposes linked to the original research questions. First, pre-PD surveys were used to determine which characteristics of STEM PD were most (and least) important to Idaho STEM educators; secondly, the resultant characteristics were compared to relevant literature. Finally, post-PD surveys were used to determine if the research-based rubrics selected PD that was identified as ‘high quality’ by the educators who participated in one of the three opportunities. Variables such as attitudinal changes in educators and students, long-term impacts on educator practices, and student outcomes should certainly be measured but were beyond this scope of work and were not factored into this study.