TIPOS DE INCENDIO EN FUNCIÓN DEL RIESGO

The evaluation of the HAND RAISE Intervention as delivered through the LIVE CHART tool focuses on three overarching research questions in alignment with the project’s goals: improving students’ confidence and skill in asking and articulating questions, improving students’ confidence in answering questions of both the teacher and those of their peers, and helping to prevent the decline of interest in STEM- related fields that is observed between kindergarten and high school. We plan to evaluate the effectiveness of the HAND-RAISE intervention along these three di- mensions through the use of surveys and quantitative field observations collected in Years 2 and 3 of the grant period. The next 3 sections detail this planned evaluation process.

Does the HAND-RAISE Intervention build student confidence to ask questions and diversify the students who do ask questions in the class- room?

It is important for students to feel comfortable in their learning environment to ask questions when they need more information, clarification, or further instruction to complete their work. It is similarly important for students to build the skill set of being able to effectively articulate their questions so that a teacher, instructor, or even peer can address the problem; it often difficult to help a student who simply says “I don’t get it” when asked to articulate a question about the material. Conversely, however, we certainly do not want students to take advantage of the system as is sometimes exhibited in the over-use of hints; asking too many questions may be indicative of a student attempting to game the system, asking for help before applying themselves to learning the assigned topic. It is for these reasons that an integral aspect of HAND-RAISE is that it requires students to articulate their

question when they use the tool; certainly some students will still give the “I don’t get it” response, but then the teacher is equipped with the ability to require a student to re-articulate the question before s/he addresses the problem. Similarly, the teacher has control over what the resulting action is when a student uses HAND- RAISE from the system; through LIVE-CHART, the teacher is able to use the reported information to determine if the student is using the tool effectively and respond accordingly by either speaking with the student, allowing another peer to answer the question, ask the student to re-articulate their question, or even instruct the student to try the problem before using the tool.

In order to evaluate the intervention on its effectiveness in improving student question asking behavior, we will use quantitative field observations collected by Dr. Kreisberg during a set of live in-class teacher trainings/demonstrations. Dr. Kreisberg will, as part of the intended teacher trainings, attend each of the Pilot Teachers’ classes at the beginning and end of the academic years coinciding with years 2 and 3 of the grant. The teachers will be instructed beforehand to assign a selected homework assignment through ASSISTments for the preceding night, and Dr. Kreisberg will lead a homework review/discussion to demonstrate effective practices. Dr. Kreisberg will be equipped with her own version of LIVE-CHART for each classroom designed specifically for data collection. During this session, Dr. Kreisberg will present the students with a poorly-formed question pertaining to the content from their previous night’s homework, in that a necessary piece of information will be omitted from the problem description; she will ask students to spend a few minutes to solve the problem, and turn her attention to her tablet so as to pretend not to see the hands that undoubtedly will begin to raise. Dr. Kreisberg will use her version of LIVE-CHART to record the students who have raised their hand and, after one minute to give students an opportunity to raise their hands,

will call on a student and take their question (likely pertaining to the missing piece of information).

In year 2, as the Pilot Teachers will not yet have used LIVE-CHART or HAND- RAISE in their classroom, the collected observations of students who raise their hands will act as a baseline measure of comparison for observations collected in year 3, but also will help indicate how hand raising behavior normally changes from the beginning to the end of the school year. Observations will, again, be collected at the beginning and end of each academic year to help control for differences in content difficulty (when comparing across years) and observe changes in individual student behavior (within student from the beginning to the end of each school year). The collected observations will be used to determine if 1) the number of hand raises in the classroom increase when students are faced with insufficiently-formed or confusing problems and 2) if the diversity of students who raise their hands increases as a result of the HAND-RAISE intervention.

Does the HAND-RAISE Intervention build student confidence to answer questions of their teacher and peers and diversify the students who raise their hands to answer questions in the classroom?

In addition to the ability to articulate questions, it is important for students to also build confidence and be able to articulate answers to questions asked by a teacher or a peer. It is important for students to feel comfortable in raising their hand to answer a question when the solution is known as it helps the teacher properly assess who understands the material as she is introducing new topics. The HAND-RAISE tool addresses and attempts to build a base of confidence and skill set focused on articulating answers through support of peer assistance. If the teacher chooses to connect a student who has their hand raised as indicated through LIVE-CHART

with another student (whether a starred student or otherwise), or if the teacher has enabled the tool to automatically choose another student, this provides another

student the opportunity to address a peer’s question anonymously. The act of

helping another student on a topic where the helper has demonstrated understanding of the topic is aimed at building confidence in not only solving problems, but actively helping others to solve problems.

To evaluate the effectiveness of the HAND-RAISE Intervention in improving students’ confidence and ability to articulate answers to questions, we will similarly utilize quantitative field observations collected by Dr. Kreisberg during a set of live in-class teacher trainings/demonstrations. As described in the previous section, Dr. Kreisberg will go to the classrooms of the 25 Pilot Teachers at the beginning and end of each academic year coinciding with years 2 and 3 of the grant. Again, she will lead a homework review/discussion based on a known assignment given to students as homework for the preceding night. From this assignment, Dr. Kreisberg will pre- select a problem to use as an example during the review session. Dr. Kreisberg’s LIVE-CHART tool, specialized to help in the collection of observations, will display each student’s performance on the pre-selected problem on the classroom display such that she is able to see which students answered the problem correctly; the problem will be pre-selected based on difficulty in an effort to maximize the num- ber of students who answered the problem correctly. With this information, Dr. Kreisberg will display the problem to the class and ask which students can provide a solution, prompting students to raise their hands. Dr. Kreisberg will then pause to give students an opportunity to raise their hands to offer a solution and record such students through her LIVE-CHART tool before then calling on a student and proceeding. With the observations of which students raised their hands, a measure of effectiveness can be calculated as a percentage of students who knew the solution

(as indicated by which students answered the problem correctly on the homework assignment) raised their hand to offer an answer.

This measure, as compared from year 2 without use of the HAND-RAISE and LIVE-CHART tools to year 3 with such tools, and also from the beginning of each academic year to the end of each academic year, will give an indication of how student hand raising behavior for the purpose of answer questions is impacted by

use of such tools in the classroom. Similarly, the described metric (percentage

of students who know the answer that raise their hands) can be observed within smaller subgroups of students to understand if there are heterogeneous effects across students; specifically, as previous works have identified female and minority students as being less likely to engage in classroom discussions and activities with the same level of interest as some of their peers (Greenfield, 1997; Bernacki et al., 2016), it is a goal of this project to improve not only the help-seeking behavior of such students but also their level of engagement in answering questions during classroom discussion.

Does the HAND-RAISE Intervention help to reduce the decline of inter- est in STEM-related fields?

The decline of interest and motivation pertaining to STEM-related subjects from kindergarten through high school has been well-studied and documented in the United States (Alexander et al., 2012; Gottfried et al., 2009; Sorge, 2007; George, 2006) as well as internationally (Potvin & Hasni, 2014; OECD, 2006; Sjberg, & Schreiner, 2005; Osborne and Dillon, 2008). Several explanations have been offered to explain this decline as listed briefly in Section A, but it is likely that there is no single cause, suggesting that there is likely no single “one size fits all” solution. However, building better teacher-student and student-peer interactions, particularly

in cases of help-seeking and answering behaviors, is a promising area to focus in order to build confidence and increase student engagement in the classroom; through such confidence and engagement, more opportunities arise to build student interest in STEM. It is for this reason that the HAND-RAISE Intervention provides the necessary tools to allow the teacher to help facilitate these types of interactions while students are working in class and providing an environment aimed at supporting student engagement and confidence when asking and answering questions.

It is the aim of this intervention to increase student engagement, confidence, and interest in STEM-related subjects to reduce the widely-documented decline of such constructs over the course of the school year. In order to measure and evaluate the impact of the HAND-RAISE Intervention on student interest, particularly in that of math as it is this project’s domain of focus, we will use a series of surveys given to the students of the 25 Pilot Teachers over the course of the academic years coinciding with years 2 and 3 of the grant. These surveys will help to gain a sense of each student’s interest and perceived engagement toward math and STEM, their level of confidence in raising their hands in class to ask and answer questions, and also their sense of belonging amongst their peers in the classroom environment. We will derive the relevant survey items from previously developed, studied, and validated sources (Mahoney, 2010; Ostrow, 2018) and distribute the surveys to students through the Pilot Teachers at the beginning and end of year 2, before use of the tool in the classroom to measure the normal decline of these measures over a single school year, and then at the beginning and end of year 3 to measure how the use of HAND-RAISE in the classroom impacts each of these measures.

Similarly as is planned for the evaluation methods described in Sections C.3.a and C.3.b, we will explore potential heterogeneous effects within subgroups of students. Particularly, as larger declines of interest have been observed in female and minority

students, we will focus on such students to measure any potential effects of the tool.