SISTEMA DE DOSIFICADO

The use of deep learning methods in the context of education and learning ana- lytics has led to a growing body of research focusing on better modeling student

behavior and performance. Within this domain, a large number of such works have begun to utilize recurrent neural networks (RNNs) [WZ89], for their ability to model complex temporal patterns of student behaviors. These models have shown great promise in recent works modeling student knowledge and short-term performance

[PBH+_{15][KLM16][XZVIB16], predicting student graduation [KVG18b] and real-}

time performance [KVG18a] in MOOCs, detecting student affective state [BBH17], and predicting long-term outcomes [SBPH18][YLYY18].

Despite the often-reported high performance of these models as applied to their respective tasks in education, the large number of learned parameters and complex model structures often make them difficult to interpret. While this difficulty ap- plies to the learned parameters of the model, this does not mean that the estimates produced by the models are similarly uninterpretable and can be utilized to explore student behavior over time at fine levels of granularity (e.g. see [BBOH18]). Some- thing as simple as observing the estimates themselves, or even model performance, over time can lead to better insights into the modeled behaviors as well as when action may best be taken through intervention.

The high complexity of deep network structures allows the model to learn rich

feature embeddings, either explicitly (e.g. [ZXZ+_{17]) or implicitly (e.g. [YLYY18]),}

that better describe the data to make better-informed model estimates. In this way, such models also support the application of transfer learning [Pra93] to better understand the relationship between outcomes of interest by providing the means to observe how learned features generalize across prediction tasks.

Feature Name Description

Action Type One-hot encoding of the action (attempt, help request, etc.) Attempt Count The number of attempts made up to the current action

Hint Count The number of hints requested up to the current action Problem Count The number of problems seen up to the current action Probability of Action The probability of the current action given the problem Probability of Action

Given Action Count

The probability of the current action given both the problem and the number of actions taken in the problem

Probability of Response When an attempt, the probability of a student answering with the specific response given the problem

Probability of Response Given Action Count

When an attempt, the probability of a student answering with the specific response given the problem and number of actions taken in the problem

Cumulative Log Likelihood of Response

The cumulative log likelihood of a student answering with the specific response on the problem

Normalized Time Taken The amount of time since the last action, z-scored within action type and problem

Used Penultimate Hint Whether the second-to-last hint has been seen before the current action

Used Bottom Out Hint Whether the student has seen the last hint (containing the answer) before the current action

Correctness Correctness or incorrectness if the current action is an attempt, or a non-attempt (as a 3-value one-hot encoding)

Preceding 3 Actions One-hot encoding describing the previous three actions taken excluding the current action

Current and Preceding 2 Actions

One-hot encoding describing the previous three actions taken including the current action (current and previous 2)

Table 4.1: Description of the generated action-level features.

4.3

Dataset

The data used in this work is comprised of students working with ASSISTments during the 2016-2017 academic year. ASSISTments is a web-based learning plat- form that provides the tools for teachers to assign classwork or homework content for which students receive immediate correctness feedback [HH14]. While working through each assignment, many problems supply students with optional on-demand computer-provided aid; hints, of which there may be from 0 up to several available, supply students with an instructional message, while scaffolding, when available, breaks the problem into smaller steps to solve. In addition to these, the system

provides a “bottom-out” hint for every problem that supplies the students with the correct answer if the student is unable to solve the problem as students are not al- lowed to progress to subsequent problems until the correct response is entered inside ASSISTments.

ASSISTments is used by several thousands of distinct students daily, most of which being in 6th-8th grade solving primarily mathematics content, providing a dataset of sufficient scale and variation to apply deep learning methods that often require such data. While the majority of students are of late-middle-school age, the dataset itself is comprised of all users of the system during the aforementioned academic year. The data is filtered to include only student interaction with mastery- based assignments, known as “skill builders” in the system, where the completion threshold is designated to simply require students to answer three consecutive prob- lems correctly without the use of computer-provided aid (i.e., without hints, scaf- folding, or bottom-out hints). In recognition of wheel spinning as an undesirable learning behavior, the system implements a “daily limit,” stopping students on the skill builder assignment for the day if the completion threshold is not reached by the tenth problem (except in the case where the student is about to reach the threshold on or directly following the tenth problem); the system provides the student with an instruction to seek additional help and return to the assignment on the subsequent day.

As teachers using the system assign a range of content, both made available through the system as well as self-built material, we include data from skill builder assignments where at least 10 students started the assignment and the overall com- pletion rate is at least 70%. These limitations help to remove outliers such as sample classes and optional supplementary assignments where the teacher does not require every student to complete. These outlier cases are excluded as we would argue that

attrition due to such factors is not stopout as we have defined it within this task (e.g. low unproductive persistence).