Carácter constructivo.

The experiment was designed to test the hypothesis that our system was more effective than a classroom setting. The definition of a classroom setting to us is that a student is given a programming problem, for example on a whiteboard, after which they would try to answer it on paper, then once the student believed they were finished the teacher would give them the answer, and the students would compare that solution and their own solution attempt.

Pre and Post tests

Two tests were created to test the participants’ Java knowledge before and after using the system. The tests were designed to be of similar difficulty, but in order to eliminate any variability between the tests, some participants received test 1 as their pretest and test 2 as their posttest, whereas others received the reverse.

Each test had 5 problems (4 short answer and 1 multichoice), and can be found in the appendices. Each question covered a different section of the curriculum they had covered in class, as well as testing a different programming skill. The tests were identical in terms of skills tested, but the questions were different. The type of each question was:

• What will a given variable be set to after the code has executed? (Understanding of value transference)

• For what reasons will this method not compile? (Understanding of syntax) • What type should the blank be filled in with? (Understanding of types)

• Into which branches will the program flow? (Understanding of conditional pro- gram flow)

• Write a line of code assigning a calculation (Basic code generation)

Group Assignment

Two types of random assignment needed to be taken into account: between the experi- mental and control groups, and between which test to give the participant as the pretest. These options give us four possible permutations - Experimental and Test 1 first, Exper- imental and Test 2 first, Control and Test 1 first, and Control and Test 2 first. The ideal situation is to have the distribution of participants across these four groups as even as possible; therefore, assignment was random, but with a constraint that the final result must have as even a distribution as possible.

The system contained 8 problems of increasing difficulty; 3 display problems, and 5 predicate problems. There were two versions of the system. Each version allowed the participant to log in, and see the problem-solving interface. A student could move through the system problem by problem, create solutions using the tiles, enter code into these tiles, and could receive a full-solution (a static image) at any time by clicking the ‘Done - Show Full Solution button’. Once the full-solution has been seen for a problem, the participant is prohibited from continuing to work on that problem, and cannot come back to it in the future. A participant is warned of this during the initial session briefing,

and also when they click the Show Full Solution button (a confirmation pop-up dialog is shown).

The experimental version of the system additionally allowed the participant to sub- mit a solution and receive feedback on it. Therefore, this version had made the Submit button available, as well as a feedback-type selector. In contrast, the only feedback the control system would give was the full solution.

Feedback Selection

For this study, the experimental group was given three types of feedback selection to choose from; ‘simple feedback’, ‘hint’, and ‘all errors’. ‘Simple feedback’ gives a sim- ple “correct”/“incorrect” response, ‘hint’ displays one of the descriptive error messages, and ‘all errors’ displays all of the descriptive error messages.

In addition to this, both groups could receive feedback in the form of the full- solution. As the control group was not submitting a solution and therefore not receiving fine-grained feedback, they could not choose the feedback type.

Procedure

All sessions were held in CSSE labs; by doing this, we were able to accommodate several people participating in parallel, giving participants more freedom in choosing session times without fear of clashing with other students. In total there were 7 ses- sions, with the minimum number of participants in a session being 1, and the maximum being 4. The sessions were held in two different lab locations, depending on what was available (the composition of these two different locations was essentially the same). Each lab had IBM-compatible computers running the Fedora Core 4 Linux operating system, with the Firefox web browser (version 1.5) installed.

A participant would sit down at a computer, log in, and run the web browser. They would then be given the information sheet and consent form and be asked to complete them. They were also assigned a non-identifiable usercode. After completing the forms, they would be given a pretest, composed of two multichoice and three short-answer questions, which they would work on for a maximum of 10 minutes (all participants completed the tests within the required time). After completing the pretest, the partic- ipant would be asked to log into the system using the given usercode. The experiment coordinator would then briefly explain how to interact with the system.

After using the system for a maximum of an hour, or completing all the problems in the system (whichever came first), the participant would be asked to log out of J- LATTE. They would then be given a posttest, with the same conditions as the first test (complete within a maximum of ten minutes), followed by a questionnaire. This marks the completion of the experiment session for this participant, and they would then receive the compensation for their time.