ESPECIALIDAD EN DEMOLICION DE OBRAS

In Chapter 7, the development and testing of the AMS questions and marking rules

using Version 2 AMS data collected in the academic year 2018-2019 was presented. The findings from the Classical Test Theory (CTT) strand of the study showed that

the questions functioned well overall. However, the findings from the Inter-Rater Reliability (IRR) strand of the study showed that the corresponding AMS marking

rules still required further development and testing in order to reach the required level of functionality; this is one of the objectives of the work presented in the current chapter. The AMS was developed and tested at The Open University through the

Moodle OpenScience Laboratory (OSL) platform. Another of the aims of the overall research was to develop conceptual evaluation tools that could be widely used, and

the potential usefulness of the AMS would be enhanced if it could be made available on a range of different platforms. The second objective of the work presented in the

current chapter was to extend the use of the AMS to a wider context by administering it on a different educational platform with a different user-base, Isaac Physics.

8.2 Methods

8.2.1 Data collection

The Version 3 AMS data set was collected in the academic year 2019-2020 by splitting the AMS into three shorter length tests, each designed to have similar conceptual

balance to the overall AMS. This data-gather approach was similar to the one employed by Han et al. (2015; 2016), who split the original FCI into two half-length tests. The

AMS questions used in each test are shown in Tables 8.1, 8.2 and 8.3 below. Note that the standardized AMS question numbering is used in these tables, and throughout the

remainder of this chapter. Further, the abbreviation FRQ is used for free-response questions that require a short phrase or sentence to answer, and the abbreviation

Question Question type

Theme Concept AMS

question

1 FRQ Car collision Newton’s Third Law Q5

2 FRQ Stone drop Newton’s Second Law Q3

3 FRQ Stone drop Newton’s First Law Q4

4 FRQ Marble in track Newton’s Third Law Q7

5 FRQ(L) Marble in track Newton’s First Law Q8

6 FRQ(L) Cannon Newton’s Second Law Q14

7 FRQ Boy on swing Newton’s Second Law Q21

8 FRQ Woman pushing box Newton’s First Law Q28

9 FRQ Woman pushing box Newton’s Second Law Q29

10 FRQ Woman pushing box Newton’s First Law Q30

11 FRQ Office chairs Newton’s First Law Q31

Table 8.1: Table showing which AMS questions were used to assemble Isaac Test 1.

Question Question type

Theme Concept AMS

question

1 FRQ Ball toss Newton’s Second Law Q15

2 FRQ Balls on table Newton’s Second Law Q1

3 FRQ Balls on table Newton’s Second Law Q2

4 FRQ(L) Bowling ball Newton’s First Law Q16

5 FRQ Elevator Newton’s First Law Q19

6 FRQ(L) Hockey Newton’s First Law Q10

7 FRQ Hockey Newton’s First Law Q11

8 FRQ Hockey Newton’s First Law Q12

9 FRQ Hockey Newton’s Third Law Q13

10 FRQ Truck and car Newton’s Third Law Q17

Table 8.2: Table showing which AMS questions were used to assemble Isaac Test 2.

Question Question type

Theme Concept AMS

question

1 FRQ Office chairs Newton’s Third Law Q32

2 FRQ Moving blocks Newton’s Second Law Q22

3 FRQ Moving blocks Newton’s Second Law Q23

4 FRQ Elevator Newton’s First Law Q20

5 FRQ Truck and car Newton’s Third Law Q18

6 FRQ(L) Rocket Newton’s Second Law Q24

7 FRQ Rocket Newton’s Second law Q25

8 FRQ(L) Rocket Newton’s First Law Q26

9 FRQ Rocket Newton’s First Law Q27

10 FRQ(L) Hammer throw Newton’s First Law Q8

11 FRQ Tennis player Newton’s Second Law Q33

These tests were put onto the University of Cambridge’s Isaac Physics platform,

which hosts physics activities and tests at various education levels. Data were collected by offering the AMS tests as an activity on the Isaac Physics site, which has a primary

user base of high school students, undergraduate university students and high school teachers; this differs from the main user base of the OSL, which is almost exclusively

Open University undergraduate students. In another difference from the OSL versions of the AMS, the Isaac Physics questions told test-takers whether their typed answer was marked as right or wrong by the computer in real-time, thus providing them

with some instantaneous, basic-level feedback on their performance. Once users had completed the tests, the data were downloaded from the Isaac Physics site, where it

had been marked by the Version 3 AMS marking rules. All non-blank entries for each question were retained for calculation of the IRR statistics.

Since a thorough investigation into human marking had already been completed

(See Chapter 6 and Chapter 7), and the response matching was further advanced, the number of human markers was reduced from five to three. Initial marking was

done by the author and one member of the supervisory team, with a second member of the supervisory team arbitrating when the others disagreed, in order to establish

the Unified Human Marker (UHM) for use in comparison with the computer marking. Both supervisors had been amongst the previous markers, so were very familiar with

the AMS questions and marking guidelines.

8.2.2 Data analysis

The objective of the study was to test and develop AMS marking rules, not AMS

questions, so no CTT analysis took place here. To test the marking rules, the IRR statistics of marking agreement and Cohen’s kappa were used, as explained in Sub-

section 6.2.2. and used throughout Chapter 6 and Chapter 7; they were used in the current study as follows.

For each of the free-response questions, the marking agreement and Cohen’s kappa

statistics were calculated for the UHM against the computer marker; the values of these were then used to identify any problematic cases where the computer marking

rules were not functioning at the required level. The number of times that the UHM disagreed with the computer marker on each question was counted, and the number of false positives and false negatives were also counted. The false positives and false

negatives were used to develop new marking rules.

After making suitable changes to the marking rules, the versions of the marking rules pre-change and post-change were further tested against the UHMs from previous academic years, by calculating the marking agreement and Cohen’s kappa statistics for

these UHMs against the computer marking rules. This extra testing was carried out in order to check for consistency between different iterations of the marking rules. The

Version 3 AMS questions used to conduct these studies can be found in Appendix E.