In order to test for conceptual understanding of physics, multiple-choice concept in- ventories are commonly used. The earliest multiple-choice questions were used to test
US army recruits in World War I (Mathews, 2006), whilst muliple-choice questions were first used in the educational context later in the 20th century (Bacon, 2003). An
advantage of multiple-choice questions is that they are versatile, meaning they can be used to test a wide range of topics (Bull and McKenna, 2004); on the other hand, free-
response questions are advantageous because they can be used to test precise content (Betts et al., 2009; Ferrao, 2010). Because of the way that multiple-choice questions
are set up, it is difficult to make use of them to assess more advanced learning out- comes (Conole and Warburton, 2005), although there have been some efforts to do so
(Itza-Ortiz et al., 2003; Gwinnett and Cassella, 2011). The main motivation for using multiple-choice questions over other question types has been that they can be marked quickly, which makes a practical difference to academics who have large class sizes
to assess (Woodford and Bancroft, 2005). However, the same authors asserted that
the quality of the multiple-choice questions needs be high to justify their use, which requires such questions to be carefully designed and tested.
A common flaw identified with multiple-choice questions is that it is sometimes
possible to figure out the correct answer by working through each of the answer op- tions to seek a least implausible option, meaning that such questions are not assessing
the skills that they are supposed to (Sangwin, 2013). Additionally, it is possible for students to give correct answers to multiple-choice questions by guessing; when this
happens, the student is not demonstrating the intended understanding (Crisp, 2007). A counter-argument to this criticism is that it is unlikely that a student could pass an
entire multiple-choice test through guessing alone (Downing, 2003); however, it stands to reason that well-placed, successful guesses on multiple-choice questions can make
a difference in the grade awarded to a student in a borderline case (Burton, 2005). In addition, there are different cognitive processes involved in answering free-response
questions and multiple-choice questions (Nicol, 2007). With this in mind, there are ways in which the effectiveness of multiple-choice questions can be increased, such as
by developing plausible distractor options (Dick-Perez et al., 2016), or by carefully con- sidering the order in which the questions are asked (Gray et al., 2002). To ensure that
questions are able to test the understandings that they are designed to, it is important to prepare high-quality questions, regardless of whether they are multiple-choice or
free-response (Bull and McKenna, 2000).
Simon and Snowdon (2014) compared multiple-choice and free-response questions
in the context of assessing coding skills. Students were given a selection of multiple- choice and free-response questions based around interpreting code, and it was found
that students in general found the free-response questions to be the more difficult of the two questions types. The authors pointed out that with multiple-choice questions,
the student needs to choose from a list of ideas constructed by somebody else, and can also employ guessing and elimination strategies to reach the answer; on the other
hand, the authors postulated that free-response questions require a student to think more deeply, since they are required to construct their own answers, and that further
insight about student thinking can be gained from students free-response answers as a result of this.
A comparison of user response to multiple-choice and free-response questions was
undertaken by Bridgeman (1991). Bridgeman used the Graduate Record Examina- tion General Test (GREGT), a standardized test taken by applicants to US graduate
schools, as the basis for the study. A set of students were given multiple-choice ver- sions of some of the GREGT questions to answer whilst a control set students were
given free-response versions of the same questions to answer. Correct and incorrect answers from the multiple-choice version of the questions were compared to the an- swers given to the free-response version of the questions, and it was found that the
scores on the multiple-choice questions and the corresponding free-response versions were similar. Bridgeman concluded that, under certain circumstances, multiple-choice
questions and free-response counterparts can be used interchangeably to assess student understanding.
Another study comparing multiple-choice and free-response questions was carried
out by Funk and Dickson (2011), who aimed to investigate the effect of these question types on student performance. To this end, the authors created multiple-choice and
free-response versions of a set of psychology exam questions. These questions were split and given to fifty students to attempt as follows: twenty-five students did 10 free-
response questions before doing a 50-question multiple-choice test; the other twenty- five did 10 free-response questions after doing a 50-question multiple-choice test. In both cases, performance on the multiple-choice versions of the questions was higher
than on the free-response versions, indicating that the students found the free-response variants of the questions to be more difficult than their multiple-choice counterparts.
This finding agrees with that of previous work by Hudson (2010), who found that multiple-choice questions were easier to answer than free-response questions for high
school chemistry students.
Woodford and Bancroft (2005) pointed out that the apparent difficulty of a multiple- choice question can reveal flaws in its construction: if a question appears easy then its
distractors may be ineffective; if it appears difficult then the distractors may be mis- leading, or the question wording may be ambiguous. Whatever the difficulty reveals
about the question, appropriate steps must be taken to improve it if there is an issue, as outlined in Woodford and Bancroft’s design priorities. On the test level, the number
of questions in the test overall, as well as the order of these questions, will affect the effectiveness of the test. On the question level, the number of options given to the multiple-choice question, the order of these options, the wording of the question itself
and the wording of the distractors all affect the effectiveness of the question. Woodford
and Bancroft argued that changing these factors on a test-level and a question-level basis can lead to improved quality multiple-choice questions.
Woodford and Bancroft proposed that provided their design priorities are applied, multiple-choice questions can be used to test for a deeper level of understanding, but
the examples that are given refer to mathematical questions. However, it can be argued that multiple-choice questions, no matter how refined or well-designed, may
be inappropriate for testing conceptual understanding (Conole and Warburton, 2005), which is more descriptive and less procedural.
Shuhidan et al. (2010) gave questionnaires to 66 computer programming instructors
to investigate their perspectives of using multiple-choice questions to assess their stu- dents. Over one third of the instructors had confidence in using high quality multiple- choice questions to assess student learning outcomes. Instructors would choose to
use multiple-choice questions with their students because they can be answered by weaker students, they are good for revision, and because they prepare students for
multiple-choice questions in other courses. On the other hand, some instructors were discouraged from using multiple-choice questions because they felt that they were too
easy, and that they encouraged students to guess. Of these instructors, some favoured the use of free-response format questions, both essay and short-answer lengths, to test
student understanding.
There also appear to be differences between how males and females perceive and perform on multiple-choice questions, though the reasons for this are not clear (Ben-
Shakter and Sinai, 1991). Gipps and Murphy (1994) found that 15-year old girls preferred answering free-response questions, whereas 15-year old boys preferred an-
swering multiple-choice questions. Ben-Shakter and Sinai (1991) found that males outperformed females on multiple-choice questions for both the ninth-grade and uni-
versity applicant age groups. Ben-Shakter and Sinai did not believe that differences in guessing patterns between males and females were alone responsible for differences
observed in performance on multiple-choice questions. This differs from the findings of Richardson and and O’Shea (2013), who found that males and females were as likely
as each other to get a question right when they attempted it, but that males were more likely to make an attempt in the first place. Such discrepancies in the literature highlight the poor understanding of demographic differences in responses to assessment
items.
The literature presented in this section has suggested that multiple-choice ques- tions are versatile and easy to use, but they are inappropriate for assessing advanced
learning outcomes, such as conceptual understanding. In addition, students can em- ploy eliminate-and-guess strategies to work out the correct answer to multiple-choice
questions, or simply guess the correct answer without any knowledge of the subject matter being tested. Further, multiple-choice questions require students to select an
option from a list of responses constructed by somebody else, so they are not required to use their own thinking to answer the questions. Use of a free-response question
format would encourage students to think deeply, and the answers to such questions would provide useful insight to educators about student thinking. However, free-
response questions are not straightforward to mark and are therefore time-consuming for human markers, which multiple-choice questions are not. If there were a way of
automatically marking free-response answers, then this could provide a viable alterna- tive to the widely used multiple-choice questions. The idea of automatic marking is
the focus of the next section.