Introducción a las Herramientas Para Evaluación

4.3.1 Test selection

This section gives an overview of aspects of the presented studies that are useful for people who are looking for a scientific reasoning test. It is clear from the review that tests with various test formats and for different target groups exist. However, populations outside of educational institutions are underserved so far. It might be hard to find a fitting test for these populations, especially since Study 2 warned us not to use a test within a population that it was not intended for. It is possible, for instance, that the item difficulties will not be in the optimal range to differentiate between test takers on higher and lower latent ability levels. Thus, the fit of the test to a specific population should definitely be checked before applying the test to a large sample. If these checks indicate a misfit it might be necessary to develop a test that is adapted to the target population.

Once a person looking for a test got an overview of potential tests that could be used, how should they go about selecting one or several tests? The review contains a useful guide for how to proceed in such a situation and Study 2 emphasizes two of the central points of these recommendations. First, it is very important that the selected test really matches the scientific reasoning construct or the subpart of scientific reasoning that one intends to measure. Study 2 demonstrated that minor item differences can matter. Therefore, not any scientific reasoning test will do when you want to measure a specific aspect of scientific reasoning and if you want to measure scientific reasoning as a broad construct the test should not only focus on one scientific reasoning skill, respectively. Similarly, it should be checked if the test measures aspects that are not part of your scientific reasoning conceptualization, e.g. quantitative reasoning. Tests or subscales of tests that do that should be avoided.

Second, it is necessary to find out which psychometric properties of the test were

evaluated by the creators of the test as well as by other authors who used the test and what the results were. Study 2 hints at some of the most pressing questions that should be answered by the literature about the test: Can you interpret subscale scores you are interested in or does only the total score carry valid information? Are comparisons between groups that you want to look at in your study free from bias, e.g. the comparison of students from different school types? Does the test tell you anything about aspects of criterion validity that are relevant to you, e.g. predicting science grades? If the test authors have not made these checks yet, try to conduct them before using the test in the final study. Furthermore, it should not be forgotten to have a critical look at the test construction process in addition to the evaluation process. For instance, it is crucial to find out which assumptions about scientific reasoning were made during test construction.

4.3.2 Basing decisions on a scientific reasoning test

Next to matters of test selection and administration another relevant aspect to think about are the decisions that can be based on the test results. In the field of secondary education it was possible to observe the influence of the outcomes of the PISA assessment (OECD, 2007) on educational decision making in recent years. Similar assessments for higher education are in demand and in development at the moment (Shavelson & Huang, 2003; Zlatkin-Troitschanskaia et al., 2015). There are states in the USA where it is required that scientific reasoning is tested in higher education institutions and these states base academic program decisions on the outcomes (State Council of Higher Education for Virginia, 2007).

However, the presented studies remind us of the limitations of the current state of scientific reasoning assessment and thus test results should be interpreted very carefully.

General Discussion 125 Using another parallel to intelligence testing, we have to avoid a situation where “consequent action affecting the welfare of thousands of persons is proposed, and even taken, on the grounds of – nobody knows what” (Spearman, 1927, p. 15). Despite the variety of tests that exist, and even when a test is selected carefully, it still seems wise to not base high-stake decisions on scientific reasoning tests at the moment. When it comes to higher education, Study 2 shows that it might be unfair to compare universities that vary in something as seemingly minor as the number of biology students. While the bias was not big enough to distort the rank order of group means in the comparison of biology and physics students, this might not be the case when biology is compared with other majors.

Selecting people for graduate programs based on a scientific reasoning test would be especially problematic. Even if measurement invariance holds, it is still possible that there will be a violation of selection invariance among groups with a different latent ability (Borsboom et al., 2008). Selection invariance is important in a situation in which a test is used for selecting qualified people. It means that there is no significant difference for any subgroup regarding the percentage of qualified people who are correctly identified as

qualified (sensitivity) and the percentage of unqualified people who are correctly identified as such (specificity). If selection invariance is violated, a test will have a different sensitivity and specificity for different groups.

Many scientific reasoning tests are not based on a model from item response theory (IRT), which should be required if high-stake decisions are made (Heene, 2007).

Additionally, when test results start to have severe consequences, the well-known problem arises that too much focus within the education system will be placed on improving test scores as opposed to improving instruction (Wiliam, 2001). In a situation in which the quality of the tests is not fully explored, which is the case for the assessment of scientific reasoning, the damages resulting from this problem would probably be even worse. Thus, it would be

good to listen to the people who remind us that it will take a long time before educational standards can lead to high-stake assessments (Pellegrino, 2013). In the meantime, we should not forget that on top of summative assessment it is also worthwhile to invest in formative assessment (Wiliam, 2011).

4.4 Limitations

The conclusions that one can draw from the results of the presented studies are limited by several factors: Both studies focused exclusively on scientific reasoning and excluded NOS and scientific argumentation. While it is a good idea to not confound these different constructs, the focus on scientific reasoning naturally limits the scope of this thesis.

Furthermore, the review did not evaluate the results of psychometric property checks in depth and instead just reported how psychometric properties were checked because the results were too different in nature to make a valid comparison. Besides, there might be tests that target scientific reasoning skills but use none of the terms that were used in the literature search when they describe what they are measuring. These tests would not have had a chance to be included in the review. However, our selection of tests should reflect the set of tests that a person who is looking for a scientific reasoning test would come across as long as they use the common search terms for scientific reasoning skills. The selection of tests for the review was further narrowed by the requirement that the test has to provide at least some form of validity check. While it might be insightful to look at tests without validity checks, this criterion was necessary to exclude any ad-hoc measurements, which were not meant for repeated use. Finally, considering that the second wave of test development probably has not reached its end, it seems advisable to repeat or update the review of scientific reasoning tests within a few years.

General Discussion 127 Study 2 only looked at university students and only used a single scientific reasoning test, albeit a very commonly used one. This reduces the width of generalizations that can be drawn from the study. The observation that item-specific aspects play such a big role further diminishes the generalizability of the results. It makes it more likely that different results will be obtained with different items. However, the importance of minor factors is also an

interesting result in and of itself. Additionally, the conclusions that can be drawn from Study 2 are somewhat limited by ceiling effects in the scales. The ceiling effects might have caused a reduction of variance which would make it more difficult to detect the accurate size of the relations between variables.