Resultados del Estudio de Avifauna

The draft assessment tool provided to the cognitive interview participants is available in Appendix F: I-STUDIO Version for Cognitive Interviews. Note that the order of the assessment tasks in the draft assessment cognitive interviews and the large-scale field test differs from the order of tasks previously scrutinized by the expert reviewers. Cognitive interviews were conducted about 6 weeks after fall semester had ended, and none of the students brought notes or other resources with them to the interview. A summary of qualitative themes observed during cognitive interviews and the resulting changes to the instrument follow.

4.3.1 Summary of feedback.

Students did not share any questions or comments on the consent form. Three out of five students skipped the directions. No students had questions or comments about the directions. In item 1 (air traffic control), some students apparently did not recognize the ellipses to indicate that this is simply an excerpt of the data; they described specific cases in the data set. Most students seemed to follow the question well enough to propose reasonable research questions and solution strategy; no major revisions to item were necessary. One student remarked that it felt intimidating to propose a detailed strategy on the spot as opposed to an environment of a take-home exam or homework which would be more similar to the context in which he had done things like this in the past.

For item 2 (note identification), several students assumed that many participants would be involved in the note identification test, or they became distracted by generalizing to a population of students. In item 3 (display screen inspection), most students described reasons why statistical inference would not be required in this scenario, but sometimes remarked that they felt like they were tempted to overthink things. With item 4 (Walleye fishing), most students seemed to follow the question well enough and propose reasonable solution strategies. No major revisions to the item

seemed necessary although some students struggled to incorporate both the length and the weight data in order to determine which brother catches larger fish on average.

Item 5 (matched-pairs study design) was a difficult item for several of the students. Several students had false starts, and decided to reread the prompt and start over. Some students described a comparison for two independent samples. In item 6 (underlying

principle of inference), several students began by describing a hypothetical sample, extended the sample to some plausible population, summarized the appropriate statistic, and then determined the corresponding parameter without having first described a realistic scenario. Overall, item 6 functioned quite well, but several students never actually typed a description of their scenario into their response when incrementally building up to their response as described. Lastly, four out of five students reread the stem of item 7 (statistical inference NOT appropriate) at least once, but they generally seemed to think about the task appropriately and arrived at a reasonable solution.

Upon completion of the instrument, two out of five students reviewed their solutions. The total time in minutes taken for each student to complete the assessment was 26, 45, 50, 55, and 63. One student finished the assessment much more quickly than the other four, but did not appear quite as invested in her responses by comparison to the others. Another one of the five students began to describe test fatigue at the end of the

assessment; that student spent a total of 50 minutes to complete the assessment tool.

4.3.2 Summary of changes to the instrument.

In light of the student feedback summarized in Section 4.3.1, a number of changes to the I-STUDIO assessment tool were warranted. The draft instrument presented during the cognitive interviews is shown in Appendix F: I-STUDIO Version for Cognitive

Interviews, and the improved instrument showing changes in response to their feedback is available in Appendix G: I-STUDIO Version for Field Test. Changes to the instrument spanned the entire assessment tool including the instructions, vignettes, and item

Since several students skipped the directions in the cognitive interviews, the

directions were modified prior to the field test such that each of the key instructions were listed and accompanied by checkboxes that the student must acknowledge before

advancing to the first item. For item 1 (air traffic control), the ATC preparation data table header was modified to state “Example ATC Preparation Data (Showing five of the nineteen students)” in order to underscore that the data shown are only an excerpt. Item 2 (note identification) was modified to specify an individual student by name in order to make clear that the problem setting is concerned with evaluating the results for only one student.

The prompt for item 3 (display screen inspection), part b was improved to clarify that the decision to accept or reject the bulk order is based on the data gathered by the

engineer in order to suggest that the student need not invent a new method in order to invoke statistical inference. A modification was made to the item 4 (Walleye fishing) vignette to state that only the length of each fish was recorded so students would not complicate their analysis proposal by incorporating more than one measurement of fish size. Part b of item 4 (Walleye fishing) was also updated to clarify that the comparison is based on the data collected on the two week fishing trip.

No noteworthy changes were made to the content of item 5 (matched pairs study design). Item 6 (underlying principle of inference) was modified to add an additional bullet point prompting students to “briefly describe your chosen scenario and state the question of interest you would explore using statistical inference in that scenario” since that information was omitted from several of the responses submitted by cognitive

interviews participants. The prompt for item 7 (inference is not required) was simplified by removing comments about deterministic results and cleaning up lengthy sentences. Also, the prompt for item 7 was modified to show bullet points listing each component the response should address as paralleled by item 6.