Tasks
During the testing, respondents were given four tasks in total, which represented different cognitive levels of work. This part covers empiric processing of the first two. Varying cognitive difficulty of the tasks was created intentionally. The first question asked mapped the basic orientation in the text and was oriented towards the central topic of the exposed page. Respondents were able to retrieve the correct information from both the verbal and the visual field. The second task posed a complex mental operation. Based on clues in verbal or visual fields, pupils needed to deduce additional information. By having to complete, compare and combine facts, respondents were forced to use demanding logical mental operations.
Which new art movements were created in the 1920s?
(surrealism and abstract art): correct solution requires simple mental operation – to reproduce the text or captions of the images;
Where did René Magritte and Josef Šíma draw their inspiration?
(being surrealists, from dreams, imagination, chance): correct solution requires complex mental operations – especially deduction.
56
Tab. 1 contains the time it took each pupil to solve the first two analyzed tasks. The solution times differ greatly. The more difficult task does not always require more time for solving.
Task 1 Task 2 Total
Ondra 288 191 479
Vojta 68 224 272
Zuzka 41 215 256
Ema 139 76 215
Viktor 75 69 144
Tab. 1: Time in seconds required for pupils to solve the task
When solving the first task, pupils split into two strategically different groups. We call them: 1. Readers, who read the text in much detail and for a long time before answering the question. These pupils had difficulties coping with the uncertainty which was caused by the relatively difficult task (by the high structure). These pupils wanted to be sure about their answers and repeatedly checked the text to verify their correctness.
2. Scanners, who scanned through visuals, captions and the text with the same amount of focus to each part. They chose the text’s semantic core (vital information), which they were able to compare, extend and validate using the images and their captions.
When solving the first task, pupils split into two strategically different groups. We call them: 1. Readers, who read the text in much detail and for a long time before answering the question. These pupils had difficulties coping with the uncertainty which was caused by the relatively difficult task (by the high structure). These pupils wanted to be sure about their answers and repeatedly checked the text to verify their correctness.
2. Scanners, who scanned through visuals, captions and the text with the same amount of focus to each part. They chose the text’s semantic core (vital information), which they were able to compare, extend and validate using the images and their captions.
Fig. 1 and 2 show how the text was processed by the pupils from the first and second group, respectively.
57
Fig. 2: Pupils who retrieved only the core information and constantly compared it with the content of the images and their captions (Zuzka, Vojta and Viktor).
The second task, more demanding on mental operations, again divided pupils into two groups: 1. Effective scanners, who only processed the part of the text which contained the necessary information to correctly answer the question (see Fig. 3). These pupils were also significantly faster in finding the correct answer (see Tab. 1).
2. Readers, who processed a significantly larger part of the text to correctly answer the question (see Fig. 4).
Pupils who were able to obtain necessary information from the verbal field spent little time in the visual part of the material. As it turns out they no longer returned to the visual components to verify their understanding. It is worth noting especially because if they used visual components before to solve the previous task, they checked the long text along with them as well, comparing and validating their answers.
58
Fig. 4 shows that some pupils spent a long time looking for the answer. These pupils have also devoted a lot of their attention to the visual part of the material. Eye movement trajectories and the number of saccades confirm that pupils read the continuous text and did not truly understand its content; therefore trying to find support in the visual parts. They were comparing the information, but kept going back to the text as the relevant authority.
Fig. 4: Pupils who studied the text for a long time and in great detail (Ondra, Vojta and Zuzka)
Text material was divided into individual Areas of Interest (AOI) based on the location of the information relevant for solving the tasks. Pupils spent 65 – 89 % of the total time looking at areas without relevant information by the task 1. All pupils spent a relatively long time watching relevant visual information. Pupils – readers spent significant time on relevant verbal information. On the other hand pupils spent 56 – 74 % of the total time looking at areas without relevant information by the task 2. All pupils (Zuzka being the only exception) nearly ignored the relevant visual information. Pupils – readers spent significant time on relevant verbal information.
Conclusion
Qualitatively oriented research gave us extensively detailed information on how respondents worked with the verbal and visual fields, how they overviewed the material, to what extent they rested on individual structural elements, which components they utilized, for how long and how often. Using the chosen technology, we were able to exactly record the eye movement trajectories, number of saccades and the interest focuses of pupils. At the same time we evaluated the correctness of answers. We discovered two basic approaches of pupils when working with text, which we named detailed reading and effective scanning. Although we cannot yet conclude the relation between the used approach of pupils to search for and their ability to focus on relevant information in the sense of reading attention due to the small number of respondents, the research brought up interesting insights, which will be the focus of a follow-up research.
The results serve as the basis for the following discussion. With detailed reading, some pupils rested primarily on verbal fields of the text, even though the part’s content did not match the given question. Certain perfunctoriness of work with the text shows lack of comprehension and probably stems from acquired reading stereotypes. Worth noting is the fact that detailed
59
reading was the preferred style even for questions with lower cognitive level requirements, e.g. simple information retrieval or its reproduction.
The table with time spent on solving individual tasks shows that almost all pupils surprisingly spent the shortest time looking for the answer for the cognitively more demanding question. This shows that the important moment during reading is the primary orientation within the text.
Pupils worked effectively if they “scanned” the text while searching for information. Mechanical reading through the text without an effort to learn to navigate it resulted in longer work times. Effective scanners were able to find the required information faster. Interestingly, these pupils worked primarily with verbal text field, not with images.
Insights produced by this research will be further examined. We expect to design the next research in such fashion as to find specific relations between executive functions (especially attention, in the sense of focusing on relevant context). For this purpose we will be employing a complementary psychodiagnostic test. The research will be further extended by semi- structured interviews with respondents that will uncover further connections concerning text comprehension, work with text material and preferences of components that facilitate pupils’ learning. Additionally, thematically different material may be introduced and work strategies of pupils on different content may be compared.
References
ČERVENKOVÁ, Iva. Žák a učebnice: užívání učebnic na 2. stupni základních škol. Ostrava: Ostravská univerzita v Ostravě, Pedagogická fakulta, 2010. 107 p. ISBN 978-80-7368-924-7. EMICK, Jessica & Marylin WELSH. Association between formal operational thought and executive fuction as measured by Tower of Hanoi-Revised. Learning and Individul
Differences. 2005, vol. 15, p. 177-188. ISSN 1041-6080.
GAVORA, Peter. (Eds.). Ako rozvíjať porozumenie textu u žiaka. Nitra: Enigma, 2009. 193 p. ISBN 978-80-89132-57-7.
CHAVES, Silvia, Patrizia VANNINI, Kay JANN, Pascal WURTZ, Andrea FEDERSPIEL, Thomas NYFFELER, Mathias LUETHI, Daniela HUBL, Roland WIEST, Thomas DIERKS, & René MÜRI. The link between visual exploration and neuronal activity: A multi-modal study combining eye tracking, functional magnetic resonance imaging and transcranial magnetic stimulation. NeuroImage. 2012, vol. 59, p. 3652–3661. ISSN 1053-8119. JANSIEWICZ, Eva, Mashock. The Relationship between Executive Functions and
Metacognitive Strategy Learning and Application. Psychology Dissertations. [online]. 2008, Paper 42.[cit. 20140530] Available from: http://digitalarchive.gsu.edu/psych_diss/42. JANSIEWICZ, Eva, Mashock. The Relationship between Executive Functions and
Metacognitive Strategy Learning and Application. Psychology Dissertations. [online]. 2008, Paper 42.[cit. 20140530] Available from: http://digitalarchive.gsu.edu/psych_diss/42. JOHNSON, Egil, Borre. Textbook in the Kaleidoscope. A Critical Survey of Literature and Research on Educational Texts. Oslo: Scandinavian Univerzity Press, 1993. 456 p. ISBN 978- 8200215066.
KARNATH, Hans Otto & Walter STURM. Störungen von Planungs- und Kontrollefunktionen. In: Wolfgang HARTJE & Klaus POECK. (Eds.) Klinische
60
KERR, Aurora & Philip David ZELAZO. Development of „hot“ executive function: The children’s gambling task. Brain and Cognition. 2004, vol. 55, p. 148-157. ISSN 0278-2626. KORŠŇÁKOVÁ, Paulína & Daniela HELDOVÁ. (Eds.). Čitateľská gramotnosť slovenských žiakov v štúdii PISA 2003. Bratislava: Štátny pedagogický ústav, 2006. 60 p. ISBN 80-85756- 96-X.
KOUKOLÍK, František. Já. O vztahu mozku, vědomí a sebeuvědomování. Praha: Karolinum, 2003. 382 p. ISBN 80-246-0736-0.
KOUKOLÍK, František. Lidský mozek. Funkční systémy. Norma a poruchy. Praha: Portál, 2002. 400 p. ISBN 978-80-7262-771-4.
PRŮCHA, Jan. Učebnice: edukace zprostředkovaná médiem. In: PRŮCHA, Jan, Moderní pedagogika. Věda o edukačních procesech, p. 269-306. Praha: Portál, 2002. 481 p. ISBN 80- 717-8631-4.
SERENO, Anne B., Shelly L. BABIN, Ashley J. HOOD & Cameron B. JETER. Executive functions: Eye movements and neuropsychiatric disorders. Encyclopedia of Neuroscience. 2009, vol. 4, p. 117–122. ISBN 978-0-08-045046-9.
61
EVALUATION OF PEER-REVIEW QUALITY IN COMPARISON
TO TEACHERS’GRADING
Veronika Dropčová, Zuzana Kubincová
Department of Informatics Education, Faculty of Mathematics, Physics and Informatics, Comenius University in Bratislava, Mlynská dolina 1, 842 48 Bratislava, Slovak Republic
[email protected]; [email protected]
Abstract
Peer-reviews were found as useful to engage students with the outputs produced by their classmates, stimulate feedback, exchange of opinions, and thus social learning. It is however often the case that teachers as well are providing their evaluation to all items that are subject to peer-review. The question whether the “doubled” teacher's evaluation is necessary, and if it can be partly or entirely replaced by the peer-reviews is therefore relevant. We investigated the relevance of peer-review outputs in our course. We showed that these outputs are not significantly affected by friendly relationships in the cohort, however, the overall reliability was not very high when compared with teachers' evaluation. We also showed that the results can be improved by selecting a subgroup of more reliable students (in our case, students with previous experience with the course topics), and also possibly by adjusting the format of the peer-reviews to be more in line with teachers' evaluation criteria.
Keywords
Peer-review, assessment, education, blogging
Introduction
The peer-review technique has been used to enhance the learning methods for a number of years. According to several studies [11, 9, 17, 15, 13] the mutual reviewing of the colleagues' work helps the students to develop the critical and analytical thinking, communication skills, constructive criticism, etc. It can also stimulate the students' interest in course subject, foster students' engagement with the output of the others (e.g., essays, blog articles, wiki pages, but also software and other products) and stimulate feedback, exchange of ideas and opinions. Peer reviewing was successfully employed not only in writing courses but also in engineering, computer science, microeconomics, etc. Here it has the same benefits for the students, giving them experience in communicating with the kinds of audiences they will need to write for on the job after graduation [3, 7]. Such an approach is highly in line with constructivist learning theories, especially with social learning [16, 14].
Contrary to the undoubted contribution to the quality of the learning process, the peer-review activities militate against the teacher since she is often overwhelmed by providing feedback, correcting and marking of both – assignments and peer-reviews. The idea of using students' feedback from the peer-reviews in evaluation and reducing teacher’s workload this way was previously explored [7, 2, 4, 5, 12, 15].
The basic problem of such an approach is the question whether the students are able to adequately judge the quality of their peers’ contributions and what criteria they should follow. Aiming to solve this problem, we conducted a research in one of our courses where blogging assignment with peer-reviews was employed. In this assignment, teachers' evaluation of
62
students' articles and reviews is binary (approved/rejected), but there are also evaluation points assigned to the students after achieving a stated number of approved articles.
As we found out that basic approaches, such as taking the average students' rating of each article are unlikely to be successful, we concentrated on the questions, if and how it is possible to identify at least certain group of reliable students, whose peer-reviews can be used to distinguish between the accepted and the rejected articles. Initially, we focused on revealing the friendly relationships among students and consecutively we studied whether these relationships influenced the reviews between friends.
Afterwards we tried to identify the group of reliable students in two ways. Firstly, based on the students’ performance in the very beginning of the assignment we selected a group of students whose reviews were well-aligned with the teachers’ assessment and studied the rightness of their peer reviews in the remaining parts of the assignment.
Then, we identified a group of students (called experts) with previous experience with the topic of the course (according to the judgement of their classmates) and studied if their peer-reviews were more reliable than the others.
During our research, we also concentrated on identifying the right criteria which could be followed by reliable students to produce the peer-reviews better corresponding to the teachers’ assessment.
Our results were not completely positive, and given the initial character of this study we did not expect that (e.g., the peer-reviews were not anyhow calibrated w.r.t. goals of this study, they were conducted as usual in the last years). We found out, to our surprise, that at least in our data the friendly relationships in the cohort did not bias the rating in a statistically significant way. We also found out, that the experts group was much more reliable in their ratings and we were able to tell apart the accepted and the rejected articles with the success rate of 75.6%. Finally, when we only considered the rejection w.r.t. topicality, the expert group achieved the success rate of 94.5%.