4. ANÁLISIS DE RESULTADOS Y DISCUSIÓN
4.3 PRUEBAS DE FUNCIONAMIENTO
4.3.2 SISTEMA DE DIRECCIÓN ENCENDIDO
6.4.1
Integrating Results
This section integrates questionnaire results with the observed students’ behavioural results, and the results from the human expert based evaluation of the similarity algorithms.
Similarly to version 2, version 3 of Umka’s visualization received more negative assessment from the students’ questionnaire replies than from the observed behavioural data. The behavioural data showed that the circle visualization was effective in encouraging students’ interaction and self-reflection on their own
positions (Section 6.3.2). But in the questionnaire the students ranked the visualisation’s ability to encourage them for these actions as not very high: 1.91-2.27 out of 5 (Table 6.5, rows 3-7). The visualization feature that received only positive comments from the questionnaire was the representation of different students’ case study resolutions by different colors (Table 6.6, row 6).
Likewise, in the questionnaire the students reported quite low trust in Umka’s judgement of differences between their arguments: average rank 1.7 out of 5=34% (Table 6.5, row 10), and provided mostly negative written comments for this feature (Table 6.6, row 5). But in the human expert based evaluation Umka’s precision reached 71% in judging differences between students’ arguments (Section 6.3.3). Similarly to other visualisation’s features, although the students didn’t rank this feature very highly, it was still helpful in stimulating desirable students’ behaviours.
6.4.2
Analyzing the Effectiveness of Students’ Interactions
Similarly to version 2 of Umka, in version 3 we analyzed how beneficial were students’ mutual interactions for their learning, and more specifically, for encouraging the students to reconsider and expand their initial positions on a given case study. We studied what interaction variables are correlated with whether students
make any changes in their analyses during collaboration, and what variables are correlated withthe number
of changes that they make in the their analyses. The changes here are mostly new arguments that students
expand their analyses with, but also include changes in case resolutions and conclusions, that is all changes the students introduced to their initial positions as a result of collaboration.
We analyzed the data from the UofT experiment as the experiment with the biggest number of students. We analyzed various students’ interaction patterns, and using logistic regression we calculated the likelihood of the students making changes to their analysis given their certain interaction pattern. We have discovered
that whether the students make any changes in their analyses during collaboration is correlated with the
following variables of the students’ interaction (Table 6.7):
1. the number of comments they made to the analyses of others: more comments made lead to changes
being introduced (correctly classified instances 56%).
2. the number of comments they made to the analyses of others with different points of view. By analyses with different points of view we mean analyses of other students that have different arguments and different case resolutions from the student’s own arguments and the student’s own case study resolutions. The more comments the students make to the analyses of their peers with different points of view, the higher is the chance that the students will introduce changes to their own analyses (correctly classified instances 68%).
3. the number of analyses of others that the students visited (correctly classified instances 68%).
4. the number of analyses of others with different points of view that the students visited(correctly classified instances 68%).
Table 6.7: Variables correlated with the whether students introduce any changes in their analyses
and the significance of these correlations. Results based on experiment #5.
N 1. Correlation 2. Significance
1 # of comments made to the analyses of others 56% 2 # of comments made to the analyses of others with different points of view 68%
3 # of visited analyses of others 68%
4 # of visited analyses of others with different points of view 68%
5 time spent in the analyses of others 68%
6 time spent in the analyses of others with different points of view 65%
7 # of comments of others assessed 59%
8 # of comments of others assessed with different points of view 68%
5. time spent by the students in the analyses of other students: the more time the students spend in
reading and assessing the analyses of their peers, the higher is the probability that the students will introduce changes to their own analyses (correctly classified instances 68%).
6. time spent by the students in the analyses of others with different points of view (correctly classified instances 65%).
7. the number of comments of others that the students assessed: the number of feedback from other
students that the students read and rated. More comments of others assessed led to changes being introduced (correctly classified instances 59%).
8. the number of comments of others with different points of view that the students assessed (correctly
classified instances 68%).
Additionally, we have discovered several numerical correlations from the UofT experimental data. Thus,
the number of changes the students made to their analyses is correlated with (Table 6.8):
1. time the students spent in the analyses of other students (linear regression: the square of the Pearson correlation coefficient r2=0.676, which means 67% of the variation in the number of changes can be
explained by the amount of time the students spent in the analyses of others).
2. time the students spent in the analyses of other students with different points of view (linear regression: r2=0.456).
3. the number of analyses with different points of view the students visited (linear regression: r2=0.532).
Consistent with the conclusions from Umka version 2, version 3 conclusions indicate that the more stu- dents are engaged in the collaborative activities of reading and commenting on the analyses of other students,
Table 6.8: Variables correlated withthe number of changes students make to their analyses and the significance of these correlations. Results based on experiment #5.
N 1. Correlation 2. Significance
1 time spent in the analyses of others 67.0% 2 time spent in the analyses of others with different points of view 45.6% 3 # of analyses with different points of view visited 53.2%
assessing comments of others on their own analyses and spending enough time on these activities, the more they will reconsider and expand their initial positions, and hence the greater will be their learning. Further- more, collaborating with students of different points of view can be especially beneficial for learners (Table 6.7, rows 2,8; Table 6.8, row 3).
6.4.3
Conclusions
The final modifications to Umka and two final experiments allows us to draw a more complete picture as to the ways to organize collaborative learning environments to support students’ ethical problem-solving. In the final treatment/control experiment we replicated the results obtained in Umka version 2. Thus, lessons learned from the experimentation with Umka version 3 are mostly similar to the lessons learned from version 2, but this time they are confirmed with more confidence:
• Umka’s support in the form of visualization and suggestions foster productive interactions among students. Thus, the students who used the Umka version with these support types in comparison with the students who didn’t have them:
– gave more comments to the analysis of their peers (not a statistically significant result)
– viewed more “helpful” analyses of their peers (a statistically significant result)
– spent more time in the analyses of their peers (not a statistically significant result)
• Umka’s support in the form of visualization and suggestions foster students’ metacognitive processes of reconsidering and broadening their positions. Thus, the students who used these support types introduced more changes to their analyses during the collaboration. We couldn’t, however, achieve statistical significance for this result.
• The precision of the WTMF algorithm for the task of finding semantically different arguments was quite decent, reaching 0.71.
• In the post-study questionnaires we could observe much more positive comments about Umka version 3 than for versions 2 and 1. The students liked most the following Umka features: peer reviews, rep- resentation of different students’ resolutions by different colors in the visualization, and representation
of the number of arguments in students’ positions by the sizes of their circles. The students found the least helpful Umka features of the system suggestions and the representation of the differences between students’ positions through the distance between their circles in the visualization.
• Similarly to Umka version 2, in version 3 we have also demonstrated that productive collaborative behaviours such as reading and commenting on the analyses of other students, assessing comments of others on their own analyses, and spending more time on these activities are correlated with students reconsidering and expanding their positions with more new arguments. Particularly, interaction with peers who hold different points of view can be especially beneficial for students.
Chapter 7
Discussion and Conclusions
The goal of this research was the development of techniques for organizing computer-based learning environments to support students in the analysis of case studies for professional ethics education.
Through the literature review we have identified the goals of ethics education as helping learners to establish their own convictions on important ethical and professional issues, broaden their perspectives and develop their skills for analysis and critique of their own and others’ convictions. We also came to the understanding (through the literature review and our own research) that learners can achieve the goals of ethical education especially well through interaction with peers over important ethical and professional issues, and especially with peers who hold different points of view on these issues.
We designed a system called Umka with these goals in mind. The main way that Umka achieves these goals is through suggesting and visualizing similarities and differences between students’ positions on a given case. We measured the effectiveness of Umka’s support types by the extent of the positive student behaviours they triggered, and the students’ own perceptions of their helpfulness.
Using three different versions of Umka and a series of six experiments we identified techniques that can effectively support students in ethical analysis. We also identified techniques that do not work as well.
7.1
Umka’s Features Across Different Versions
Over the course of this research Umka was deployed in three versions. Table 7.1 demonstrates what support features were implemented in each version of Umka.
Umka version 1 was the only version that supported students in their individual analysis, but this support came at an extra cost of domain model construction. The support feature that was found to be the most helpful in the individual stage was system hints on new arguments that a student had not yet considered for a given case. Another feature that was unique to Umka version 1 was presenting arguments of all students in the class semantically clustered. This feature was rated as the most helpful in the collaborative stage by the students in the questionnaire (Section 4.5.2).
Umka version 2 supported students by suggesting to them certain helpful arguments of their peers, and a useful visualization of students’ positions. The system suggestions of helpful arguments did not find much uptake among students (Sections 4.5.2, 5.2.3).
Visualization was enhanced for supporting students in Umka version 3. System suggestions were changed in version 3: instead of suggesting to students certain isolated arguments of other students as in version 2, Umka version 3 suggested students certain helpful peers to interact with. These suggestions were more followed up by the students than the suggestions of version 2, as evidenced in the number of helpful analyses the students visited (Section 6.3.2).
If a next version of Umka were to be developed, the following features from the previous versions will be integrated into it, as these are the features that have been shown to render the best support to students: 1) hints on arguments that a student hasn’t yet considered (provided that a domain model for a case is available); 2) presenting arguments of all students semantically clustered; 3) suggestions to consider helpful peers; 4) visualization of students’ positions.
Table 7.1: Umka’s features across different versions
N Support feature Umka
v.1
Umka v.2
Umka v.3
1 Individual: guidance on the steps of the ethical analysis
X
2 Individual: feedback on argumentsX
3 Individual: hints on arguments a student hasn’t consideredX
4 Collaboration: presenting arguments of all students semanticallyclustered
X
5 Collaboration: system suggesting to consider similar, different,
and contradictory arguments of peers
X
X
6 Collaboration: system suggesting helpful peers to interact with