Aplicación “Autoescáner”

Abstract

Experience, knowledge, and assumptions: This male TA has no experience with LA and has been coordinating his course for about one year.

Satisfaction with course: (unknown)

Way of using tool: He uses the tool the first time after reading our email and two other times spontaneously. His usage is focused on the monitoring overview (dashboard).

Level of surprise: The data of the indicators are not always as he expects it. He feels uncomfortable and is surprised, but since he does not really trust the prototype, he does not reflect further about possible reasons.

Involvement, interest, curiosity, and lack of interest: He is only interested in simplified access summaries, such as the number of students who have logged into the system at least one time. But he also likes some filtering options.

Tool reliability: He does not like to wait for visualizations several minutes. Trust in LA tool: His level of trust in the tool is very low because it is a prototype. This might have been influenced negatively by the long loading times of some indicators.

Support, qualification: The TA explicitly wishes for short online tutorials for each indicator because he feels that explanations on how to use them could help him a lot.

Impact: The user wants to be aware about the amount of active students

and eLAT helps him to achieve this goal, although he wishes for a simpler indicator with respect to his question. There is no sign for reflection about the course design because he does not trust the accuracy of the data collection. However, he mentions that the final tool (not the prototype) could influence his own activities in future courses.

Course Description and User Profile

This course was a weekly lecture with additional exercises. All learning materials were provided within the VLE. Additionally, there were lectures of practitioners, giving the students insight into practical examples of ‘quality management’ in industries, which was the topic of the course. For exam preparations, there were office hours and sample exam questions.

The TA was the coordinator of the lecture and the exercise course. He had been teaching for one year and had no previous experience with analytics tools or spreadsheet calculations (experience: LA newcomer). His teaching role was to coordinate all the people involved in the lecture and also to conduct the exercise course and planning of the exam. He stated that students should actively participate in the lecture and exercises for successfully finishing the course. It was his goal to monitor this active behavior. In his opinion, communication and collaboration among students was helpful, but not necessary for the course.

Impact Forecast

Based on the status quo analysis, we concluded that the TA would be most interested in the indicators concerned with the students behavior, e.g., ‘activity behavior’, which allows for defining, what an active students is and having a weekly overview on the number of active students.

Pilot Phase Review

The TA told us that he had used the LA tool about three times during the pilot phase. His first usage was initiated by our email about the tool availability. The other two sessions were more spontaneously at the end of the semester, when he

was doing something else in the course room anyhow (way of using tool: initiated by email and spontaneously). When asked, how he would use such a tool in future courses, he guessed that he would use it on a monthly or quarterly basis, which was conform to his usage behavior during the pilot phase. He also explored the system a bit, e.g., he opened the analysis view of at least one indicator, but because of slow loading times (tool reliability: performance issues), he canceled to work with it. He also did not understand well how to use the filters in the analysis view, so he mostly relied on monitoring the dashboard overviews (way of using tool: explored featured, but used mainly monitoring view).

Overall he thought the tool was interesting, and he was mainly focused on the access statistics. Although he was skeptical about the overall number of accesses, which was lower than the number of registered students, he found that it was a good feedback to know that some students were actually accessing the materials (surprise: uncomfortable feeling, data does not meet assumptions). His main question was: How many students access L²P at what times? He told us:

“I looked at it at times and I found it really quite interesting ... the statistics on access were the most exciting. Especially for us as a feedback that people access it. However, I'm a bit cautious in total with the results that came out of it because the number of participants is significantly higher than the access numbers and I so am a bit skeptical because everyone is - so my expectation would be that the students use the L²P at least once - I am accordingly not quite sure if everything was recorded.”36 When asked about his reaction, he also said: “I was especially surprised because numbers seemed to be very low”.37 However, since he assumed that every student would at least login once and since he knew that the system was a pilot, he did not quite trust the unexpected results (trust in LA tool: low). He concluded: “To be honest, I thought: this is a test phase, who knows if everything is really recorded.”38 His distrust distracted him from reflection about his teaching on several occasions during the interview.

His main goal was the planning of the course. LA feedback was supposed to tell him how many students were active learners, e.g., because he wanted to estimate how many students would come to the exam. Therefore, he was not interested in detailed visualizations. We talked about a new indicator, which was described as a

36 _{Translated from: “Ich hab ihn mir mal angeschaut und ich fands eigentlich ganz interessant... die}

Statistiken über Zugriffe, die fand ich eigentlich so mit am spannendsten. Auch so für uns mal als Rückmeldung, die Leute greifen drauf zu, wobei ich in Summe etwas vorsichtig bin mit den Ergebnissen, die dabei herauskamen, weil die Teilnehmerzahl deutlich höher ist also die Zugriffszahl und ich deswegen so ein bisschen skeptisch bin, weil jeder sich ja – also meine Erwartung wäre gewesen, dass die Studenten den Learning Analytics, also das L²P überhaupt mal nutzen – entsprechend bin ich nicht so ganz sicher, ob da alles erfasst wurde an der Stelle.”.

37 _{Translated from: “Ich war vor allem erstaunt, weil die Zahlen mir sehr niedrig vorkamen”.} 38 _{Translated from: “Also wenn ich ehrlich bin, hab ich gedacht: das ist ja ne Testphase, wer weiß,}

single bar, which presents the number of students, who have at least logged in once to access something, related to the overall number of registered students. This would be a key indicator for him to measure how many students are getting all relevant information. If this indicator would show him a low number of active students he would take action and tell students during an exercise meeting to use the VLE.

TA: “That is definitely an important thing. So I assume that a student who needs the learning materials... that he finds it and all is ok. At the same time, we must ensure that the students ... that we have done everything to ensure that the students have all the organizational relevant information. Then it would be definitely good to know, ok, I've ... it is now Christmas and I've of 100 students with whom I count I only have 20 or only 10 active in L²P. This is definitely an important info.” Moderator: “And then you would possibly become active and in your course...”

TA: “…in the exercise meeting I would point to it on several occasions, yes.”39

This example shows that such indicators would have some potential for action. The only other indicator, besides ‘access rates over time’, that was interesting to him was ‘adoption rate’. But he only understood its purpose after we explained it to him (support: explanation by moderator). Then, he liked it because it might help him to ‘get a better feeling’, when to upload resources in relation to the times students download it. Our explanations helped him to understand the purpose of the tool and how to use the analysis view and its filters. Therefore, he suggested providing tutorials to users, which tell them what to expect of the analytics tools (support: short tutorials).

He showed no interest in other indicators. He argued that access rates are no indicator for the quality of learning materials, since students have to access first before they can decide, if a file is useful to them. The indicators regarding communication and collaboration did not provide any data because there was no activity in the discussion forum and wiki of his course. If there had been active participation by students, he would have been interested in these indicators. On our way out after the interview, the TA told us that he also liked the university-wide evaluations for getting feedback regarding the quality of his

39 _{Translated from: TA: “Das ist ja auf jeden Fall eine wichtige Sache. Also ich geh davon aus,}

dass ein Student, der die Lernmaterialien braucht, sich ... das er die die dann entsprechend findet und das alles ok ist. Gleichzeitig müssen wir aber sicherstellen, dass die Studenten... also, dass wir alles dafür getan haben, dass die Studenten die organisationsrelevanten Informationen haben. Dann wäre es auf jeden Fall gut, zu wissen, ok, ich hab... es ist jetzt Weihnachten und ich hab erst von 100 Studenten mit denen ich rechne erst 20 im L²P oder erst 10 Aktive. Das ist auf jeden Fall ne wichtig Info.” Moderator: “Und dann würden Sie evtl. aktiv werden und in der Veranstaltung...” TA: “…in der Übung noch mehrfach darauf hinweisen und ja.”.

course. We talked about integrating these kinds of students’ feedback into LA, and he thought it was a good idea.

Improvement Suggestions

The user test showed several issues that could be improved in the next version of eLAT. Of course, the slow performance was an issue again (tool reliability: performance issues). Furthermore, the TA did not have any expectations based on the term ‚learning analytics’ and he sometimes confused the wordings “accessor” und “access”40 because they look similar at a quick glance. Because of his interest towards ‘usage over time’, he ignored most of the other indicators. So, it might be better to have more personalized selections of indicators in order to create more space for those, which are relevant to the user. The participant also specifically wished for bigger visualizations and making filtering menus more prominent (maybe put them on the dashboard). Additionally, he wanted less granularity in the graphical presentations. Monthly overviews opposed to daily or weekly would be more adequate for his purposes. The filtering items regarding ‘field of study’ were most interesting to him. Furthermore, we developed a new filter idea concerned with grouping data by degree type, such as ‘bachelor’ or ‘master’. During the usage of the filters, it took too much time to select all of them. We concluded that they should be pre-selected in a standardized way and there should be standard one-click options to select or de-select all checkboxes at once. Also, automatic adjustment of warning regarding filter selections was suggested to avoid illogical calculations. As an analytics beginner, the TA wished for short tutorials for learning how to use the tool and each indicator. And finally, a notification subscriptions counter, which shows how many students are receiving email notifications about new items in the course, was a new indicator idea besides the ‘active students’ bar chart indicator mentioned in the previous section.

Discussion

This use case revealed the following behavior pattern of the TA (Figure 42). He mainly wanted to be aware about the amount of active students. He was also interested in diversity filters, especially ‘field of study’ differentiations. Accordingly, a state of surprise was be triggered by low access statistics of certain groups of students. He stated:

“I think it would be most important for me to realize ... So, we use L²P not only for learning materials, but also for all organizational information; i.e., it's all about: When is the exam? When is the inspection? How is the grade distribution? So the grade distribution is there, how is it? We would like to control everything with the L²P. So, the most important statement for me is, I've got a certain amount of active students that also corresponds approximately to the amount of registered participants or

not. If I don’t have it, then I would have to think about it heavily. I would need to point out in the lectures that L²P has to be used.”41

His actual reaction to the low access statistics observed during the pilot phase was to mistrust the system. He thought that it was probably a mistake of the data collection tool. A possible reason for his doubt in the reliability of the tool was the pilot phase declaration. His suspicions clearly kept him from reflecting about students learning behavior or his own teaching. So, there was no impact observed. However, he stated that normally, he would be initialized to think about the data and talk to students, in case that something was going wrong. Although he conveyed the impression in us that he feared changes in didactical plans during an ongoing course because students might not get it.

Figure 42. Expectation and reaction cycle of UC5.

During the interview analysis, we noted that he talked about “getting the feeling” several times, which we interpreted as ‘learning how students usually behave’ and

41 _{Translated from: “Ich glaube, das für mich wichtigste an der Erkenntnis wäre... also wir nutzen}

L²P nicht nur für Lernmaterialien, sondern auch für sämtliche organisatorische Informationen drum herum. D.h. da geht’s vor allem darum: Wann ist die Klausur? Wann sind die Einsichten? Wie ist die Notenverteilung? Also die Notenverteilung ist da, wie ist die? Das würden wir alles über das L²P gerne steuern. Also dh. die für mich wichtig Aussage ist, ich hab ne gewisse Menge an aktiven Studenten, die auch etwa der Teilnehmermenge entspricht oder eben nicht. Wenn ich die nicht hab, dann müsste ich mir sehr große Gedanken machen, dass ich in den Vorlesungen nochmal darauf hinweise, dass das L²P auf jeden Fall zu nutzen ist.”.

‘being able to predict the outcomes of indicators’. So, we wondered what would happen, when he got the feeling eventually? Would he stop using the tool? We could not make any conclusions about this based on this use case’s data, but UC4 had given some impressions related to this. Advanced user might ‘get bored’ with looking at the same indicators again and gain. This finding will be explained in more detail in the overall discussion and conclusion (see section 7.5).