Estudios Experimentales y Ensayos clínicos

Multimodal learning can be defined as the learning process that involves more than one modality in the representation of the learning material [196]. More specifically, the content knowledge can be presented to the learners using verbal communication means (such as text and speech) in company with related non-verbal illustrations such as photos, graphics, videos and animations [197]. Multimodal approach could have positive implications on the learning environments. When only one sensory channel (mostly visual) is used to communicate the presented information, overloading cognitive capacity of the learner is more likely to happen as opposed to the use of multimodality where both visual and auditory channels can be involved to acquire the knowledge contained in the learning material [198, 199]. In addition, learners’ ability to understand that material could be improved by the multimodal presentation. Therefore, multimodal learning environments are considered to be more effective as compared to that which incorporates single modality [198, 200-202]. In fact, this has been demonstrated by a series of experimental studies where the research on the impact of multimodal interaction metaphors on learning performance as well as the usability of e-learning interfaces have gained a considerably interest.

The use of speech and non-speech sounds to enhance users’ learning has been investigated in different domains. For example, the experimental work performed by

35 Bonebright et al. [9] tested the effectiveness of earcons to communicate data sets represented in two types of graphs from subjects studied by students during the academic coursework. Flute sound was used for bivariate graphs whereas both flute and bassoon sounds were used for multivariate graphs. For bivariate graphs, monophonic sound was presented to both ears while stereo sound for multivariate graphs was distinguished by different timbres, one variable to the right ear and the other variable to the left ear. While a sound was played, four visual graphs were displayed on the screen and the student has to click the graph that matches the communicated sounds. It was found that students were able to successfully match the auditorily conveyed graph with its corresponding visual representation on screen [13].

A learning-training simulator developed by Li et al. [203] incorporated audio and visual feedback for students while they were trained for lathe operations. The inclusion of environmental sounds to communicate warning messages in addition to different basic operations in lathe machine was found to be useful in providing safer, efficient and satisfactory technical training for students.

Speech technology in combination with earcons, text, graphics and animation was investigated in a multimedia online-learning system by Rigas and Hopwood [11] to examine its use with learning topics of increasing complexity and to evaluate as well its effects on users’ performance in different complexity learning tasks related to these topics. In addition to text, recorded speech was utilised to communicate instruction to users whereas earcons, animated arrow and graphical transparent icons with animated text were used as navigational cues to communicate the part of the interfaces to be clicked during the interaction. Users’ of the multimedia learning interface outperformed their counterparts who have been exposed to the same content with only the text with graphics in terms of responding to the learning tasks more accurately, particularly those

36 related to intermediate and advanced learning topics. Speech and earcons were also investigated in note-taking applications for learning purposes. Specifically, recorded speech was combined with textual and graphical metaphors for taking notes about the displayed content and shown to be contributing to reduce the note-taking time in addition to improving the accomplishment rate and users’ satisfaction in such activities compared to visual-based note-taking in e-learning applications [105]. These findings are supported by another study [89, 204] which explored the use of speech and earcons to support students in recording their own notes about the learning content and to make use of these notes in reviewing that content. It was concluded that the usability of note-taking could be substantially promoted by multimodality where users’ learning performance and experience in terms of question answering time and correctness of answers as well as users’ attitude have been enhanced [205].

The use of avatars in e-learning environment to serve educational purposes has been explored by several studies which demonstrated the benefits that could be gained due to the inclusion of such metaphors in e-learning interfaces. The presence of lifelike avatars could enhance learners’ motivation and engagement in the learning activities [206, 207].

When they interact with a lifelike agent, learners may find their learning more entertaining [208] and taking place in an actual learning environment [7]. In fact, the inclusion of avatar as a learning agent could reinforce the social nature of the learning process [209, 210]. For example, Robertson et al. [211] carried out an experimental investigation on the effect of animated avatars on students’ attitude when these avatars were incorporated to help them in story writing, and found that students who used the avatar-based interface expressed stronger tendency to use the interface as compared to those who interacted with a traditional graphical interface without avatars. Also, a study by Baylor [8] investigated three different roles of avatars as pedagogical agents and

37 concluded that the one who combines both expertise and motivation features was evaluated by learners to be more engaging and facilitative of learning in comparison with those that played as either motivator or expert agents.

It is believed that the motivating effects of avatar-based interaction are most likely to result in enhancing learners’ understanding and their learning outcomes [207, 212]. A study by Moreno et al. [201] compared the learning performance between two groups of students who learned about botanical physiology. The first group received the learning material by speaking full-body animated agent whereas the other group communicated the same material by an on-screen text in the absence of an agent. They found that the group who had interacted with agent-enhanced interface performed better than the other group in applying what they had learned to solve new problems; however the addition of the avatar did not make any difference in learners’ performance in retention questions. On the other hand, contradictory results were observed in another study [213]

which found that the use of a speaking avatar with gaze and pointing in explaining human cardiovascular system had outperformed the use of speech or text in retention tasks only and no difference among the three metaphors was found in transfer questions.

However, a study on the influences of facial expressions: neutral, happy, sad, scared, surprised, angry and disgusted on students’ motivation and their learning outcomes was carried out by Theonas et al. [12, 214] who investigated the use of facially expressive virtual lecturer against another one without any facial features in conducting a set of four virtual lectures of two complexity levels: easy and difficult. The results from their experimental work indicated that the students were more motivated, attentive and excited when they attend the lectures presented by the facially animated lecturer.

Furthermore, they noted that student’ learning performance was improved with the presence of facial expression particularly in difficult lectures. In a further experiment,

38 Theonas and his colleagues explored the impact of a smiling expression when depicted by the virtual lecturer and found that this expression could attract the students and improve their performance if used appropriately.