The third phase of the study was used to test the theoretical framework with the design artefact in practice and to collect the data required to address the research questions. In other words, this phase conducted “rigorous and reflective inquiry to test and refine” the innovative multimedia learning environment as well as to test and modify the existing principles and “define new design principles” (Reeves, 2000, p. 26). The methodology used to investigate the proposed solution is described in detail below.
Research context
This research focused on pre-university level English for General Academic Purposes (EGAP) students in an EFL context. For this reason, it was appropriate for the research to be conducted at the Eastern Mediterranean University (EMU) English Preparatory School (EPS), which is an English medium university in North Cyprus. Students who apply to study at EMU need to provide documentation of their English language proficiency level, that is, students must accomplish a band of 500 in TOEFL and 5 in IELTS or they are required to pass the Proficiency Test given at EPS. Those students who are below the required level study at EPS until they reach the proficiency level. EPS prepares students for academic life by giving them competency in the target language, English, and required academic skills. The program is intensive and while the school offers courses in A1, A2, and B1 levels (Council of Europe, 2001), successful students can study up to two courses in any one academic year.
Figure 3.2 illustrates the course alignment of EPS according to the Common European Framework of Reference for Languages (CEFR) (The Council of Europe, 2001). In this respect, the first course, EPS101, is designed for beginner level students and targets the can do statements (The Council of Europe, 2001) indicated in A1 and A2 levels. The second course is EPS102 and it focuses on A2 and A2+. EPS103 consists of A2+ and B1 and EPS104 focuses on B1 and B1+ can do statements.
Figure 3.2: EPS course alignment with CEFR
Students who complete EPS103 level have the right to take the Proficiency Test. Success in this examination, scoring 60% or more, enables them to go into the mainstream instruction program in the departments of their chosen field of study. Students who score below 60% go to their departments conditionally, in which case their score in the Proficiency Test determines how many hours of English support courses they take a week.
Researcher’s role
The researcher did not assume the role of a participant teacher in the course of the study and thus was not in any contact with the learners while implementing the
learning environment. Kervin et al. (2006) identified five different possible roles that a researcher can be engaged in while observing (see Figure 3.3).
Figure 3.3: Roles of observers (Kervin et al., 2006, p. 85)
These roles range from being a non-participant to full participant. Throughout this research, while observing the participants, the identity of the researcher was known to
Full participant Non-participant
Passive presence
Limited interaction
the participants and the role of the researcher was one of limited interaction (Kervin et al., 2006, p. 85). The researcher interacted with the participants only if clarification on any technical aspects of the web-based program was required.
Selection of participants
A theoretical or purposive sampling approach was used for selecting the participants. This approach facilitated the identification of information-rich cases and enabled the researcher to study a case in-depth (Mertens, 2005; Patton, 1990). In this research, it was also believed that, since the accessible population represents the target population, the participants of this research would generally reflect how other students or teachers might respond to the activity if they had been involved in a similar task.
Participant teachers’ roles
Three practitioner teachers and their classes participated in the research, two for the first cycle and one for the second cycle. The selection of the participant teachers and students were made, not only on theoretical grounds, but also to some extent on practical grounds in order to fit in with the practical timetabling requirements of the School.
During the use of the program in the computer laboratory, in both cycles, each
teacher’s primary role was as a facilitator and students consulted them if they required further assistance. For example, the teachers helped students to locate resources, such as sample reports, or provided guidance on using tools such as online concordance so learners could see the different uses of the language they were learning. In addition to these roles, teachers also wrote initial questions on the discussion forums to enable learners to maintain meaningful discussions and develop relevant knowledge and skills. Teachers observed the progress of the students throughout the activity and provided scaffolding and feedback where necessary and appropriate. Support and scaffolding are reported and discussed in-depth in Chapter 7.
Student participants
Class teachers used the learning environment as part of their education curriculum. However, due to the large number of students in each class (the average class sizes at
process, although all completed the activities. During the first cycle, six students (three students from each class), and during the second cycle, four students, were asked to participate in in-depth interviews and focus group discussion. The participant students were selected on the recommendation of the class teachers, as they were thought to be information-rich cases representing the target population (Mertens, 2005; Patton, 1990).
Consent of the participants
Participation in this research was voluntary. All participants (teachers and students) were given information about the study (see Appendices 3 and 4) and their written consent was gathered (see Appendices 5 and 6) before the research started. The consent form informed the participants about the research and its objectives and how their participation would be kept confidential. It was clearly mentioned to the
participants that they had the right to refuse to participate, that they were free to withdraw from the research at any time, and that they had the right to withdraw any data they had contributed. It was also explained to the participants that their
participation or refusal to participate, or their withdrawal of consent (and data), would not affect their treatment in any way or their relationship with EMUEPS.
Ethical review
Ethical approval to conduct the research was sought from Murdoch University. Following the completion of the required procedures, this study was approved by the Murdoch University Human Research Ethics Committee (approval number 2012/028). Further ethical approval from EMU was not formally required; nonetheless, the school administration was informed in writing and written permission was given to conduct the study at the institution (see Appendix 7).
Procedure
Both cycles were designed to last for six weeks. The first cycle commenced in the eleventh week of the semester and the second cycle commenced in the tenth week. However, because one week in the second cycle was affected by a religious holiday, it lasted for seven weeks. The project was administered as a substitute to the normal semester project and was initiated with teachers’ informing students in their
classrooms that they would do a different project than the other classes of the same level. All students in three classes completed the activity even though not all of them were interviewed.
The project was implemented in the computer laboratory of the Student Self Study Centre (SSSC). Each class had a scheduled class hour each week in the SSSC and learners were expected to study at their own pace according to their needs by using the paper-based and electronic resources available at the centre. In the first week of the activity, teachers gave learners information about the activity. Teachers mainly focused on informing learners about their roles and demonstrating the use of the program. While demonstrating the program, learners were informed about the links on the home page (see Figure 4.2) and what each page was about, for example, that they had the opportunity to access information about the key dates of activities on the calendar. They were neither given information about the content of the tasks on the agenda nor the content of the resources, as they were expected to read, understand, and respond accordingly in the target language. Moreover, learners were told that they could ask any questions to their teacher or classmates by using the discussion forums; however, only the target language was permitted to be used.
On the first day, learners were informed that they could work in pairs or groups of three; however, due to some personal reasons (e.g., one student preferred to work individually because of religious beliefs and his article was about a religious event– zakat), a few learners preferred to work individually and teachers did not interfere with the learners’ decision. After learners were formally introduced to the activity and the learning environment, and groups were formed, the activity commenced.
During the following weeks, learners were expected to complete the contributory or sub-tasks required (such as participating in online discussions) and the major task of writing an article for the newsletter. The research activity was completed upon the publication of the newsletter and the learners’ presenting their findings in class. Following the completion of the activity, interviews were conducted and each
interview lasted approximately 45 minutes. Interviews were conducted within a week at a mutual time that best suited both the participants and the researcher.
The same procedures were employed in both cycles. However, based on the outcome of the first cycle, the learning environment was refined between iterations (details are provided in Chapter 5). However, no changes were made to the design principles until the end of the study due to the fact that it was aimed to obtain consistency in the outcome. This has helped to achieve dependability in research (see dependability below in the trustworthiness in data analysis). New design principles are reflected in Chapter 9 at the end of the research process.
Once the ethics approval was obtained and the research was put into practice, it was time to collect data. Data was collected through observations, teacher journals, interviews (both individual and focus group), and analysis of work samples. Procedures followed for each is described below.
Observations
During the observation process, the researcher walked around the room, observed the students, and took handwritten notes. In order to prevent any data loss, during both iterations, two video cameras were used to record two pairs of students’ voices and the screen of the computer they were using. This data was later transcribed by the
researcher for analysis. Additional notes and reflections were added in relation to relevant insights, such as discussions with teachers after class.
Teacher journals
The participant teachers were asked to keep a journal of detailed notes and anecdotal records on the support and scaffolding provided to students during the study. All three teacher journals were kept in hand written form. These were collected and also used for analysis.
Interviews
Following the completion of the task (i.e., writing articles, publishing them in the newsletter, and presenting the findings to the class) students and teachers were interviewed. While the teachers were interviewed individually, students were interviewed both individually and in groups. An in-depth interview is a useful data collection technique as the researcher has control over the questions asked. There were also times when the participants could not be observed, such as while students were
collecting data outside the school to be used in their articles, or while the participant teacher was facilitating the use of the program. Thus, interviews provided the opportunity to gather valuable data about these situations that were not able to be observed.
Interviews were semi-structured to allow informants to freely articulate their opinions and enable the interviewer to probe the answers to obtain additional information (Kervin et al., 2006). While individual interviews were used to obtain student’s personal opinions, group discussions were used to raise a genuine discussion among learners in order to obtain diverse opinions and allow discussions on group-initiated concerns (Mertens, 2005).
In order to help participants feel at ease, the individual interviews took place in a quiet standard office at EPS and group interviews were conducted in the meeting room of the school. McMillan and Schumacher (1984) suggest that “to provide honest answers to questions, the respondent must feel comfortable with the interviewer” (p. 154). To facilitate this, interviews started with an explanation of the purpose of the interview to establish a friendly and comfortable relationship with the researcher and to inform learners of the purpose of the interview.
Interview questions
Qualitative interviewing aims at gathering information about how participants view their world (Patton, 2002). In this regard, Patton (2002) and Fraenkel and Wallen (2006) suggest a number of interview question types that can be used during an interview. Experience and behaviour, opinion and values, feeling, and demographic are the main interview question types that were used in this study. Each is described below:
Experience and behaviour questions target information about the behaviours, experiences, or activities that the respondent is doing or did but can or could not be observed by the researcher. For example: Have you ever used any web-based program in your course?
Opinion and values questions can uncover what the participants think about the topic. Answers to these questions reveal the respondent’s goals, beliefs, attitudes,
or values (Fraenkel & Wallen, 2006) about the topic or issue. For example: What were the strengths of the web-site?
Feeling questions aim at finding out how the respondent feels about the topic or issue. Feeling questions should not be confused with opinion questions. While opinion questions ask the respondent’s opinion about the topic or issue, feeling questions elicit how the respondent feels about the topic or issue. For example: How did you feel about taking on the role of a journalist with a complex task to complete?
The use of demographic questions elicits background information about the participant being interviewed. Such questions can be about the participant’s name, age, occupation, education, and the like.
The interview questions that were used in the study with the targeted participant (S for student and T for teacher), together with a rationale for the use of each question, are given in Table 3.1 below:
Table 3.1: The interview questions
Questions
Type of question Participant
Rationale 1 Exp. 2 Opin 3 Feel 4 Dem S T Background
The information that you provide will be used for research purposes (my PhD) and the purpose of this interview is to get some information that will help designers of web-based learning environments to design more effectively. As someone who has experience in EFL, you are in a good position to describe your experience and how you found it.
[Statement on right to withdraw].
Explanatory and introductory comments.
The interviewer asks the participant his/her name, years of experience etc.
Brief demographic information.
Have you ever used any e- learning environments for your own education? If so, which programs or sites?
Background questions to ascertain the level of experience with web- based learning environments.
Have you ever used any e- learning environments in your course? If so, which programs or sites?
Questions
Type of question Participant
Rationale 1 Exp. 2 Opin 3 Feel 4 Dem S T
What did you think of the web- based learning environment on learning a foreign language?
Question seeks opinion on the overall concept of the program.
What did you think of the activity
on developing academic skills?
Question seeks opinion on whether this program is a good way to develop academic skills.
Effectiveness of the web-based learning environment and pattern of use. When you were working with the
web-based learning
environment, how did you find what you were looking for? What strategies did you develop?
Experience questions to encourage the respondent to review the program before offering more detailed opinion.
What were the strengths of the environment?
What were the weaknesses of the environment?
Presupposition questions (i.e. the questions assume the web-based learning environment has strengths and weaknesses, and can thus elicit useful information).
What else do you want to see in the web-based learning
environment?
Presupposition question (i.e. the question assumes that the web-based learning environment has missing information).
How effective do you think the
environment is?
Opinion question which seeks summary comments and reinforcement of previous answers.
What have you learned from this
environment?
Open-ended, opinion question on the students’ assessment of learning rather than a knowledge question.
What are some of the things you really liked about the web-based learning environment?
What are some of the things you disliked about the web-based learning environment?
Feeling questions which aim at finding out the respondent’s emotional response to the web-based learning environment.
If you had the power to change the learning environment, what would you make different?
Opinion question which seeks
recommendations for change or
improvements to the learning environment.
Questions
Type of question Participant
Rationale 1 Exp. 2 Opin 3 Feel 4 Dem S T
If a colleague of yours was about to use the environment for the first time, what advice would you give?
Projective question which asks the respondent to take on the role of ‘expert’.
Perceptions on authentic activity We have been talking about your experiences with the web- based learning environment in general. I would like now to ask your opinion on some of the specific features of the activity.
Transition statement to move onto the
discussion of each of the critical elements of authentic activities.
Authentic activities have real world relevance How did you feel about taking
on the role of a journalist with a complex task to complete? Did you feel like a real staff of the city newsletter?
Feeling questions to elicit emotional
response to the activity.
How did you feel about taking on the role of an editor of the city newsletter with a complex mission to publish articles? Did you feel like a real editor?
Feeling questions to elicit emotional
response to the activity.
What did you think of the activity you were given to do within the environment?
Open-ended opinion question on the activity.
Authentic activities are ill-defined, requiring students to define the tasks and sub-tasks needed to complete the activity.
How did you go about completing the activity? What tasks did you do to complete it?
Questions aim at eliciting information on how the complex task was broken up.
Did you think that the activity was too demanding or easy for students to complete?
Question seeks opinion whether the difficulty level of the activity is appropriate.
Authentic activities comprise complex tasks to be investigated by students over a sustained period of time.
How did you find the time allocated to complete the activity? Too long? Too short?
Question seeks opinion on whether sustained thinking is possible within time allocated.
Authentic activities provide the opportunity for students to examine the task from different perspectives, using a variety of resources.
The activity required you to consider it from a number of different perspectives. For example, the mayor, readers of the newsletter and the editor. How did you feel about this