3.6.5.1 The verbal report instructions
The participants were provided with clear instructions for the think aloud task following the sample instructions provided by Green (1998, p. 46-47), based on Ericsson & Simon (1993), and by Lumley (2005). These instructions also followed the guidelines provided by Bowles (2010) for better instructions:
Minimally, this set of instructions should include (1) a description of what is meant by “thinking-aloud,” (2) the language(s) participants are allowed to use to verbalize their thoughts, and (3) the level of detail and reflection required in the think-aloud. (p. 115).
The sample instructions given by Green were in line with these criteria although the ‘language’ criterion (2) was not specified in Green. Sanz et al. (2009) did not mention
the language(s) that participants were allowed to use in the think aloud because their learners were monolingual native English speakers. Likewise, this feature was largely redundant for our instructions as all informants used English either as L1 or L2. Teachers, particularly those accustomed to communicative language classrooms, may in any case assume that they are to speak their thoughts in the language they normally use if they are not otherwise instructed (Bowles, 2010; p.115). Interestingly, one informant whose English is L2 used it in his think-aloud aloud instead of his Arabic L1 without being instructed.
For the sake of obtaining rich information and discovering details of the rating process it was important to make a decision on the level of detail (3) requested by the instructions. Lumley (2005, p. 119) for example refers to questions requiring deep awareness by the rater of what they are doing, such as ‘explain why you give the score you give’, ‘why do ignore this aspect’, ‘what does this mean’. However, we preferred to adopt the forms of verbalization which were characterized by Ericsson & Simon (1984, 1993). In the type which we adopted, verbalization takes the form of subjects vocalizing all their thoughts that pass through their heads naturally while they perform the task without analyzing them. In this way, the emphasis is on the performance of the task with the verbalization as an incidental activity. The subjects in this type of TA are trained not to add other information in the form of explanation or justification (Ericsson & Simon 1984, 1993; Green 1998). Our aim was to reassure to the participants that we wanted their thinking, not what they think about what they are thinking (Brown & Rodgers; 2002).
Our actual instructions were as follows, provided both verbally and in written form. (See Appendix D)
Purpose
These instructions are written to help guide you and others producing think-aloud protocols for this research, in a consistent and informative manner. Think-aloud protocols ask people to say everything they think about while they perform a task, with the aim of documenting and better understanding what you pay attention to and consider important when you do a task. The purpose of the think-aloud protocols for this study is to find out in as much detail as possible what you, as a teacher/rater of ESL compositions, are thinking about, deciding, and doing while you rate a sample of ESL compositions. The most important thing to emphasize is, say everything you are thinking about, and make certain this is recorded clearly onto the recorder. What you say will become important data for our research.
I would like you to talk and think aloud as you rate your students’ written compositions while this recorder records what you say.
First, you should identify each script by the ID number at the top of the page and the task.
Then, as you rate each script, you should vocalize your thoughts.
It is important that you keep talking all the time, registering your thoughts all the time. If you spend time reading the scripts, then you should do that aloud also, so that I can understand what you are doing at that time. In order to make sure there are no lengthy silent pauses in your rating, I propose to sit here, and prompt you to keep talking if necessary. I will sit here while you rate and talk. I will say nothing more than give you periodic feedback such as ‘mhm’, although I will prompt you to keep talking if you fall silent for more than 10 seconds. Thanks!
3.6.5.2 The rating task
The actual task was rating seven of their own students’ written compositions, in whatever order they chose.
Following Cumming et al, (2002), Cumming (1990), Connor-Linton (1995b), Kobayashi and Rinnert (1996) and Erdosy (2004), in this study, no criteria nor analytical categories nor rating scale nor rules for arriving at an overall rating from ratings of separate features were provided, with the purpose of finding out what scale and criteria these teachers followed themselves. In fact, this was quite realistic since, when the researcher asked the course director about the scale that is used for the EELP course, he stated “no rating scale is provided; this is among the issues that I will investigate in the near future and might include one” (personal communication 2013). No scale seemed to be imposed in CESC either.
Following Brown & Rodgers’ (2002) suggestion about avoiding being too directive in the instructions to the participants, the researcher did however give some general instructions about the rating:
• Assess these scripts in the way you normally do. • You are free to select what script to start with.
• You are free to give feedback as in your normal practice essay situation. • You are free to write or draw on the script if this is part of your feedback
practices.
• You are free to provide a score or not.
After this, all the informants followed their normal response behavior without being controlled by the researcher’s instructions or prompting shaping their responses.
3.6.5.3 Presence of the researcher during TA data collection
The researcher's presence in the room is considered to be an important issue in data collection. The researcher was in the room while the raters produced the TA protocols, since as Ericsson & Simon (1993) have suggested, it is necessary for the researcher to be there to prompt the subjects if it is necessary, if there is a pause in the talking exceeding 10 to 15 seconds. Likewise, Green (1998) emphasized the need to inspire the subjects by asking them to ‘keep talking’ or ‘can you tell me what you are thinking?’ if they fall silent for a period of time (p. 42). The Pilot had shown that such silence can happen.
In this sort of TA, the researcher is invisible to the participant (behind the subject or at the corner side of an L-shaped table) and her role is to provide no more explicit prompt than ‘keep talking’ or ‘think aloud’, when the subject falls silent during the performance of the task. In the present study, the researcher sat on the long arm of an L-shaped desk whereas the teacher sat on the short side.
The TAPs and following retrospective interviews were recorded using two digital recorders. One was a Sony digital recorder with new battery which was tested twice: before the beginning of the task and at the beginning of the task. The other device was an iPhone 4 which accepts voice memos. It is an important arrangement to have a spare recorder in case the main recorder went wrong or the sound quality was weak. Meanwhile, the researcher took some observation notes. These notes made were for two reasons: the first reason was to make the participants feel relaxed and not observed or looked at, as would occur with video-recording. Secondly, these notes might yield some details that might not be captured by recordings.
After we finished recordings we made copies of the original recordings by backing up the files.