The study was conducted by means of a series of investigations: firstly a questionnaire survey was carried out to examine teachers’ beliefs of classroom-based assessment; secondly, an interview was undertaken with each volunteer teacher focusing specifically on why they do what they do regarding the issues suggested in the first investigation; then these findings were reviewed by referring to the relevant materials that had been developed and used for assessment activities. The data collection for the first investigation was carried out from September in 2007 to December in 2007 during the second semester of the primary schools. The data collection for the second investigation took place from mid-March 2008 to the beginning of July 2008.
Ⅳ.5.1.Questionnaires
The research tools were developed based on the theoretical arguments of the procedural principles of classroom-based assessment developed previously in the literature review (see, chapter Ⅲ. 4) and were carefully examined through discussion with the thesis supervisor; this took account of the particular consideration of the English teaching and the learning context in Korean primary education (see Chapter Ⅱ) in which the research would be conducted.
Ⅳ.5.1.1.Pilot study
The questionnaire was piloted with fifteen primary teachers of English to check whether it was valid and worthwhile from teachers’ point of view. Ten teachers have Master’s degrees in English language teaching and are experienced English teachers, one teacher is reading for a Master’s degree and the other has a Master’s degree in primary education and has received sufficient teacher education program for English language teaching. Feedback from twelve of them was received. The other three agreed to scrutinize the questionnaire but they did not return it.
A summary of the feedback to the pilot is as follows. First, providing the teachers with a Korean version of the questionnaire would help the teachers to understand exactly what the researcher intends and would assist them in answering the questions precisely. Secondly, it would be a very good idea to explain the terminology used in the questionnaire such as ‘classroom-based assessment’, ‘validity’, ‘reliability’, and ‘ethical
issues’. The reason is that some teachers would not answer the questions simply because their knowledge of the terminology is inadequate. Third, some questions do not represent a particular area adequately so they need to be replaced with other questions.
In response to the feedback, the research tools were revised and also carefully scrutinized through discussions with the thesis supervisor. After that, they were returned to the same monitors to see if they could detect any further problems. Taking into account further feedback, it was decided to carry out Part 4 of the questionnaire one week later than Parts 1-3. The reason for this was that the participants’ opinions might be influenced by what they read in the questionnaire as well as by their responses in Part 3; alternatively, had they answered Part 4 first, it might have affected their responses to Part 3.
Ⅳ.5.1.2.Data collection and analysis
The questionnaire and the consent form were mainly distributed by post and by hand. Email was used to contact the teachers who work in Gwangju because they are quite a distance from the researcher’s workplace. In accordance with the feedback of the pilot, a Korean version was provided to help participants understand clearly and the explanations of a number of terms were given to the teachers to enable them to give more exact and straightforward responses. They might have provided the researcher with unclear or wrong responses if they had misunderstood the terms used in the questionnaire which would have been harmful for the study, for ‘the difficulty in studying teachers’ beliefs has been caused by definitional problems [and] poor conceptualizations…’(Pajares 1992: 307). This communication also explained the objectives and the contents of the
questionnaire. In the case of Parts 2 and 4, the teachers were allowed to write their replies in English or Korean as they could thus save time and elaborate on their ideas easily; some teachers wrote their opinions in English, and others did it in Korean. In addition, it was emphasized that the findings of the research would help teachers to understand themselves as assessors, and that the findings might be a foundation for the development of the pre- and in-service assessor programs for the teachers.
Finally, data were collected by hand or by asking the teachers to return the completed questions via internal document delivery system of the Incheon educational authority, by post or by email attachment. The investigation took over three months as all the questionnaires could not be distributed to the teachers at the same time. For example, the time the researcher visited the teachers in Seoul was different both from when he met the teachers in Incheon and sent them the questionnaire through internal document delivery system, and when he sent the emails to the teachers in Gwangju. The timescale was affected by the plan to distribute part four of the questionnaire one week later than the other parts. The timetable was not adhered to as closely as the researcher had hoped because, in some cases, he received the responses late. He had not anticipated these delays. The respondents did not require any further clarifications, but some did have difficulty expressing their views when completing the open-ended questions.
Further analysis of the data of Parts 1 and 3 (etc) was carried out using SPSS (15.0 for Windows) in order to produce the frequency, percentage, mean and standard deviation of all the items. In order to estimate if the gap between teachers’ beliefs and their practice is
statistically significant, a T-test was executed. According to Brown and Rogers (2002: 205), this is the most frequently used measure in second language research for comparing mean scores and a very useful measure for various group sizes. In addition, Cronbach’s alpha was also used to examine the internal consistency of the questionnaire. The data of parts 2 and 4 were translated then analyzed through classification and listing.
Reliability Statistics Reliability Statistics
Figure 3: Reliability of 3-1 Figure 4: Reliability of 3-2
Ⅳ.5.2.Interviews
The aims of the interview were to investigate the gaps between the teachers’ beliefs and their practice, and to gain some understanding of the reasons for perceived divergences
Cronbach's
Alpha Alpha Based Cronbach's on Standardized Items N of Items .936 .938 42 Cronbach’s
Alpha Alpha Based Cronbach's on Standardized
Items
N of Items
between principles and practice in their assessment activities. Interview questions were developed based on the analysis of the first investigation and this was carefully examined through discussion with the thesis supervisor, taking account of the particular circumstances of English teaching within the learning context of Korean primary education and especially how the teachers would be contacted and the interviews recorded. Also, a procedure was devised which would be adopted to guide the interviewer.
Ⅳ.5.2.1.Data collection and analysis
Advance notice of the interview was distributed to the teachers by email and by phone; then the volunteer teachers who were willing to be interviewed were provided with a full introduction to the interview.
Since they had taken part in the first investigation, all the teachers readily understood what they were supposed to do and accepted the demands involved. However, one problem which revealed itself was the difficulty of making a schedule for the interviews, because the interviewees usually worked in their schools from 8:40 to 4:40 and, moreover, they had regularly been very busy. As a result of this, a number of interviews were carried out after school and sometimes on a Saturday. The researcher visited ten teachers and interviewed them face-to-face, he also interviewed six teachers by phone; this applied particularly to those who worked in Seoul and Gwangju which are geographically distant from where the researcher resides. All the interviews were recorded
by voice recorder and saved in an audio file in MP3 format. Then they were selectively translated and analyzed through classification and listing. That is, the teachers’ responses were classified based on such criteria as: whether or not they represent the reality of teachers’ classroom-based assessment in a given context; whether or not they might contribute to the development of the body of theory of classroom-based assessment; and whether or not they might provide the teachers with any clues as to how to improve their assessment practices.
Ⅳ.5.3. Assessment batteries
The collection and analysis of the assessment batteries was used to investigate the teachers’ assessment practice; this was done because they normally contain assessment specifications, criteria, assessment tasks, and marking scales; and in order to understand the extent to which the teachers’ personal principles of classroom-based assessment were applied to their practice.
Ⅳ.5.3.1.Data collection and analysis
The approach adopted was to analyze the assessment batteries and thus to determine the criteria by which the results were analyzed. Were they based on such criteria as: the extent to which they comply to the assessment cycle presented in the study; the degree to which they relate to the students’ learning tasks given in their textbooks; the coverage of four skills of English; the level and nature of any encouragement given to students which might improve their performance while using English language; and finally, the
practicality of whether the assessment activities was really manageable within the limits imposed by planned time restrictions. The results are evaluated when the teachers’ assessment practice is analyzed at each stage of classroom-based assessment in chapter 5.