Research questions two and three yielded qualitative data, and both involved the same analyses. The analyses consisted of open or substantive coding, which is when researchers “read their data and begin to code all different types of actions, events, processes… and write analysis notes in the margins of the documents they are working on” (Dillon, 2013, p. 3). This type of coding is also referred to as content analysis. In content analysis, key points are identified from the data. “Content Analysis is an interpretive approach and therefore includes extracting examples from the data” (Ho, 2012, p. 2). Thus, two separate files were created in NVivo (v. 10.2.2) for Mac, to analyze questions two and three and their subquestions.
After combing through the focus group and semi-structured transcripts, each line(s) or paragraph was coded for research question two: What do Uzbekistan EFL university teachers report they do to assess their students (i.e., assessment tasks and scoring procedures)? This question began the qualitative data analysis process, which started with coding for how teachers reported how they do Assessment-of-Learning and Assessment-for-learning practices. Then, subcategories were created from the broad categories after the data had been read multiple times. Subsequently, to address the inquiry – What are their cognitions surrounding those assessment practices – I went back through the introduction phase of each focus group and semi-structured interview and charted each teachers’ self-introduction. I noted down the following: pseudonym of each teacher, what province s/he came from, what type of university s/he taught at, what type of curriculum s/he are a part of, and as much of their background as they revealed throughout the
interviews. From these identifiers, I reread the transcripts and matched who said what at what point to discern patterns in the teachers’ cognitions and their reported assessment practices. To address the third subquestion – How do their cognitions about assessment shape how they assign scores to their students’ work – I combed back through the transcripts and identified how each teacher (or a group of participating Uzbekistan EFL teachers) scored students. Based on the results of the three subquestions I determined how the teachers’ use assessment for the service of student learning.
After I addressed research question two (and its subquestions) I analyzed research question three. I began with the following inquiry: What are the macro-environmental factors Uzbekistan EFL university teachers report? Lines and paragraphs in the focus group and semi- structured interview transcripts was coded for social, cultural, economic, and/or political factors that influenced the Uzbekistan EFL teachers’ assessment practices. Following the initial coding of the data, I read through the data six more times for possible coding categories that could be merged, modified, or clarified, and then identified two core codes. To make sure the core codes were consistent, I utilized the card-sort technique, initially developed by Lincoln and Guba (1985) as a quality control mechanism. I placed individual statements from each category onto three-by-five index cards and placed them randomly in a pile. After I explained the definitions of each category to two additional coders (one for research question two and three respectively), I had them distribute all the cards into the respective categories. Then, intercoder agreement was calculated. Then, to answer the second subquestion - What are their cognitions surrounding these (macro-environmental) factors – I used the charted categories from research question two (see above) and matched which teacher said what about each macro-environmental factor and when. After identifying the patterns in the teachers’ cognitions, I answered the overarching research
question that inquired about the macro-environmental constraints and/or affordances in Uzbekistan that could shape how EFL teachers provide meaningful assessment situations for their students.
4.4 Conclusion
In this chapter, I discussed the research methodology used to address three overarching research questions (and nine subquestions): (1) To what extent does the Language Assessment Literacy Survey (Kremmel & Harding, forthcoming), provide valid and actionable information about teachers’ language assessment literacy; (2) How do Uzbekistan EFL teachers talk about their assessment practices and justify the scores they provide for their students; and, (3) What are the macro-environmental constraints and/or affordances in Uzbekistan that could shape how EFL teachers provide meaningful assessment situations for their students? I began with an explication of the research participants, which included descriptive statistics of their ages and number of years of experience teaching English in university settings. Then, I discussed research question one, and how Harding and Kremmel’s Language Assessment Literacy survey was adapted for this study, followed by the procedure for its distribution and quantitative analysis. Furthermore, I explained the necessary steps to validate the Language Assessment Literacy survey.
Subsequently, I discussed the methods of research questions two and three. First, I explained the recruitment process for the focus groups, the focus group interview protocol, and the procedure for carrying out the focus group interviews. Then, I discussed the semi-structured interviews, the recruitment process, and the interview protocol. Because research questions two and three yielded qualitative data, I explained how they were both analyzed similarly, which began with open (substantive) coding. The results from the three research questions will present a picture of
if and how the participating Uzbekistan EFL teachers provide assessment at the service of learning for students.
5 RESEARCHER POSITONALITY AND REFLEXIVITY
In this chapter I will address my positionality, which I believe is a valuable and necessary exercise for any researcher who collects and analyzes empirical data. One of the challenges of writing this statement is to be ethically honest and open, so I would be able to acknowledge biases that are inherent in my orientation toward research, the research process, the content under investigation, the participants, and the analyses. The purposes of creating a positionality
statement cannot be overstated because it will inform me, the researcher, and you, the reader, that I am cognizant of my own biases and I am trying to address them so I can achieve something as close as possible to what is commonly referred to as ‘truth’ from the research process.
Smith (2013) explains that “research is not an innocent or distant exercise, but an activity that has something at stake, and it occurs in a set of political and social conditions” (p. 15). The conditions she referred to are all related to issues of power. Those of us who serve as researchers are exercising a form of power when we engage with participants, the questions that we write or ask, and particularly when we assign categories to data collected. To address my positionality and to analyze such power relationships, I will locate myself in relation to the phenomenon under investigation – the cognitions of language teachers related to assessment literacy. Then I will locate myself in relation to Uzbekistan EFL teachers, who are the study’s focal participants. Finally, I will locate myself in relation to the research context and process of conducing
qualitative research. Throughout, I will address how I have tried to overcome the barriers of my position as a researcher in the field of Applied Linguistics.