There is a relationship in the factor structure of Chiesa’s EFA and Kremmel and Harding’s EFA. (See Figures 18 and 19 for visual representations of the relationship between them.)
Factor Factor Name Mean Standard Deviation Min Max Range α
Factor 3 AfL 2.87 0.17 2.57 3.23 0.66 .93
Factor 1 TSA 2.78 0.08 2.61 2.97 0.36 .93
Figure 18 Chiesa, Factor 1: Technical Skills of Assessment w/ Kremmel and Harding’s EFA
Note. The [ ] indicates the number of items in K&H’s factor that were not included in Chiesa EFA (e.g., K&H’s first Factor contained 14 items. Of these, 11 items were also in Chiesa’s first Factor, and 3 were only in K&H.)
Sixteen of 19 items overlap with Chiesa’s Factor 1 – Technical Skills of Assessment, and Kremmel and Harding’s Factor 1 – Developing and Administering Language Assessments; Factor 5 – Statistical / Research Methods; and, Factor 9 – Scoring and Rating. However, three items were identified in Chiesa’s Factor 1 but not in Kremmel and Harding’s EFA: Item 50 – using specifications to develop items (questions) and tasks; Item 53 – developing portfolio-based assessments; and, Item 66 – selecting appropriate ready-made assessments. Also, three items from Kremmel and Harding’s Factor 1 were not in Chiesa’s first factor and were all items I removed: [Item 60 – aligning tests to proficiency frameworks (e.g., the Common European
Framework of Reference [CEFR], American Council on the Teaching of Foreign Languages [ACTFL]); Item X – Accommodating candidates with disabilities or other learning impairments; Item X – How to train others about language assessment.]
Additionally, two items were a part of Harding and Kremmel’s Factor 5 and not Chiesa’s Factor 1: Item 45 – using statistics to analyze the difficulty of individual items; and, Item 46 – using statistics to analyze overall scores on a particular assessment. All items in Kremmel and Harding’s Factor 9 were a part of Chiesa’s Factor 1. These results suggest there is a relationship between Chiesa’s Factor 1 and Kremmel and Harding’s Factors 1, 5, and 9. Kremmel and Harding’s results shows the nuanced nature of the different types of technical skills one would need to be language assessment literate. Additionally, the close relationship between Chiesa and Kremmel/Harding shows that the participants in both studies were responding similarly to survey items that proposed technical skills of language assessment. Also, Kremmel and Harding’s larger sample (N>1000) might have allowed for more factors than my study.
Ten of 16 survey items overlap with Chiesa’s Factor 2 – Social Aspects of Assessment, and Kremmel and Harding’s Factor 3 – Assessment Policy and Local Practices; Factor 4 – Personal Beliefs and Attitudes; Factor 6 – Assessment Principles and Interpretation; and, Factor 7 – Language Structure, Use, and Development (see Figure 19 below.)
Figure 19 Chiesa, Factor 2 (blue): Social Aspects of Language Assessment; and, Chiesa, Factor 3 (black): Assessment-for-Learning
Note. The (<) symbol indicates that the number of items in K&H’s factor also belong to Chiesa’s Factor 2; the (>) symbol indicates that the number of items in K&H’s factor also belong to Chiesa’s Factor 3. The [ ] brackets indicate that the number of items belong in K&H’s factor but neither Chiesa’s Factor 2 or 3.
Six items from Chiesa’s second factor were not present in Kremmel and Harding’s EFA: Item 14 – how to communicate assessment results and decisions to students or parents; Item 18 – how to find information to help in interpreting results; Item 27 – how pass-fail marks or cut-scores are set; Item 32 – the philosophy behind the design of a relevant language assessment; Item 33 – the impact language assessments can have on society; and Item 40 – different forms of alternative assessments (e.g., portfolio assessment). Additionally, all four items from Harding and
Kremmel’s Factor 4 were a part of Chiesa’s second factor. Three items [Item 35 – the assessment traditions in a local context; Item X – how to determine if a language assessment aligns with a local system of accreditation; Item X – how assessments can be used to enforce social policies (e.g., immigration, citizenship)] were a part of Kremmel and Harding’s third factor, but neither in
Chiesa’s second or third factor. Additionally, Item 26 – how social values can influence language assessment design and use, was a part of Kremmel and Harding’s seventh factor, but not in either Chiesa’s factor 2 or 3. These results suggest that the relationship between social aspects and assessment are somewhat difficult to capture in a survey for the Uzbekistan EFL teacher population. Thus, the underlying dimensions between the measured variables (e.g., the scores given by the participants on survey items that correspond to social aspects) and latent constructs (i.e., underlying variables) are not as clear-cut as seen above in the extracted factors of Kremmel/Harding that correspond to Chiesa’s first extracted factor.
Fifteen/Sixteen items of 22 survey items overlap among Chiesa’s Factor 3 – Assessment- for-Learning and Kremmel and Harding’s Factor 2 – Assessment in Language Pedagogy; Factor 3 – Assessment Policy and Local Practice; Factor 6 – Assessment Principles and Interpretation; Factor 7 – Language Structure, Use, and Development, and Factor 8 – Washback and
Preparation. (See Figure 18 above for a visual representation.) (One item in the original survey on self- and peer assessment was split into two items for Kremmel and Harding’s EFA.)
However, six items (2, 8, 9 11, 13, and 16) are a part of Chiesa’s Factor 3 and not Kremmel and Harding’s EFA: Item 2 – how to use assessments to evaluate progress in language learning; Item 8 – how to interpret measurement error; Item 9 – how to interpret what a particular score says about an individual’s language ability; Item 11 – how to determine if the content of a language assessment is culturally appropriate; Item 13 – how to communicate assessment results and decisions to teachers; and, Item 16 – how to recognize when an assessment is being used inappropriately. Additionally, Kremmel and Harding’s Factors 2 and 8 are completely
submersed into Chiesa’s Factor Three. These results suggest that there is a relationship between Chiesa’s Factor 3 and Kremmel and Harding’s extracted factors (2, 3,6, 7 and 8). Also, Kremmel
and Harding’s extracted factors show the nuanced nature of assessment literacy for classroom language teachers, particularly in the separation of Assessment in Language Pedagogy from Washback and Preparation. These extracted factors (and the items within them) reflect the
different assessment practices classroom language teachers do inside and outside the classroom.