From the point of view of this investigation, the CEFR scales have limitations for both LSP and ELF. This section presents its shortcomings in relation to these two aspects.
8.3.1 Limitations for LSP
For assessing LSP, the CEFR scales have limitations, a point raised in previous literature (e.g. Huhta 2010; Toepfer and Virkkunen-Fullenwider 2005). A central problem is the lack of focus on professional vocabulary and discourse features for professional communication. Contributing to this discussion, Table 8-3 summarizes five limitations of the CEFR scales for assessing LSP.
The left-hand column presents five categories into which the limitations have been grouped. The right-hand column outlines the limitations. One shortcoming includes the treatment of assessment in generic terms, with descriptors based on
Table 8-3. Some CEFR limitations for assessing LSP.
predetermined and fixed norms that are directed at a ‘learner’. This approach lacks descriptors that conceptualize the social context in which an assessment occurs. The de- contextualized approach omits situational factors relevant to communicative competence, excluding facets such as situation, role, and appropriateness of language and communication skills relevant to determining the successful completion of professional tasks. In addition, the scale for vocabulary range lacks harmonization, assuming a wide range of categories that do not reflect measures for capturing the range of professional vocabulary acquired. Similarly, the scales do not consider communicative factors related to communities of practice (Huhta 2010). For example, what features of discourse are important to delivering a lecture successfully? On measurement for LSP, the literature suggests applying indigenous assessment criteria (e.g. Jacoby 1998), a feature missing from these scales. These limitations illustrate the inadequacies of the CEFR scales for assessing LSP.
For lecturing, the accounts suggest the descriptors are missing some aspects that would be relevant to assessing language competence for this target purpose, as summarized in Table 8-4.
In the table, the left-hand column shows four categories into which the missing aspects have been grouped. The right-hand column outlines the aspects drawn from the interview data that appear to be missing from the CEFR scales. The first category, speaker credibility, was raised as important for lecturing, where having good grammar is relevant, being knowledgeable about the subject matter is central, and having an ability to answer questions is pertinent. The second category, engaging the
Table 8-4. Some aspects to consider for assessing lecturing.
listener, arose as important to the professional context, where monologic discourse is prominent. Aspects of engaging the listener include showing enthusiasm for the subject as well as pausing accommodatingly and naturally in a way that enhances listening. The next category, accessible content, is essential to lecture comprehension, especially when content is cognitively demanding. This category refers to the ability of the lecturer to present challenging content in an accessible manner, where accessibility means expressing ideas clearly through explanations and examples at an appropriate level. The third category, linguistic aspect, includes the ability to use correctly specialized terminology. These findings indicate the focus of the five CEFR scales to be too general for assessing LSP, as the scales lack performance criteria that include contextual aspects specific to the target purpose. Overall, the findings point toward the five scales as being inadequate for assessing professional English.
8.3.2 Limitations for ELF
For assessing ELF, the CEFR scales have some major limitations, a point also raised in earlier critiques (e.g. Ahvenainen 2005; Hynninen 2006; Seidlhofer 2003). Particularly problematic is the NS as the primary measure of language competence. This ‘learner’ goal is contrary to lingua franca usage, which primarily entails interaction among NNSs using English as a tool to accomplish tasks. Although the term ‘native speaker’ does not appear in every CEFR scale, the accounts suggest the scales were interpreted as NS based.
Related to the NS limitation is the notion of acculturation to a NS target culture, where monolingua-culturalism appears to be the objective. Although the CEFR acknowledges the need for competence in plurilingualism and pluriculturalism (Europe 2001: 4, 168), its scales concentrate on language competence for a target culture of native speakers, as emphasized by NS referencing. Such criteria for language assessment fail to consider the need for plurilingual and pluricultural skills in lingua-franca contexts. While this approach to language assessment may suit other European languages, it is ill fitting for English as a global lingua franca, which is comprised of more L2 than ENL speakers, who communicate in English outside a monolingual ENL world.
For spoken ELF, the following aspect seems to be missing from the descriptor scales: assessment criteria for a competent international SLU. The accounts suggest that suitable criteria for SLU would exclude a NS-bias. In other words, they would be neutral, reasonable, and fair for usage of English in international contexts. A related aspect, implied in the accounts, is the concept of speech being co-constructed and highly variable.
Table 8-5. Some suitable criteria for ELF – missing from the CEFR.
As shown in Table 8-5, one criterion appears to be NNS fluency that is accessible to ELF users. In other words, fluency is determined by the degree to which NNSs interact successfully to achieve a given task. Another criterion seems to be a reasonable accent, which was defined as one that is intelligible (regardless of the accent being native or non- native). A third criterion was the acceptance of some NNS features in lexicogrammatical constructions. The accounts suggest that NNS language is not perfect. They also suggest the use of plurilingual strategies to be a positive attribute in ELF communication.
8.4 Conclusion
In this chapter, I set out to investigate the extent to which five CEFR descriptor scales used in a pilot-certification assessment were suitable for evaluating professional English in an international context. The data were gathered during interviews with EMI lecturers, who were reading and interpreting the scales. The findings from the data analysis suggest the descriptor scales do not target skills relevant to the spoken professional English of
lecturers in a lingua franca context, which lends support to H2. Support for H2 was already evident in lecturers’ self-assessments (in section 8.1), which pointed to different evaluations of their professional English and general language skills. Section 8.2 also revealed a clear difference between lecturers’ and (language and communication) experts’ perceptions of a sufficient CEFR level for EMI lecturing, as well as the reasons for lecturers’ choices and what they considered to be adequate measures of their spoken professional English.
With regard to lecturers’ perceptions of the suitability of the CEFR scales for measuring their professional English, the key findings are summarized here. In regard to the coherence-cohesion scale, the accounts suggest L2 lecture discourse to be perceived as similar to conversational discourse, a view that aligns with previous literature on L1 lecturing (e.g. Biber 2006). The accounts also view some ‘jumpiness’ in lecture discourse to be normal and telling a story logically to be important. The view of jumpiness included a positive function, perceived as an act that may elicit a dialogue that serves to restore logical progression. This view suggests some descriptive phrases in the lower half of the scale to be more suitable for describing lecturing than the upper half. In relation to the
fluency scale, the accounts imply accessible (NNS) fluency to be central to professional
communication rather than native-likeness. For lecturing, this fluency involves engaging the listener and presenting content accessibly. These criteria include an ability to present conceptually difficult content at an appropriate level using clear examples and explaining ideas clearly, as well as pausing naturally and accommodatingly in the given context. Here, ‘naturally’ may mean pausing due to an occasional lexical gap that is repaired in a non-distracting manner, for example. In other words, ‘pausing naturally’ is not described as native-like, but rather in terms of accessible NNS fluency.
On phonological control, the scale itself draws attention to distinctions between L1 and L1 use. The accounts viewed NS targets as biased, and a reasonable, intelligible accent as fair. In short, intelligibility was stated as the goal, not native-like English. As the accounts also rightly point out, scales that pit ‘intelligibility’ against ‘foreignness’ as a measure of communicative success (as this scale does) display overt prejudice. This concept has no place in language testing scales. Similar to the previous scale, accounts related to grammatical accuracy also viewed the scale as NS-biased. The accounts also imply some NNS features to be acceptable at high levels of proficiency. Good grammar was viewed as important for speaker credibility and so was in-depth knowledge of the subject.
The view of the scale for vocabulary range was that it is suitable for describing general language competence rather than professional language ability. The accounts also viewed the scale as applicable to SLL for an ENL context, but not to SLU where terminology for professional English is relevant in an international context.
The analysis also indicated some limitations in the CEFR scales for assessing ELF, a point previously critiqued in the literature (see discussion in 8.3.2). The central problem lies in the NS as the measure of language competence. The accounts suggest the scales to be interpreted as NS based, irrespective of whether they contain the term ‘native speaker’. This term also represents a ‘learner’ goal of acculturating to an ENL target culture, a goal not shared by ELF users. This aspect of the CEFR renders it inappropriate as a tool for
measuring lingua franca competence. For assessing ELF, the present study indicates some suitable criteria as including accessible NNS fluency, an intelligible accent, accommodation skills, and plurilingual strategies.
For assessing LSP, the analysis also discerned several shortcomings in the CEFR
scales, a point also raised in previous literature (see discussion in section 8.3.1). In the present study, the main drawback points to a lack of contextualization. More specifically, this drawback is characterized by the following: assessment of the general language abilities of a ‘learner’, exclusion of situational appropriateness, omission of communities of practice, minimization of professional (i.e. specialized) vocabulary, and absence of indigenous assessment criteria. For the purpose of assessing lecturers, the analysis also identified some aspects missing from the scales, such as speaker credibility, engaging the listener, making content accessible, and linguistic use of specialized terminology.
The present study also found the CEFR scales to lack harmonization relevant to assessing ELF, as follows: the scale for grammatical control is inconsistent in its focus on communication, and all five scales are NS-biased (whether implicitly or explicitly stated). The latter aspect was shown to cause ambiguity in self-assessing spoken fluency, lexicogrammatical ability, and intelligible pronunciation. For LSP, the scale for vocabulary range was shown to lack harmonization for assessing professional vocabulary.