Materiales de Partida y Métodos Experimentales
II.3. TÉCNICAS DE CARACTERIZACIÓN DE LOS MATERIALES
This second research question was raised to examine lecturers’ judgments of the generalizability of five CEFR scales to the assessment of their professional English for the purpose of lecturing in EMI. (For details on CEFR scale selection, see Chapters 1 and 5.) Including this aspect is important since meaningful testing should reflect the target situation. Thus, what the lecturers say in direct response to the CEFR criteria presented to them is extremely useful for pointing the way towards the central elements upon which relevant tests for this type of environment should be based. Including this research angle also adds pragmatic value to the present study. In other words, the aim was to investigate the extent to which a prescriptive, generic language assessment tool is suitable for measuring spoken professional English in an international context. For the investigation, a hypothesis was formed: the CEFR descriptor scales do not tap into the skills relevant to EMI lecturing.
To investigate the five CEFR scales, data were collected during semi-structured interviews. EMI-lecturers’ self-assessments were based on their interpretations of the five scales intended for measuring the generic linguistic skills of SLLs – not the professional language of SLUs – as depicted by the prescriptive tones and deficit views in the scales (as presented in Chapter 5 and disclosed in the analysis in Chapter 8). The findings from the investigation support H2, indicating the scales to be inadequate for assessing the professional English of EMI lecturers.
What EMI lecturers considered as suitable criteria for assessing their professional English was not found in the CEFR scales. What they perceived as relevant to the assessment of their professional English is summarized here. For lecturing, important to
coherence and cohesion is the ability to speak about conceptually more demanding
matters with well-reasoned arguments using ‘semantically dense’ (cf. Fillmore 1979: 51) utterances. This skill entails the ability to use cohesive devices for linking utterances into coherent discourse as related to telling a story (logically). In a long contribution, some ‘jumpiness’ is perceived as normal. The view of jumpiness also includes a communicative function: it may serve as a repair strategy for retrieving logical progression, for example, through a dialogic act triggered by a question, which then either restores the line of thinking or takes it in a new direction. The cohesive devices were also perceived as being similar to those found in conversational discourse.
Central to perceived fluency is (NNS) accessibility. This criterion entails an ability to
comprehend (NNS) fluency, where variability may be high. Some features of (NNS) accessibility include pausing when time is needed for message conceptualization as related
to formulating ideas and managing lexical gaps. Another criterion was perceived
intelligibility. Important to its description is the concept of a reasonable accent (as defined
by speech perception) that approximates what the intended audience could expect (whether ENL or not). In other words, an unreasonable accent includes speaking in a way that causes difficulty in comprehensibility. Closely related to this is the concept of a
‘correct’ accent, described as the ability to pronounce the sounds of English and to utilize
nuclear stress to enhance communication. This view seems to support the notion of a lingua franca core as defined in Jenkins (2000). Another feature of perceived intelligibility includes the ability to adjust or to accommodate others, such as speaking slowly and clearly as opposed to fast-paced native-like speech. A fourth criterion is acceptable NNS
lexicogrammatical features. Even at high levels of proficient English such features are
present, and those that do not hinder comprehensibility or meaning are viewed as natural and acceptable.
For the purpose of assessing lecturing, the analysis also identified some relevant aspects to be missing from the CEFR scales. For example, the findings indicate important criteria to be speaker credibility, defined as including an ability to demonstrate in-depth knowledge of the subject matter, to handle questions well, and to speak with good grammar. Another important item is making content accessible, which includes an ability to present cognitively demanding content in a way that is easily understood. This criterion was further defined as pausing naturally and accommodatingly, as well as expressing ideas clearly through explanations and examples at an appropriate conceptual level. Having an ability to engage the listener was another criterion, defined as showing enthusiasm for and interest in the subject matter as well as speaking interactively. Correct use of specialized
vocabulary was also considered important as well as knowledge of vocabulary from other
fields when lecturing in a cross-disciplinary context.
For assessing LSP, the analysis also detected several shortcomings in the CEFR scales, some of which have been previously raised in the literature. Earlier research has pointed out the lack of descriptors for professional vocabulary and discourse features for professional communication. The present study targets generic prescriptive scales as the main drawback. In other words, the descriptors aim to assess general language abilities and lack strong performance measures rooted in real-world criteria that are necessary for assessing professional English in work-related tasks. This drawback results in scales that de-contextualize the ‘learner’, exclude situational appropriateness, omit communities of practice, minimize professional (i.e. specialized) vocabulary, and overlook indigenous assessment criteria. Thus, while a generic approach to language assessment may adequately evaluate general linguistic correctness, assessing LSP with such criteria creates a weak basis for adequately assessing the communicative character of work-related meanings and functions in a given real-world task.
The present study also found some limitations in the CEFR scales for assessing ELF. As already indicated in previous studies, the NS as the measure of language competence is a major drawback for the assessment of ELF. In the present study, lecturers’ accounts suggest that the scales are interpreted as NS based, irrespective of whether they contain the term ‘native speaker’. This term also represents a ‘learner’ goal of acculturating to a target ENL culture, a goal not shared by SLUs. As the yardstick of SLL attainment, the
CEFR emphasizes NS language ideologies and prescriptivism as the basis for assessment, which enforces both learner-related language ideologies and correctness as the measure of success. This measure evolves around a NS-NNS axis, where the aim for SLL is to assimilate to a given ENL target culture. These ideologies are prominent in the CEFR scales (as illustrated in Chapter 5), and contrast with its discourse related to plurilingualism and pluriculturalism. In brief, the emphasis on native-speakerism and correctness over the communicative character of the language fails to capture the nature of ELF spoken discourse.
All in all, the findings support H2, indicating that a generic tool, such as the CEFR, cannot adequately assess professional English (i.e. LSP). Nor can it assess English for use in an international context (i.e. ELF). Its prominent basis in NS language ideologies and prescriptivism contrasts with findings from the present study, which indicate that non- nativeness, variety, and plurilingua-culturalism are key factors for spoken professional English in an international context. These factors point toward the notion of a competent
international SLU (see Table 4-2) as a concept against which to evaluate ELF usage. This
notion contrasts with the idealized educated NS as an attainment target. While the target of an educated speaker seems plausible (see, for example, Mauranen 2012), the ELF literature casts doubt on the idealization aspect as being relevant to successful communication. This point crystallizes in the acceptability of (broad) variation in ELF interaction (see, for example, Hynninen 2013; Jenkins 2000; Mauranen 2012; and Ranta 2013). For LSP, the present findings indicate that the CEFR needs some sort of extra option. This option could include scales for assessing different genre, such as English for lecturing, advertising, or consulting.