Generación de autoempleo en Vinchos-Ayacucho, 2007-

A key element in the inquiry’s evaluative focus was the extent of participants’ L2 English development as an attributed result of the program’s L1/L2 awareness-raising focus. Because of methodological concerns centring on the futility of attempting to establish meaningful controls within the naturalistic setting of the program, I felt it was essential that the facilities for initial and follow-up achievement testing be augmented by extensive reference to the perceptions of the participants in this regard – hence the important qualification, ‘attributed’. The methodology in respect of this 87

Table 1. Tabular Summary of Data Generation Methods

Primary Purpose

Data Generation Tool

Diagnostic Base-Level Measurement

Initial Cross-Linguistic LA Test

Program Achievement Measurement

Follow-Up Cross-Linguistic LA Test

Recording of LA Module Delivery and

Developing Researcher Cognitions

Researcher’s Journal

Recording of Participants’ LA Learning

Participant Learning Journals

Advance Organizing of Participant

Cognitions Prior to Focus Group Interview

Participant Cognition Elicitation Worksheets

Discussion of Participant Perspectives on

Aspects of the Japanese EFL Context

Video-Recorded Focus Group Interview

Recording of Participants’ Evaluation

of LA Program Materials

WENS Materials Feedback Schedule

Recording of Participants’ Evaluation

of LA Program and Its Relevance

Written Evaluative Feedback Assignment

Table 1. (continued) Tabular Summary of Data Generation Methods

Primary Purpose

Data Generation Tool

To Allow Detailed Retrospective Access

to LA Class Content/Interaction

LA Class Audio Recordings

To Record Participants’ Evaluative Feedback Commentary on LA Module

Audio-Recorded Exit Interviews

To Elicit Stakeholder Perspectives on LA Module and Maintain Researcher/

Participant Contact Post Module

Personal E-mail Communications

Recording of Participants’ Awareness of their Interlanguage Development

Interlanguage Development Worksheets

To Elicit Participants’ Cognitive

and Affective Responses to the Focus Group Interview

Focus Group Interview – Participant Response Schedule

To Record Participants’ Planning of and Reflections on the LA Micro-Teaching Sessions

Lesson Plans / Reflective Lesson Evaluations

To Place the Evaluation of the LA Module within the Context of the Overall 2010

Program Evaluation

Director’s 2010 Program Overview Report

To Record Participants’ Rating of the LA Module Relative to the

Other Program Modules

2010 Program Feedback Questionnaire

91 aspect of the evaluation was therefore a combination of basic quantitative, and more nuanced qualitative analysis. With regard to the former - the administering of pre- and post-tests - the procedure was as follows:

The initial diagnostic test, which was administered on April 28th_{, 2010 (week one, day} three of the program), was oriented to the notion of trainee L2 teachers as L2 learners, and, as such, constituted a basis for ongoing experiential learning as the program unfolded. The test essentially involved the elicitation of L1 Japanese to L2 English translation equivalents using photocopied pages from the WENS materials, i.e. the – as yet unseen - program content. The L1 Japanese test cues for the elicited L2 English translation equivalents on each photocopied page were, in fact, the L1 Japanese translation equivalents of my reformulation of the fifteen attested interlanguage utterances which featured on the opening page of each of the WENS materials’ thirty- six units.

The need to balance such factors as available time and anticipated test-taker fatigue with adequate and representative coverage of the program content material resulted in my decision to select twelve worksheets, i.e. photocopied pages (unit numbers: 05, 07, 11, 12, 14, 16, 17, 18, 19, 20, 22 and 30), from the thirty-six WENS units, that is, one third of the worksheets potentially available, and to allow a total of ninety minutes for the completion of the test. These parameters gave the participants an average of seven and a half minutes to complete each of the twelve worksheets – this averaging out to thirty seconds for each of the fifteen elicited translation entries on each of the twelve standardized worksheets.

The participants completed the exercise individually under test conditions and without the aid of dictionaries. I felt that the marking scheme adopted in respect of both the initial diagnostic test and the follow-up achievement test had to be sufficiently finely calibrated so as to permit a finely-grained analysis of the test answer formulations (i.e. L2 English translation equivalents) produced by the participants. For this reason, I decided to allocate a potential maximum of ten credits for each of the one hundred and eighty test entries - fifteen entries times twelve worksheets, reflecting a composite of multiple evaluation criteria – facets such as grammatical accuracy, semantic equivalence and transparency, socio-pragmatic appropriateness, correct spelling and punctuation, syntactical awareness, the use of suitable registers and the appropriate 90

92 degree of formality. Acceptable alternative L2 English translations of L1 test cues offered by the test-takers – alternative to the L2 reformulations that I myself had produced and included in the WENS materials, that is – were of course equally eligible for potential maximum credits.

The potential maximum total test score for each participant was therefore one thousand eight hundred credits – a figure which I felt provided the requisite degree of calibration; enabled the establishment of a realistic initial test baseline score for each participant; allowed for a degree of intra-group differentiation, and facilitated the subsequent quantification of initial to follow-up test differentials.

The follow-up test was held in the afternoon session on July 28th (week fourteen, day three of the program). The format adopted for the follow-up test was exactly the same as that used for the initial diagnostic test three months previously on April 28th_{, with} identical test items (this fact was not revealed to the participants in advance of the follow-up test) and identical test conditions and requirements. The one notable difference concerned the scheduling: whereas the initial test had taken place in the morning, when the test takers might be expected to be relatively alert, the follow-up test was held in the afternoon, immediately after lunch, when the test takers might be expected to be relatively less alert. In addition, the scheduling resulted in a clash between any revision that the participants might have been intending in respect of the follow-up test and their preparation for the second micro-teaching practice session, and thereby precluded an exclusive focus on test preparation. The participants were, if anything, disadvantaged by the scheduling of the follow-up test relative to that of the initial test.

The criteria adopted for the grading of the follow-up test were identical to those used in the grading of the initial test. I graded the follow-up test ‘blind’ to avoid any undue influence which might exert itself as a consequence of my being aware of whose test paper I was grading. The procedure was as follows: I printed out the numbers one to nine on adhesive label tape, and had the nine participants each select a number while I was out of the room. The participants committed their code numbers to memory and/or secretly recorded them and attached their numbered adhesive labels against a name sheet which I had prepared for the purpose. The participants - I nominated Yuya - then sealed the name list containing the test codes in a prepared envelope, which I 91

93 kept unopened until I had completed all the grading. The participants used their code numbers in lieu of their names on their follow-up test papers. The assigning of code numbers took place a week or so before the test to ensure that I did not overlook this step on the actual day of the test. I made a point of checking that all the participants remembered their code numbers on the Monday of week fourteen, two days prior to the test, and again on the day, immediately prior to the test.