ESPACIO PÚBLICO PRECARIEDAD Y POBREZA

Detailed explanation of the methodologies considered for each study and the reason for why specific methods were selected rather than another is described in the following chapters. The reasons for choosing both qualitative and quantitative methods can be briefly explained in the context of each research question as follows.

RQ1: To what extent can the automated rhetorical parser XIP be used to identify indicators of good academic writing in undergraduate student essays from different disciplines, as judged by the essay grade?

As XIP was originally developed for the analysis of experienced researchers’ journal articles in the discipline of science, the first sub-research question explores how XIP performs in the analysis of student essays. The aim of this research question is to

understand whether XIP’s analysis criteria can match those required by human markers for undergraduate essays in various disciplines. In doing this, quantitative analysis was

conducted to explore how successful XIP analysis is in identifying the quality of student essays in terms of essay grades. The data collected for this research question therefore includes a range of argumentative student essays from different disciplines (hard

knowledge fields like physical sciences, and soft knowledge fields like arts and

humanities, social sciences) with their essay grades. All these student essays were rendered through XIP and the results were quantified in terms of the number of salient sentences for each XIP category. The dataset was then analysed using the quantitative method of

regression analysis, as described in detail in chapter 5.

RQ2: How do educators define the attributes of good student writing and to what degree can the automated rhetorical parser XIP identify the presence of these attributes?

The second sub-research question requires educators’ understanding of what makes good student writing and to what degree this overlaps with how XIP judges good writing. RQ1 used a quantitative measure of essay grades as a way of identifying good quality student writing. For the RQ2, qualitative measure of the educators’ definition of a good quality student writing was used. In order to understand and explore the opinions of educators on this matter, individual interviews were conducted with educators after they had assessed a student essay. In order to answer this question, it was necessary to observe educators in their real-world context of essay assessment and to comprehend their way of thinking about what makes a good essay; therefore, a qualitative method of data collection was chosen. As educators mostly assess student essays alone, the observation needed to be in an individual setting, which also allowed further questions to be asked. This facilitated the exploration of what educators think makes good student writing, what the attributes of good student writing are and what they value most when assessing student writing. Interview recordings were transcribed by using an intelligent verbatim approach6

(McLellan, MacQueen, & Neidig, 2003). The emerging themes from the thematic analysis of interview data were used to evaluate to what degree XIP output can meet the educators’ expectations. Following this, an overlap study between educators’ and XIP’s highlights

6 _{Leaving out background noises, ‘um’, ‘eh’, ‘you know’ etc. for both interviewer and respondents but includes}

((laughs)) to show emotion of respondent. No repetitions, descriptors or tripping over words included. The transcript was tidied up but without losing or adding to any of the important data.

was used to measure levels of accuracy and how well XIP’s highlights aligned with the educators’. For a detailed description of the study, data collection and analysis see chapter chapter 6.

RQ3: How congruent is the XIP analysis of student essays with educators’ judgement of quality?

The third sub-research question explores how reliable educators find the XIP output in terms of the quality of student essays, whether this type of analysis by XIP is acceptable to them, and whether XIP results are reliable and trustworthy enough to embed within a pedagogical context in higher education.

Focus group interviews with educators and senior researchers in the area of academic writing were carried out not only to add to the reliability of the quantitative data analysis results, but also to observe and comprehend the ideas and the interaction between experts about the applicability of XIP in an educational context, and to discuss possible actions to make it better for the purpose. Group interaction is the essential feature of the focus group method; it is distinguished from one-to-one interviews or questionnaires by the

opportunities it provides to reflect on the interaction between research participants. The ability to observe participants’ agreement and disagreement is an important strength of the focus group method. Therefore, in order to answer this research question, the focus group method was used. Focus group recordings were transcribed by using a full verbatim

transcription (Poland, 1995). The qualitative method of thematic analysis was used for data analysis of the transcribed focus group session. In quantitative data analysis of any sort of transcribed talk, the analytic possibilities are reduced at an early stage and researchers work with pre-defined categories, and this would have limited the analyst’s sensitivity to what was taking place in the talk. Instead, a qualitative approach was used, so that emerging themes and categories were generated based on the data rather than on prior

assumptions underlying the coding scheme used (Mercer, 2010). For a detailed description of the study, data collection and analysis see chapter 7.

RQ4: In what ways should the output of the XIP analysis of student essays be delivered to educators so that they make use of this output to give feedback on student essays?

The final sub-research question explores the ways in which XIP output can be presented to educators so that they make use of this output to give valuable feedback on student essays. This research question was answered through a pilot study followed by a main study. The pilot study was the first attempt to make sense of the raw XIP analysis output through a user-friendly dashboard.

In doing this, it was necessary to collect user requirements for the possible output. This was done through discussions about paper prototypes with PhD students at the Open University in the pilot study, and with educators (module team chairs who take part in the production of the module) in the main study. This elicited initial reactions from experts as well as their requirements. Co-designing a prototype tool with experts led to a working prototype. The aim was not to create a product but to use a working prototype which used a combination of previous findings to develop guidelines that may inspire future

researchers to develop a fully working tool. Therefore, the prototype was not tested in terms of the user experience or usability. For a detailed description of the study, data collection and analysis see chapter chapter 8.

The table below summarises the mixed-methods approach used for each research question. Table 4.1 Data collection and analysis methods used for each research question

Research question Data collection Data analysis

methods

1. To what extent can the

automated rhetorical parser XIP be used to identify indicators of good academic writing in undergraduate student essays from different disciplines, as judged by the essay grade?

 Qualitative data: student essays  Quantitative data: essay grades  XIP analysis of student essays  Quantification of XIP analysis results  Regression analysis with quantified XIP

analysis results and essay grade 2. How do educators define the

attributes of good student writing and to what degree can the automated rhetorical parser XIP identify the presence of these attributes?  Qualitative data from individual interviews with educators  Quantitative data of the number of sentences that educators highlighted  Qualitative thematic analysis of the transcribed interviews  Quantitative measurement of the degree of overlap between XIP highlights and educators’ highlights on student essays 3. How congruent is the XIP

analysis of student essays with educators’ judgement of quality?

Qualitative data: focus group session with educators

Thematic analysis of the transcribed focus group session

4. In what ways should the output of the XIP analysis of student essays be delivered to educators so that they make use of this output to give feedback on student essays?

Qualitative data: focus group session with educators to elicit the user and system requirements with educators

Thematic analysis of the transcribed focus group session

Overall the data collection methods described above were selected because they were the most suitable for answering the specific research question; data analysis methods were selected considering their suitability to the data as well as the research question.