This study presented a number of methodological challenges. The expansive focus (e.g. in terms of the amount of content and the detailed categories to be analysed) raised concerns about its cross-national comparability and its ability to record and understand the rhetorical style, the narratives and counter-narratives of multiculturalism and ethnic relations in the UK and the framing and agenda setting functions of the press. These issues are discussed in detail in the Final Technical Report, while in the last section of this report we highlight a number of recommendations for the future development of this study.
Some of the methodological issues and concerns raised early on – especially in terms of the huge amount of data needed to be analysed – were negotiated and dealt with in close communication with the FRA. As regards the selection of newspapers selected for the UK case (The Guardian, The Daily Telegraph, The Sun and The Daily Mirror), we feel that this was the most appropriate choice. They represent a good sample of partisanship diversity (two left and two right-wing), and type of publication (two broadsheets, two tabloids). However, the original plan to code all the editions from a week’s sample (Monday to Saturday) was found to be a task unmanageable according to the current budget and deadlines. One of the reasons for this is the length and density of the British press – big editions, including several supplements and different news sections. According to UK team estimates, the time needed to read the newspaper, number it, photocopy relevant articles and code them according to the database was a task unmanageable if all six editions of the week were to be included. In order to manage the task, the UK team negotiated with the FRA to reduce the sample to three editions per week, rotating the days selected per phase. In Phases 1 and 3 the Monday, Wednesday and Friday editions were coded, whereas in Phases 2 and 4 the Tuesday, Thursday and Saturday ones were coded. It should be noted that Saturday’s editions are particular long, with almost twice the number of articles, supplements and content. The extensive amount of newsprint to be scanned and the amount of data to be coded has meant that this project has had to be subsidised by the participating university departments.
The UK team excluded the data collected during Phase 1 on several grounds. Firstly, because the criteria for the selection of relevant articles were modified: after Phase 1m coders were required to read ALL newspaper articles, irrespective of whether the headline or the first paragraph indicated relevant minority content. This inevitably resulted in an increased sample of relevant minority content articles. Secondly, the coding scheme was significantly changed for the following variables. Thirdly, because the criteria to code minority issues was modified minority issues coded on Phase 1 are considerably outnumbered as well as miss represented.
Throughout the progress of this project the collaborators in the different pilot countries have developed a shared comparative methodological tool. Through dialogue with each other and with the FRA they have honed a coding schedule that has become a common analytic instrument .However, it is apparent that this efficacy of this instrument is dependent upon the shared acquired habitus of the national teams as in working together they build a common repertoire of understanding that enables them to employ the coding schedule that has been developed internationally. This is both necessary and inevitable given the complexity of the coding schedule and the need to bring a national perspective to bear in the nuanced interpretation of the data. Consequently we must acknowledge the entry of shared subjective routines into the generation of the data. We have no way of knowing how compatible, or divergent, these distinctive habituses would be across the teams operating in this project.
This is in fact a generic problem in developing a workable comparative framework that is committed to a micro analysis of news print. Some of the pragmatic compromises, of acceptance and rejection that have had to be taken in the dialogue between the FRA and specific teams in responding to the teams’ concerns have illustrated this tension. We can note, for example, that our team’s concern regarding the potentially ‘Eurocentric’ nature of the coding of geographic areas illustrates how the development of agreed categories may have unintended consequences. Similarly we can note that the implications of mixing ‘domestic’ and ‘international’ context have remained unresolved and will leave a residue of questions about what is being subsumed within the agreed coding routine. Or, again , we may note that not choosing to include a category that explicitly tapped issues that related to the ‘suppression of freedom of expression and speech’ has left this issue irredeemably lost within a much wider coding category. These painful pragmatics of generating a manageable coding schedule may be intrinsic to the chosen methodology: but the ambiguities they consequently build into the available categories need to be held in mind as the data is
subsequently interpreted. This means that the analysis and interpretation must be carried out with the intimate involvement of those team members most closely associated with the actual content coding. This would have direct implications for the guidance given by the FRA in terms of team building and team practice for future contracts for a wider comparative study. We strongly recommend that teams, including experts and coders have more direct involvement in the development of the methodology and the study throughout its various stages.