Proyección de las mejoras propuestas

The evaluation project was commissioned by the British Council, who needed to have an independent assessment of their performance for internal purposes as well as to report to the joint funders of the programme, the GREAT Britain Campaign – the UK Government’s international promotional campaign first launched in 20122 _{(see Section 3.2.3.1 for more details on the}

relationship between the British Council and the project team). The project was run by a team of investigators:

• The principal investigator, who initiated and designed the project and oversaw its overall execution. She was responsible for administering the project funding and ensuring that the service agreement between the project team and the British Council was fulfilled ethically and responsibly. She was actively involved in supervision of all the ongoing work on the project, contributed to the project reports and directly led and conducted much of the research work. As will be shown below, some of it was outside the scope of this case study.

• The co-investigatorwith a background in computer science, responsible for oversight of the computational aspects of the involved work.

• The co-investigatorwith a background in political science who oversaw the issues surrounding soft power(Nye, 2004) – an approach to international relations focusing on establishing a positive image of a country among the populations abroad.

Aiming at comprehensiveness, the evaluation project included several research strands each pursuing its own objective. As the work on the three strands was, to a high degree, independent, the strands provide a useful framework to give a brief introduction to the project. I will outline the aims of each strand and briefly introduce their core methods and research teams.

However, before looking at each of the strands in detail, it is worth noting that I was deeply involved in the ongoing research- and reporting work conducted for Strand 1 during the

1_{https://www.facebook.com/LearnEnglish.BritishCouncil/}

3.2. EVALUATION OF THE SHAKESPEARE LIVES CULTURAL PROGRAMME 39

Shakespeare Lives cultural programme itself and shortly after it finished. This work fits the definition of social data science (see 1) most closely due to the use of novel forms of social data (primarily social media data) and the associated challenges. It is the subject of discussion within this case study.

There was much work done for the overall evaluation project as part of the other strands. Moreover, the project investigators carried a lot of work after the social media analysis team was adjourned and the cycles of ongoing reporting were finished. All of those are outside the scope of this case study and are only introduced inasmuch as is required to better understand the work on Strand 1. Therefore, this case study should not be considered a full account of the Shakespeare Lives evaluation project, but rather a rich and comprehensive account of one of its particular aspects3.

3.2.1.1 Strand 1. Qualitative analysis of social media interactions across different languages

Strand 1 of the evaluation project looked for evidence of the programme’s influence on public perceptions of the British Council, Shakespeare and, ultimately, the UK on social media. For each of the five studied languages of social media interaction – English, Spanish, Arabic, Russian, Mandarin – and for interactions with the visual content on Instagram, the strand team investigated whether the social media users recognised the values that the British Council promotes (mutuality, diversity, creativity, innovation and welcoming nature). The team also provided an in-depth assessment of the public’s emotional response to the Shakespeare Lives programme and examined whether the social media users found the programme’s content enjoyable, useful and of high quality.

The key research method that the strand’s team employed was content analysis actualised primarily through human coding of social media data and qualitative interpretation of the findings derived from the coding exercise. On each of the three rounds of reporting, a researcher coded a sample of social media entries from Twitter, Sina Weibo4 _{or Instagram via a coding}

framework. The framework had been developed before the start of the project by the project investigators and was subsequently shaped by the strand team at the start of the project. In addition, the researchers took ethnographic approach to studying two social media platforms – Facebook and VK.com5_{. They were observing the public- and group pages relevant to the}

3_{More on the overall evaluation project:http://www.open.ac.uk/researchprojects/diasporas/cvp/} shakespeare-lives-2016

4_{Sina Weibo is a Chinese microblogging platform that resembles Twitter and Facebook.http://weibo.com/} login.php

Shakespeare Lives programme and the discussions that were emerging there over time straight through the platforms’ web interfaces.

Given the high number of its research outputs, Strand 1 required a significant investment of labour. Indeed, for each round of reporting, five reports for data in each studied language and one report for the visual content analysis were produced, and a number of overarching final reports were prepared later. Data in each language was analysed by a single researcher (hereinafter referred to aslanguage researchers) with a separate researcher being responsible for the visual content analysis.

The work of the six researchers was coordinated by a project manager. The project manager was not one of the project investigators, but rather another team member who provided their services to the project on a consultancy basis. Within the course of the evaluation project, the role was occupied by two different researchers of seniority similar to that of the regular researchers. The project managers oversaw the data collection from the third-party social media monitoring platforms (see Section 3.2.6.1) and played the key role in development of data acquisition criteria. In this capacity, they were a point of liaison with the British Council concerning some of the operational research decisions. At the end of the project, the project manager was responsible for meta-analysis of the findings derived across different languages.

The eighth and final member of the strand team was thetechnical coordinator. I took this role on the team. Within this capacity, I liaised with the analysts (i.e. the language researchers and the visual content analyst) on quantitative analysis and interpretation of data. I was responsible for supplying the analysts with data visualisations and for facilitating large-scale data collection when this needed to go beyond the social media monitoring platforms. The work of the strand team was supervised the project’s investigators, who played the lead role in developing the methodology.

As mentioned above, due to the nature of Strand 1 and my personal involvement in it, the vast majority of the discussion on the Shakespeare Lives case study will be dedicated to the work on this particular strand.

3.2.1.2 Strand 2. Cultural Value of Shakespeare Lives: learning, monitoring and evaluation

The second strand of the research project aimed to provide an overarching evaluation of the Shakespeare Lives programme using the Cultural Value Framework previously co-developed by the Open University, the BBC World Services and the British Council (Gillespie et al., 2014).

3.2. EVALUATION OF THE SHAKESPEARE LIVES CULTURAL PROGRAMME 41

While the framework was adapted to the project needs, it retained its essence of assessing a soft power endeavour from the perspective of four groups of stakeholders: funders of the endeavour, collaborators, delivery teams within the organisation and the audience.

This research strand collated data collected with a wide variety of research methods and from different sources. The strand heavily relied on secondary data. For example, the “audience” component was predominantly informed by the work on Strand 1. Some of the data came from the British Council’s internal reports and evaluations. However, this strand also generated new primary data – for example, through surveying British Council employees. The bulk of the research for this strand was done by the principal investigator and a senior consultant who had both previously led the development of the Cultural Value Framework.

3.2.1.3 Strand 3. Visualising Shakespeare Lives: interactive overview of the programme

The last strand of the project was concerned with creating visualisations to provide a high-level overview of the social media data that the Shakespeare Lives programme had generated. These visualisations were mostly aimed to enhance the presentation of the project deliverables and to support the qualitative findings of Strand 1 with relevant figures derived from larger-scale quantitative analysis. While playing an important role of illustrative support for the findings of the other project strands, these visualisations did not generate major new findings by themselves. The work on the visualisations was done by a contracted third-party commercial organisation whose co-founder closely liaised with the project’s research team. Additionally, sentiment analysis – i.e. automatic detection of the strength of tone underlying a piece of text (low to high) and of its direction (i.e. positive or negative) as expressed in a piece of text – of a large corpus of tweets containing the programme’s #ShakespeareLiveshashtag was carried out by one of the project investigators who used a readily available sentiment measurement algorithm, SentiStrength (Thelwall, 2017). Alongside the project’s technical coordinator (myself), the investigator also lent support to preparation of the visualisations at the later stage of project. Example visualisations included maps and timeline diagrams depicting the spatial and temporal distributions of the relevant tweets and of their sentiment, an info-graphic demonstrating the most actively tweeting users and the most commonly mentioned Twitter accounts and an interactive calendar of the programme.