ANALISIS DE LA HIPOTESIS

The Shakespeare Lives programme evaluation project can serve as an excellent example of the importance to understand who the stakeholders are and to ensure that their goals are aligned with those of the team. I will show how different parameters such as a project setup and the differences in backgrounds of the project team’s members and of the stakeholders may potentially make this non-trivial. I will also show how our team mitigated these obstacles.

3.2.3.1 Aligning stakeholder interests and team background

The first aspect to look at is the relationship between the British Council and the studied evaluation project. As is quite often the case in projects such as this one, the key stakeholder – the British Council – was simultaneously the subject of the independent evaluation carried by our project team and the body that had commissioned the evaluation. While our team was deriving the findings that (among other purposes) were meant to be used in reporting on the British Council’s achievements to the organisation’s funding partners, those further reports were the British Council’s responsibility to prepare, and our team had no relationship (contractual or otherwise) with those funding bodies.

By the baseline guiding principles of independent evaluation, the British Council had no power over the findings that our team produced and could not reject our outputs if they satisfied the agreed level of rigour and quality. Being a publicly funded organisation with a reputation to maintain, the British Council was interested in outputs that complied with these principles. This was supported by our team for whom it was crucial to be compliant with good academic practice. The vast majority of the team members were either academics or doctoral students affiliated with various universities in the UK and abroad. Therefore, academic rigour was not only a matter of principle – it was also a matter of reputation and of the ability to subsequently use the findings to publish academically.

On the other hand, the British Council had had a long history of prior collaboration with the project investigators (e.g. during the development of the aforementioned Cultural Value Model); therefore, if the research outputs presented by our team had caused a conflict between the team

3.2. EVALUATION OF THE SHAKESPEARE LIVES CULTURAL PROGRAMME 45

and the British Council, it could have jeopardised a long-established partnership. Additionally, the British Council could havein principleused some instruments of resistance such as coming up with formal reasons to not accept the work – even if, based on past experience, the team did not expect the British Council to do so. Indeed, the issue of pressure from funding bodies (including the well-minded ones) is not unknown in research. For example, Smith (2010) shows that UK researchers in health inequality who are funded to contribute to health policies feel bound to produce ideas compatible with existing policies. The following paragraphs will show how our team managed to maintain the integrity of the research within the project’s setting – primarily through the efforts of the principal investigator.

The primary challenge faced by the principal investigator was in making sure that the research outputs were appropriatelyrecipient-designed. Recipient design is a term coined by Sacks et al. (1974) for the process of formulating the message in a way that is sensitive to its intended audience. It is a recommended practice in qualitative social research as it helps to “avoid losing the audience” (Silverman, 2017). The purpose of recipient design is not to change the essence of the message, but to make sure thatthe way the message is conveyedfits the recipients’ needs. For example, our team had to make sure that the findings were appropriately contextualised and that the overall tone of their discussion and presentation was tailored to the non-academic audience of the British Council representatives who might have been accustomed to less critical tone than the one often employed in academia.

As a result, the principal investigator often secured meetings with the British Council to demonstrate preliminary findings and to get their comments and suggestions as to what kinds of evidence our team might have overlooked in the research or what kind of questions could be answered in more detail. The meetings with the British Council were held in a close circle of people: the investigators and the project manager of Strand 1 from the project team side and the Shakespeare Lives programme manager from the British Council side. While this was not necessarily done specifically to reduce risks of confrontation between the British Council and the team, it achieved this as a side effect. The continuous feedback from the British Council allowed the project to stay relevant to its interests throughout. It was specifically helpful that the meetings with the British Council were consistently held in short advance of the project team meetings. Therefore, the principal investigator could communicate the British Council’s relevant feedback to the team members in person and let the team thoroughly discuss it.

The principal investigator always maintained her strict academic integrity and was respectful to the aspirations of the project team. She never pushed to mask the reported findings – even the negative ones. Rather than that, she motivated the project team to ensure that those findings were appropriately framed, so if there were objective obstructions to more successful performance,

those would have to be discussed. Likewise, she made our team ensure that no evidence that viewed Shakespeare Lives in a positive light was missed. During the project meetings, she metaphorically encouraged the team members to “put two hats on”, referring to the academic and the consultancy roles respectively. Given that the analysis mostly answered quite open questions with a support of predominantly qualitative research techniques, effectively the principal investigator achieved to motivate the team to dig deeper and come up with inventive yet appropriate research angles. As will be shown further when discussing the research methodology, this only strengthened the quality of the research (e.g. see Section 3.2.5.1).

3.2.3.2 Interpreting stakeholder needs

The long-term relationships between the project team investigators and the British Council were rooted in the investigators’ interest and deep expertise in studying soft power organisations, i.e. the organisations that strive to establish a positive public image of a particular country among the populations abroad (Nye, 2004). Soft power organisations often strive to influence their audiences in indirect and tacit ways that ask for corresponding approaches to evaluation of their effectiveness. The British Council is a soft power organisation and the investigators’ ability to theorise them as such was of extreme value for the Shakespeare Lives evaluation project. For example, it lay at the core of development of the evaluation framework for Shakespeare Lives, especially in the context of Strand 2. Therefore, both our team who assessed Shakespeare Lives as a soft power initiative and the British Council who acted as one attended to theirproper responsibilities.

That said, in addition to having responsibilities (and thus interests) as a soft power organisation, the British Council had responsibilities asjustan organisation. For example, the British Council had to report back to its funding bodies. The soft power nature of their activities sometimes made it difficult to pinpoint what their specific contributions were in such external reports. This occasionally led to potential difficulties of implementing some of the recommendations provided by our team. For example, during one of the data collection sessions designed to observe the team members at work, the project manager said the following about the tweets containing the British Council-initiated#ShakespeareLiveshashtag:

“There was stuff that wasn’t necessarily directly related to the programme, but still used the hashtag; [the British Council would] be happy to see that, because for them that’s still evidence of impact. All they want to see is that their programme shapes anything to do with Britain in any shape or form”.

3.2. EVALUATION OF THE SHAKESPEARE LIVES CULTURAL PROGRAMME 47

cultural organisations, as well as the local events and activities that were done with very active contributions of the local partners. Further development of such activities was also one of the persistent recommendations across multiple project reports. While the British Council did not reject this recommendation, the organisation’s feedback on our reports suggested that implementing this recommendation to the suggested degree could potentially create difficulties for external reporting. This example shows the importance of understanding the complexity and heterogeneity of interests of a single project stakeholder when reporting to them and providing them with recommendations.

Another point to make in regard to interpretation of the stakeholder needs is that the stakeholders themselves are heterogeneous. In the case of the studied project, while our points of contact were within the British Council headquarters in the UK, the list of the real beneficiaries of the evaluation project also included the local British Council teams across the world. There was one example of not accounting for the particular interests of such a local team (namely, the Russian team) in the initial stages of the project design, which was repaired in the early stages of conducting the research. This example directly links to the subject of data selection and thus is discussed elsewhere (see Section 3.2.4.1).

3.2.3.3 Expecting stakeholder engagement

Finally, to stress the importance of continuous engagement with the stakeholders, it is worth mentioning that lack of it once was a reason for miscommunication within the project team. The original project proposal promised a visualisation representing a “macro view on Shakespeare Lives in the format of a digital calendar” that should have been prepared by a third party contractor for Strand 3. There were no more stated requirements for this digital calendar, so the contractor expected collaboration with the British Council to elicit and refine those. However, the British Council focused its attention on the interim reporting done for Strand 1 and on the on-going work done for Strand 2, leaving Strand 3 out of the loop. This was most arguably due to the fact that Strands 1 and 2 actually produced output on a continuous basis, and the British Council’s representatives felt more comfortable to engage in providing continuous feedback on interim results rather than actually participating in planning activities.

Simultaneously, some of the other work done for Strand 3 – specifically, the sentiment analysis performed by one of the project investigators – could also benefit from longitude visualisations, but in a form of tweet timeline diagrams rather than a digital calendar. The the focus of effort of the third party contractor naturally morphed into preparation of those visualisations with an implicit assumption that the timeline diagrams had replaced the calendar. It was quite a surprise for the contractor when, already after the internal deadline for delivering a working version of the

project website, they were contacted by the project principal investigator for the digital calendar. As it appeared, the plan to prepare the calendar always stood intact. The contractor did manage to deliver the calendar, but at high personal costs and on a hectic schedule.