Primera S: (SEIRI – Clasificación) 117

As mentioned in the Shakespeare Lives case study (see Section 3.2), the research project’s team leader has a long track record of collaborations with cultural- and media organisations. Running evaluation projects like the one our team performed for the British Council is business-as-usual for her. Therefore, it would not be surprising for a reader that soon after completing the work on Shakespeare Lives, she started to assemble the team for a new evaluation project – this time, for InfoMigrants: a multi-platform news and information resource for those seek asylum in the European Union or are in the process of migration to the EU (Gillespie, 2017). InfoMigrants is a collective effort of three international media organisations – France MÃl’dias Monde, Deutsche Welle and ANSA – that delivers its content through a multi-language website and a number of public Facebook pages, Twitter accounts and WhatsApp channels.

The circumstances of the InfoMigrants case mirrored that of Shakespeare Lives in many respects. Not only did the project have similar objectives (such as the Cultural Value assessment, see Section 3.2.1.2) and employed some of the same data sources (e.g. Facebook public pages, see Section 3.2.4.1) and analysis methods (such as content analysis and online ethnography, see Section 3.2.5) – the teams of the two projects also had four people in common (aside from the PI and myself, another investigator and the lead qualitative analyst). The relationships with the key stakeholder (InfoMigrants being simultaneously a funder and a subject of the evaluation, see Section 3.2.3) were the same. Due to this overlap, many observations made during the two project overlap. Such purely congruent observations are not discussed in this section, although some of them are mentioned directly when discussing the contents of the designed project management tool (see Section 4.3). However, the InfoMigrants evaluation case study did bring some qualitatively new insights that came from the“benchmarking excercise”(as our team got

to call this process), which is discussed below.

3.4.1.1 Benchmarking exercise: introduction

As outlined in Section 3.2.5.3, one of the greatest challenges in the Shakespeare Lives evaluation project was lack of benchmarks – i.e. other similarly scoped international cultural programmes – to compare the programme’s performance with. Therefore, when assessing the level of the programme’s success, the project team had to construct baseline expectations rather than infer them from the data. While the team succeed in doing that, additional assumptions had to be made and the analysis required careful comparison of performances of the programme’s own events across different countries and events, which was often non-trivial. Overall, the lack of benchmarks was something that arguably everyone on the team would have preferred to avoid if there had been a chance.

The issue of benchmarking arose again in the InfoMigrants evaluation project, but this time there was more that we could do. To begin with, the representatives of InfoMigrants provided us (as well as the funding organisations) with target values for several key quantitative indicators of online success such as the number of website visits and reach on social media platforms (cf. Stephen et al., 2015). For most modes of quantitative analysis, our team used exactly those figures as the benchmarks. However, in an attempt to further increase the rigour of our research, we decided to complement our core analysis with some form of external comparison of the InfoMigrants’ performance – even though, given the modest resources of the project, that had to be done on a limited scale. Conducting this piece of research raised several interesting issues, especially in relation to research design and data acquisition.

3.4.1.2 Designing the exercise

From the research design perspective (cf. De Vaus, 2001), the benchmarking exercise required our team to obtaincomparable data on online performance ofcomparable entities, i.e. other initiative groups, organisations and campaigns that provided information support to asylum seekers and migrants from economically disadvantaged territories and that operated in the same languages as InfoMigrants. As some of the analysts of the team had prior knowledge of the field, identifying a long list of candidate peers for InfoMigrants was relatively straight-forward. However, upon closer examination, all of them differed from InfoMigrants in one or more fundamental ways:

• Many peers targeted narrower groups of migrants and asylum seekers (e.g. those in refugee camps specifically in Greece);

3.4. ADDITIONAL FIELDWORK EVIDENCE 105

• Many peers offered information to migrants as part (and often as a by-product) of other services (e.g. help in getting asylum);

• Some peers did not only have offline presence in addition to the online one (which InfoMigrants did not), but specialised on offline activity (e.g. on volunteering in the refugee camps);

• Some peers, who, like InfoMigrants, distributed information in several languages, did not keep their presence in different languages separate (e.g. InfoMigrants had a different Facebook page for each language, while others would post in multiple languages).

This dimensions of variability could influence the differences in online performance of InfoMigrants and its peers. For example, one may expect a long, sustained engagement (and thus repeated interactions) with an organisation that offers direct help to specific migrants and refugees that would come from those who receive the help. This is quite different compared to the sporadic engagement often observed in the media sector. Our team dealt with these discrepancies in two ways. First, using formal (a non-zero intersection in the proposed mission of a candidate benchmark peer and that of InfoMigrants) and informal (expert judgement of the team members) criteria, we shortlisted a fraction of InfoMigrants’ peers for whom the online performance data were to be analysed. Second, for each short-listed organisation we provided our stakeholders with a summary that highlighted their differences with InfoMigrants and included appropriate caveats while reporting on our analysis, thus accounting for potential sources of performance discrepancies.

3.4.1.3 Data availability

Our proximity to InfoMigrants provided us with elevated access to their online performance data – some of those data could be obtained from the administrators of their website and of their social media profiles, while others – from their marketing team who used commercial analytical software focused on internal evaluations. While these data were of great help for the bulk of our team’s research, most of those data were of little value for the benchmarking exercise since comparable data on the InfoMigrants’ peers could not be obtained with this tool set. Our team could only rely on publicly available benchmark data. For example, this implied discarding such a crucial measure of InfoMigrant’s success as website visits, since only crude estimates of that metric such as an approximate Alexa Ranking17 _{without elevated access to some proprietary}

data.

After much consideration, it was decided to focus the benchmarking exercise specifically on the Facebook engagement. With the exception of the website itself, the Facebook presence was the one into which InfoMigrants invested by far the most efforts and resources. Moreover, specifically with Facebook, our team was confident in our ability to estimate one of the key success metrics from publicly available data: engagement (cf. Stephen et al., 2015) of individual posts on the Facebook public pages – i.e. a number of interactions Facebook users make with a post. We could operationalise this metric as a summation of reactions, comments and shares that a post had received – something observable through Facebook’s user interface and accessible programmatically through the Facebook’s Graph API – the publicly available programming interface for reading and writing Facebook Data18_.

It is worth mentioning that our initial plan was to only use the publicly available routes to access the benchmark pages’ data, since a more convenient (and allegedly at least as complete) access to the post engagement data for InfoMigrants was available with Facebook Insights – a set of monitoring tools that Facebook provided to administrators of its public pages (cf. Spiliopoulou et al., 2014). However, the Insights data appeared to be inconsistent with those available publicly, with the interaction metrics received through Facebook Insights being systematically higher. To the best of our knowledge, Facebook had never documented the definitive source of this disparity. However, our sporadic observations suggested that the engagement metrics obtainable through Facebook Insights were strictly non-decreasing and thus arguably represented all interactions a post had ever received, while the publicly available ones represented only the currently active post engagement – i.e. these metrics would decrease if someone, say, removed their like or a comment. Thus, for a fair comparison, it was instrumental to treat the InfoMigrants’ own public pages the same way as the benchmark sample and to acquire the respective data using the same methods.

3.4.1.4 Data acquisition

Given the set-up of the benchmarking exercise, it was a straightforward decision to develop a web application that would connect to the Facebook Graph API to acquire the data. Compared to scraping, it was more reliable and returned data in a conveniently structured JSON format that did not require much post-processing. The practice of using the Graph API for research had also been well-established over the years of the API’s existence. There were only two practical limitations. First, the same data were available to a developer through the API and through the user interface – so, for example, no data from users who had hidden their profiles from non-friends could be accessed. Second, judging from my prior experience, after a number of

3.4. ADDITIONAL FIELDWORK EVIDENCE 107

consecutive requests the Graph API tended to start delaying its response. Therefore, the time required to obtain a dataset grew faster than the dataset’s size19. Both limitations were acceptable for the benchmarking purposes, since only a limited number oforganisational public pageswere studied.

Our data collection application successfully acquired data through the Graph API for the first round of benchmarking exercise in late 2017. However, in spring 2018, when the application was used again, the Graph API only returned error messages instead of data. As it turned out, on March 21, 2018 Facebook put access to the platform’s data via the API on hold, unless the requested data belonged to a Facebook user who had used the data collecting web application in the past three months (Xu, 2018; Facebook, 2018a). This condition arose from the “default” scenario that Facebook implied for the use of the Graph API: i.e. developing applications that allowed third party web-services to leverage their user interactions by connecting to a user’s Facebook profile. Examples of such enhancements might include providing a “sign in with Facebook” functionality or providing a customer with promotional deals in exchange of posting something on their Facebook timeline. All such interactions require explicit interactions between a Facebook user and the third-party web-app. Thus, the Facebook’s decision effectively rendered research-purposed data collection through the Graph API temporarily impossible20 _{– unless}

the research was done privately by an entity that had active app users. Facebook’s decision was motivated by the then-recent scandal around the Cambridge Analytica data leakage and seemed quite surprising to our team. Indeed, even if restricting API access to the individual users’ data could be considered consequential to the scandal, this was at least questionable for the organisational public pages’ data, especially given that such pages often explicitly sought wider exposure and access to their content.

While this posed difficulties for our research it also taught us an important lesson about data acquisition: one should be not operate under an implicit assumption that a publicly available method of data collection would remain accessible indefinitely in the future. Relaxing this assumption means approaching data acquisition strategically, planning the data needs ahead and trying to collect data as soon as they get available rather than when the research actually calls for them.

Overall, the InfoMigrants evaluation project improved our team’s understanding of the issues related to designing social data science studies, dealing with inconsistencies of different

19_{To the best of my knowledge, this was never officially documented or tested.}

20_{In May 2018 Facebook announced launch of a new version of the Graph API that required web application to}

through a review process in order to access Facebook data; it is worth noting that no specific procedure for reviewing academic apps was provided (Papamiltiadis, 2018). Later Krishna (2018) reported that Facebook had put additional restrictions on data access again.

representations of the same data and planning data acquisition routines, all of which is valuable for designing the project management tool for social data science.