Los Mecanismos de Desarrollo Limpio (bonos de carbono):

We recruited participants through our individual research contacts in different institutions. They were mailed individually or asked in-person for their willingness to participate in the study. Part of the evaluation was carried out in-person while visiting the Centre for the Study of Manuscript Cultures (CSMC) at the University of Hamburg to present unrelated work. Others were scheduled through video conferencing (Skype) and asked to access the framework through a remote desktop software (TeamViewer). The evaluation results were recorded using an online questionnaire-based survey (Qualtrics).

Participants were initially given a brief overview of the study and asked to provide non-personal details such as their position, institution and other details to assess their suitability for this study. They were then presented with a manual of the prototype of the framework, which described the functionalities of individual modules and ways to interact with the framework for character analysis. This was typically followed or preceded by a live demonstration of various modules of the framework and their features. They were then asked to use the prototype and experiment with it briefly to familiarize themselves with its interface and operations.

After they felt comfortable with the system, we asked the participants to perform three different tasks on their own. However, they were al- lowed to ask for assistance and guidance if required. The first task was to perform an end-to-end character analysis through the framework, from importing images to extracting features, for two different charac- ters. To maintain uniformity, we preselected a character set from which participants could choose any two. For the second task, we skipped the importing of image and asked them to select any five distinct characters from the script repository (of the Indic script development dataset) as we wanted them to interact more with other modules. They were suggested to experiment with different kinds of analysis by selecting different tra- jectories. For the first and second tasks, they were also asked to verify the automatic results, and encouraged to interact and override the system whenever they thought it was appropriate (i.e. when they thought the

results were incorrect and needed to be updated). For the final task, they were asked to select any 10 characters from the Indic development repository and validate the trajectory and stroke segmentation results of the characters (see §4.1).

Once participants’ predefined tasks were completed, they were re- quested to fill the questionnaire regarding the usability, usefulness, user interaction and module-level evaluation of the framework along with the validity of the sampled dataset. To evaluate usability and usefulness, we used standard frameworks that were applicable to our current study (see §5.3.1 & §5.3.2 for more details). We asked them to rate the usefulness specifically in the context of identifying/analyzing characters. For most of the questions, the extent of their agreement to statements was recorded

in a five-point Likert scale as follows : Strongly Disagree, Disagree, Neither

Agree nor Disagree, Agree, Strongly Agree. The questionnaire also allowed them to give extensive textual feedback.

After evaluating the framework, participants were presented with the descriptions of the various metrics proposed. The descriptions were simplified, avoiding jargon as much as possible, taking into consideration that the audience was mostly non-technical from the perspective of com- puter science. After presenting the descriptions, they were asked to rate the metrics in terms of relevancy, usefulness, potential for future/frequent use. Similar to the above, they rated the metrics on a five-point scale with respect to each of these factors. They were then asked to rank the metrics and give specific feedback if any.

Finally, to evaluate the results of our case study we first presented participants with various charts that showed the development of Indic scripts (as seen in figures 4.2, 4.3, 4.4 and 4.5) and briefed them regarding the source and assumptions of the dataset. They were then presented with a subset of results that were obtained by analyzing the visual metrics of the dataset. The extent of their agreement with the results was recorded using a five-point Likert scale as described earlier. We did not perform a complete evaluation of the analyses in our case study as they were too detailed and would require a substantial amount of time to evaluate and check their correctness. Also, for many methods, if we are to accept the underlying data as correct, the results of the methods can be considered

to be valid. Hence, we chose a subset that was easier to validate from the point of view of the participants.

The whole evaluation exercise took about two to three hours to com- plete. If participants required additional time to evaluate the metrics and results, they were asked to fill in those parts of the questionnaire at a later time convenient to them. The appendices A and B consist of the university ethics approval for the evaluation and the complete question- naire used by the participants. It is to be noted that the questionnaire was also translated to the Tamil language for participants who were not sufficiently proficient in English.