University
of Cape
Town
University
of Cape
Town 8. EVALUATION:
Sustainability, Success and Impact
ABSTRACT
Sustainability, Scalability and Impact are some of the key words when assessing the success of an ICT4D intervention. While there is no one way of measuring these metrics, there have been a number of frameworks put forward that measure the impact of the intervention against the process through which it was built. In this chapter we present two pieces of data on overall evaluations conducted at eight months and on Ummeli’s second ‘birthday’, in July 2012. Further, we give insights on some of the lessons learnt from building Ummeli first as an academic project, and then turning it into a social venture. Finally, we give a roadmap for Ummeli in the future.
What results in an ICT4D project being regarded as successful? More specifically, how does one determine that this project was successful? In scanning ICT4D literature on ways to evaluate ICT4D projects, one theme is clear: the success of an ICT4D project is mainly determined by the process through which it was developed, as well as by measures that have been put in place to ensure that it is sustainable and able to scale. It is widely believed that the process of planning, designing, developing and deployment of an ICT4D (and any other development project, for that matter) is the probable determinant of its success. It has been argued that the processes by which projects are introduced or deployed into a
University
of Cape
Town
developing community determine their appropriateness, usefulness, and adoption by the community.
A number of frameworks and approaches have been proposed to ensure the successful implementation of development projects. In Chapter 1 we mentioned the sustainable livelihoods approach (SLA) proposed by Chambers and Cornwell of IDS and the International IISD, later adopted by DFID, which provides a useful checklist for the design of monitoring development projects [25], [26], [27]. Another highly acclaimed approach is the Outcome Mapping (OM) approach, which was proposed by the Canadian-‐based International Development Research Centre (IDRC) [223–225]. Outcome mapping provides a framework where the project under development is evaluated for and by the people involved. Outcome mapping does not focus on the quantifiable impact17 of development programs on the community in terms of more schools, or better healthcare; rather it looks at outcomes in a community and their environment as a consequence of the project. Here, outcomes are viewed as changes in the behaviour, relationships, activities, or actions of the people, groups, and organisations with whom a project works directly. These outcomes can be logically linked to a project, without necessarily directly causing them. The changes are aimed at contributing to specific aspects of human and ecological well-‐being by providing the community with new tools, techniques, and resources to contribute to the development process [224–227].
The one thing common to both these approaches is their emphasis on beneficiary participation as a key to the success of any development project. In setting the theoretical framework of this work we introduced participatory development, an approach that has been highly acclaimed as resulting in successful implementation of development projects [17], [228], [229]. By looking at all the other approaches, one could argue that they are all indeed a form of participatory development approaches, whose difference is only found in their emphasis. Other than the involvement of the project’s beneficiaries, the frameworks encourage developing projects within the context of a lived-‐in experience; the process should be flexible enough to cater for the needs of the community, and should ensure
17 defined as changes in state
University
of Cape
Town
support as well as a strong sense of leadership from both the interventionist and the community.
Due to the lack of a definitive evaluation technique for ICT4D, these frameworks have been repurposed in the design and deployment of technology-‐based interventions. For instance, Walton and Heeks [230] proposed what they termed a Process-‐Driven Approach to evaluating ICT4D. Their framework, which is primarily borrowed from the SLA framework, provides the ICT4D interventionist with a checklist through which to approach the design and implementation of an ICT4D project. They propose five elements that would ensure that an ICT4D approach leads to a successful intervention: beneficiary participation;
flexible, phased implementation; learning from experience; institutional support; and programme management. Similarly, Heeks [231], [232] proposed the compendium approaches to impact assessment, which is comparable to outcome mapping. This looks at the lifecycle of an ICT4D initiative and how the impact changes over time, with various measurements at each change, where ultimately, impacts in terms of Outcome, Output and Development are measured in the same way as in Outcome Mapping. An important point to note is how similar the development approaches are to EAR.
Hence, a more appropriate means of assessing the success of Ummeli is by analysing the process that brought it about. In the following section we look at EAR and UCD in the context of Process Driven approach as well as the ICTD compendium approaches, in order to address our first research sub-‐question (evaluation of methods).
As indicated and demonstrated throughout this dissertation, the process of designing Ummeli lies at the nexus of EAR and UCD (Figure 44).
University
of Cape
Town
Figure 40: Designing Ummeli as a process of EAR and UCD
Therefore, the best way to evaluate Ummeli is by first evaluating the process that brought it into reality. In both her articles, Hayes ([102], [218]) notes that AR and HCI – and by extension, EAR and UCD – have a lot of similarities. They are participatory, cyclical, and iterative processes that encourage the early and continuous engagement of users. When weighted against each other, their stages of execution correspond (Figure 45).
Figure 41: Weighing EAR phases against UCD stages
This means that they can work well together, which indicates a predisposition to success in the process, due to their similarities.