This section describes work related to ICT4D evaluation from the perspective of development studies. Development studies comprise a wide array of organisational entities ranging from NGOs to research institutes and global entities like the United Nations. M&E approaches are defined by abstracting common attributes from multiple ICT4D projects, just as for the computer science and information systems approaches described above. M&E approaches offer either explicit (bridges.org, 2004a; Oyedemi, 2004) or implicit (Bachelor & O'Farrell, 2003; Michiels & van Crowder, 2001) frameworks for combining technical and
social inter-relationships in order to measure impact. Unlike the approaches described in Section 3.2.1, these holistic approaches are higher level and consequently do not delve into performance and implementation details of underlying ICT4D systems. Instead, M&E approaches examine ICT4D interventions within broader contextual issues. M&E tools can be used either up front for requirements gathering and/or after-the-fact for post-mortem project examination. M&E tools can also be used continuously in a long-term iterative fashion throughout a project's life cycle.
Much work related to ICT4D evaluation fits within the Real Access/Real Impact (RA/RI) criteria offered by bridges.org listed in Table 3-2 (bridges.org, 2004a). RA/RI criteria did not emerge from formal academic research literature. Bridges.org is a non-governmental organisation (NGO) engaged in the ICT4D space. Thus, we use other related research literature to backup bridges.org's findings. RA/RI criteria will be appropriated into the innovative socially aware framework discussed in later chapters. Chapter 8 will argue, based on empirical evidence and reflection from Chapters 6 and 7, for the appropriation of RA/RI criteria by pulling on the related technical work from this chapter, especially the lessons learnt described below in Section 3.3. This section uses a series of tables and figures to introduce RA/RI criteria and other related frameworks. The text accompanying and between the tables and figures describe the frameworks in brief and demonstrate how other frameworks' components fit to the RA/RI criteria.
The bridges.org RA/RI criteria construe an explicit framework that emerged from a wide survey of ICT4D projects from around the world (bridges.org, 2004a). The bridges.org NGO originated in America led by Teresa Peters. They relocated to South Africa from 2000- 2006 and then moved to Uganda. Their website has an extensive collection of reports on their methods and the projects to which they applied the methods. RA/RI criteria-oriented reports tend to be post-mortem examinations that employ RA/RI criteria as a lens to evaluate social impact and the extent to which target communities adopt ICT4D interventions. Reports on significant African ICT4D projects include analyses of SATELLIFE (bridges.org, 2003a), Tsilitwa telehealth (bridges.org, 2004c) and Cell-life (bridges.org, 2005c).
Physical access
Is technology available and accessible to people and organizations? Appropriateness
Is the technology appropriate to the local needs and conditions of the community, and how people need and want to put technology to use?
Affordability
Are the technologies and services affordable for local people to access and use? Human capacity and training
Do people have the training and skills necessary to use technology effectively? Do they understand how to use technology and its potential uses?
Locally relevant content
Are there locally relevant content and services available to people through the technologies, especially in terms of language?
Integration to daily life
Is technology use an additional burden to the lives of people already burdened by daily tasks, or is it integrated into their daily routines?
Socio-cultural factors
Are people limited in their use of technology because of their gender, race, age, or other socio- cultural factors?
Trust in technology
Do people have confidence in technology use, and do they understand the implications of the technology they use? Do people understand issues like privacy, security, or cyber crime? Local economics
Will the local economic environment sustain long-term technology use? Macroeconomics
Does the macroeconomic environment in the country or region, such as issues like deregulation, investment, or labour issues affect technology use?
Legal and regulatory
Laws and regulations affect the use of technology? Public support and political will
Do people support the widespread use of technology in their communities? Do they understand the links between their local technology use and the government's overall political strategies to promote ICT for development? Does the government have the political will to drive needed change?
Table 3-2 RA/RI criteria from bridges.org
RA/RI criteria are social concerns related to ICT4D interventions. Bridges.org studied a worldwide array of technology projects to distil these characteristics to help determine the 'real' access and 'real' impact of ICT4D interventions. RA/RI criteria are intended to evaluate ICT4D projects.
Overall, RA/RI criteria provide a holistic lens to view a range of social issues involved with ICT4D interventions. This section does not go into a detailed discussion of RA/RI
criteria and how to use them. Rather, Table 3-2 briefly describes the twelve RA/RI criteria. The components of other M&E frameworks described below easily align with RA/RI criteria. The intention is to establish that RA/RI criteria sufficiently cover a representative set of relevant social ICT4D criteria from a development studies perspective.
Figure 3-2 Universal access wheel
Universal Access Wheel offers a clear visual depiction of how various non-technical components work together to influence ICT4D scenarios (Oyedemi, 2004). If one of the components is faulty, the whole wheel does not turn so well. Oyedemi (2004) also stated that more spokes could be added to the wheel as required.
Like RA/RI criteria, the Universal access wheel (see Figure 3-2) provides another explicit framework that emphasises non-technical aspects of ICT4D. Oyedemi (2004) stated "the challenge for policy makers and national governments in Africa is to develop appropriate policy – policy that enhances the universal diffusion of and access to ICT services by adopting an holistic approach that takes cognisance of the social, cultural and political needs of the community". With respect to the wheel itself, "The universal access wheel does not project totality; rather it provides flexibility and dynamism typical of the information and communication technology sector. Consequently, as other elements and issues arise, they may be added to the wheel" (Oyedemi, 2004). Each Universal access wheel component aligns with one or more RA/RI criteria: policy and reforms fit 'legal and regulatory'; universal service funds fit in 'macroeconomics'; service platform is related to 'accessibility' and 'applicability'; other social utilities mean non-ICT utilities like water and electricity and are also tied to
Service platform Content and language relevance Techno -logy and social relevance Universal access Literacy Other social utilities Universal service funds Policy and reforms
'macroeconomics'; literacy fits 'human capacity and training'; content and language relevance are both clearly 'local content'; technology and social relevance involve 'applicability' and 'socio-cultural issues'. Oyedemi (2004) also noted that the Universal Access wheel could be extended to include more categories, including RA/RI criteria that are not there.
- Community driven - Innovative - User-friendly - Sustainable - Transferrable - Gender/youth-sensitive - Has a training component
Table 3-3 Lessons from the Magic Box
An early survey paper of ICT4D projects synthesised lessons and provided a categorisation scheme (Michiels & von Crowder, 2001). The 'Magic Box' identified issues closely aligned to RA/RI criteria. A follow up paper on the same projects two years later found the issues still relevant (Bachelor & O'Farrell, 2003).
The 'Magic Box' contains the examination of a collection of projects in order to distil general lessons about ICT4D. According to Michiels and von Crowder (2001), "Local appropriation of ICTs is about communities and groups selecting and adopting communication tools according to the different information and communication needs identified by them and then adapting the technologies so that they become rooted in their own social, economic and cultural processes". The 'Magic Box' themes are shown in Table 3-3, and align with RA/RI criteria as follows: community-driven involve 'local economics' and 'integration to daily life'; innovation reflects factors in multiple RA/RI criteria; user- friendliness fit well with 'appropriateness'; transferrable also implies 'appropriateness'; gender/youth-sensitive fits to 'socio-cultural factors' and finally, the training component is clearly 'human capacity and training'.
A follow up paper re-examined the original case studies to gauge how well they were doing (Bachelor & O'Farrell, 2003). Out of 24 case studies, six had no information available, four shared significant problems, and only one of the remaining 14 had evaluation data available, but those 14 still appeared to be ongoing. Bachelor and O'Farrell (2003) found the original 'Magic Box' principles still relevant. They also feel that the only way to share information about successes/failures is via case studies. They also noted that there still are no
proper methodologies for evaluating impact. Therefore, the 'Magic Box' is an example of an implied impact framework.
According to Bachelor and O'Farrell (2003), there are four guiding principles for ICT4D evaluation, all of which clearly dovetail with RA/RI criteria:
1. The need for empirical evidence or analysis of actual experiences of applying ICTs locally and their impact on the lives of the poor.
2. The process and approach for development interventions was key to appropriate development activity, and had in the recent past been overlooked in many ICT- related activities.
3. A focus on the benefits of the new technologies was needed rather than the quantity of technologies available.
4. Access to ICTs must be accompanied by the strategic creation of content for it to be locally appropriated and affect development.