CAPÍTULO II: MARCO METODOLÓGICO
2.5. Técnicas de Procesamiento y Análisis de Datos
and their Associated Problems
Problems of Measurement at the Sectoral Level: Examples of Indicators and their Associated Problems
The intention of this section is not to provide an exhaustive overview of what needs to be measured at the sectoral level. We have already explained that the detail of
performance indicators has to be developed in conjunction with in-country
representatives. Instead, the purpose is simply to draw attention to some problems of definition and data where the performance indicators are to be based on data from the entire system.8
Enrolment Ratios
The problem here is 'simple': the quality of the data. There are several aspects to both the numerator and denominator:
• what is actually meant by enrolment: registration of child (for whatever reason), appearance in class at beginning of school year (if that can be
identified), regular (rather than sporadic) attendance, inscription for (or sitting) annual examination;
• the relevant population: the 'decennial' censuses in many developing countries are unreliable for a variety of reasons so that the estimated size of the relevant age group has to be treated with extreme caution (Murray, 1988).
Attendance
No one suggests that this will be easy to monitor in developing countries. Yet, after enrolment, it is the next most important statistic because, without that data, we cannot sensibly assess what the enrolment figures mean in terms of childhood exposure to school. It might give some perverse comfort to know that this has also been a problem in developed countries. Ruby (1992) explains the difficulties of operationalising
attendance as part of the OECD project set of indicators.
Measuring Quality
This is probably the most contentious area. Most of the heat has been generated around school effectiveness research because of the growth of achievement monitoring in order to identify "improved environments and educational aids which lead to detectable gains in knowledge, skills and values acquired by students" (Ross and Mahlck 1989).
In addition there are arguments over what is meant by 'quality' (e.g. Cheng, 1994) and over who should decide what is meant by quality (e.g. Hoppers, 1994 ;Stephens, 1991). Together, this would suggest that it is foolhardy to propose a system intended to be valid across all countries and systems
Instead, it might be appropriate to consider the more cautious approach of the USAID research project on Improving Educational Quality (IEQ) (with offices in Mali, Ghana, Guatemala, South Africa and Uganda) with the following objectives:
• to understand the processes through which classroom interventions in different countries influence student performance;
• to demonstrate a process whereby classroom research on improving educational quality is integrated into the educational system;
• to create opportunities for dialogue and partnership among researchers and educators who are seeking to improve educational quality at local,
regional, national and international levels.
In this way, the intention, presumably, is to develop sets of quality indicators that are consensually agreed at the country level. Whether this is feasible, and whether the conclusions of such groups actually do generate national consensus is unclear.
Disadvantaged groups
Both the Jomtien agenda and the ODA Education Strategy Paper (1993) highlight the importance of monitoring the situation of disadvantaged groups. Possible indicators are:
• participation and success rate of ethnic, religious or language minority students;
• number and status of teachers and administrators in the educational system from those groups;
• appropriate curriculum and textbook content;
• provision of teachers familiar with non-mainstream cultures; and
• linguistic information on teachers and students.
This is very obviously a case where no hard and fast suggestions can be made. It depends on the specific situations in each country or region. Introducing a term or concept from one country or region to another may lead to entirely inappropriate conclusions.
What Happens After School?
The ODA Education Strategy Paper (1993) points to the difficulties here of
institutionalising verification and accountability mechanisms for the wider objectives and longer term outputs of projects9:
• on exit from education system, 'no one is responsible';
• the management and design of tracer studies is rarely specified clearly in project design;
• interest in longer term outputs/outcomes often diminishes.
From the societal point of view, this is the most important outcome; yet the 1999 Policy Framework Document is much vaguer, talking about strengthening capacity (p40) and
rights and responsibilities (p33).
Decentralisation and Devolution
Indicators of the devolution of financial responsibility can include: number of distinct school systems; proportion of key education decisions that are made locally; existence of school boards, their methods of selection and financial mandates; percent of locally generated revenue that stays local.
In the health sector of Scandinavian countries, Mills suggests assessing the extent of decentralisation in terms of the following two sets of indicators:
( ) revenue raising in devolved systems (thus: percentage of public health care centrally funded; local authority tax powers; controls on local taxes; central sanctions if expenditure is exceeded; and the local right to take out loans);
( ) planning controls in devolved systems (thus: the existence of a planning process linking levels; the initiating level; whether it is compulsory; and whether government approval is required.
Democratisation/Beneficiary Participation
The World Bank (1996) now argues for monitoring beneficiary participation to increase client investment in project success. Developing joint monitoring and evaluation
systems work towards Bank goals of teaching new skills, but one must note their own caveat that these require: continuity of personnel from both government and donor agency; a network of supportive government personnel; avoidance of partisan politics; community leadership; and a sense of community and investment in project goals (Uphoff, 1992). Clearly, not an easy task.