At partner level (3 e,f,g)
Monitoring and evaluation are in principle performed regularly by the partner organisations. According to the PME manual,131 the purpose of monitoring and evaluation is to continuously verify progress towards specific results, inform decision-making for intermediate remedial action, and enable lessons learned and results to be incorporated in the one-year and four- year plans. The PME manual offers detailed formats, suggestions on how to get organised, instructions for monitoring intermediate results and schedules for monitoring and evaluation as well as reporting formats. Partners, including affiliate representatives, have all had training on how the materials provided in the manual are to be used. Each partner has a PME committee that takes care of all PME activities, including the internal monitoring and plan preparation:
- In Ghana and Indonesia, the partners’ PME process is in the hands of a PME committee, which is in fact the CNV-financed programme management unit at confederation level. Each committee has a permanent staff paid by CNVI. E.g. CNV is financing the SBSI- Indonesia PME committee for 5 years (2007-2012). It employs two fulltime employees
(€26,620 per year) and one gender expert (€15,000 per year). The heavy dependence on external financing is necessary because the organisations would be unable to cover such costs themselves, as they hardly have sufficient own resources to pay for ongoing internal operations. CNV wants to be sure that monitoring and evaluation are done properly and also wants to stay informed and comply with the donor’s requirements. CNV also sees it as a learning process for the organisation.
- The Committees follow the steps outlined in the PME manual as well as the formats. In Ghana, the GFL PME committee is coached intensively by BCPA,132 the CNV-WSM permanent regional consultant. These consultants are highly qualified, well versed in the PME system, have an intimate knowledge of labour-related issues and unions, and assist the organisations through coaching and knowledge transfer. Due to relative proximity, there is real influence on the quality of the committees’ work. The ‘BCPA formula’ thus has major added value for GFL.
The GFL committee follows a detailed and clear procedure133 to coordinate the step-by-step development of the work plan, the assessment of affiliates’ proposals against outcomes, the provisional approval and contracting, the monitoring of implementation (including field visits to affiliates) and its regular evaluation – prepared first by the affiliates and aggregated by the committee, which then reports to CNV. This is done in a participative manner, so that all nine affiliates are included.
While similar methods were introduced in SBSI Indonesia through training courses run by CNVI, improvements must still be made, if it is to have the same internal learning function. A regional consultant to represent CNV has not yet been found.
According to CNV, which can compare the present situation with the past, the quality of PME has generally improved since 2008. In particular, there is a much stronger outcome orientation compared to the previous VMP period and this is apparent in reporting.134 Effects of this way of working as signalled135 by the confederations are improved joint decision- making and transparency in the allocation, disbursement and use of funds at the various levels (accountability!). The procedure is considered user-friendly and facilitates informed decision-making. Planning and implementation have become results-oriented, showing the way to results at worker level and thus helping unions make their activities more concrete. In general affiliates and the confederation are now better organised, so that reporting has improved, although exceptions remain. We would point out that learning is a built-in function and is shared between the affiliates and the committee; this was especially apparent within GFL and less so with SBSI which committee is more hierarchical due in part to a different management style.
Both CNV and the federations/affiliates agree that the timing of the annual planning evaluation procedure, preceded by an evaluation of intermediate results achieved, can be improved. This has several causes, illustrated by the Ghana and Indonesia examples:136 the affiliate leaders are too deeply involved in implementation, inadequate organisational capacity (small number of permanent staff or work on a volunteer basis), in some cases a poor level of training and comprehension, late budget implementation/disbursements and so delays in work, and budget rigidity which makes it difficult or time-consuming to adapt to changed circumstances. These are practical problems that may require the annual planning evaluation process to start even earlier in order to meet CNV/Ministry planning deadlines. Bringing deadlines forward has limited value however, as it could also mean the process is started halfway through the implementation period, diminishing the value of monitoring and evaluation, and of budget accuracy. The alternative may be more intensive coaching, more ‘discipline’ or even sanctions.
Annual reports of partners
The partners use the format prescribed in the PME manual for their annual137 reports.138 This information is used to keep track of the implementation process and indicate the effectiveness of the partner in the field. In Ghana, this is mainly checked by BCPA; in Indonesia, this is the task of the programme manager in the Netherlands.
The reports concentrate on activities and outputs. Little is said about ‘what happens next’, even in the case of failure. The confederation will also report on impact as and when achieved. E.g. when a collective labour agreement has been concluded (outcome) and is actually implemented (which largely falls outside unions’ influence). The GFL affiliate NUTEG for instance reported that ‘10 workers were reinstated’ (after unlawful dismissal was rectified).
It should be noted that organisational strengthening activities and services provided to workers are often both ‘translated’ into their (intermediate) impact in terms of the strengthening of the affiliate and/or the confederation139 – like increased membership or increased payment of dues.
CNVI annual reports
CNV’s annual reports to the Ministry are based on partners’ reports, assessments, internal memorandums written by the portfolio managers, and internal memorandums consolidating the results of contextual and organisational analyses, baseline surveys and country evaluations. All this information is provided in CNVI’s general annual report (jaarverslag). In addition, since 2008, CNVI also publishes a more ‘popular’ account of its international work. In response to the Ministry’s request to differentiate between regions and countries, a separate overview is given for each continent. This gives structure to how the information is presented. There is no added value in analytical terms, as there is no real connection between country projects on a continent.
The format in the annual/multiannual plan and Monitoring Protocol is used for reporting on results. Results at outcome level are thus presented by programme for selected countries. Illustrative examples (anecdotes) are given to present results at impact level. The examples given clearly show how and to what extent contributions can be made to strengthening labour-related organisations at both confederation and affiliate level and to improving conditions for workers – the two main objectives of the VMP.
It is positive that CNV has adopted a system in which outcome is the leading factor. At the same time, it could be tempting to report only positive outcomes in citing examples. In practice, this has not been a problem: the 2009 annual report, for instance, also described less successful activities, using them as examples of lessons learned.140
The Ministry’s responses have invariably been positive and issue-specific. Few questions were posed.141
Portfolio management (3c)
A major factor in CNV’s PME system is the role and functioning of the portfolio managers. The emphasis placed on the PME system, including on the instruments used, ensures that all ‘speak the same language’. A key role in plan preparation is for portfolio managers to look closely at the proposals and prevent the practice of giving ‘every affiliate a project, whatever’. One of the main tasks of portfolio managers is to make an independent analysis and connect with other experiences, get an overview and provide partners with ideas, stimulate innovative thinking (e.g. including the informal sector), link up partners with other organisations (e.g. Fairfood). At the same time there is a ‘control’ on the functioning of the unions/confederations
through frequent formal and informal contact, which also provides for the possibility to intervene, stimulate timely action and so on. Supervising research and initiating additional research is another important task.
Functioning is largely determined by seniority, personal and technical competencies, and feeling for the subject matter and types of organisation. The requisite competencies are present in the CNVI staff establishment. Performance is currently reported to be very good, an improvement on underperformance in 2009 which may have been due to personnel problems.
The way BCPA assists the unions and effectively operates as an extension of the CNVI office is an approach that could be considered for other regions, with an emphasis on coaching/permanent training and knowledge transfer (less ‘control’). This type of assistance is much appreciated by the unions and may be cost-effective if various partners could be supported and cofinancing could be arranged in the same way as for BCPA.
Baselines (3d)
In order to be able to measure changes/outcomes, it is necessary to measure the baseline situation before projects are implemented. There is not ‘one’ baseline and opinions differ within CNV as to what constitutes a baseline. In fact, baseline information is available from various sources and for various purposes:
- Contextual analyses, such as for Cambodia, provide a lot of general information that can be used to measure major changes, e.g. changes in the institutional environment or policies.
- Organisational analyses measure the organisational capacity and problems through a SWOT workshop and additional assessments by the workshop and by a consultant. This could also be done in a more general sense by the portfolio manager in order to identify, together with the organisation concerned (the confederation or the affiliates) what aspects should be addressed in the framework of organisational strengthening.
- A specific study has been carried out into the financial situation of GFL Ghana as input for preparations for a special multiannual project to achieve financial independence (self- financing). The study has provided baseline information on, for instance, the present level of self-financing and membership per affiliate.
- Concise contextual and problem analyses and mapping of labour-related characteristics. These analyses are performed by the affiliates or the confederation in relation to proposals for a project or series of interventions for a specific sector, in order to get an idea of the labour situation at plant level or in a (sub-)sector.142
- Ex ante evaluation such as the analysis of the cocoa sector and child labour in the cocoa sector (done in collaboration with Fairfood).
- Gender audits may also be performed. CNVI conducted a gender audit in Indonesia to measure the gender sensitivity of the two organisations concerned (unions) and their activities. Gender audits may provide insight into the gender situation at plant level or into salaries and other labour conditions.
- Other studies. In recent years, CNVI has contributed to a study into the labour situation in Senegal at the request of RDA Senegal in order to determine the prospects for improving social dialogue together with the Employers Federation/DECP.
The use of the instruments listed above generates a lot of specific information. We did not examine to what extent such research is done systematically in all countries and for all projects. The mix of studies conducted should in any case provide sufficient information for the measurement of most of the specific short-term changes and the identification of contextual elements and risk factors in relation to the interventions.
Much also depends on whether and to what extent baseline instruments are used to verify changes:
- Given the character of most of the outcomes and impacts, verification is rather straightforward. E.g. in the case of negotiating collective agreements, it is only necessary to know whether the agreement has been signed and to follow up on its implementation. Having negotiated and verified implementation at plant level, the improvements are apparent.
- Organisational strengthening can be checked in-house, for instance through qualitative research workshops and comparing the results of partner self-assessments.
- Repeat studies and further data collection may be needed to measure other types of outcomes and impacts. Funding set aside so far for repeat studies or data collection may be insufficient. Without repeat studies, the added value of baseline studies is diminished.
Evaluations (3h)
CNVI has so far commissioned143 two country evaluations, including Indonesia,144 as part of the PME system and agreed outputs in the framework of the VMP:
- The title suggests a country study, while the aim was apparently to look more broadly at the first year of implementation of the current VMP.145 KSBSI serves as an in-depth case study into the efficiency and effectiveness of overall programme implementation so far. The study’s shortcoming however is that findings relating to KSBSI can hardly be extrapolated to the programme as a whole; the case is not representative for the whole programme. In this case, we would have expected a more in-depth analysis of the outcomes and impacts at worker level as well as of the different work methods. This would have provided an answer to the question, to what extent the programme led to improved performance on the part of affiliates and the confederations in negotiations and social dialogue.
- The findings, conclusions and recommendations on partner KSBSI are still largely recognisable and applicable today.146 CNV’s approach, prioritising the development of better management systems in order to improve performance, probably cannot be maintained. There should be a two-pronged approach, addressing aspects such as organisational sustainability, financial soundness and a strong support base, also in relation to positioning and roles in social change. In this regard, we reiterate the recommendation to CNV147 in the country evaluation report to think more in terms of a ‘whole organisation approach’ or organisational change, which may be relevant for more partners. In its management response CNV indicated that it has followed up on this recommendation.148 While CNV has the in-house competency to advise on PME systems development, it may have to make adjustments with regard to other aspects of organisational change.