Capítulo III. Marco Metodológico
3.5. Técnicas e Instrumentos de Recolección de Datos
Criteria and Sub-Criteria Evidence Rating Benchmarks Guidance
1. Relevance
1.1 Strategic Relevance Definition: Assesses the degree of relevance to Bank strategies and policies (non-transactional TC), or degree of integration with the related investment (transactional TC).
Non-transactional TC: Assess whether the TC’s alignment with corporate strategies and policies/initiatives stated in the design occurred in practice.
(i) Were the stated strategic links confirmed during implementation?
(ii) How did the TC match and therefore address the client’s actual capacity gaps?
(iii) Did the TC contribute to other policies etc.
not envisaged in the design?
Transactional TC: Assess how the TC’s alignment with the related investment stated in the design occurred in practice. Questions might include:
(i) Did the TC integrate into the related investment by contributing to the results achieved by the investment, or did the TC become an ‘add-on’ largely disconnected from the related investment?
(ii) Did the TC address the client’s actual capacity gaps?
(iii) Did the TC contribute to other aspects of the related investment not envisaged in the design (e.g. did it support achievement of other project results or transition impact)?
Excellent: Evidence from implementation confirmed that the TC was aligned in all aspects with the related investment or relevant strategies and policies.
Fully satisfactory: Evidence from
implementation confirmed that the TC was reasonably well aligned with the related investment or relevant strategies and policies.
Partly unsatisfactory: Evidence from implementation confirmed that there was limited alignment between the TC and the related investment or relevant strategies and policies.
Unsatisfactory: Evidence from
implementation confirmed that the TC was not aligned with the related investment or relevant strategies and policies.
Strategic alignment analyses key TC design features, judged against the related investment or relevant Bank policies and strategies.
The evidence provided for strategic relevance should demonstrate how the TC actively helped EBRD to deliver on its policy and strategy intentions, rather than happening to be in alignment (or not being in conflict) with (often very generally-worded) strategies and policies.
Annex 2: Rating criteria and sub-criteria for technical cooperation
Criteria and Sub-Criteria Evidence Rating Benchmarks Guidance
1.2 Relevance of Design
Definition: Focuses on: the soundness of the design logic (implicit or explicit theory of change); the evaluability of the operation (i.e., the adequacy of the specification of expected results and the indicators for measuring their achievement);
the adequacy of identification and incorporation of the lessons of past experience; and the adequacy of risk
identification and mitigation.
For both transactional and non-transactional TC, assess the following design features:
(i) The completeness and clarity of specification of expected results (outputs and outcomes) and the soundness of the embedded design logic.
(ii) To what extent the TC was evaluable and whether the results matrix facilitated effective monitoring during implementation.
(iii) Whether the lessons identified in design proved to be relevant during
implementation.
(iv) Whether risk factors were adequately identified and taken into account in design through identification of coping strategies.
Excellent: All four aspects of project design proved to be fully appropriate during implementation.
Fully satisfactory: Three of the four aspects of project design proved to be fully appropriate during implementation.
Partly satisfactory: Only one or two of the four aspects of project design proved to be fully appropriate during implementation.
Unsatisfactory: None of the four aspects of project design proved to be appropriate during implementation.
The analysis focuses on whether the TC and/or policy dialogue would plausibly have produced the expected results.
Evaluability is defined as the extent to which the expected results of a project are verifiable in a reliable and credible manner. Evaluability is assessed using the checklist prepared by EvD for transactional and non-transactional TC. Assuming that the checklist was used prior to TC Com approval to confirm the project’s evaluability, the evaluation should assess how well the results matrix supported TC implementation and monitoring.
All projects, TC, strategies and policies should demonstrably take account of the lessons from past experience whether sourced from EvD findings, the team’s experience or elsewhere. Identification of lessons in the approval document is not a sufficient test – something must have been done differently as a result if there is a relevant lesson.
Finally, all key risks that reasonably could have been foreseen should have been identified, and plausible coping strategies proposed.
Annex 2: Rating criteria and sub-criteria for technical cooperation
Criteria and Sub-Criteria Evidence Rating Benchmarks Guidance
2. Effectiveness (Results)
2.1 Achievement of Outputs
Definition: The extent to which the TC’s expected outputs were achieved.
Applies to both transactional and non-transactional TC.
Compares achieved outputs with expectations as at approval, or as revised if any revisions were formally approved during implementation.
The delivery of outputs against the planned timeframe for achievement is also taken into account.
As achievement of outputs is largely within the control of the project, the boundaries for the rating categories have been raised compared with other aspects of performance, which are subject to greater risk of realisation.
Excellent: Progress report – Project is on track to achieve all outputs within an acceptable timeframe. Completion Report – All outputs are achieved within an acceptable timeframe.
Fully satisfactory: Progress report – Project is on track to achieve all or most outputs (=>50%), but with significant delays in the timeframe necessitating action. Completion Report – All or most outputs (=>50%) are achieved, but with delays that might have resulted in an extension to the project timeframe.
Partly unsatisfactory: Progress report – Project is likely to achieve few outputs (<50%) and is incurring delays necessitating significant remedial action. Completion Report – Few outputs (<50%) are achieved within an acceptable timeframe.
Unsatisfactory: Progress report – Project is unlikely to achieve any outputs necessitating a substantial redesign or closure of the project. Completion Report – No outputs are achieved within an acceptable timeframe.
For evaluation purposes in EBRD, outputs are defined as the products, capital goods and services that result from a TC project.
Outputs are described in the Results Matrix for all TCs approved after 1 July 2013.
As a majority of TCs include some form of capacity building for staff and/or institutions, the evaluator should ensure that all outputs and outcomes relating to such capacity building are assessed and given due weighting.
As not all outputs have equal value in scope and scale with regard to influencing outcome achievement, and since some outputs will only have qualitative targets, the evaluator should use discretion in reaching an overall output rating.
Annex 2: Rating criteria and sub-criteria for technical cooperation
Criteria and Sub-Criteria Evidence Rating Benchmarks Guidance
2.2 Contribution to Expected Outcomes
Definition: The extent to which expected outcomes described in the project design or revised during implementation have been achieved, plausibly as a result of the TC.
Applies to both transactional and non-transactional TC.
Outcomes are assessed against the TC’s expected results using the associated
indicators, their baselines and target values, as described in the TC submission document or as formally revised during implementation.
Evidence would include: (i) baseline and at-completion values of performance indicators defined for the TC; and (ii) other quantitative and qualitative information relevant to the expected outcomes.
The rating reflects the TC’s incremental contribution to observed outcomes, regardless of whether this represented movement in the right or wrong direction (positive or negative).
When a positive outcome is achieved but there is evidence that this is primarily due to other factors, the rating may be adjusted downward accordingly. This requires consideration of the counterfactual, i.e. what would have happened in the absence of the project.
For example, if outcome indicators meet or exceed their target values, but there is evidence that this was due mainly to external factors unrelated to the TC, a partly
unsatisfactory rating may be warranted.
Conversely, if outcome indicators deteriorated, but there is evidence that the decline would have been worse in the absence of the TC, a fully satisfactory rating may still be warranted.
Excellent: Progress report – Project is on track to meet all outcome, project and/or donor indicators within the planned timeframe. Completion Report – All outcome, project and/or donor indicators are met within the planned timeframe.
Fully satisfactory: Progress report – Project is on track to meet all or most of outcome, project and/or donor indicators (=>50%), but with significant delays in the timeframe necessitating action. Completion Report – All or most outcome, project and/or donor indicators are met (=>50%), but with delays that might have resulted in an extension to the project timeframe.
Partly unsatisfactory: Progress report – Project will meet few outcome, project or donor indicators (<50%) and is incurring delays necessitating significant remedial action. Completion Report – Few outcome, project and/or donor indicators are met (<50%) within an acceptable timeframe.
Unsatisfactory: Progress report – Project is unlikely to meet any outcome, project and/or donor indicators, necessitating a substantial redesign or closure of the project.
Completion Report – No outcome, project and/or donor indicators are met within an acceptable timeframe.
For evaluation purposes in EBRD, outcomes are defined as the short-term and medium-term improvements to systems, institutions and/or participant’s behaviour that are a consequence of delivering the TC outputs.
Outcome statements can be found in the TC Results Matrix since 1 July 2013.
Prior to rating results, and in the absence of an adequate results matrix in the TC’s design, the evaluator prepares a results framework. This should identify the TC’s outputs and outcomes with
appropriate indicators, baselines and associated target values, derived from the information contained in the TCR, Terms of Reference and (in the case of transactional TC) the Operations Report. The results framework is appended to the evaluation.
As a majority of TCs include some form of capacity building for staff and/or institutions, the evaluator should ensure that all outputs and outcomes relating to such capacity building are assessed and given due weighting.
If the expected outcomes, indicators or targets were formally revised during implementation, the project will be assessed against these revised expectations.
Outcomes are compared against a without-project counterfactual with plausible causality being
established using a theory-based method based on a results framework. The evaluator should present evidence on achievement at all levels in the results chain and test the validity of assumptions.
Annex 2: Rating criteria and sub-criteria for technical cooperation
Criteria and Sub-Criteria Evidence Rating Benchmarks Guidance
2.3 Unanticipated Results
Definition: Other positive or negative results that are not covered in the above sub-criteria.
Applies to both transactional and non-transactional TC.
Assesses positive or negative results that were not anticipated in the TC design and not, therefore, covered by the output and outcome sub-criteria.
The rating reflects the incremental results, whether positive or negative. Taking account of the counterfactual might reveal that the quantum of an unanticipated positive or negative result could have been different in the absence of the TC project.
Excellent: The project made a substantial and plausible contribution to significant positive unanticipated results.
Fully satisfactory: The project made a plausible contribution to net positive unanticipated results.
Partly unsatisfactory: The project contributed to net negative unanticipated results.
Unsatisfactory: The project contributed to significant negative unanticipated results.
Not applicable: There was no evidence of unanticipated results.
To be included, unintended results must be truly unanticipated, attributable to the project, quantifiable, of significant magnitude, and at least as well evidenced as the project’s other results.
Note that unanticipated results are rated, but the rating is not a determinant of the overall performance rating.
Annex 2: Rating criteria and sub-criteria for technical cooperation
Criteria and Sub-Criteria Evidence Rating Benchmarks Guidance
2.4 Sustainability of Results
Definition: The likelihood that achieved results will be sustained or extended.
Applies to both transactional and non-transactional TC.
Sustainability considers the likelihood that human,
institutional, financial, and other resources are sufficient to maintain the TC’s outcomes over the economic life of the investment for transactional TCs or the long term for non-transactional TCs.
This forward-looking assessment aims to highlight risks (if any) that the results achieved will not be sustained, or the potential for circumstances to change such that the extent of results achievement could be enhanced beyond the point of evaluation. Of particular importance is whether the results of any capacity building initiatives will be sustained.
Consider:
(i) whether appropriate policies, procedures and financial structures are in place to ensure client continues operations;
(ii) whether client staff successfully
demonstrate application of new skills and knowledge from training;
(iii) the adequacy or otherwise of policies, institutions, markets and regulatory conditions, and risk of change in these;
(iv) the political will to ensure client ownership and commitment to TC results;
(v) the existence of an appropriate legislative structure to support the continued implementation of TC-generated actions, programs or other recommendations; and (vi) the political will on the part of the client to
maintain support of key stakeholders.
Excellent: Evidence confirms that all TC results are likely to be sustainable against the relevant determinants of sustainability.
Fully satisfactory: Evidence confirms that most TC results are likely to be sustainable against the relevant determinants of sustainability.
Partly unsatisfactory: Evidence confirms that only a limited number of TC results are likely to be sustainable against the relevant determinants of sustainability.
Unsatisfactory: Evidence confirms that none of the TC results is likely to be sustainable against the relevant determinants of sustainability.
This assessment is forward-looking and so to some extent speculative. However, it should be rooted in the conditions that exist at the time of evaluation and which could influence sustainability or enhance results achievement.
While the rating of this sub-criterion is not incorporated in the overall rating (unless deemed appropriate to do so by the evaluator), it provides additional narrative evidence to support the overall TC project rating.
Annex 2: Rating criteria and sub-criteria for technical cooperation
Criteria and Sub-Criteria Evidence Rating Benchmarks Guidance
3. Efficiency
3.1 Bank Handling
Definition: The extent to which the Bank efficiently managed implementation, including whether it proactively identified and resolved problems, managed risk, and monitored and reported on progress during implementation.
Applies to both transactional and non-transactional TC.
Includes:
(i) the timeliness of the engagement of the consultant(s);
(ii) the quality of the consultant management process;
(iii) the quality of monitoring, including sufficiency of field missions;
(iv) the completeness of progress reports in documenting TC status and risk;
(v) the adequacy and timeliness of the Bank’s response to emerging problems or opportunities (risk management and mitigation);
(vi) the reasons why scope changes, if any, became necessary;
(vii) the effectiveness of hand-over procedures where there were changes in Bank staff management and monitoring
responsibilities; and
(viii) whether the TC was good value for money.
Excellent: The Bank met its operational standards, there were no significant shortcomings due to the Bank’s supervision performance and the TC represented good-to-excellent value for money.
Fully satisfactory: The Bank met most of its operational standards, there were some relatively minor shortcomings due to the Bank’s supervision performance and the TC represented fair-to-good value for money.
Partly unsatisfactory: The Bank met few of its operational standards, this had a limited effect on the achievement of results, and the TC represented poor-to-fair value for money.
Unsatisfactory: The Bank met few or none of its operational standards, this had a significant effect on the achievement of results, and the TC did not represent value for money.
The assessment of value for money is not about least cost. Rather, it requires a judgment about the value derived from the outputs produced by the consultants in relation to the cost of their services. This has two dimensions: (i) the delivery of specified outputs; and (ii) the extent to which those outputs were used to produce value.
Where the project scope changed, the reasons why this was necessary and the impact of the scope revision on results should be commented upon under Bank Handling. For example, if the causes of scope change could not reasonably have been foreseen at approval, and changes to put the TC operation back on track (successfully) were made in a timely manner, this aspect of Bank Handling should be assessed
positively. Conversely, in the case of inadequate project design and/or a response that was not timely or did not bring the TC operation back on track, this aspect of Bank Handling should be assessed negatively.
Annex 2: Rating criteria and sub-criteria for technical cooperation
Criteria and Sub-Criteria Evidence Rating Benchmarks Guidance
3.2 Client’s Handling Definition: The client’s
contribution to achieved results.
Applies to clients of both transactional and non-transactional TCs.
The client’s non-financial performance might include commentary on how the client
supported TC implementation. Where the client is the contracting entity, this includes:
(i) engagement of consultants efficiently in accordance with EBRD PP&R’s; and (ii) effective management of consultants.
Where either the client or the Bank is the contracting entity, it includes the client’s:
(i) provision of timely reports;
(ii) role in steering the TC activities to meet its requirements, including formal
management arrangements (e.g. TC steering committee);
(iii) frequency and extent of interaction by senior client staff with consultant(s);
(iv) execution of key actions in a related investment to ensure TC implementation is not delayed;
(v) ensuring that appropriate staff participate in TC activities; and
(vi) provision of agreed cost-sharing arrangements.
Excellent: The client met its agreed
responsibilities, and there were no significant shortcomings due to the client’s performance.
Fully satisfactory: The client met most of its agreed responsibilities, and there were some relatively minor shortcomings due to the client’s performance.
Partly unsatisfactory: The client met few of its agreed responsibilities, and there were shortcomings due to the client’s performance.
Unsatisfactory: The client did not meet its agreed responsibilities, and there were significant shortcomings due to the client’s performance.
The evaluator should identify client strengths and weaknesses, which contributed to achievement or otherwise of the project’s results. Lessons from implementation such as good practices illustrating effective client ownership should also be identified.