Theory as well as good practice as defined in the literature position a planned, systemat-ic and rigorous evaluation process as a key element of successful management of change (Skinner, 2004a). However, Skinner (2004a) notes that this rarely takes place.
As a consequence, decisions are made based on assumptions and do not consider reali-ty in form of experiences and lessons learned from the change (Skinner, 2004a).
Increasingly dynamic and more complex developing environment are leading to more comprehensive changes and transformations in companies and organisations (IBM, 2010). This makes it even more challenging to monitor and evaluate respective change programmes. This is one of the key challenges those responsible for organisational change have to deal with today and in future (Rank & Scheinpflug, 2010).
According to Russ-Eft & Preskill (2009), the arising question is: What are the challenges and barriers impeding evaluations in organisations? Russ-Eft & Preskill (2009), being in the field of evaluations in organisations for around two decades, list a variety of potential reasons not to undertake an evaluation. In addition to those, Skinner (2004b) distin-guishes between primary and secondary barriers to evaluate change in organisations.
She defines primary barriers as those factors acting against an evaluation being under-taken (before an evaluation) whereas secondary barriers arise during an evaluation pro-cess (Skinner, 2004b). The use of evaluation findings is emphasised by Patton (2008) as a key element to capitalise fully on the evaluation and its findings, a third category is in-troduced to differentiate and categorise challenges and barriers to evaluation in organisa-tions. It is about barriers after an evaluation was conducted impeding the use of the findings. Subsequently, the reasons, challenges, and barriers that might occur are cate-gorised in these three phases of a potential evaluation process – before, during, and af-ter an evaluation. These factors are not always distinct to one or the other as some of them might also be prevalent in other evaluation stages. The table below presents the main factors assigned to respective phases of an evaluation and they are described in more detail afterwards.
Table III-7: Challenges and barriers before, during or after evaluation
Challenges and barriers in the evaluation process BEFORE – hindering the setup and start of an evaluation
• Perceived lack of need
• Inherent value of programme to be implemented and evaluated
• Negative experience from previous evaluations; fear of negative effects/outcomes/impacts
• Time pressure, too time-consuming, estimated effort higher than benefit
• No resources, experience, skills BEFORE and DURING
• Politics on managerial levels
• Preference for informal evaluation
DURING – impeding the evaluation process and the usefulness of the findings
• Conflicting research/evaluation perspective
• Absence of clear purpose, scope, objectives
• Credibility of evaluator
• Changes during evaluation
• Limited involvement/participation, suspect results will be shared/used
• Changes in evaluand/programme
AFTER – impeding acceptance of findings, practical application, and implementation
• Insufficient communication; restricted target group, not open/honest/transparent
• Timeliness
• Fear of dealing with negative outcomes
• Findings challenged/suspected/questioned and therefore not used/applied
Evaluation as such not valued
Source: Own table based on Skinner (2004b), Millmore et al. (2007), Patton (2008), Russ-Eft &
Preskill (2009)
3.3.1 Before, during, and after evaluation
The following challenges and barriers might inhibit evaluations being set up and/or compromise the evaluation process, its effectiveness, and the meaningfulness of re-sults. A very dominant influencing factor is politics, often on senior management levels.
It centres on power struggles, personal self-interest, hidden agendas, trying to influ-ence management thinking, decision-making, and behaviour at higher organisational levels (Fox, 1989; Easterby-Smith, 1994; Skinner, 2004b; Norris, 2005). Skinner (2004b) identifies a preference for informal evaluation as a continual, ongoing activity occurring at many levels in an organisation creating a key barrier to formal evaluation.
Further, Easterby-Smith (1994) recognise senior mangers’ preference for receiving in-formation via their own selected sources and channels and rating it as more valuable and influential than information received via formal evaluation channels.
Finally, this might result in stakeholders’ suspicions when evaluation takes place, as to whether the findings might be shared honestly and transparently and applied accord-ingly.
3.3.2 Before evaluation
One of the main reasons not to evaluate a change programme is a perceived lack of a need to evaluate it (Millmore et al., 2007). It is recognised very often that people being responsible for a certain initiative act on the assumption of an inherent value in the ini-tiative potentially to be evaluated (Brunsson, 2009) and that anticipated benefits will in-evitably result from the programme implementation, irrespective of whether it is evaluated or not. Skinner (2004a) supports this argument, the unquestioned assump-tion leads to a reduced perceived need for formal evaluaassump-tion as those responsible
“know” already the positive outcome. Moreover, some organisational leaders think they also already “know” what does and what does not work (Russ-Eft & Preskill, 2009). As a logical consequence based on a perceived lack of need to evaluate Russ-Eft & Pre-skill (2009) argue that another strong reason not to undertake evaluations is that no-one requires or asks for it. On the other side, they also recognise organisation mem-bers’ misunderstanding with regard to evaluations’ purposes, role and perceived effort and costs outweighing perceived benefits.
The literature highlights negative experiences (e.g. broken promises, misuse of findings, ignorance of findings on critical issues, feel of waste of time), poorly executed evalua-tions, “blame cultures” (Skinner, 2004b) and organisations members’ fear of the impact of the findings and dealing negative with outcomes as further major barriers to undertake evaluations (Skinner, 2004b; Russ-Eft & Preskill, 2009). Leaders might be afraid of com-ing in for criticism and appointcom-ing of responsibility for failure when negative results are reported (Millmore et al., 2007).
Very often evaluation is seen as an additional burden, a time-consuming and laborious task. Perceived or ongoing real time pressure resulting in short-term oriented views ra-ther than mid- or longer-term thinking might also cause a shifting focus from evaluation towards other more “urgent” or dominant tasks (Russ-Eft & Preskill, 2009). Consequent-ly, evaluation is often seen as an afterthought appearing towards the end of a pro-gramme, if at all, and not central to its implementation process (Millmore et al., 2007).
The time issue may result in a reluctance to free up and allocate resources to plan, or-ganise, implement and conduct an evaluation (Skinner, 2004b). Another reason for not assigning resources and responsibility to evaluation tasks is a real or perceived lack of required skills, knowledge and experience for undertaking evaluations (Millmore et al., 2007; Russ-Eft & Preskill, 2009).
3.3.3 During evaluation
Once agreement has been reached to conduct an evaluation another set of challenges and barriers can emerge. Different epistemological and ontological understandings also have a strong influence towards whether and/or how to set up, undertake and what to monitor and evaluate in an organisational change programme (Butler, Scott & Ed-wards, 2003). Skinner (2004b) notes that a barrier for evaluation can be caused by the incompatibility between methodological bias of an organisation, manager, or evaluator and the pragmatic needs of the evaluation research. The conflicting perspectives build on the preferences to use quantitative or qualitative measures (Hughes, 2010), de-pending on the data focus to be evaluated (elements, items, success factors, or indica-tors), like rather hard facts (e.g. numbers) or soft facts and intangibles (e.g.
understanding, satisfaction, readiness).
Having no clear understanding about concrete purposes, scope, limitations, objectives, and success criteria of the evaluation to be undertaken represents another key challenge (Skinner & Mabey, 1997; Skinner, 2004b).
The absence of these features complicates the identification and allocation of required resources. The lack of assigned responsibility of an evaluator / evaluation team, or if available a lack of expertise, professionalism, reputation and honesty compromising its credibility represent additional challenges to evaluations (Russ-Eft & Preskill, 2009).
Phillips & Pulliam Phillips (2007) recognise a trend towards fact-based or evidence-based management. Although many key decisions are evidence-based on intuition, managers are increasingly using sophisticated and detailed processes to claim value (Phillips & Pulliam Phillips, 2007). However, evidence in terms of hard data, like financial data, is a challeng-ing subject for a discipline like managchalleng-ing organisational change, where many intangibles are decisive for its success (Burke, 2011). Measures of organisational change are far too complex to be expressed just in numbers (Burke, 2011). Burke further points out that there is a need to learn much more about how to define and measure organisational change and performance than is known today. Even in those organisations where change and respective activities to manage it are given a high priority, the monitoring and feedback process often focuses more on financial measures than on indicators with re-gard to the people side of the change process and key cause-and-effect relations to the achievement of the overall targets (Hayes, 2010)
Kaplan & Norton (2004) state that in organisations very often attention is paid to what is measured and that organisations are not good at managing that which is not measured.
The following quotation highlights this by referring to the McNamara fallacy:
“The first step is to measure whatever can be easily measured. This is OK as far it goes. The second step is to disregard that which can’t be easily meas-ured or to give it an arbitrary quantitative value. This is artificial and mislead-ing. The third step is to presume that what can’t be measured easily really isn’t important. This is blindness. The fourth step to say that what can’t be easily measured really doesn’t exist. This is suicide (Handy, 1994, p.219).”
Furthermore, Millmore et al. (2007) emphasise the difficulty of gathering reliable and valid data and isolating the effects of certain activities from other influences (Phillips & Pulliam Phillips, 2007).
Another critical aspect emerges when the features of an evaluation are clear, but not widely shared with those who should participate and be involved in the evaluation. This will inevitably lead to scepticism, mistrust and decreasing interest in being involved and participation (Skinner, 2004b).
Furthermore, an aspect to be avoided deals with any changes during an evaluation, for instance stakeholders having a vested interest in the programme who are transferred to another part within or even leave the organisation or any changes in the programme be-ing evaluated (e.g. new sponsor, changes in scope or objectives). This may result in col-lecting data on something that does not exist anymore for its original purpose or where the initial evaluation questions are no longer appropriate.
As noted by Toracco (1997, p.121) “evaluation is usually not included in long-term change processes”. Thus, this lack of evaluation of change in practice is due to long time frames and ongoing changes in different parts of an organisation impairing the relation-ship between change interventions and their outcomes (Toracco, 1997; Millmore et al., 2007).
3.3.4 After evaluation
Essential to the success of any evaluation is the timeliness of the evaluation results.
Usually, there are time constraints if an evaluation is conducted to solve or understand some problems. When evaluations miss these constraints their findings are of limited use (Russ-Eft & Preskill, 2009).
Insufficient communication is another area creating barriers in terms of acceptance and the use of evaluation findings. Findings must be made available to all relevant stakehold-ers, and if possible, in a variety of formats. This requires using appropriate communica-tion channels and systems for disseminating and accessing the findings. Problems of acceptance among organisational members occur when an evaluation report is only shared among management levels, programme staff, or any other selected and restricted audience.
As a consequence organisations miss an opportunity for individual as well as organisa-tional learning for a better future (Russ-Eft & Preskill, 2009).
Although evaluation findings might be communicated and shared with all relevant stake-holders, due to all the challenges and barriers mentioned, stakeholders might challenge, suspect, question and therefore not use and apply the findings.
Finally, the main barrier to using the findings of an evaluation rests in the fear of dealing with outcomes leading to negative impacts and consequences (Skinner, 2004b).
All these aspects, as discussed in Section 3.3, potentially limit the usefulness of whatev-er evaluation findings are genwhatev-erated. Refwhatev-erring to Russ-Eft & Preskill (2009) again, they state that organisation members do not value evaluations and the number one reason to neglect them is an aggregate of all factors presented in Table III-7 above. In combination, all these factors provide significant obstacles and disincentives to undertake evaluations in the context of a change programme. Organisations need to understand the value of evaluation (Skinner, 2004b). However, finally the area of evaluation in organisations is of a highly political nature (Clarke & Dawson, 1999). Consequently, the required power and resources to prompt a formal evaluation process and to apply the findings accordingly remains with the dominant stakeholder group on (senior) management levels (Skinner, 2004b).
Despite these difficulties, Toracco (1997) emphasises the importance of evaluating the changes against key organisational goals, even if effects cannot be fully assigned to a particular change intervention. Finally, Hughes (2010) states that despite potential evalu-ation pitfalls the real need remains to evaluate managing change initiatives to inform the developing body of knowledge about managing change.