PurPose of assessMent
Assessment is any process that allows us to receive feedback on or make judgments about or in some way analyze a set of activities. It can take many forms. In designing a performance assessment process it is useful to think in terms of what, when, where, why and how, as well as who will conduct the assessment and for whom is it being done.
Measuring performance requires effort both on the part of assessors and those who are being assessed or are responsible for what is being assessed. To ensure that the assessment process is useful, meaningful and resource efficient we need to understand the purpose of the assessment. Why is it being undertaken? Possible purposes are:
• Ensuring alignment with strategy;
• Determining achievement of objectives or outcomes;
• Determining effectiveness or efficiency;
• Reviewing progress;
• Ensuring adequate competence or capability to support performance;
• Providing feedback for improvement.
unit and Context of assessMent
In order to satisfy the specific purpose, what needs to be assessed? Answering this question requires an understanding of both the unit of assessment and its context. As shown in Figure 7.1, the focus of assessment may be on outcomes, processes and practices or the competence and skills of people.
Figure 7.1 context for performance assessment: a guiding framework
107 M e a s u r i n g P e r f o r m a n c e
The context may be a single project, multiple projects or programs or organizational capability including portfolios.
PerforManCe and suCCess
When measuring performance we need to be very clear about the terms we use and what we are assessing in order to avoid confusion and ensure effectiveness of the assessment process. Although the term ‘measuring performance’ is widely used it is actually open to a wide range of meanings and interpretations. Terms such as performance and success are often used interchangeably but different people at different times may interpret them in different ways. It is now generally accepted that success is in the eye of the beholder and may be a matter of timing. The Sydney Opera House is often used as an example of this. At the time of its design and construction it was considered a failure because it significantly failed to meet time, cost and quality criteria, but decades later it is recognized as the most representative man-made monument in Australia and a masterpiece of modern architecture. This illustrates a difference between project management success and product success. In the case of the Sydney Opera House, while the management of the project may be considered unsuccessful, the final product is an acknowledged success.
This distinction between project management success, concerned with internal measures of project performance such as time, cost and quality and product or project success, measured against the overall objectives of the project (Cooke-Davies, 2002;
Jugdev and Müller, 2005), is useful, although success remains an ambiguous concept that may be judged differently from different stakeholder perspectives over time. Nevertheless, for practical purposes the definition provided by Baker, Murphy and Fisher (1988) in their landmark study reflects a generally accepted understanding of the concept. They concluded that success is a matter of perception but that a project will be most likely to be perceived as an ‘overall success’ if:
the project meets the technical performance specifications and/or mission to be performed, and if there is a high level of satisfaction concerning the project outcome among key people on the project team, and key users or clientele of the project effort.
Further, although it is generally agreed that time and budget performance alone are inadequate as measures of project success, they are still important components of the overall construct. Project management success can therefore be seen as a component of project success. In examining the literature on project success, Patanakul, Iewwongcharoen and Milosevic (2010) proposed that dimensions for judging success could be categorized into internal or project related criteria (time, cost and performance), customer related criteria (satisfaction, actual utilization and benefits) and organization related criteria (financial, market and benefits).
When thinking about assessment metrics it is useful to distinguish between success measures which relate to the achievement of objectives or outcomes and performance measures which relate to the processes and practices used to deliver the outcomes. In terms of timing, performance measures relating to processes and practices are most likely to be used during the course of a project or program while success measures relating to the achievement of outcomes are most likely to apply at closure. Customer related criteria
such as actual utilization and benefits are most often measured some time, generally three to 12 months, after project or program completion and handover.
Criteria and faCtors
Two other terms that often cause confusion and lack of clarity in performance measurement are ‘criteria’ and ‘factors’. Essentially, criteria are measures or metrics that can be used as a basis for judgment and factors are elements or causes that contribute to a result. This may seem confusing because factors that affect a result may also be used as criteria. For instance, delivery on budget may be used as a criterion for judging success, but delivery on budget may also be a factor contributing to achievement of other criteria for judgment of success such as return on investment or client satisfaction.
BenCHMarks and standards
In order to interpret and make use of performance measures it is necessary to have standards or guidelines of acceptable performance against which actual performance can be compared. These will vary according to the purpose of assessment and the performance measures being used. Basic performance measurement for monitoring purposes compares actual performance with planned performance. However, it is still necessary to have guidelines that indicate what level of variation is considered acceptable. For instance, if we are assessing time and cost performance of a project at completion, acceptable performance may be ±5% of agreed baseline. Acceptable quality may be set at zero defects or 90% customer satisfaction rating.
Other forms of performance measurement may be used to identify gaps or drive improved performance. In this case standards can provide indicators of minimum achievable outcomes, process checklists or targets for improved performance or ‘best’
practice. An example of this would be assessment of organizational project management capability, where we may decide to use one of the many capability maturity or excellence models available from professional associations and consulting organizations (PMI’s OPM3, the UK Cabinet Office’s P3M3, IPMA’s Delta, Human Systems’ 4Quadrant Model, see Chapter 5) both as a basis for measurement and a guide for improvement.
Benchmarks are a particular form of comparative indicator. They are measures that represent the best performance of a particular outcome, process or practice either within or across projects or industries. Internal benchmarking can be done within an organization by identifying the best performance against one or more metrics across a number of projects, divisions or other units of assessment. The benchmark demonstrates that a specific level of performance is achievable and provides a realistic target for improvement across all projects or divisions. External benchmarking involves looking outside the organization for ‘best practice’ comparators. This is most effectively done if there is access to a database of performance data than can be used as a basis for comparison.
Such databases are generally developed over a period of time through collaborative arrangements between groups of companies. Examples are the Benchmarking & Metrics program of the Construction Industry Institute (CII) which enables member companies to compare performance of their capital and maintenance projects against a large sample of projects from industry leaders; the extensive database of process industry projects maintained by Independent Project Analysis (IPA) as a basis for assessing and comparing
109 M e a s u r i n g P e r f o r m a n c e
project elements and results and the cross industry project management benchmarking network facilitated by Human Systems International Ltd (HSIL) that focuses on assessment and improvement of organizational project management capability.
Measures
Performance measures are generally considered to be quantifiable expressions of the amount, cost or result of activities that provide information on quality of capabilities, capacity, processes, practices and outcomes during a given time period. They should, as far as possible, be objective in the sense that they are not influenced by emotions or open to multiple interpretations. Ideally, measures will be direct rather than indirect.
A measure is direct when it measures, directly, what you want it to measure. For instance, if you want to know the number of bricks laid in a specified time period, you can count the number of bricks laid. A measure is indirect when you measure something by measuring something else. For instance, you may wish to measure commitment of team members. You could ask them directly to tell you how committed they are but the answer would be subjective. Alternatively you could measure the number of days sick leave taken in a specified time period and use this as an indirect measure of team member commitment. This example illustrates one of the difficulties in using indirect measures. The relationship between the measure and the phenomenon in which you are interested is not clear and straightforward. While it is reasonable to assume that there is a link between team member commitment and attendance on the job, there are many other factors that may contribute both to number of days sick leave and to commitment.
This is referred to as ‘noise’ in the system. Clearly, direct measures are preferable but not always possible. Similarly, objective measures are preferable but not always possible.
Customer satisfaction is a commonly used measure that is difficult if not impossible to measure both directly and objectively. A measure of customer satisfaction may be repeat business. This measure is objective but indirect. Asking the customer to rate their level of satisfaction on a scale of 1 to 5 is a direct measure but is subjective. Although the answer will be a number and will appear quantitative, it is in fact subjective. It is important to remember that numbers are not always objective. For instance, measures that use any opinion based rating or ranking scale are inherently subjective.
In projects, time, cost and other financial measures are popular because they are easily determined, objective and direct. Other aspects of performance are less amenable to direct and objective measurement. Tangible outcomes are easier to measure directly than intangible outcomes and so benefits, which are usually intangible, can be particularly challenging. The more indirect the measure, the more difficult it becomes to ensure that the measures are objective and meaningful indicators of the outcome or quality that is being measured.
Characteristics of good measures
Apart from striving wherever possible for measures that are direct and objective, and understanding the limitation of measures that are indirect or subjective, there are other characteristics to look for when selecting performance measures. The following
characteristics are adapted from the requirements for performance information set by the Government Accounting Standards Board (USA) (2008).
Relevant measures: satisfy the purpose and requirements of the audience for which they are intended. They will also be useful in providing feedback that is required for governance, control or improvement.
Understandable measures: will be clear to those required to provide, analyze, receive and interpret the information they provide.
Timely measures: will be current and provided with sufficient frequency to support decision making and action.
Comparable measures: provide feedback on changes in the level of performance over time and whether or not performance expectations are being met. They also enable comparisons with performance against the same measures in similar projects, divisions or organizations for benchmarking purposes.
Consistent measures: support comparability by ensuring that the same items of performance information are reported over time to enable tracking of variation or improvement in performance. Measures should however be reviewed from time to time to ensure continuing relevance.
Reliable and valid measures: are verifiable, free from bias and measure what they are meant to measure. Of course, this can be more challenging where it is necessary to use indirect and/or subjective measures, but triangulation, which involves using a number of measures to track the same item of performance, may improve validity. Internal control systems for collection of performance data can assist with reliability.
For all measures it is also important to ensure feasibility, cost and resource effectiveness in collecting the necessary data. In other words, the data are available and will not require unreasonable levels of additional effort to collect and analyze. This raises the issue of how many performance measures to use. Collecting, storing, reporting, monitoring and analyzing data can be expensive in terms of time and effort. Tracking too many performance measures at any one time may dilute the overall impact but too few measures may not provide a useful picture of performance. One suggested rule of thumb is the use of 10–15 measures at any given level of the organization (Office of Financial Management, 2009).
Each level of the organization (executive, division, portfolio, program, project) may have 10–15 measures that include some measures used by lower levels.
success criteria and factors as measures
A useful source of potential measures for project performance are success criteria and critical success factors. A considerable amount of research has been done over the last 40 or so years providing useful checklists. If we are measuring performance to determine whether objectives have been achieved, then success criteria may be a useful source of generic measures. Factors that have been found to contribute to satisfaction of success criteria are useful as progress measures and indicators of likely achievement of desired objectives and outcomes. Fortune and White (2006) reviewed 63 research publications that had focused on critical success factors for projects. The results of their analysis have been divided into necessary conditions, practices and project characteristics and are presented in Table 7.1
111 M e a s u r i n g P e r f o r m a n c e table 7.1 critical success factors for projects identified by Fortune and White
(2006)
Necessary conditions practices project characteristics support from senior
management strong/detailed plan kept up
to date Proven/familiar technology
Clear realistic objectives Good communication/
feedback
Environmental influences
Strong/suitably qualified/
sufficient staff/team user/client involvement Political stability Competent project manager effective change management
/control
Project size/complexity/# of people involved/duration strong business case/sound
basis for project Good leadership different viewoints
(appreciating) management support; clear and realistic objectives and production of an efficient plan.
Through my own work, I have identified four sets of practices (engagement of stakeholders, communication of change, ensuring business integration, making informed decisions) and two contextual variables (stability of context and stakeholder cohesion) as key predictors of project success. Other factors considered as influencing success include organizational culture; behavioral competencies of the project manager; fit between the project manager and the project; leadership; vision; knowledge sharing; social embeddedness and cultural characteristics; change readiness and change implementation capability.
CateGorization
Comparability of measures and benchmarking of performance, whether internally or externally, requires categorization to enable like to be compared with like. Categorization facilitates analysis and enables data to be used in a variety of ways for reporting purposes.
Categorization is done by assigning labels or attributes to distinguish between items of different types. For performance measurement on projects, the first level of categorization is identification of projects as a unit of measurement by differentiating projects from
operations. Measurement may then be done at the level of the individual project, at the program level or the level of portfolios of projects and programs. A corporate project portfolio may be broken down by business unit or division, by geographic location or industry sector.
Categorization systems are rarely simple. They are usually hierarchical in that projects are categorized or labeled one way (for example small, medium, large) and then each category is categorized further, in different ways (Crawford et al, 2006). For instance, small, medium and large project types may be broken down into different groupings based on other attributes. In parallel systems of categorization, several sets of attributes are assigned to every project. Both parallel and hierarchical systems may be combined so that, for instance, all projects may be categorized by division, by size and by phase (initiation, planning, execution, close-out).
In some cases it is useful to use composite attributes for categorization. The most common example is complexity. Although projects may be categorized as being of high, medium or low complexity this is open to a wide range of interpretations. Most organizations that categorize the level of complexity use several attributes, between two and 12 (with an average of five). These attributes include project scope, technical complexity, number of functions and skills and level of ambiguity and uncertainty (Crawford et al, 2006). A useful approach to categorization of complexity is provided by the Global Alliance for Project Performance Standards (www.globalpmstandards.org) in their CIFTER, a table used for categorizing the management complexity of projects. The CIFTER uses seven dimensions of management complexity, rated from low to high or very high to arrive at a score that can be used for comparing projects with similar levels of complexity. A similar instrument (ACDC Table) differentiates management complexity of programs, taking into account governance, stakeholder relationships, program definition, benefits delivery and resourcing.
The most commonly used attributes for project categorization are: application area, nature of work, client/customer, complexity, cost, size, strategic importance, risk level, organizational benefit, deliverables (Crawford et al, 2006). Attributes are selected according to the purposes for which they are to be used but each attribute may be used for different purposes. For instance, the composite attribute, complexity, may be used to guide choice of governance structure, risk mitigation strategy, contract type, selection of methods and tools or choice and assignment of personnel. For performance measurement, two major purposes for categorization are tracking of efficacy of investment in projects and developing project delivery capability within the organization. Measures of time and cost performance could be used for both these purposes.
Categorization is a pre-requisite for reference class forecasting which is based on theories of decision-making under uncertainty (Kahneman and Tversky, 1979). Reference class forecasting relies upon information about actual performance in a reference class of comparable projects and has been proposed as a way of mitigating the effect of optimism bias and strategic misrepresentation on estimates (Flyvbjerg, 2006) in infrastructure projects.