The basic principles and building blocks of performance measurement can be applied in various project contexts (Figure 7.2).
113 M e a s u r i n g P e r f o r m a n c e
Some ideas for measuring performance of single projects, multiple projects including programs, portfolios, organizational project management capability and people in projects are provided here.
ProjeCts
Projects are the primary unit for measurement of project-related outcomes, processes and practices. Assessments at the program, portfolio and organizational capability levels will all draw upon one or more aspects of individual project performance so it is appropriate to deal with this first. Measures used at the project level are primarily concerned with monitoring and reporting performance against scope, time, cost and quality baselines.
The focus of measurement is internal to the project rather than for comparative purposes.
For the individual project, standards are only likely to be used to set tolerances for acceptable variation from baseline. Ideally this information will be used to monitor and control project performance but in some cases the focus for collection of performance data at the project level may be less for the benefit of the project and more to meet the demands of reporting cycles and other requirements.
Project managers notoriously have difficulty with cost reporting because many organizations do not have integrated systems that provide timely cost data that are relevant at project level. Where enterprise resource planning (ERP) systems are in place they may help or hinder performance measurement at the project level. If the systems are well tailored to gathering and reporting of project level data then this can increase accuracy and reduce the effort required by project managers for data collection, analysis Figure 7.2 reporting and data flows for project-related performance
measurement
and reporting. If not well tailored and integrated, ERP systems can increase the data collection burden at the project level while reducing timeliness.
Usually the performance data collected at project level will be dictated by requirements for upward management reporting. However, the project manager may choose to use particular measures in order to drive aspects of performance or behavior. For instance, if a project manager is concerned about resource utilization on projects perhaps with a view to ensuring good work/life balance, they may choose to monitor overtime on the project.
As long as it is not overdone, some measures can be changed from time to time to focus attention on particular aspects of performance.
An important tool for measuring performance at the project level is Earned Value Management, EVM (Chapter 16), which integrates scope, time and cost measures in a way that enables forecasting of estimated cost and time performance to completion. EVM is most useful for measuring performance of projects with clear scope definition and tangible outcomes where the amount of the scope delivered at any point of time can be objectively measured. It is less suited to projects with intangible outcomes or unclear scope definition.
Project Health Checks (Chapter 6) may provide the opportunity to benchmark the use of processes and practices on the project with other projects within the organization or industry sector. These forms of assessment are clearly intended to both monitor and drive performance at the project level. Benchmarked project health checks are generally designed to encourage improvement in the way projects are managed.
For effective performance measurement it is important that the metrics used at the project level are meaningful to the project manager and the project team. If they understand how the data is to be used by others they are more likely to ensure accuracy in reporting. There are dangers however. In the absence of a no-blame culture, performance metrics may be manipulated to provide protection or pursue personal political agendas.
MuLtiPLe ProjeCts and ProGraMs
In measuring performance of multiple projects, while the majority of the base data is collected at the level of the individual project, the focus moves from performance against baseline for each project to performance trends across groups or categories of project. This is where categorization, standards and benchmarking become important for purposes of comparison. When organizations look at performance data across all of their projects they will be looking for metrics that provide feedback on issues such as overall financial performance, resource utilization and productivity, benefits realization, customer satisfaction, aggregate risk, accuracy of forecasts, safety and sustainability. They may compare performance of different types of projects, or projects within different divisions, business units or regions. They will set tolerances for acceptability of variance on key measures to be applied across all or groups of projects being undertaken in the organization. This type of information will be reviewed by senior managers and distilled for presentation to the executive and board as a basis for decision making and setting improvement targets.
Measuring performance in programs is similar to general management concern for the performance of multiple projects as described above. The difference is that the program manager will be particularly concerned with progress towards achievement of the specific objectives for the program including delivery of program benefits. At the program level,
115 M e a s u r i n g P e r f o r m a n c e
there is an opportunity for comparison and aggregation of results across the program and assessment of features that are more significant for programs than for projects such as resource utilization, program risk, change management, stakeholder satisfaction and benefits realization. Although there is increasing evidence that effective benefits delivery begins with progressive monitoring at the project level, this is generally considered to be more effectively done at program or divisional/business unit level. Performance metrics in programs will include project specific metrics rolled up as well as metrics that relate to the program as a whole. The focus is on monitoring and controlling performance of the program against baseline as well as utilizing performance measures to drive performance improvement.
PortfoLio LeveL
Measuring performance at the portfolio level is in many ways an extension of performance measurement in the context of multiple projects and programs. Projects remain the primary unit of measurement and there will be a concern for overall financial performance, resource utilization and risk. The specific difference at the portfolio level is the need for performance measures that provide feedback on the alignment of project based activity with corporate strategy. Areas of interest will include alignment of commitment with capabilities, balancing of the portfolio to support strategy delivery and manage risk exposure and performance data that will support prioritization when allocating resources. Performance measurement at this level is critical to good project and corporate governance.
orGanizationaL PM CaPaBiLity
Performance measurement at the individual, multi-project and portfolio levels is primarily concerned with results. However, performance measurement is also important in assessing and driving the improvement necessary to ensure the capability is in place to consistently deliver results at acceptable standards of performance. To develop and improve project management capability you need to know the status of your current capability, and have a plan for where you want to be. In fact, evidence suggests that just maintaining current capability means that you have to be involved in some form of continuous improvement or it will go backward (Thomas and Mullaly, 2008). All of this requires an understanding of the components of organizational project management capability, ways of assessing or measuring those components and standards and benchmarks against which to judge how well you are performing as a basis for planning improvement.
If we think about project management capability as encompassing everything within an organization that relates to delivery through projects, we realize that it involves the whole of the organization and not just the project management community and the projects themselves. It involves the capability of the board of directors and senior management who set the corporate strategy, decide on the portfolio of projects and programs they will undertake and how they will resource them. The executive team also provide the vital link between corporate, program and project governance, between the permanent and temporary or project based parts of the organization, through their role as sponsors (Crawford et al, 2008). The permanent organization or business as usual (BAU) needs to have the capability to accept the fruits of projects and programs
and exploit them to achieve the planned benefits. Organizational project management capability encompasses portfolio, project and program management processes, systems and practices, as well as the competencies and skills of the people who use them. When thinking about assessment of an organization’s overall project management capability there is a tendency to think in terms of models for assessment. A number of models are available for this purpose and they tend to be in the form of ‘Excellence’ or ‘Maturity’
models (Chapter 5).
General management is familiar with business excellence models such as the European Foundation for Quality Management (EFQM) Excellence Model and Baldrige Award which they use to identify, describe and then improve business processes. Similar models, focusing on project and program management, can be used to assess the current status of organizational project management capability, to guide development and for periodic re-assessment to determine and demonstrate progress. The IPMA International Project Excellence Awards and IPMA Delta (www.ipma.ch/2012/ipma-delta/), and the Human Systems 4 Quadrant Model (www.humansystems.net/services.html), which has been continuously developed over the last 15 years by the companies that are members of the Human Systems knowledge networks are excellence models developed on principles similar to those driving the EFQM and Baldrige awards. The focus here is on identifying and assessing those working practices that lead to improved performance (Cooke-Davies, 2004).
The large number of available maturity models have been strongly influenced by the Capability Maturity Model (CMM) developed by the Software Engineering Institute (SEI) of Carnegie Mellon University for the software development process (Chapter 5). Many of these have combined the concept of maturity as established by the SEI CMM with the view of project management presented in the Project Management Institute’s PMBOK Guide (Project Management Institute, 2013), which tends to present a specifically project centric or tactical view.
In 2003 the Project Management Institute released its Organizational Project Management Maturity Model Knowledge Foundation (OPM3) which identifies more than 600 best practices, more than 3,000 capabilities and more than 4,000 relationships between capabilities. They have since developed an OPM3 Product Suite (Project Management Institute, 2008a), and a process for certification of assessors. The UK Cabinet Office has also developed a Portfolio, Programme and Project Management Maturity Model, P3M3 (Office of Government Commerce, 2010), in recognition that maturity questionnaires provide an effective tool for identifying areas where an organization’s processes may need improvement.
Different organizations in different industries will place emphasis on different aspects or components of project management capability. Each organization should therefore choose the model and process best suited to their need as a basis for establishing their current baseline, formulating and driving an improvement plan and conducting periodic reassessment.
Project, Program and Portfolio Management Offices (PMOs) of various types and sizes have become an important aspect and repository of organizational project management capability (Chapter 31). Although research clearly indicates that there is no simple categorization or formula for PMOs (Chapter 31), there is a growing interest in assessing their performance. Available research (Hobbs, 2007) and benchmarking initiatives such as that of Human Systems provide a basis for comparison and baselining and this is often combined with assessment of stakeholder satisfaction through questionnaires and interviews.
117 M e a s u r i n g P e r f o r m a n c e
PeoPLe in ProjeCts
Discussion up to this point has focused on measuring performance in terms of outcomes, processes and practices, but, as indicated in Figure 7.1, the competencies and skills of people are also subject to measurement with a view to on-going performance improvement.
Assessment and development of competence of the people on projects is also a vital aspect of overall organizational project management capability. Assessment of competence requires standards or guides against which measurement can be made. Knowledge and demonstrable performance (use of project management practices) are the two aspects of competence that are most widely addressed in project management standards or guides produced by project management professional associations or standards setting bodies.
Knowledge is the easiest to assess and least controversial aspect of competence and is generally assessed using multiple choice questionnaires such as those used for the Project Management Professional (PMP) exam. Indeed, measures often used by organizations to demonstrate capability to deliver projects are number of training days and number of project personnel who hold qualifications in project management which may include both academic qualifications and those offered by professional associations.
Demonstrable performance or use of project management practices can be assessed against performance-based competency standards such as those of the Australian, British and South African Governments and the Global Alliance for Project Performance Standards (GAPPS). Assessment against performance-based competency standards is generally undertaken by trained workplace assessors. Candidates are required to gather evidence of the use of practices in accordance with performance criteria specified in the standards. The workplace assessor works with candidates, advises and assists them in achieving recognition of competence. Candidates are assessed either as competent at a particular level, or not yet competent. If assessed as competent, a candidate may be awarded a qualification recognized within a government endorsed framework. The Global Alliance for Project Performance Standards (GAPPS) does not award qualifications. It audits and endorses assessment processes and standards as a basis for mutual recognition and transferability of project management and related qualifications. It provides very useful mappings of the comparability of different standards and assessment processes.
Behaviors are the most difficult, expensive and controversial aspects of competence to assess although it is behaviors that differentiate between threshold and superior performance. A number of organizations have developed corporate competency models, identifying the behaviors that are considered desirable and associated with superior performance within a specific corporate context. However these are usually applicable across the organization and not project management specific. Assessment Centers are sometimes used to assess behavioral competencies and can be extremely effective but they are also very expensive.
When dealing with individuals, assessment and development of competence can often be addressed through the certification and qualification processes of professional associations and academic institutions. Professional certification will generally ensure a consistent level of basic knowledge, shared language and threshold performance across a project team.
Superior performance may emerge as a result of the reflective processes encouraged in some academic programs. NASA has found that team assessments provide them with the best returns in terms of performance and capability improvement on projects and information about this is available on the NASA APPEL website (http://www.appel.nasa.gov).
From an organizational perspective, a resource effective approach to assessment of project management competence is to use knowledge tests and self-assessment as a first cut assessment across the entire resource pool, performance based assessment for a subset of more senior practitioners and then behavioral assessments for those people seen to have potential as superior performers. Results can be aggregated to baseline competency profiles across the resource pool, and provide guidance for training needs.
A number of organizations encourage development of individual project management capability through internal accreditation processes and some larger organizations have established relationships with academic institutions to provide a range of development and assessment programs for more senior project personnel.
There is growing recognition of the role of the Program Manager (Chapter 27). The Project Management Institute has produced standards for both Program and Portfolio Management (Project Management Institute, 2008b,c), and offers a Program Manager qualification, the Program Management Professional (PgMP). The UK Government was first to recognize the program manager role and produced a guide titled Managing Successful Programs (Cabinet Office, 2011). Qualifications, including MSP (Managing Successful Programs), MoP (Management of Portfolios) and P3O (Portfolio, Program and Project Offices) are offered on behalf of the UK Cabinet Office by APMG-International.
usinG assessMent to drive ProjeCt ManaGeMent CaPaBiLity iMProveMent
As the saying goes: ‘What gets measured gets done’. To maintain and improve capability, you first need to assess or measure your current capability. You need to know where you are. Using the guidelines discussed above you can determine which aspects of capability you will assess and how to carry out that assessment. Once you have a baseline you can develop a plan for improvement, initiate and resource an improvement program.
Benchmarking is important to ensure that your improvement targets are realistic, achievable and relevant. They are also important to ensure that you aren’t fooling yourself about your capability. Networking with other organizations can help you to identify improvements that have worked for others.
Assessing once is not enough. You need to have a regular program of assessment whether this is for single projects or programs, for the PMO, for individuals and teams or for overall project management capability. These assessments drive project management capability improvement providing evidence of success and targets for moving forward. As research into the value of project management has demonstrated, organizations that stop focusing on value, or believe that they are done, stop demonstrating value, and the act of not enhancing value appears to destroy value (Thomas and Mullaly, 2008). Assessment demonstrates both value and improvement. A regular program of assessment maintains improvement momentum.