There are several issues that should be discussed concerning the research pre- sented in this doctoral thesis.
10.2.1 Has the objective been met?
One of the more important questions to address when critically reviewing a thesis is whether the objective has been met or not. The main academic objec- tive of this doctoral thesis was formulated in chapter 1 as:
“To investigate and clarify how to evaluate and revise performance measurement systems.”
Without a doubt, the research has indeed investigated this issue as well as sug- gested a method that explains how to evaluate and revise a PMS (i.e. meeting
On the one hand, the proposed method has several strengths when it comes to evaluating and revising a PMS. First, it is rather simple to understand and to use, which makes it suitable for assisting measurement practitioners. Second, it separates the system level from the individual measure level, which makes it easy to handle different forms of requirements. Third, it allows the company to slowly improve its PMS without making any dramatically changes. Fourth, it can be used independently from how the PMS is structured or on what concep- tual framework that has been used to design the PMS.
On the other hand there are also several limitations with the method. A major constraint is, of course, that the method has not yet been fully tested in reality. Furthermore, from a practical point of view, it also is necessary to describe the method more detailed before it can be directly used by the industry. Many of the tools within the method force its user to rely on subjective analysis. The method also leaves much to the user, which must come up with own ideas about how to make improvements. It can also be very time-consuming to use all available tools within method on a large number of performance measures. However, it is here important to decide what performance measures that need to be thoroughly analysed or not.
In conclusion, it is believed that the research has met the objective to a high degree and answered the formulated research questions. However, there are still some issues within the research area that needs further attention and they are explained in future research (section 10.3).
10.2.2 Academic and industrial relevance
The academic relevance of the research is believed to be significant since it deals with important problems that are still not solved in a satisfactory manner. The scope of thesis is rather unique due to the fact that it concerns an area (i.e. the continuous updating of performance measures) that has been neglected in the performance measurement literature (Neely, 1999), (Medori and Steeple, 2000), (Bourne et al, 2000), (Kennerley and Neely, 2003), (Kuwaiti, 2004). Three important main contributions have been made within this area:
1. The research has from an overall point of view showed how to system- atically evaluate and revise PMS. Opposite to most research within the field of performance measurement, this work has developed means to improve existing PMS, instead of means to design completely new PMS.
2. The research has made it easier to deal with different requirements that a PMS should fulfil by identifying them and separating them into sys- tem level and measure level criteria. Furthermore, the research has also
suggested new ways to deal with the requirements, such as: the concept of system classes and the concept of measure types.
3. This work has also covered an area that few researchers have consid- ered, namely, how to actually work with the design of performance measures in practise. For example, the research has showed what per- formance measure parameters that should be specified. It also has dis- cussed problems related to forming a suitable equation for the perform- ance measure. Finally, it has emphasised many important positive and negative measure properties.
The industrial relevance of the research can be described from various perspec- tives. To begin with, the confusion surrounding terms such as performance and productivity is enormous in industry. Generally, managers have different ideas of what they mean, how to measure them and what that are affecting them. By studying these problems, this research has made important contributions to the industry by, for example:
• Describing how to use the terminology within the field
• Describing strength and weaknesses of many frequently used perform- ance measures as well as of common conceptual frameworks for de- signing PMS
• Describing various key-factors that influence a company’s productivity Furthermore, the research has also developed several tools to be used in prac- tice when evaluating and revising PMS. These tools are in turn simple to un- derstand and to use and make it easier to conduct the process of continuously updating the performance measures in a company.
10.2.3 Relevance of performance measurement
An important discussion is the relevance of performance measures and how they should be used. Many authors describe measurement as an important part of productivity improvement, which is, of course, supported in this research. However, productivity improvement is also highly dependent on other issues. Appropriate measures must be used as well as the acceptance of the measure- ment process is essential to its success as an improvement tool (Sink and Tut- tle, 1989). As described by Daniels (1997), productivity improvement does not for its success rely on the application of specific productivity techniques; it depends on the commitment and creativity of all members of the organisation. In other words, a perfect productivity tool alone cannot increase productivity. Without motivated workers the tool becomes useless and will fail. The ap-
There are also researchers that disagree with the usefulness of performance measurement and argue that knowledge about processes instead of performance measures should guide the improvement work (Johnson and Bröms, 2001). Their point of view is understandable, since the subject of performance meas- urement has in many cases gone too far. For example, many consultants are using the phrase “You are what you measure”, and argue that measures can solve all types of problems in a manufacturing company. Which is, of course, not true. Another problem is that the number of performance measures used tends to increase to an extreme level. To use numerous measures will usually not give a better picture of a manufacturing situation, just result in confusion. Measurements succeed when they are relevant and carefully used as a support tool. Even though appropriate measures have been chosen, they should not be followed blindly, since measures are always just a simplified version of the reality (Tangen, 2003b). Knowledge of the processes in a company is vital to the improvement work. It must therefore be kept in mind that using measures, without knowledge of the processes that are being measured, will most cer- tainly be unsuccessful.
10.2.4 Critical review of the research
There are also some critical comments that should be made to the presented research and how the four research questions (RQ1-4) have been answered. RQ1: Concepts and glossary of terms
To begin with, criticism can be raised to the thoroughly investigation of the terms and concepts used within this field from two perspectives. First, one could say that such a discussion is pointless, since there is no correct definition of terms like productivity. Second, many people would also not agree that the Triple-P model reflects the terminology in a correct way. However, the reason for investigating this issue is significant. Thus, if the definitions were ne- glected, this work would then contribute even more to the existing confusion on the subject. As stated by Sink and Tuttle (1989):
“We have worked with organisations that have given up trying to define productivity. We have sat through numerous presentations where high-level managers and leaders of the productivity and qual- ity movements in their organisations showed slides that said “Pro- ductivity = Quality”. We have read too many articles in prestigious journals that talk about productivity and quality, but failed to opera- tionally define the terms meaning and indicate that the authors do not know what the terms really mean in operational sense.”
Moreover, the Triple-P model should not be seen as an absolute version of how these terms should be defined. Compromises are unavoidable in order to ex- plain the terms and their relations. The advantage with the Triple-P model is that it is a simple model that can erase most of the misinterpretations that sur- round the terms and has commonly been adopted both in industry and acade- mia. In conclusion, people who are not willing to give up their view will of course not accept the suggested terminology.
RQ2: The review of the performance measurement literature
Criticism can be raised to the review that is presented in chapter 4, since there are probably many existing performance measures and conceptual frameworks that have not been considered. However, from a practical point of the view it is always important to limit the contents of a review. Otherwise, it will not be enough space in the thesis for presenting other research results. The intention has been to give a comprehensive overview of the field as possible.
RQ3: Key-factors affecting productivity
The key-factors presented in chapter 5 can be criticised in several ways. First, they are presented from a general point of view and the research have made no attempt to clarify what key-factors that are of most importance or explained what hierarchical levels they should be related to. Second, the key-factors have mainly been identified through literature research. Studies should therefore be made in industry to clearly recognise their relevancy. Third, no tool is given for how to actually determine existing key-factors in a company.
This chapter obviously has its limitations. However, one of the other ongoing sub-projects in the productivity project (Performance improvement - towards a method for finding and prioritising potential performance improvement areas in manufacturing operations (Grünberg, 2003)) has focused this issue in more detail. This is a major reason for only briefly considering RQ3.
RQ4: System and measure level criteria
Criticism can be raised concerning the presented requirements that a PMS should fulfil in several ways. One could first question if it is even possible to satisfy all of the presented requirements? Probably not, but the important thing to remember here is that the specified requirements should be used to identify current weaknesses of a PMS and point out directions of what to strive for. Without knowing a PMS’s weaknesses, there will always be a risk that it is interpreted inaccurately. Furthermore, one could also question how the re-
than that these choices had to be made when the proposed method was devel- oped and that they are highly influenced by practical issues.