5. RESULTADOS DE LA INVESTIGACIÓN
5.1. Resultados del análisis de las WebQuests de Primaria
As stated above, Rodriguez et al. (2009) found the frameworks they investi- gated to be inadequate for the objective identification and quantification of inter-KPI relationships. The reasons for these dismissals are founded on basic flaws and shortcomings in the two analysis techniques used by these frame- works. These two analysis techniques are subjective analysis, and pair-wise correlation analysis.
Although these analytical methods are viable and employable methods in other applications and research, they are inappropriate for the requirements of this study; requirements that are adopted from Rodriguez et al. (2009) and stated in Section 1.2. It is noted that there are tools available at the PM and PMa context that may be used in support of multi-criteria decision making, and can possibly be employed to identify the aforementioned relationships. According to da Silveira (2005), these tools are referred to as Multi-Criteria Decision Analysis (MCDA) methods and are classified as follows:
• Objective programming. • Scoring models.
• Hierarchical techniques. • Deployment techniques.
MCDA methods generally develop a ranking of varying competitive at- tributes or priorities, according to predetermined criteria, to maximise per- formance (da Silveira, 2005). However, Rodriguez et al. (2009) state that all MCDA methods commonly involve, or are dependent on, subjective decisions at any point, or are inadequate for the objective identification and quantifi- cation of inter-KPI relationships. It is therefore important to explain the problems found with subjective analysis and pair-wise correlation analysis to identify other suitable methods to accomplish the aforementioned.
2.5.1.1 The error in subjective analysis
Subjective analysis, according to Rodriguez et al. (2009), is an inadequate tech- nique to employ in a framework which aims to identify inter-KPI relationships between a set of KPIs in an objective manner. Rodriguez et al. (2009) state subjective analysis is easily influenced by the biased opinions of analysts, and therefore cannot be considered a mathematically accurate and reliable analy- sis technique. Any introduction of subjective analysis into the computational elements of a framework compromises the mathematical validity of the results; the framework cannot claim to yield objective results (Rodriguez et al., 2009). 2.5.1.2 The deficiency of pair-wise correlation analysis
Cai et al. (2009) attempted to categorise inter-KPI relationships into three groups: parallel, sequential, and coupled. However, the assessment completed by Cai et al. (2009) only considers the strong cause-effect relationship between two KPIs. As stated in Section 2.4.6.2, there may be other influences caused by third party KPIs that may have changing affects on the relationship between the first and second KPI. This problem is magnified when such a pair-wise correlation analysis technique is employed to identify relationships between a
large set of KPIs. It is for this reason why Rodriguez et al. (2009) deems pair-wise correlation analysis as an inadequate technique for employment in frameworks that aim to identify inter-KPI relationships between a set of KPIs.
2.5.2
Inadequate Frameworks For Identifying Inter-KPI
Relationships
As mentioned earlier, Rodriguez et al. (2009) found very few frameworks that attempt to identify and quantify relationships between performance elements. The following discussion briefly covers each of the frameworks investigated by Rodriguez et al. (2009), highlighting their objectives and the reason for their dismissal.
Youngblood and Collins (2003) developed a methodology to quantify trade- off issues between performance measures used on a Balanced Scorecard. This methodology employs Multi-Attribute Utility Theory (MAUT), a quantita- tive analysis technique which expresses the advantages or disadvantages of multiple-attribute outcomes in terms of the advantages of each attribute con- sidered alone (Torrance et al., 1982). MAUT was employed by Youngblood and Collins (2003) in a BSC framework to evaluate trade-offs between perfor- mance measure options and their respective effects on performance objectives. Rodriguez et al. (2009) state, however, that the methodology is limited in its analytical capability due to the use of correlation analysis. Due to the reasons stated in Section 2.5.1.2, it is dismissed.
The methodology developed by Cardona Siado and García (2005) also im- plements the BSC, and aims to identify inter-KPI relationships using two of BSC’s perspectives: internal perspective, and innovation and learning per- spective. The proposed methodology is composed of four steps: formulation of quality strategy, strategic map design, verification, and strategy execution (Cardona Siado and García, 2005). The MICMAC method is employed in the verification stage, and is described by Elmsalmi and Hachicha (2013) as a structural modelling technique. It describes a system using a matrix linking up its constituent components, identifying the influential, dependant and es- sential variables critical for system evolution. According to Rodriguez et al. (2009), the MICMAC method is a subjective process and is therefore dismissed. Bauer (2005) suggests a framework for the reduction of a large set of perfor- mance metrics to the most important, or useful, measures; measures that are also uncorrelated. The initial description of this framework showed promise, however, the significant use of correlation analysis to “sort” the set of measures makes it an inadequate framework, according to Rodriguez et al. (2009). It is acknowledged that using correlation analysis would aid in the understanding
of inter-KPI relationships, but as mentioned in Section 2.5.1.2, a large number of possible relationships are overlooked.
Rodriguez et al. (2009) expands on another framework, called the Quantita- tive Model for Performance Measurement System (QMPMS), fearing possible confusion between its deliverables and the those of the aforementioned frame- works. Considering this, the relative ease of sourcing this framework from the literature compared to those previously discussed, deemed it beneficial to discuss this framework in greater detail.