=TAMR
2. TASAS DE APALANCAMIENTO
5.3 VISIÓN
Here, we present the data collected from quantitative results. IBM SPSS Statistics software (version 20)7is used to analyse the results. Given three sections of the questionnaire, collected data is categorised into three distinct sub-categories as follow.
7http://www-03.ibm.com/software/products/en/spss-stats-standard
Results and Discussion 112
Background Knowledge of the Participants
Prior to using the CoS software, participants were asked to answer the first section of the questionnaire which discusses their background knowledge about sustainability concepts, reporting organisations (e.g. GRI, OECD, UN) and CoS methodology. This section contains five Likert questions with the answers: “Very Familiar”, “Somewhat Familiar”, “A Little Familiar” and “Not Familiar”, which are ranked from 1 to 4 respectively. The responses to each question totalled 26, which indicates the fact that all users participated in this section.
Since participants’ responses provides a distribution of a single variable, for data analysis, we used descriptive statistics of the frequency of responses while each response is correspon-dence to a numeric value. The results are shown in the histogram in Figure 5.4, where the x axis represents four types of responses in each question and the y axis summarises the corresonding mean value.
Turning to the type of the responses: “Very Familiar” was chosen the least (zero) in Q3 and Q4. In contrast, “Somewhat Familiar” was the most frequent response with a mean value of 15 in Q1, which is followed by “Not Familiar” as the second most frequent response with a mean value of 13.50 in Q3 and Q4. The other type of response, “A Little Familiar”
scored differently ranging from 5 to 10 of the mean value in all questions.
Looking at the questions, almost two-thirds of the users ranked their knowledge level with Q1 as “Somewhat familiar”, whereas the others ranked it as having various degrees of familiarity. Similarly, on average fewer than two-thirds of users ranked Q3 ad Q4 as “Not Familiar”, while the others had “Somewhat” or “A Little” knowledge about these questions.
There were no users who had been very familiar with the aforementioned questions. Finally, the participants’ knowledge about Q2 varied in four types of the responses, where almost half of users were “A Little Familiar” or “Not Familiar”, and the other half scored this question as “Somewhat Familiar”; a few users had ranked their knowledge towards Q2 as
“Very Familiar”.
These results can be interpreted in a way that most participants are familiar with sustain-ability concepts at a reasonable level, whereas, only a few – almost one-third of participants – had known about reporting organisations specifically. In addition, two-thirds of the
par-Results and Discussion 113
Q6 Q7 Q8 Q9 Q10 Q11
N Valid 20 20 20 20 20 20
N Missing 6 6 6 6 6 6
Mean 1.800 1.400 1.900 1.650 1.600 1.700
Median 2.00 1.00 2.00 2.00 1.00 1.500
Std. Deviation .767 .502 .718 .587 .820 .864
Range 3.00 1.00 2.00 2.00 3.00 3.00
Minimum 1.00 1.00 1.00 1.00 1.00 1.00
Maximum 4.00 2.00 3.00 3.00 4.00 4.00
Table 5.2: Descriptive Statistics of Six Questions - Perceived Usefulness
ticipants had some familiarity with CoS methodology.
These findings prove our participant hypothesis (Section 5.2.3) that most participants satisfy the condition of familiarity with sustainability domain and the CoS methodology.
However, only a few experts who know about sustainability indicators sets. It can be con-cluded that the user study experiment is conducted for the proper sample size of participants who have some prior knowledge about the domain. Therefore, their responses are accurate and their feedback can lead us to a more reliable analysis of the data.
Overall Usefulness
The second section of the questionnaire measured perceived usefulness of the CoS software.
This section had six questions with five response types: “Extremely Likely”, “Slightly Likely”,
“Neither”, “Slightly Unlikely” and “Extremely Unlikely”, which ranked from 1 to 5 respec-tively. Lower values represents higher satisfactions of the participants with regards to the overall usability of the software. As discussed in Section 5.2.2, the questions’ topics are corre-spond to to the usability variables adapted from the study by McGrenere et al. [2002] which are discussed in Section 5.2.2. We conducted the Spearman’s rho test to identify significant correlations between the usability variables (given in Section 5.2.2) whose the questions are designed based on.
Tables 5.2 and 5.3 illustrate the descriptive and correlation results in a tabular form. The Null Hypothesis exists when there is no significant correlation and alternative hypothesis is
Results and Discussion 114
Table 5.3: Spearman rho Correlations for Six Questions - Perceived Usefulness
Q6 Q7 Q8 Q9 Q10 Q11
Q6
Correlation Coefficient 1.000 .656∗∗ .515∗ .486∗ .715∗∗ .745∗∗
Sig. (2-tailed) – .002 .020 .030 .000 .000
N 20 20 20 20 20 20
Q7
Correlation Coefficient .656∗∗ 1.000 .694∗∗ .685∗∗ .895∗∗ .747∗∗
Sig. (2-tailed) .002 – .001 .001 .000 .000
N 20 20 20 20 20 20
Q8
Correlation Coefficient .515∗ .694∗∗ 1.000 .657∗∗ .549∗ .477∗
Sig. (2-tailed) .020 .001 – .002 .012 .034
N 20 20 20 20 20 20
Q9
Correlation Coefficient .486∗ .685∗∗ .657∗∗ 1.000 .710∗∗ .369
Sig. (2-tailed) .030 .001 .002 – .000 .109
N 20 20 20 20 20 20
Q10
Correlation Coefficient .715∗∗ .895∗∗ .549∗ .710∗∗ 1.000 .705∗∗
Sig. (2-tailed) .000 .000 .012 .000 – .001
N 20 20 20 20 20 20
Q11
Correlation Coefficient .745∗∗ .747∗∗ .477∗ .369 .705∗∗ 1.000
Sig. (2-tailed) .000 .000 .034 .109 .001 –
N 20 20 20 20 20 20
∗Correlation is significant at the 0.01 level (2-tailed)
∗∗Correlation is significant at the 0.05 level (2-tailed)
valid when there is a strong correlation. The sample size is shown by N , which is 20 in our user study indicating that only 20 participants out of 26 completed this section. The results indicate that Q7 shows strong correlations with other five questions. In particular, the highest correlation value is between (Q7,Q10) of 0.895, which is significant at the confidence level of .01. The second highest correlation values are seen between pairs of (Q6,Q11) and (Q7,Q11) of 0.745 and 0.747 respectively, which are also significant at the confidence level of .01. In contrast, the weakest correlation exists between pairs of (Q9,Q11) and (Q6,Q9) with the values of 0.369 and 0.486 respectively.
These figures support some of our hypothesises given in Section 5.2.3 but reject others.
Q7 is about improving participants’ performance using the CoS software and it has strong
Results and Discussion 115
N Mean Std. Deviation Minimum Maximum
Q13 19 2.47 1.35 1.00 5.00
Q14 19 2.63 1.06 1.00 5.00
Q15 19 2.74 1.19 1.00 5.00
Q16 19 2.90 1.15 1.00 5.00
Q17 19 2.26 1.15 1.00 5.00
Q18 19 2.32 1.21 1.00 5.00
Q19 18 1.72 .83 1.00 4.00
Q20 19 1.80 .79 1.00 3.00
Q21 19 1.79 .63 1.00 3.00
Q22 19 1.79 .85 1.00 4.00
Q23 19 1.68 .88 1.00 4.00
Q24 19 2.32 1.20 1.00 5.00
Table 5.4: Signed Wilcoxon Descriptive Statistics
correlations with all other questions, which focus on individual aspects of the usability of the software. This supports the satisfaction hypothesis that says most participants will be satisfied with the CoS software overall. Furthermore, Q11 suggests using the CoS software is a useful tool at sustainability assessment and has strong correlations with two other ques-tions, which break down the usability aspects into two sub-tasks namely: Accomplishing tasks more quickly (Q6), and improving performance (Q7). This partially supports the us-age hypothesis that the CoS software will improve the participants’ ability at sustainability assessment. However, the weak correlations between pairs of (Q9,Q11) and (Q6,Q9) reject the hypothesis that using CoS will enhance the effectiveness of sustainability assessment, which is the content of Q9. The reason could be the “effectiveness of the software” was not clear for all participants, who ranked it very low. We think the reasons for this are related to the technical limitations of the current implementation of the software which are listed in Section 5.3.2.
Ease of Use of Browsing Indicator Sets
The third section of the questionnaire aimed at comparing users’ satisfaction with the per-ceived ease of use of two ways of browsing indicators: Tabular view versus Circular view.
Twelve 5-point Likert scale questions are paired for each view using the variables discussed
Results and Discussion 116
Table 5.5: Signed Wilcoxon Test - Sigfinicant Difference and Z-value - Perceived Ease of Use
N Mean Rank Sum of Ranks
Q13-Q19
Negative Ranks 7a 7.50 52.50
Positive Ranks 4b 3.38 13.50
Ties 7c – –
Total 18 – –
Q14-Q20
Negative Ranks 10d 6.90 69.00
Positive Ranks 2e 4.50 9.00
Ties 7f – –
Total 19 – –
Q15-Q21
Negative Ranks 11g 6.41 70.50
Positive Ranks 1h 7.50 7.50
Ties 7i – –
Total 19 – –
Q16-Q22
Negative Ranks 13j 7.35 95.50
Positive Ranks 1k 9.50 9.50
Ties 5l – –
Total 19 – –
Q17-Q23
Negative Ranks 8m 5.19 41.50
Positive Ranks 1n 3.50 3.50
Ties 10o – –
Total 19 – –
Q18-Q24
Negative Ranks 2p 2.50 5.00
Positive Ranks 2q 2.50 5.00
Ties 15r – –
Total 19 – –
a. Q19<Q13, b. Q19>Q13, c. Q19=Q13 d. Q20<Q14, e. Q20>Q14, f. Q20=Q14 g. Q21<Q15, h. Q21>Q15, i. Q21=Q15 j. Q22<Q16, k. Q22>Q16, l. Q22=Q16 m. Q23<Q17, n. Q23>Q17, o. Q23=Q17 p. Q24<Q18, q. Q24>Q18, r. Q24=Q18
in Section 5.2.2. The questions in this section also have the same type of the responses from previous section. For analysing the results, we used the Signed Wilcoxon test for the paired responses, for example Q13 (Tabular view) versus Q19 (Circular view).
Tables 5.4 and 5.5 feature descriptive and rank results of the Wilcoxon test. The results suggest that questions related to the Circular view rank lower than Tabular view, in which, lower values indicate greater satisfaction. As a result, four pairs out of six questions – (Q14,Q20), (Q15,Q21), (Q16,Q22) and (Q17,Q23) shown in Table 5.6 – ranked positively at significant difference of .05. This indicates that users were more satisfied working with the
Results and Discussion 117
Table 5.6: Signed Wilcoxon Test - Perceived Ease of Use
Q13-Q19 Q14-Q20 Q15-Q21 Q16-Q22 Q17-Q23 Q18-Q24
Z -1.749b -2.434b -2.516b -2.745b -2.326b .000c
Asymp. Sig. (2-tailed) .080 .015 .012 .006 .020 1.000
a. Wilcoxon Signed Ranks Test b. Based on Positive Ranks
c. The Sum of Negative Ranks Equals the Sum of Positive Ranks
CoS features using the Circular view rather than the Tabular view. Those features in which the Circular view is more preferred are: (i) Easy to understand what to do, (ii) Interaction is clearer and understandable, (iii) It is more flexible to interact with and (iv) It is easy to become skilful in its use.
Further analyses of the results indicate several findings. First, the Circular view is the preferred interface for browsing heterogeneous indicators by most participants. This confirms the functionality hypothesis, given in Section 5.2.3, which is using Circular view facilitates most features of the CoS software. Second, the learnability hypothesis of the Tabular view is not supported because the Wilcoxon signed test did not find any significant difference between the scores of the two views for the question of “Learning to operate with which view is more easier?”. These findings conclude that the question, “Which view assists users more to use CoS software?” does not have a definite answer. It depends on which feature of the CoS software are considered. Although, the Circular view is preferable for most reporting procedures, the Tabular view is also ranked higher for some of the tasks. Subsequent analyses of qualitative participants’ feedback clarify these points.
Results and Discussion 118
Tabular view Circular view
1. The Tabular view is easier to browse and compare among indicator sets.
2. The Tabular view is better for specific searching.
3. The Tabular view is less visually ori-ented and more complex to use. Its ex-actness makes it ideal for reporting spe-cific values, while simultaneously chal-lenging to understand during the first use.
4. For complicated models with numerous indicators, the Tabular view would be more efficient and usable.
5. The Tabular view is more preferred be-cause there are many text related to each indicator and would be useful to view them all together at one.
6. The Tabular view as it represents the environment in a more holistic and re-alistic manner.
7. In Tabular view it would be helpful if indicators could be ’refined’ and sorted based on criteria such as ’tags’, key-words, themes etc.
8. In the Tabular view browsing is slow, and a keyword search did not throw up the kind of indicators.
9. There was too much text involved with the Tabular view, which made it diffi-cult to obtain an overall picture of the different indicators.
1. It provides a comparative visual expe-rience of the assessment process.
2. The Circular view is better for a graphic display which then could be use for re-porting more easily, quicker and more pleasant way.
3. It is visual and communicates to the user more easily and it is visually easier to navigate.
4. It is easier to see several different in-dicators simultaneously. This is im-portant for comparison of indicators as some are very similar.
5. The Circular view appears to assist gen-eral indicator browsing and give short details. Also indicators can be viewed according to the individual frameworks e.g. OECD.
6. The Circular view is far easier to use because it only requires the user’s in-tuition to figure out. Each piece of the circle can easily be identified and ma-nipulated.
7. It would be useful to have the option to select additional areas of information per each section of the circle by sim-ply clicking with the mouse on a single segment of the circle.
8. The Circular view may be more appeal-ing for users with a language barrier.
9. The Circular view works well as long as the indicator can easily be categories against the four parameters.
Table 5.7: Users’ Insight of Working with Two Views
Results and Discussion 119
Which View is Preferable?
Q25. The software ultimately aims to help an organisation to select indicators for measuring a series of critical issues that a stakeholder group has identified.
In addressing this goal, which view is preferable?
This question investigates the insight of users’ experiences of working with the two ways of browsing indicator sets at the end of third section of the questionnaire. In Table 5.7, nine features are listed for each view extracted from users’ responses. While it is not appropriate to compare individual responses, some conclusions can be made after reviewing the responses from both views.
The Tabular view is preferred for its simplicity in presenting indicators in a table format, having a search functionality, and being less visually oriented. However, this view fails in providing users with information about the structure of indicator sets, and displays too much text at a time, which is not visually pleasant. On the other hand, the Circular view is pre-ferred by most users as providing several advantages including: comparative visual experience of assessment process, better graphical display, being easier to navigate and communicate visually and providing short details about indicators with distinguishing their domains and sub-domains. The downside with this view is: indicators must be correlated to at least one of the Circles of Sustainability.