Que cuenten con Certificación en materia de IVA e IEPS bajo la modalidad AA o AAA en

6.3.1 Questions to Require Business or Information Knowledge

The experiment questions required either business or information knowledge to answer. A total of ten questions were selected with each type of knowledge having five associated questions. A question that related to the understanding of a banking concept or the interpretation information in a report was classified as requiring business domain knowledge. A question that related to the data model, classification hierarchies or performing calculation was classified as requiring information structure knowledge. Examples of both types of the questions are provided in Table 6.1.

Table 6.1 : Comparing Business and Information Questions

Business Domain Questions Information Structure Questions

Understanding a business concept

Example Question: What are Risk Weighted Assets (RWA)?

Understanding the classification and calculation of information

Example Question: If a €1,000 loan is past due payment by more than 90 days, what would its Risk Weighted Asset value be using the Standardised Approach?

Interpreting information in a business report

Example Question: Does the change in the bank’s Core Tier 1 Capital Ratio between 2013 and 2012 indicate an improvement in the financial stability of AIB?

Identifying which attributes in a data model support a business report.

Example Question: Which of attributes from the data model fact entities would you use to calculate the Core Tier 1 Capital Ratio shown in Table 2?

The level of difficulty of the questions was designed so that participants would be challenged when completing the tasks but would still be at a level where all participants should be able to attempt the majority of the tasks. The questions were tested to validate their classification and they could be answered with the content in the model. This testing identified a number of gaps in the model content and so became part of the iterative process of semantic model development described in Chapter 5 . The ten experiment questions are listed in Table 6.2 and the full experiment questionnaire is provided in the supporting material.

Table 6.2 : Experiment Questions

# Experiment Questions Knowledge

Type

Difficulty

1 Which of the following is the primary audience of the Pillar 3 Disclosure Reports published by AIB?

Business Easy

2 What is the target Core Tier 1 Capital Ratio set by the Central Bank of Ireland for AIB?

Information Easy

3 What are the different types of risk that must be managed by AIB?

Business Easy

4 What are Risk Weighted Assets (RWA)? Business Medium

5 In relation to Table 2, does the change in the bank’s Core

Tier 1 Capital Ratio between 2013 and 2012 indicate an improvement in the financial stability of AIB?

Business Medium

6 What types of Risk Weighted Assets are used in the calculation of Capital Ratios?

Business Easy

7 Which of attributes from the data model fact entities would you use to calculate the Core Tier 1 Capital Ratio shown in Table 2?

Information Medium

8 If a €1,000 loan is past due payment by more than 90 days,

what would its Risk Weighted Asset value be using the Standardised Approach?

Information Medium

9 In Table 4, would you include a mortgage to a retail customer in the calculation of total Retail Exposure in the Standardised Approach to Credit Risk?

Information Hard

10 The information in Table 6 is supported by the entity Credit Exposure Weekly Fact. Is this entity modelled correctly to support the calculation of ‘average exposures over period’ if corporate customers can change country of operation in the middle of the year? If not, why not?

6.3.2 Metrics of Model Usefulness

Metrics were used to measure and observe the overall and relative usefulness of the ontology and concept maps to the data modellers. The quality framework suggested by Moody and Shanks (2003) emphasises the use of a small number of quantitative metrics when attempting to measure model quality. On completion of each question the participant was asked to identify which of the models was most useful for completing the question and to rate how useful they found that models in answering the question. A five point rating was chosen to allow for comparison with the Osman’s et al. (2011) study. The three metrics chosen are listed in Table 6.3.

Table 6.3 : Metrics for Model Usefulness

Metric Measure Collection Method

Most Useful Model Single choice of Concept Map, Ontology, Both and Neither

Participant was asked ‘Which semantic model was most useful for this question?’

Helpful Rating Scale of 1 to 5, where 1 is not helpful and 5 is very helpful

Participant was asked ‘Please rate how helpful this model was in answering this question.’

Answer Correctness Yes or No Answer provided by the participant evaluated after the experiment session.

The selection of a simple measurement approach also had the benefit of being easy for the participant to understand and complete. An example of the questionnaire format used for all ten questions is illustrated in Figure 6.2.

In addition to the three qualitative metrics on usefulness, a set of five questions were designed to elicit qualitative feedback from the participants once they had completed the tasks. The intention of these questions was to get the participant to reflect on the overall usefulness of the knowledge in the models and to share their thoughts on how the implementation could be improved. While the questions were included in the questionnaire it was intended that they would initiate a discussion with the participant rather than providing a simple yes or no answer. The questions were designed to allow for a comparative evaluation with the qualitative results reported by Osman et al. (2011), Vieritz et al. (2013) and Moody and Shanks (2003).