6.5.1. Bias and subjectivity
There was only one researcher involved during data collection, modelling, merging, analysis, and synthesis, and though this contributed to semantic consistency, it is possible that the study encountered investigator bias (Shenton, 2004). Every effort was, however, made to maintain objectivity: peer scrutiny, as recommended by Shenton (2004), was invited during
Page 123
the research project from other PhD researchers, through on-demand reviews and quarterly group meetings. Questions to the respondents were posed in an open and professional manner, and the researcher endeavoured to maintain self-awareness to avoid any actions that could bias the responses to ensure they were the results of “the ideas and experiences of the informants” and not those of the investigator. Also, second and third party reviews were invited, as mentioned earlier.
The stakeholders identified were involved in this research from the start of the study at each airport till the end of the project. This meant that to quite an extent, the person providing the data at the beginning was also the person providing feedback on how accurately that data was interpreted for that case study. While this contributes in minimising misunderstanding, it limits the extent to which the correctness of the data could be tested.
Also, in some cases, two stakeholders fulfilling a similar role at two different airport cases would describe a process segment differently, but on closer scrutiny during the merge and subsequent questioning, the researcher found no significant difference, and corrected the model to ensure semantic consistency. While introducing a slight complexity from a correctness perspective, regarding model validation, this does add a layer of corroboration for those segments that are similar. This research has attempted to minimise biases and subjectivity through repeated checks of quality and validity, as described in earlier among other measures taken to ensure the research is as objective and correct as possible.
6.5.2. Research implementation
A limitation for implementation was that part of the software required to enable configurability in the artefact had been unavailable for this research at the time it was conducted. As explained in Chapter 6, the complete Synergia toolset did not yet support BPMN. Part of the toolset was available and successfully utilised – namely the modified model editor Signavio; Questionnaire Designer for developing the questionnaire model; and Quaestio for translating this model into a questionnaire interface from which selected answers are captured in an output file. The software unavailable for use with BPMN at the time included C-Mapper to map the configurable gateways to appropriate facts from the questionnaire model; Process Configurator to apply the answer file from Quaestio to the merged model via the mapping from C-Mapper to conduct configuration; and Process Individualizer to extract the individualised process model from the merged model and present it to the user. While developing the missing software was beyond the scope of this
Chapter 6 Evaluation of quality
Page 124
project, the principle of configurability was successfully demonstrated manually using real processes and real scenarios from the domain. As a result, the working concept could be evaluated ex-ante.
6.5.3. Usage
The domain knowledge on which the artefact reference model is based is Australian airports for international departures, therefore it is likely that there exist different needs and contexts relevant to the global perspective have not yet been captured. The model also needs to be extended to other parts of facilitation such as international arrivals, transit/transfer, and domestic processes. In addition, while version management is out of scope of this research, it is central to the notion of an extensible and reusable tool. Any significant process changes would render the tool out-of-date without suitable measures, which must be explored in future research as discussed later in section 7.4. Such issues do not impede the research objective, however, as it should be remembered that the intention of this project (stated in Chapter 1) is to provide the means to develop a global artefact, rather than to provide the completed artefact itself.
As the artefact is a working proof-of-concept rather than a complete product, product adoption has not been studied. In order to do this, the researcher would have needed to witness and measure - e.g. through observation of its usage or via usability studies - the implementation and utilisation of the final artefact over a period of time, which was not possible. Usability and ease of use were, however, assessed through ex-ante evaluation and anecdotal feedback from business representatives indicated potential usefulness.
Finally, using the artefact requires knowledge in both the application domain and BPMN, as mentioned by La Rosa (2009). Being relevant to the airport passenger flow, the user requires an understanding of what problem area they are addressing in order for the tool to be useful.
They also require at least basic training that enables them to read and interpret BPMN, as the entire reference model is represented in that notation.
Page 125
6.6. Chapter summary
Evaluation of a design science artefact is a critical activity (Johannesson & Perjons, 2014).
This chapter described the ex-ante evaluation of the artefact according to the criteria established in the research requirements of Chapter 3. It explained how the design of the reference model is injected with checkpoints and considerations to ensure syntactic, semantic and pragmatic quality. Some of these included ensuring that all configuration points and related dependencies were shown, and that the models were as concise and correct as possible, balancing this with the completeness of content and expressiveness of the models. It also discussed aspects around technological dependence of the tool, which could only be demonstrated in concept and be manually configured until the completion of the appropriate software, which was not part of the scope and occurred after project completion.
Many measures were taken to ensure that the content of the artefact is reliable and valid.
Reliability was ensured by using dependable sources of data and drawing methods from extensive scientific literature. Validity of the research outcome was ensured through involving a broad selection of domain experts who provided the data and corroborated it.
The approach used is well-documented and it is expected to yield similar results, if repeated by another researcher under the same conditions and with the same checks.
The final and most important indicator of quality is to determine whether the tool can successfully fulfil the purpose it was created for. This was done extensively in the final section of this chapter, explaining how the artefact assists or supports decision making for the airport user. In summary, the artefact efficiently and precisely elicits the contextual needs of the user based on a set of modelled empirical evidence, and with the specific contextual needs (identified by that particular user) efficiently customises the evidence and presents it in a format that is a combination of visual and textual information. The user can now use this tailored set of processes as functional requirements to inform a broad range of design decisions for the airport with their envisioned contextual characteristics, in terms of spatial, strategic, administrative, or resourcing decisions, to name a few. This artefact reduces complexity for the user, can be used by anyone who has learned the global standard notation BPMN that is used to describe the model, and finally can be adapted and extended to accommodate any number of airport process models and their contextual factors.
Chapter 6 Evaluation of quality
Page 126
While the study provides many benefits, some limitations have been identified and addressed as far as was possible. Data bias and subjectivity of the researcher conducting the study was mitigated by numerous reviews, validations, and quality checks performed. Issues were encountered in implementing the configuration, in that part of the toolset was notation-dependent and unavailable for use with BPMN at the time, therefore the evaluation by users of the full automated configurable product could not be performed, though evaluation was performed for the design and development of each component of the artefact. Lastly, the model needs to be extended to other airports and facilitation processes to be of benefit as a reference; product adoption has not been tested; and basic training in BPMN is a prerequisite to understand the model.
This chapter discussed the usefulness of the artefact in how it answers the requirements posed in the overarching research objective in Chapter 1. The next chapter discusses how the research questions are answered through completion of the contribution, along with providing reflections and overall conclusions from the work done in this project.
Page 127