A complete overview of the system with the Bridging Application, FlightGear instances, switch, server and all transport layer protocols between nodes is depicted in figure 5.19.
Figure 5.19: Overview of the MVP with the Bridging Application, FlightGear instances, switch, server and all transport layer protocols between nodes
5.4
Conclusion
The MVP is realised according to the specifications along with further capabilities to enable multiple pilots to utilise the MVP. The sub research question How can this be translated into a MVP? can be answered; The Ideation and Specification phases have led to a design of the MVP and in this chapter, the MVP is realised according to the requirements with the aid of the open source flight simulator ”FlightGear”, the programming language Python and UNIX based operating systems. This complete setup will be subject to testing and evaluation in the next chapter: Evaluation.
6
Evaluation
This chapter forms the final phase of the Creative Technology Design Process, the Evaluation phase. This chapter describes the evaluation of the realisation of the MVP and aims to elicit experiences and observations from the interaction with the MVP that help answering the main research question. This chapter commences with a description of the user test and the results of the user test. Furthermore it will evaluate to what extent the “Must-have” requirements are fulfilled. In addition, the “Should-have” requirements are also evaluated.
6.1
User Testing
The user test performed in this phase serves two purposes: the evaluation of the fulfilment of the re- quirements set for the MVP and the evaluation of the user interaction of the MVP. The main advantage of this user test is that, next to problems that are known to the designer, also problems that the de- signer is not aware of may reveal them self. Inviting users and allowing them interact with the MVP and observing their possible struggles gives an indication of problems that need to be solved, next to any problems that have already arisen during other phases of the project. Feedback from the user tests along with ideas generated during every phase of the project serve as a base for a set of well-formulated and relevant Recommendations at the end of this thesis.
6.1.1
User Test Setup
The Evaluation is done in the form of a structured interview. It is deliberately decided not to record the sessions in audio or video, as from experience of the author it may occur that the interviewee may hold back his or her feedback. An interview that is not recorded but merely documented allows interviewees to feel less ”watched” and feel inclined to provide more feedback. The manner of assigning tasks to the user has impact on the outcome of the interview, therefore during this user test, the instructions on how to realise a task that are given to the interviewee are kept to a minimum. This is to realise that the interviewee finds a way to realise a certain action by him or herself, which contributes to determining the usability of the product. The user test setup document that has been designed and utilised during this phase, is included in appendix K.
The complete setup of the MVP is set up in a quiet room at the NLR, free from any distractions. Images of the setup are included in figure 6.1 through 6.4.
Figure 6.1: Complete MVP test setup
Figure 6.2:Three main screens of the MVP: From left to right: Delayed UCAV view, map overview and the adversary view
Figure 6.3:Right side view of the MVP
Figure 6.4:Left side view of the MVP
6.1.2
User Test Main Stakeholders
The user test is performed with the stakeholders of the NLR, Jan Joris Roessingh and Gerald Pop- pinga. They were subjected to the user test set up in appendix K in one session in which the MVP was presented.
The choice of the project stakeholders to be the only interviewees during the evaluation has pros and cons. An evaluation with the stakeholders is necessary to evaluate to what extend their wishes are met. Performing this user test on the stakeholders provides an interactive way to evaluate the MVP. However, the stakeholders are highly likely to have a biased standpoint towards the MVP and they fur- thermore possess a substantial amount of inside-information about the MVP. This may give distorted results regarding the research question and the outcomes of this user test. Since the stakeholders are the customers of this project, they will be included in the user test. The documented results of the user test interview are included in appendix L
6.1.3
Flight Test Tjebbe Haringa
In addition to the user test with the stakeholders, a flight test has been conducted with Tjebbe Haringa. Tjebbe is F-16 pilot and has been chief test pilot of the Royal Netherlands Airforce. His expertise in piloting and air combat make him a valuable test subject. In addition, Tjebbe has no further knowledge of the SDCS and will be less biased than the stakeholders regarding the MVP. Tjebbe did not perform the complete user test along with the qualitative interview, but has performed the same tasks as in the user test. Namely flying with joystick with 0, 250, and 1000 ms delay respectively and flying with the SDCS Interface with a latency of 250ms. During Tjebbe’s test, a map of the airspace was placed on the middle screen of the test setup so that it was easier to find eachother during flight. Tjebbe mentioned that during the joystick flying with delay used a different approach to controlling the aircraft. Instead of using a ”closed loop” method where every action is countered, he anticipated the result of the aircraft and corrected with small ”pulses” of joystick movement. Tjebbe was successful in following the adversary aircraft with joystick control under a latency of 250 ms, at this point, the stakeholders (who have no highly developed skill in flying jet aircraft) had difficulties in even flying the aircraft. Tjebbe mentioned to anticipate more on the behaviour of the aircraft, thinking one move in advance. Tjebbe was able to follow the adversary aircraft under latencies of 250ms. A delay of 500ms made following difficult. He mentioned that the latency was not so much a problem in following another aircraft, but for what he called ”high-gain” tasks, such as air to air combat, the latency of 250ms was already too much. He acknowledged that there should be a different form of control than joystick control when there is latency present on the data link. Tjebbe was therefore introduced with the SDCS and was presented the SDCS control interface in which he flew with a 250ms delay. Tjebbe acknowledged that this form of control helps mitigating the increased workload of actually controlling the aircraft under latency altough, with the current state, performing the tasks was not less difficult due to the limited control options. Tjebbe would like to see an option where you can set a heading and consequently, the aircraft will initiate a bank towards that heading. However, the need for a control system such as the SDCS was illustrated.