Different patterns in interaction could be found during the experiments. When looking at the interaction some problems came forward. The main confusion that occurred in both conditions was that of the usage of the draw and edit functionality. The draw function is tap based: the users start the function and have to tap on the map to create waypoints and have to finish the waypoint by tapping the last waypoint. The edit func- tion is drag based, when the edit function is started the waypoints become draggable. Waypoints can be deleted by tapping them and can be created by tapping inactive ones. What happened is that when participants wanted to finish the route in the edit mode they tapped the last waypoint resulting in a deletion of that waypoint, while they had
to use the save button to finish the route. This happened frequently during the experi- ments.
Another problem that occurred was that when finishing a route in the draw mode by tap- ping the last waypoint the sometimes double or triple-tapped the last waypoint resulting in a faulty detection and the creation of new waypoints, most participant corrected this with the edit mode.
When looking at patterns in the interaction it was found that most participants used the draw mode to roughly draw the route on the map and used the edit mode to create and fine-tune more waypoints. This can be explained by the fact that the edit mode directly updates the route statistics when the waypoints are dragged as opposed of the draw mode where the statistics are updated when a new waypoint is added.
The system warned the participants in the draw and edit mode when the endurance of an asset was exceeded by coloring the route red and showing a warning message with the amount of exceeded endurance in nautical miles. The participants took more notice of the warning message than the visual feedback. The warning message would only dis- appear when the route was corrected.
Another pattern that could be seen was that of the time-slider. The time-slider was used much more in the AM condition than in the MM condition.
It was noticed that there were more discussions between the team members in the MM condition about route plan choices to be made. The usage of the set slider and focus buttons was low in both conditions, participants sporadically used the functionality and the show/hide functionality of the scheduler and intel-box was only used when the plan was finished. The reset map view and reset time view buttons were used the most in both conditions. No real differences between the usage of these features in the two con- ditions could be noticed.
In both conditions different collaboration styles could be observed during the experi- ments. For instance the active discussion style, where both users were discussing the plan without using the system. The active work style, where one person was working and the other would watch and give comments. The disengaged style where one person would work on the plan and the other person would be watching passively. The last style is the thinking style where both users were quietly thinking for themselves without using the system.
occurred more in the MM condition. No differences were noticed for the other styles.
6.11
Conclusion
In this chapter, the results of the different measurements were covered. The discussion of the results, as well as the conclusion of the study can be found in the next chapter.
Discussion and conclusion
7.1
Discussion
In Chapter 1 we introduced the following research question:
“How can one design a multi-touch tabletop mission planning system that supports decision-making, collaboration and shared situational awareness?”
The findings can partially answer the research question.
More specific questions were defined to help answering the main research question. Be- low we discuss each of the questions based on the findings from our study.
Research questions The aforementioned questions were introduced to cover the
components that should lead to better mission plans such as usability, workload, au- tomation and situation awareness [4].
For all cases that are discussed it seemed that the low number of participants had a great influence on the results since there were hardly any significant differences. Explanations will be given for the trends that were found.
The first question investigated the high-level subjective view of usability in the two pro- totypes. To investigate this the SUS (system usability scale) was used. Although the overall SUS scores in both conditions did not show significant differences, the scores were very high, the MM condition scored 75.42 and the AM condition 77.08, meaning that the scores are positioned between good (71.4) and excellent (85.5) usability [117]. We expected that the AM condition would score better on all individual items, but this
was not the case. In 5 items the AM condition scored poorer or slightly poorer: “I think that I would like to use this system frequently”, “I found the various functions in this system were well integrated”, “I found the system very cumbersome to use”, “I felt very confident using the system” and “I needed to learn a lot of things before I could get going with this system”.
Since the AM condition with its automation approach should allow the users to focus more on planning instead of measuring, it should be more easy to use. The item “I found the system very cumbersome to use” could have scored lower in the AM condition because of more visual data and maybe more cluttering on the map, where the MM condition only displayed a line for the measurements. This could also apply for the items “I found the various functions in this system were well integrated”, and “I felt very confident using the system”. Since the measurement-tool in the MM condition was a function that could be manipulated by user input they could have the feeling of having more control over the system in the MM condition.
The second question focused on the workload in the two conditions. This was inves- tigated with the NASA-TLX workload scale and the Team Workload Awareness scale. The NASA-TLX results did not show significant differences in workload between the two conditions. With a score of 5.25 in the MM condition and 5.23 in the AM condition the workload was virtually equal in both conditions. As the scale ranged from 1 (low) to 10 (high), it can be stated that the overall workload was medium in both conditions. We expected that the workload would be lower in the AM condition, since fewer actions must be performed for the measurements in this condition as was also seen during ob- servations. Nonetheless it seems that the participants from the MM condition did not think that they had a higher workload. A reason could be that people compensated by spending more effort on the planning in the AM condition, which could be connected to the better scores of the plan quality in the AM condition.
When looking at the results of the individual items it can be seen that the frustra- tion level scored (insignificantly) higher in the AM condition. I do not have a good explanation for this, but it must be mentioned that the multi-touch tabletop, although frequently calibrated, was sometimes very sensitive and thus touch actions could be mis- interpreted and triggered something else on the screen than was intended. That this was frustrating was also stated by some participants on the open questions.
As an extra indication of the shared workload, the subjects had to estimate the overall workload of themselves and the overall workload of the other participant. The MM
condition and the AM condition scored, again, almost the same on the self-estimation (5.39 and 5.33). But the AM condition scored higher on the workload estimation of the other participant with 5.39 versus 5.11 in the MM condition, which could be explained by the fact that some participants collaborated better than other users. This outcome could be explained if there was better collaboration and teamwork in the MM condition, which is discussed next.
The third question investigated the effect on the difference of teamwork in both con- ditions. The teamwork scores were high with a score of 83.07 in the MM condition and a score of 82.54 in the AM condition. This can be explained by the fact that the experiment and tasks had a collaborative setup. Although it was expected that the AM condition would score better on teamwork, this was not the case. It was expected the AM condition would score better because the subjects in the AM condition had to focus less on measuring. We assume that as a consequence they would spend more time on discussing the asset planning, which should lead to higher collaboration and more team- work. From the observations this seemed to be the case. A higher (insignificant) score of the AM condition was found for the question where the participants had to indicate how similar their understanding was.
The fourth question covers the influence on situation awareness of both prototypes. It is was again expected that the subjects in the AM condition would show a better situa- tion awareness. This was expected because the animated furthest-on-circle should give better insights in time and space, which is crucial for situation awareness [4]. The AM condition scored higher on both the scales and on the independent items, nonetheless these differences were not significant.
In order to gain additional data to answer the main research question, the sensor cov- erage of the plans and the duration of the routes of the assets of all participants were calculated for both conditions. It was expected that the AM condition would have a higher coverage, since it should be easier to see the overlap of the sensor range with the possible position of the smuggler in time in that condition. Although without a significant difference, the AM condition had a higher average sensor coverage with 2737 nautical miles than the MM condition, which had an average coverage of 2112 nautical miles. When the assets are examined separately the AM and MM condition scored equal on the OPV - P110 asset, and the AM condition scored better with the Ship Borne Helo and the MPA asset, where the Ship Borne Helo showed a significant difference. The MM condition scored better with the OPV - P114’s sensor range showing a significant
difference. This can not be really explained.
That the overall coverage is higher in the AM condition can be explained by the fact that the users in this condition can see all possible locations of the smuggler at a glance, where the users in the MM condition have to first measure the smuggler’s location and have to edit their asset’s routes to improve the plan. In both conditions the scores were expected to be bigger for the Ship Borne Helo and the MPA since their sensor coverage is greater. Expected scores can also be seen in the outcomes of the interface, interaction and work flow questions where the participants in the AM condition found it less difficult than medium to estimate the location of the smuggler, whereas the participants in the MM condition found it slightly more difficult than average. It is remarkable that the participants in both conditions found it less difficult than average to estimate the overlap of the smuggler’s position with the sensor coverage. In both items the participants in the MM condition mentioned that they found the measurement-tool easy to use, which might be a socially desirable answer because they had nothing to compare the prototype to.
Another reason for the higher coverage in the AM condition could be that although the measurement-tool in the MM condition could be used very precisely, it would take much more time to measure. Since the participants had a limited time (on average 30 minutes) to deliver a mission plan the usage of the measurement-tool could lead to more imprecise measurements, leading to more mismatches with the smuggler and thus a poorer planning of the assets.
It is noteworthy that the Ship Borne Helo covered twice as much in the AM condition than in the MM condition where the other assets did not show such big differences. This could be explained by the fact that the Ship Borne Helo was planned as last in most cases. The participants in the AM condition could at this point have a better insight in the functioning of the furthest-on-circle and thus plan the Ship Borne Helo with a higher coverage.
That the animation of the furthest-on-circle helped in covering the smuggler’s position, and thus led to better coverage, is also supported by the answers on the question where the participants were asked whether moving the time with the time slider helped them in estimating the location of the smuggler. The results on this question differed signifi- cantly where the MM condition scored average and the AM condition scored very high on the 5 point Likert scale. Although the use of the time slider was essential in creating a mission plan, it might be more used in the AM condition because the furthest-on-circle
could only be controlled with the time slider, whereas the measurement-tool in the MM condition had to be used without the time slider.
Besides the coverage in the plan outcomes, the amount of scheduled time of every asset was also calculated. The overall scheduled time was significantly higher in the AM con- dition than in the MM condition. Looking at the independent assets this was also the case for OPV - P110, the Ship Borne Helo and OPV - P114, where only the Ship Borne Helo did not show a significant difference. The MPA was significantly scheduled more in the MM condition than in the AM condition.
This shows that the plans were more elaborated in the AM condition than in the MM condition. An explanation for this could be that the users in the AM condition had a better (insignificant) situation awareness resulting in a better overview of what hap- pened, was happening, and was going to happen on the map, thus could plan the assets to get the most out of them.
On the whole many of similarities could be seen in the outcomes, e.g. the subjects in both conditions strongly agreed that the time-blocks were useful in the creation of the plan and that the scheduler helped them in arranging the timing of the assets. All participants strongly agreed that changing the time with the time-slider was useful for getting a plan overview and to make predictions in the plan. This also happened for other items and might be due to the simplicity of the scenario. The scenario was defined in such a way that the average user would understand it. If the scenario had been more complex with longer experiments and more participants the results would probably have differed more.
The open questions showed some contradictions in the comments. It seemed that some participants explained the opposite of what they had answered on the Likert scale on the item “It was difficult to estimate the location of the go-fast in its time course between its departure at 20:00 and the arrival line” and the item “Estimating the overlap of the sensor coverage with the go-fast’s position was hard for me”. It might be that they had misinterpreted the scale. To give an example, a participant from the MM condition who agreed with the statement that it was difficult to estimate the location of the smuggler mentioned the following on the accompanying open question: “We could just move the cursor to know where and at what time the go-fast vehicle would be, this makes the interception pretty easy”.
In the majority of the results the MM condition scored better than expected when tak- ing into account that its prototype could be more cumbersome to use due to the manual measuring. The participants in the MM condition rated the usability almost as high as the participants in the AM condition, which also came back in the reactions on the open questions, where the opinions of the MM and the AM conditions were both very positive.
Hypotheses revisited The hypotheses that were introduced in Chapter 5 can be
revisited.
Based on the results, H1 (It is expected that a mission planning system that automati- cally visualizes the possible position of a unit will lead to better mission plan outcomes than a mission planning system where the users have to measure the possible position manually with a measurement-tool) can only be partially confirmed. In the experiment an indication of a good mission plan was the amount of coverage of the sensors with the location of the smuggler, which showed no overall significant differences. There was a significant difference in the overall duration of the routes where they took longer in the AM condition, which means that the plans in the AM condition on average were much more elaborated. Because of the automation more time could be spent on planning which is a positive effect for mission planning on itself.
H2 (It is expected that a mission planning system that visualizes the possible position of a target will give the users a higher situation awareness than a mission planning system where the users have to measure the position manually with a measurement-tool) is false. Both scales to measure situation awareness did not show significant differences, although a small trend could be seen on the content scale. It could mean that the dif- ferences between the two prototypes were not great enough to influence the situation awareness scores.
H3 (It is expected that the workload will be lower in a mission planning system that vi- sualizes the possible position of a target than a mission planning system where the users have to measure the position manually with a measurement-tool) is false. The results