1. Which one of the two presented dashboards do you think you would use more for your work?
The answers provided by the participants can be divided into two separate categories and they are describing either an approach or a preferred dashboard according to the role. Some of the participants stated that they would start with the “Activity review” screen for a better overview and that they would move next to the “System status” dashboard, which offers them more in-detail information. They declared that more time would be spent on the second dashboard as it provides more information that would offer a clear image of the context and of the steps that have to be performed next. Participant 5 even affirmed that “I am a technical guy who likes details”
The other participants explained that in their vision and from their experience, the preference depends upon the role that the person performs. They stated that the managers would prefer the “Activity review” and the analysts would prefer the System status dashboard. This answer validates our hypothesis and proves that the persona method was appropriate to use. Participant four answered to this question:
“Depends on the role. If I am in the SOC, on a daily basis I will use the System status more as my daily tool, really seeing what has happened but as a manager, or if I want just a quick overview, then I would use the Activity review dashboard. Exactly when I come to work, as a landing page I would like to see the Activity review page. I think that we need dashboards as they have a different level of details.”
2. What are the main problems and difficulties you have found while using this prototype?
One problem that was identified by a high number of participants is that the compliance graph was missing a timeframe which would make it easier for them to identify the exact time when the drop and the fix happened. Another issue that was confusing for the users was that the results of the compliance check that had scores in a certain boundary. This feature has to be studied further to understand what the true value of it is and if the problems were in the way the information was displayed or at the core of the functionality.
Another issue that was identified by the participants in the System Status page is the lack of interactivity on the graph and that clear relationship was not defined. They would expect that if something is selected in the event history graph, that would be reflected also in the other tiles and the information would be updated accordingly. Also, in the same chart, they would like at each given point in time to be able to find out the number of incoming events.
3. What or which functions have you liked most of the prototype? Why?
Participants really appreciated that the urgency of the issues were colour-coded in the interface so that they could identify quickly what was the first thing that they should start working on. They have also liked the ease of creating reports by only selecting what to include from the dashboard. The functionalities that were offered were appreciated but the most appreciated elements were the ones presented in a graphical format, such as a pie chart showing the status of the tickets and the graphs displaying the compliancy history, events history and types of events.
“I like these elements that resemble a traffic light and that from the first sight I can identify if is good or bad. I like also that we have some graphs and I think it has to be a mix of different elements, not just lists, tables or graphs. If everything’s the same, it gets boring and if I have only similar things, like lists, then I get so tired that I do not even see the changes anymore” (Participant 4)
“I like that the urgency is colour coded. That makes it easy to see what is wrong and also, that you can sort the events by risk, by severity and by the number of affected devices. Also, from the “Activity review” page it is really good that you can see the total number of tickets, but I think that the details are more important for this.” (Participant 3)
4. Can you describe your overall experience with this prototype?
Overall, all participants described their experience as good. Some participants mentioned that they liked the division of elements into different blocks as it makes it feel well organised and clear. Some participants stressed the good integration of many useful functions and some mentioned that it is clear that a lot of thinking and work has been put into the design. However, some of the participants would like to customise the system, to make it more familiar and useful.
5. How did you like the user interface design?
The users praised the design and found it simple and good looking. They liked the new implemented design in the dark theme and the only improvement that was requested by them was to accentuate more with the accent colours the urgent issues of the critical events. They found that the division by tiles helped them to identify easily and quicly what they were interested in. One of the participants even appreciated that the relevant metrics were in a bigger format so that they can be observed faster: “I like in general the percentages in bigger figures I can spot some items really because of big numbers.” (Participant 4)
6. How would you evaluate the set of functions being offered in the prototype? The set of functions that were offered in the prototype were very well received by the participants. They evaluated the functionalities according to their daily work and to their experience. The opinions are quite diverse. Some functionalities such as the average ticket were not understood by all of the participants but some of them said that that functionality is very useful to have. A full answer of one participant: “It is at the border of being too much. The UI is well structured by blocks which gives it a sense of clarity and the views do not feel too overcrowded by information. If we would add even one more feature, then I think it would be too much.” (Participant 4)
7. What would you improve in this prototype?
The thing that was mentioned numerously by the participants as an improvements that they would like to see is customization of the dashboard, regarding the positioning of the elements, the content that is displayed, and the personal settings. Another wish that was expressed by the majority of the participant is a clear relationship between the elements on the “System status” page. They asked for a clear identification of the events that cause the peaks of incoming events in the other sections, such as the list of real time events.
8. If you had one wish free, what or which function would you like to have for such a system?
None of the users had any suggestions for additional separate feature that should be integrated into the design, only on how the existing features should be improved.