NAVEGANTE DE VUELO –NDV Requisitos para expedir la licencia.

We invited the domain experts that collaborated with us to use and evaluate FacIt. These experts were selected based on two criteria: (i) they must be familiar with the domain of the data; (ii) and they must have experience of using tensor-based analysis to analyze multi-way association patterns. We performed an in-depth interview with each expert one-on-one. Each

interview lasted about 1.5 hours, and consisted of three sessions: (1) interface explanation and initial feedback (30 minutes): we introduced experts to the dataset and key modules of FacIt, and collected feedback on their first impressions; (2) using the system to explore patterns (30 minutes): a session where experts used FacIt to explore tensor factorization, and (3) a semi-structured interview (30 minutes): a post-study interview session where experts discussed the usability and suggested improvements. The rest of this section presents their aggregated feedback.

Visual Design: Experts were particularly impressed by the design of pattern presenta- tion. They found that the pattern presentation in FacItdramatically improved their efficiency when understanding, comparing and recognizing meaningful patterns. One of the experts told us, “It usually takes a long time for my colleagues and me to examine the patterns,..., before locating meaningful ones, especially when the number of them is very large”. With the help of pattern circular view and projection view in FacIt, they felt it was “much eas- ier to gain a comprehensive overview of patterns and understanding pattern relationships”. They highly valued the design of the pattern glyph. They explained that, “the information jointly encoded in the glyph, such as pattern dominance, descriptor informativeness, etc. is really helpful to understand a pattern”. Moreover, experts approved of the functions of selecting patterns for comparison and highlighting similar and discriminative items. All of these features make pattern comparison easier and more intuitive.

System Interaction: Experts felt that several interaction tools supported by FacItwere quite useful. With the help of those tools, they were able to easily incorporate their prior knowledge, feedback, and specific target into the final presentation of patterns. First, all experts appreciated having a model inspection module in FacIt. They pointed out that their pattern exploration process with tensor factorization usually starts by selecting a proper rank. Although they have data fitting evaluation metrics to consult with, they agreed that “the whole process takes a considerable amount of time and effort because this rank is not all about fitting to the data, but rather finding a set of interpretable patterns.” With the FacIt’s inspection module, one expert reported that “the rank selection then becomes a trade-off problem among a transparent set of objectives”, which makes the process much easier and faster. Second, the feedback-based model fine-tuning was very well received by our experts.

They confirmed that most of the time they all had some prior intuition or domain knowledge before they leveraged any tool to explore patterns. Prior intuition sometimes helped them locate meaningful patterns, but was unreliable, leading to unreasonable discoveries. One of the experts emphasized that “this interactive feature will be useful and phenomenal.” He confirmed that “it visualizes patterns after being re-tuned with my prior intuition; this speeds up the process of locating meaning patterns when my intuition is correct, and helps me recognize much earlier if my intuition is far from the fact.” Third, the experts gave espe- cially positive feedback to the interactive pattern query module, where they “could explore patterns relevant to an explicit set of interests.” Our experts tried different combinations of queries. “[I] needed to go through each component to find [ones] that are most relevant to my interests,” said by one expert. He also commented, “This module significantly speeds up this process”. He particularly appreciated the query book because it allowed him to quickly switch between visualizing of relevant patterns from different queries, allowing him to compare them more efficiently.

System Usability: According to the experts’ feedback, they were satisfied with FacItand considered it a comprehensive visualization and analysis system that fulfilled their require- ments of understanding, exploring and interpreting patterns from a multi-aspect real-world dataset. For instance, the experts were confident that the system could be effectively ap- plied to areas not limited to: (1) semantic analysis of knowledge base composed of tensors like (subject, verb, object), (2) community detection in multi-view, large-scale social net- works, where each view corresponds to an aspect in tensor, and (3) discovery of road spatio- temporal relationships from traffic datasets, which play a critical role in determining traffic management strategies. The experts also expressed that the system could be efficiently and effectively applied not only to search for meaningful patterns, but also to generate new pat- terns with users’ domain knowledge as input. The experts believed this would be extremely useful for researchers who want to verify their intuition with pattern visualization or hope to incorporate their domain knowledge into pattern generation.

Improvements: Although our experts agreed that FacItwas easy to use in general, they suggested some improvements and new features: (1) Domain-specific visual design: While our experts understand FacIt’s value as a generic visualization tool for multi-aspect

data in different domains, they suggested the system could use domain-specific visual design to make patterns more intuitive to users, e.g., encoding descriptors with makers whose shape and color have more semantic meanings in the corresponding domain. (2) Comparing different modules: All the experts highly valued the feature of interactive feedback-based model fine-tuning. However, they mentioned that it would be more convenient to compare patterns generated with different intuition. If the system could retain model results and allow experts to compare multiple models, experts could verify the validity of their intuition. (3) Active feedback collection: The experts suggested that it would benefit the pattern exploration process if feedback collection was two-way instead of one-way. Currently, only users of the system are allowed to re-tune model results and update pattern presentation. Two-way feedback collection would allow the system to actively collect feedback from users for parts of results it has low confidence in. The system could crowd-source and store the feedback from different users and update the default pattern display.

5.8 SUMMARY

In this chapter, we present FacIt, a visual analytic system for Tensor Factorization. The system is built to meet the common requirements of real-world applications, such as model selection, model refinement, and results scrutinization and interpretation. We have developed a suite of model scrutinization and inspection tools to empower the model selection process. A novel weakly semi-supervised tensor factorization algorithm is proposed to allow human- in-the-loop tensor factorization discovery. In addition, we provide an interactive design that caters to experts’ different exploration strategies, such as characteristics- and content-driven pattern discovery. The effectiveness and usefulness of FacIt has been evaluated in usage scenarios across different domains, followed by in-depth interviews with domain experts.