Registros Electrofisiológicos (Electroantenograma)

Insight has become an important performance indicator in the VA community, though challenges remain as to how to evaluate and measure such outputs [16]. Thus a deeper understanding of the evaluation methods currently in use and how to measure insight is needed.

In cognitive science insight are measured via studies that are constructed around creating puzzles that lead to a gridlock situation where participants have to change their assumptions or frame of mind to arrive at the solution, therefore generating an insight by measuring the time it has taken for the solution to be revealed [36–38]. These approaches are biased towards the ‘aha!’ sudden insight phenomena as reviewed earlier in [34], and. Further, reinforced by studies by Metcalfe and Wiebe [39], where they found that insight problems differed from non-insight ones by the sudden and unexpected nature of the solution, these, cognitive science approaches are good, but do not give a full comprehensive picture.

Using a neurological approach, Bowden et al., [40] based their analysis on current limitations in insight research and the inability to detect insight unambiguously. They define a framework for insight studies using a large set of problems that are quick in their resolution and that can be solved with or without insights with an unambiguous solution. The approach uses recent neuroimaging advances in functional magnetic resonance imaging (fMRI) combined with Electroencephalogram (EEG) readings, to map and measure brain activity. With this method, they have created a neurological model that establishes that enables some predictability of insight by analysis of the mapping and activity patterns in the brain. Bowden et al. conclude that this kind of neurological approach together with classical cognitive science methods can help understand the brain functions and demystify the origins of insight. This framework has clear tangible benefits, but it also has major drawbacks. This approach can be intrusive and requires very specialised and expensive equipment, hence not the most readily available of method to use. Further, the detection is as good as the problems themselves, thus suffering to an extent from the same insight categorisation restrictions.

Other, objective measures of insight include, Riche [41], who proposes to investigate physiological methods to detect insights. The approaches considered use body sensors to monitor eye position and pupil dilatation as well as heart rate, muscle and brain activity. Riche aims to create a physiological model to predict insight by associating these physiological measures to observed insights and as such also suffers from the same drawbacks than the work by Bowden et al. [40]. Additionally physiological measures are hard to interpret unambiguously to assign a psychological value, although the measures

are objective the interpretation and inferences towards insight as a psychological phenomena can be subjective.

In the information visualisation field, Lam et al. [16] developed an evaluation taxonomy. Insight measurements have been identified as particularly useful in two of the seven scenarios of their classification: ‘Evaluating visual data analysis and reasoning VDAR’; and ‘Evaluating collaborative data analysis CDA’. VDAR studies concentrate on how user generate actionable knowledge from insights, where as CDA studies are focused on the collaborative nature of the analysis. Both these evaluation categories are mainly conducted as either controlled experiments [29, 31, 42] or as longitudinal studies [30, 32, 43, 44] and take different formats either as a case study [45, 46] or as a laboratory experiment [47] and generally use observation as the exploration progresses or post- experiments interview or both to measure insight.

North et al. [48], compare different information visualisation evaluation methods using previously evaluated visualisations [49]. The first method is based on a benchmark task, the second is coined the insight method. The benchmark task method is composed of structured tasks executed by levels of complexity and collects the answers to a multiple choice questionnaire and measures the time to provide the answers, as well as their accuracy as dependent variables. Whereas the insight method is an open-ended, think- aloud protocol experiment, where the researcher silently records the timings of the insights. The dependent variables are insight count and post experiment categorisation. Due to the difference in nature of the two methods, the comparison metrics in the context of visualisation are broader and include task taxonomy and associated effort spent in the analysis. North’s et al. research found that the number of insights were positively correlated to the time spent on the task; where, more time spent generated more insights. Thus, North et al. suggest that limiting the time of the study when performing a task-based method can bias the results, by limiting the number of insights generated. Interestingly, an important breakthrough was that the type of visualisation used facilitated the generation of certain type of insights. Hence there was a relationship between type of insight and visualisation type. Equally depending on the visualisation one or the other method was favoured. Yet, interaction based visualisation acted equally on both methods. Based on the findings of the insight method they hypothesised, interaction played a role in generating more insights. Additionally, participants engaged in the insight

method gave immediate feedback during the analysis, wishing there was more interactivity in the visualisation to deepen their analysis. Table 2.2 provides a comparison of the empirical results, showing the difference that evaluation has on the interpretation.

Benchmark task method results Insight method results

More Insights Less Insights

Fast and accurate Confirms Refute

Slow and inaccurate Refute Confirms

No difference detected Expand Expand

Not tested Extend Extend

Table 2.2. – Comparison of Empirical Results, (adapted from [48])

North et al. conclude that although the insight method is more time consuming, complex and subjective to analyse, the benchmark task is more complex and time consuming to design and requires deep domain knowledge. This suggests that to reduce the complexity and subjectivity of the insight method, a generalised categorisation would address these issues. Table 2.3 gives a summary comparison of the evaluation methods, showing the benefits and drawback in each method, providing a good and valid comparison of the evaluation of the currently available methods in information visualisation. The results take a dichotomist view, and conclude that there are benefits in both methods, but implying that either one or the other must be used. Nevertheless, a viable approach might be to merge both methods leveraging the benefits of the union.

Comparison

factor Benchmark Task-based method Insight-based method

Purpose Evaluate specific research question about task performance

Evaluate insight generated in realistic analytic scenario Design prep Prepare benchmark tasks and scoring scheme

Better with simple data, tools, tasks

Prepare problem scenario

Better with complex data and tools Experiment

design

Benchmark task protocol Form based

Time and accuracy Can be multiplexed Short-term study only Longer preparation time

Open-ended protocol Think aloud

Capture insights Interaction with user Can be longitudinal Variable procedure time User tasks Determined by experimenter Determined by user

(user identifies insights) Participants Any users

Many users

Expert, motivated users Motivation is detectable Train without biasing Empirical

data analysis

Processing scores data

Quantitative statistical analysis

Coding rich insight and usability data Statistical analysis

Higher variance Longer analysis time Primary

outputs

Identify tasks supported by a visualization Perceptual, mechanical task efficiency (time, accuracy)

Statistical differences

Feedback on selected tasks only, ensures coverage of those tasks

Low-level tasks

Identify tasks promoted by a visualization

Cognitive, interactive learning efficiency (amount of insight) Statistical differences Detects new tasks, ignores unneeded tasks

Higher level tasks, user hypotheses, Summary

Qualitative feedback and analytic process

Subjective bias

Choice of benchmark tasks and scoring scheme Coding of insights and categories

Bias threat Ecological validity Repeatability

Table 2.3. – Summary of Benchmark Task and Insight Methods [48]