Our first design iteration was initiated with defining the general issues of investment advisory encounter (problem-centered initiation); the second iteration targeted the same research problem with extended solution objectives and revised design requirements. Based on the objectives and requirements of the second iteration, the third iteration mainly focused on improving particular design aspects of the application, corresponding to a
“Design & Development Centered Initiation” according to the taxonomy of possible research entry points by Peffers et al. [2007].
The demonstration of the third iteration’s design requirements and evaluation results builds on their presentation in Nussbaumer et al. [2012b].
7.7.1 Solution Objectives
While the second iteration demonstrated promising results towards accomplishing its solution objectives, the designed artifact could not significantly improve the client-advisor encounter. Ascribing the artifact’s failure to meet its solution objectives in the second iteration mainly to flaws in the interaction design, we attempt to accomplish the initial solution objectives of increasing process transparency (SO1), information transparency (SO2) and controllability (SO3) as well as overall client satisfaction (SO4) with extended design requirements and their adapted implementation.
7.7.2 Design and Development
The general artifact design of the second iteration showed some promise in improving the client-advisor encounter as compared to the traditional setting.
Thus, the design and implementation of the third iteration focused on alleviating artifact issues related to its interaction design.
7.7.2.1 Design Requirements
Based on the results of our second design iteration, we suggest that transparency of advisor-application interaction may influence the comprehensibility of the encounter as perceived by the client. This
“interactional transparency” is not to be confused with the concept of interaction transparency in HCI research [Bardram and Bertelsen 1995], referring to users wanting to think about their tasks and not about the computer artifact they are using. In our context of dyadic interaction mediated by IT artifacts, we mean the comprehensibility of one party’s system interaction as perceived by the other party.
We have suggested above that the second iteration’s use of error-prone gestures might have affected the comprehensibility of the advisor’s actions.
The client’s comprehension may have been further limited by the gestures’
implicit or hidden nature – as learning the meaning of different gestures (one finger vs. two fingers, “single tap” vs. “double tap”) was already difficult for the trained advisor, it is improbable that the incidental client user may easily map gestures to their proposed effects.
Therefore focusing mainly on the interaction with the artifact, we define the requirement of providing “interactional transparency” between users and the artifact to further the comprehensibility of interaction. As this requirement addresses the particular implementation of the shared information spaces, we add it as a constraint to our first design requirement:
DR1. Provide shared information spaces
Constraint 1: Enable sociable use of the shared information spaces Constraint 2: Provide interactional transparency
7.7.2.2 Implementation
In our revised interaction design, we aimed at more comprehensible system interaction for both the user and the immediate observer. We introduced a physical token that – when being placed on the screen – could be used to more explicitly change information levels and application states and thus allow for increased attention and awareness of the observer. Placing the token on the screen activates a radial menu, mostly containing functionalities that were previously only accessible with hidden gestures.
For example, the menu allows switching between the abstract process representation level (see also Figure 7-7.1 in Appendix A1) and the overview level (Figure 7-5) as well as “cleaning up” the screen, i.e., resetting orientation and placement of the information widgets. Rotating the physical tag will rotate the screen, allowing for more intuitive screen orientation.
Also, we included access to the advisor tools (calculator, information browser, session summary) into the radial menu, as their prior position (top-left and lower-right corner of the screen) caused frequent unintentional interaction.
To improve interaction with information items, we also replaced the “double tap” gesture with a tree menu on top of every widget that allows switching between detail view and overview.
Figure 7-5: Basic front end design of third iteration 7.7.3 Evaluation
To test the third iteration’s prototype application against our solution objectives, we conducted evaluations similar to the second iteration. We sought to test the same hypotheses as in the previous iterations, relating to improved transparency, controllability and client satisfaction in the artifact-supported investment advisory encounter compared to the traditional setting.
7.7.3.1 Method
Participants. We recruited 24 client participants by convenience sampling, 14 of them were students. Clients were between 20 and 52 years of age (M = 28.04, SD = 9.12), whereas 7 of them were female and 10 reported to have some experience with investment advice. Regarding their proficiency in
computer use, 11 participants characterized themselves as professional users, another 11 participants as advanced users and only two participants reported to use computers only occasionally.
The advisory sessions were carried out by professional investment advisors of a single Swiss bank (10 male, 2 female) with an age ranging from 27 to 55 years (M = 38.00, SD = 8.55) and advisory experience ranging from 2 to 30 years (M = 11.58, SD = 8.20). In respect of their proficiency in computer use, three advisors categorized themselves as professional users, seven as advanced users and two as occasional users.
Procedure. The evaluation applied the same procedure as the previous iteration, with two notable differences. To further increase the available time in the artifact-supported setting, we adapted the client’s scenarios such that it included only one goal (e.g., buying a car) rather than two. In the same vein, the scenario specified that the client was already having an account with the FSP (so her existing assets were already available in the system) but utilizing the FSP’s investment advisory service for the first time.
Apparatus. The same apparatus was used as in the second iteration evaluation. The artifact-supported setting was provided with the Microsoft Surface tabletop system (1st generation) running the software prototype.
Additionally, advisors were provided with notepad and pen. For the traditional situation, advisors again used their own advisory material and were additionally provided with notepad and pen.
Design and Analysis. The evaluation mirrored the design of the second iteration, using the very same questionnaires and metrics as well as observation and debriefing guidelines (see above). Table 7-7 provides the metrics’ scale reliabilities. All Likert scale items showed high reliability for both settings with all Cronbach alpha scores greater than .731. We therefore computed the scale averages of the participants’ responses. Furthermore, qualitative assessment of construct validity showed high convergent validity of participant responses from questionnaires and interviews.
Shapiro-Wilk tests on the differences between the settings’ scores revealed that all Likert scales were normally distributed, except for information transparency II (comprehension of use). We therefore used dependent t-tests (two-tailed) to compare the scale data of process transparency, information transparency I (information provision), controllability and satisfaction, while we applied a Wilcoxon matched-pairs signed-ranks test (two-tailed) to compare the ratings for information transparency II.
Table 7-7: Scale reliability of metrics used in the third iteration evaluation
Perceived process transparency 5.27 0.926 6.05 0.919
Information transparency I
(information provision) 5.02 0.945 6.38 0.923
Perceived information transparency II
(comprehension of use) 5.46 0.917 6.42 0.818
Perceived controllability 4.83 0.731 5.96 0.833
Satisfaction 5.16 0.890 6.32 0.924
All p-values were corrected for multiple hypotheses testing using the Benjamini-Hochberg procedure [Benjamini and Hochberg 1995]. The correction also accounted for the tests with non-significant results. As for the previous evaluations, we also calculated the effect size for all scales revealing significant differences.
Analysis of the qualitative feedbacks followed the same strategy as the first two iterations. Again, the debriefings were audiotaped, transcribed, summarized and the answers analyzed for central themes.
7.7.3.2 Results
With improving usability and interactional transparency of the system, we observed advisor-system interaction to be much more fluent and comprehensible, allowing for more attention and focus on content rather than on how to operate the system. This is also reflected in the client’s assessments of the artifact-supported encounter, showing all ratings being significantly higher as compared to the traditional setting.
Perceived process transparency. Clients assessed process transparency of the artifact-supported encounter (M = 6.05, SD = 0.98) to be highly increased compared to the traditional situation (M = 5.27, SD = 1.28), showing a statistically significant difference with large effect size (t(22) = -3.847, p = .011, d = 0.69). Similar to the second design iteration, clients found the shared representations and its visualizations greatly improving comprehensibility of the situation.
Perceived information transparency. Also, the assessment of perceived information transparency was increased. As in the second iteration, the quality of information provision (information transparency I) was significantly improved when using the IT artifact (M = 6.36, SD = 0.65) compared to the traditional situation (M = 5.02, SD = 6.36). The difference
between the settings was statistically significant with large effect size (t(22)
= -5.100, p < .001, d = 1.41).
Furthermore, and in contrast to the second iteration, clients also reported increased comprehensibility of information gathering and its purpose (information transparency II) for the artifact-supported setting (M = 6.42, SD
= 0.75), rating the traditional situation comparably low (M = 5.46, SD = 1.44). This difference showed to be significant with a large effect size (Z = -3.222, p = .004, r = -.47). Clients argued that this was mainly to the information representation that allowed them to enter client information whenever and they were relevant.
Perceived controllability. Regarding controllability, clients perceived increased influence and opportunities to participate for the artifact-supported setting (M = 5.94, SD = 1.04) compared to the traditional setting (M = 4.83, SD = 1.27). The difference was statistically significant and showed a large effect size (t(22) = -3.553, p = .003, d = 0.96). In their feedbacks, clients related their increased influence to the interactive development of the solution as well as the artifact’s “playfulness” in allowing changing and adapting “everything” to their preferences.
Satisfaction. Overall, clients were more satisfied with the artifact-supported (M = 6.32, SD = 0.74) than the traditional encounter (M = 5.16, SD = 1.30), showing a significant difference with large effect size (t(23) = -3.564, p = .003, d = 1.14). Furthermore, 87.50% of the clients reported that they preferred the artifact-supported setting over its traditional counterpart (4.17% reporting no preference). Clients brought forward the following reasons for their preference: better comprehensibility and overview, innovativeness and “fun”. However, the two clients who preferred the traditional setting argued that they were granted more speaking time, making the encounter more pleasant.
As compared to the second evaluation, the third evaluation’s sample of participants was younger (mean age of 28.04 vs. 38.17 in the second evaluation) and consisted of more students (14 of 24 participants vs. 6 of 24 in the second evaluation). Thus, we investigated a potential sample bias towards IT support based age and occupation (student/non-student).
To investigate effects of the former, we computed the correlations between age and the main scales. For both the second and third evaluation the tests revealed no significant Spearman correlations (two-tailed) for our main metrics.
Regarding the influence of occupation, we tested both data sets (second and third evaluation) for different ratings between students and non-students (using independent t-tests where both groups were normally distributed, and Mann-Whitney U tests otherwise). The tests revealed no significant differences.
7.7.4 Discussion
Looking at the results, improving usability and interactional transparency greatly improved advisor-system and client-advisor interaction. Simplifying advisor-system interaction and replacing hidden gestures with interaction primitives more easily to be followed also increased the client’s comprehension of the advisory encounter.
Introducing the physical token and the information widgets’ tree menus served two purposes; firstly, it allowed us to increase usability of performing previously error-prone actions. Secondly, and maybe more important, the token allowed more explicit enactment of the underlying actions and provided stronger interactional cues; in this way, the token drew the client’s attention to important advisor interactions and thus may have increased her comprehension of interaction effects and consequences. The revised interaction may also have led to our observed increased sociability of the client-advisor encounter, as the explicit interactional cues allowed the advisor to better shift and focus client attention.
Overall, the results suggest that the design rationales of process and information transparency are only effective if coupled with sufficient interactional transparency, i.e., sufficient client comprehension of the advisor’s interaction with shared information representations. Interestingly, the majority of participants of the second iteration did not deem such transparency to be important – introducing this requirement was mainly based on our own sense-making of observations of advisor-system and client-advisor interaction.