The DABA prototype was evaluated within the framework of an online workshop [Wan09]. Similar to the evaluation of Similarity Search (SiSe) and MISTM services (Subsection 4.5.5), information system (IS) Success Model, adapted for communi-
ties [RKS08], was applied. Quantitative evaluation of the dashboard usage was combined with qualitative evaluation based on user questionnaires. The evaluation results were afterwards compared with the DABA impact on community in terms of productivity.
Online Evaluation Figure 7.3 shows the concept of the DABA evaluation. The
evaluation participants were divided into two groups. The first group was supposed to use BAT only. Whereas, the second group was recommended to use both BAT and DABA. The participants of each group included students and researchers from the domain of computer science and other scientific fields like architecture, electrical engineering, etc. The evaluation was performed in the form of a two-day online workshop. The workshop goal was to improve an existing social network analysis tool “PALADIN” developed by our group. Different screenshots of the system were then uploaded into the databases of each group before the evaluation. During the workshop, users annotated those screenshots in the form of bubbles by using recommended software: BAT and/or DABA. Moreover, participants were allowed to upload and share other pictures in additional to those provided, for example, a screenshot of another existing social network analysis tool. After the workshop, each participant was asked to fill out an online questionnaire. The questionnaires contained similar questions for both groups except for one additional question, which had been provided solely to the second group.
To investigate the effect of DABA on community-oriented processes, we compared the system usage, the information outcome, amount of the generated content and the satisfaction of each group. By means of questionnaire analysis it was possible to measure the degree of subjective satisfaction experienced by both groups. The investigation of the system usage was executed with help of the monitoring service MobSOS [RKS08]. The quantitative analysis was conducted by studying the community generated content.
In accordance with the study, 12 people participated in the BAT group and thereby submitted 11 feedbacks, whereas 11 participants took part in the BAT&DABA group and offered nine feedbacks. As it turned out, most participants of both groups were familiar with the most basic concepts: Web 2.0 and social network. They also reported to have some previous experience with the non-business dash- boards like Apple Dashboard and iGoogle. Hence, we concluded that the evaluation group was well-captured.
Quantitative Analysis consists of two parts: (1) analysis of community-generated
Figure 7.3: DABA Evaluation: Workshop Workflow Concept.
Community generated artifacts Participants of both groups had uploaded nearly
the same number of the new screenshots. However, the number of the bubbled discussions created by the BAT&DABA group was about four times bigger than the number of discussions started by the BAT group (cf. Figure 7.4). Thus, the group equipped with DABA had showed considerably higher community data utilization in terms of content creation.
Frequency of the system usage Analysis of the system usage also confirms a
stronger activity within the BAT&DABA group. The number of sessions of the BAT&DABA group is more than four times higher than those generated by the BAT group (188 against 43 sessions, cf. Figure 7.5). Moreover, the request number of the BAT&DABA group runs up to even nine times those invoked by the BAT group (7839 requests as opposed to 829 requests, cf. Figure 7.6). Further investigations show that the users of the group using BAT&DABA created 49 sessions (1219 requests) from BAT and 139 (6620 requests) sessions invoked by DABA. Hence, users of the BAT&DABA group
Figure 7.4: DABA Evaluation Results: Community Generated Content.
Figure 7.5: DABA Evaluation Results: Session Number.
worked much more with DABA than with BAT which they had not been obligated to do.
To summarize, the DABA dashboard fostered significant increase of community activity and productivity.
Figure 7.6: DABA Evaluation Results: Request Number.
Quality Analysis The subjective assessment executed by means of a question-
naire explains and enhances the results of the quantitative analysis. First, most participants specified that DABA reduced the complexity of the system. It became especially easier to understand what other members were referring to. Second, it was suggested that DABA had solved some of the communication problems. For instance, the users could more easily get an overview of the current situation in their community. Third, the investigation about communication enjoyment and participant motivation revealed the importance of CA for RE in Community Infor- mation System (CIS). The last and most significant question in the questionnaire was related to the communication enjoyment of the system. The result showed that the DABA dashboard increased the users’ process enjoyment. Although, according to participant analysis, the users were not familiar with the concept of CA, they enjoyed the exploration of the presented information. Many evaluation participants wanted to use dashboard-like tools in the future. Thus, a navigation dashboard presents a good solution for supporting CA within communities.
Collected Requirements Analysis of the generated content showed that the group
which used BAT& DABA produced much longer discussions about the uploaded screenshots, than the group which only used BAT. It is important to note that the same screenshot set of a new version of PALADIN software was provided to both groups. On average, the number of the bubble messages pre picture was 5.4 and 2.14 in the BAT& DABA and BAT groups respectively. In the second case, there were many one-bubble long discussions, meaning that the addressed issues did not receive feedback from other community members. To visualize this difference, we
present the collected discussions for a screenshot with a new PALADIN feature for color coding:
BAT&DABA group :
Anna: Color coding makes sense
ChristianH: Color coding is fine but where can I find the legend? renzel: Can I get more information about Layouts?
ChristianH: What about a set of self explanatory icons instead of color coding + legend?
renzel: Many edges overlapping with nodes. Is edge layouting also supported?
BAT group :
Yan: Color Encoding -> nice idea!
In the second case, the topic of “color encoding” remained as is without any further negotiation. Based on the discussions from the BAT&DABA group, we were able to extract five functional requirements (zooming function, node naming, node previewing, edge layout, color coding). Whereby, the analysis of the BAT group artifacts led us only to one functional requirement (zooming option). During personal talks after the evaluation, several participants of the BAT only group complained that it was difficult to find out what was currently under discussion within their group and where changes had recently happen. Communication is one of the binding forces in a CoPs. Therefore, it was the process which we wanted to foster in the first place.