To evaluate the tool we conducted experiments in two phases. The first phase included general evaluation by a number of users for their opinions and is subdivided into two sections: a) general feedback and b) task based feedback. The second phase included the actual validation of the outcomes of the tool mining data by the expert developers.
General Feedback
In this part of the user study, we have set some general questions in order to get feedbacks on the overall opinions of the users based on their familiarity with software development and various program comprehension tools they have used. One of the main motivations of this thesis is to build a tool support for understanding the implementation details of a system. In this part of the user study, our intent was to get the feedback from the participants about their opinions towards different ways of understanding a system and the tools they have used to support their comprehension.
From the feedback where 93% of the participants had the experience on modifying or fixing or optimizing a system developed by other developers, we have come across that when they were asked whether they wrote any documentation for a system they developed, 14% answered “no”, 29% said “partially” while 36% and 21% replied “yes but did not maintain the documentation” and “yes and maintained the documentation” respectively. On the other hand, 64%of the participants also did not attempt to use any documentation tools such as doxygen or javadoc. To further verify their feedback they were asked, if they were given a system and asked to modify some of its functionalities, then what would they do initially to understand the code? Interestingly, we came across the fact that 31% were in favour of reading documentations, 22% would ask the developers to help them out, 22% would run the system and 20% would use an automated tool to them out. However only 4% would try to create a new documentation of the existing system. To support this we found that 43% of the participants have experience with profilers or debuggers out of which 38% of them used to find errors in code, and 31% to understand a specific portion of code. From here we can conclude that majority of the users were in favour of using documentation followed by other techniques such as discussing with the developers, run the system or use other automated tools.
Task Based Feedback
In this section we will describe how the tool is evaluated based on its usefulness. Generally, the usefulness of a tool is determined by the following criteria:
C1 How effective our tool is in order to meet users’ requirements during the program comprehension? C2 How simple and easy the tool is to operate for source code analysis?
C3 How presentable the information is in the tool for the users to interpret and analyse?
We then further conducted an intensive study on three different systems developed in our lab. The results were presented to the actual developers and their opinions were noted.
4.7.1
User Study
Our main goal is to evaluate the overall usefulness of the tool and for that we had to design a methodology to conduct the study. The main challenge in this study was availability of the users and the type of users we had to consider. To fulfil the experiment we have conducted a series of steps starting from experimental setup to analysis of the result.
Experimental Setup
To evaluate the usefulness of our tool we required a set of participants, a questionnaire, methods of evaluation, and analysis of the result. We also selected an interactive user interface based subject system to present the participants with the implementation details to comprehend a system. The following are the steps that constitutes our setup for the experiment.
1 Design a questionnaire: We have designed a total of 24 questions to cover responses of the par- ticipants from different perspectives. The questions in the questionnaire are classified into different sections. The first section of the questionnaire is designed to acquire the background of the partici- pants with respect to their knowledge about the domain of software development. The second section extracts information about the common scenarios that they have faced and how they have dealt with them. This part also allows them to do specific tasks using the tool and answer the related questions. The questions in this section are solely designed to get user satisfaction level. The satisfaction level is represented using a likert scale. Five different levels such as strongly agree, strongly agree, neutral, disagree and strongly disagree defines how satisfied the participants are about the tool’s usefulness after the tasks are done. Therefore, as a part of the questionnaire design phase we have also put forward a set of tasks for the participants.
2 Demo session: It is the second phase of our experiment modelling which initiates the user participation for the study. The users were given introduction about the nature and purpose of the study. Then they
were briefed about the tool that they would use and the motivations behind it. We then ran the tool over a subject system and presented every feature of it. We also described the architecture of the tool including the visualization and the information management in particular so that the participants can operate it easily. The participants were then handed with the tool and allowed to operate for a while to get familiar with it.
3 Task Performance: After the demo session the questionnaire was handed to the participants. Each individual task was followed by a question that recorded the participants’ opinion based on their level of agreement on the performed tasks. It was an open discussion session and the participants had the liberty to ask questions regarding any difficulty they faced during the task completion. All the participants had to answer the questions on-line. We selected 14 participants with backgrounds in software development. The study session was carried on Windows 7 workstation on which the TrAM was installed as a client environment. The client side was connected with the database server to fetch data from it and complete the mining process.
4.7.2
Task Design
To evaluate our tool and address the three evaluation criteria we have set four tasks that will allow the participants to express their opinions accordingly. Each task intends to gather statistics about the usefulness and the usability of the tool. Usefulness signifies how the tool can bridge the gap between the complexity of a system’s implementation and the program comprehension by the users. Usability portrays how simple and easy the tool is to operate and therefore assists the users to interpret the result during the comprehension process. Table 4.5 provide a summary of the task the users were provided with.
Table 4.5: Summary of Tasks
Task Description
Locate feature The participants were provided with a scenario and then were asked to find the portion of code that was responsible for that particular task.
View Statistics The participants were asked about a scenario such as how the behaviour of a particular method can be found from the trace.
Filter methods The participants were asked about a scenario such as how specific information about methods can be obtained using the tool.
Categorize groups The participants were asked about a scenario such as how a group wise information can be obtained.
To address the evaluation criteria we have set different questions specific to those criteria. The participants were asked questions on usefulness, usability and the design of the tool. The following are the sample questions that are designed for each criteria.
C1: How effective our tool is in order to meet users’ requirement during program comprehen- sion?
1 It was very difficult to isolate the feature given a bug using the source code. 2 The tool clearly presented the statistical information of the selected methods.
3 The tool very easily allowed to retrieve the information of active clones in the system. 4 The filtering mechanism was very quick to filter out the methods that was required. C2: How simple and easy the tool is to operate for source code analysis?
1 What difficulty did you face in using the tool?
2 What features do you think are the most significant in the tool that guided you towards your goal? 3 How satisfied are you with the filtering by type, searching, categorization by groups, categorization by
active clone class, image and code viewer and metrics view?
C3: How presentable the information is in the tool for the users to interpret and analyse? 1 What portion of the tool you think requires improvement?
2 The information in the tool was well organized. 3 Overall the tool was very satisfying.
4 The overall performance (delay when a user clicks on any item) of the tool was very good.