Tipos de medios

Gaucho is a visual programming environment (VPE) which makes use of spatial relationships, data abstractions, and a high level language of interaction for performing programming tasks, such as renaming a class, or adding a method to a class. Gaucho is in the middle ground between visual programming (VPLs) and pure textual languages [61].

In this dissertation we focused on investigating an alternate interface for the construction of OOP environments, nevertheless we decided to retain the use of text to specify behavior because of the complexities of devising a VPL for fully replacing textual descriptions of code [62; 63]. Nonetheless, we believe that Gaucho could easily be modified to incorporate other means of specify- ing behavior, including a shape that manipulates methods via a VPL instead of editing textual descriptions of statements. For instance, the VPL could use an iconographic language to denote operations, and specify the method body using a data-flow language such as Coherence [64], which relies on the abstrac- tion of a virtual tree, similar to the abstract syntax tree of a parsed method.

Chapter 4

Evaluation

In the previous chapter of the thesis, we introduced Gaucho, which is our ap- proach to the design of a development environment for OOP, and described its application to several modeling and programming activities.

Gaucho is an object-focused environment that allows developers to write programs by creating and manipulating lightweight and intuitive depictions of object-oriented constructs.

On the other hand, traditional integrated development environments (IDEs) include many tools that provide the means to construct programs. Coinciden- tally, the very same IDEs are a primary vehicle for program comprehension. We claim that IDEs may be an impediment for program comprehension be- cause they treat software elements as text, which may be counterproductive in the context of program understanding—where abstracting from the source text to the level of structural entities and relationships is the key.

The research question we investigate here is how does an object focused en- vironment such as Gaucho compare with traditional IDEs when it comes to per- forming program comprehension tasks.To answer our question, we conducted a preliminary controlled experiment with eight subjects, comparing Gaucho against a traditional IDE.

We acknowledge that software is created by humans, for humans, thus soft- ware engineering is—above all—a human activity. However, the intrinsically non-deterministic nature of humans introduces a number of threats to the va- lidity of controlled experiments, performed by researchers to evaluate software tools. One of them concerns how to record information without influencing the behavior of the subjects involved. Another one relates to providing means to assure the correctness of the gathered data, for further analyses and replication.

66 4.1 Instrumenting the Evaluation of Software Tools Furthermore, instrumenting an evaluation of a software development envi- ronment, such as Gaucho, places a heavy burden on the experimenters, due to the short duration of the actions programmers perform to fulfill the tasks com- posing the experiment. The brevity of the tasks makes it hard to track the time and effort of the subject under observation, which hampers with the correctness of the ulterior observations regarding the experiment.

To alleviate this burden, we focused on improving the tool support that en- ables researchers to perform controlled experiments with human subjects to evaluate the performance and usability of novel approaches and software engi- neering tools. We devised Biscuit, a toolset to specify tasks to be undertaken by the subjects of the experiment, and precisely time the subjects and record their complete behavior as they perform the tasks.

Structure of the chapter. In Section 4.1 we discuss often overlooked issues

that come up during controlled experiments with human subjects, and we present Buiscuit, our toolset to alleviate some of the issues. In Section 4.2 we discuss the context of the evaluation. We describe the experimental design in Section 4.3, and the operation and results in Section 4.4. We conclude the chapter by reflecting on the outcome of the evaluation in Section 4.5, and dis- cussing the advantages of instrumenting controlled experiments with tools such as Biscuit in Section 4.6.

4.1

Instrumenting the Evaluation of Software Tools

If software engineering is above all a human activity, then taking the human aspect out of the loop would be definitely wrong, when it comes to evaluating software engineering approaches.

Consequently, these past years have seen a steady increase of evaluations based on controlled experiments performed with human subjects. For exam- ple, Cornelissen et al. evaluated a tool for visualizing execution traces [65], Stasko et al. contrasted two visualization tools for depicting hierarchies using different space-filing layouts [66], Marcus et al. analyzed the support for pro- gram comprehension tasks of their sv3D tool [67], and Wettel et al. assessed the validity of the city metaphor for visualizing software systems [68].

Controlled experiments are prone to a large number of pitfalls, due to the one and only uncontrollable element of any controlled experiment involving human subjects: humans. Humans have individual talents, skills, behaviors,

67 4.1 Instrumenting the Evaluation of Software Tools and quirks. In any experiment humans behave in a non-deterministic way, which introduces various threats to the validity of any experiment of this kind.

The pitfalls in question regard an issue that might be seen as a corollary to Heisenberg’s uncertainty principle: How is the information that one wants to record actually being recorded, and does the fact that one records information in an intrusive way influence or modify the behavior of subjects?. If so, how can this be minimized?.