Table 3.4. Experiment III Selected Methods Name Method Class (method)
M1 Interface REmailLightweightDecorator (decorate)
M2 Regular MapEditor (buildMenu)
M3 Regular MapEditor (buildWest)
M4 Regular MarkerInitActionDelegate (prepareSQLite)
M5 Regular Parser (parseFunction)
M6 Regular SpreadsheetReader (loadFile)
M7 Regular SpreadsheetReader (removeDoubleQuotes)
Table 3.5. Experiment III Results (0 = Not Relevant, 4 = Highly Relevant).
Method D D D D D D D D D D D D D D D Name 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 M1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 M2 1 0 4 4 4 4 0 0 0 0 0 0 1 4 1 M3 2 0 2 0 2 2 1 3 3 2 0 0 0 2 0 M4 2 3 0 2 0 2 0 0 0 0 0 0 0 0 0 M5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 M6 2 1 2 0 0 0 4 4 2 2 0 0 0 4 0 M7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Experiment III: Results
In Table 3.5 we report the results for the tested methods.SeaHawk performs well in this scenario. On the one hand this is influenced by the higher amount of useful information that can be fed to the query engine. On the other hand the fact that methods have (or should have) a single responsibility limits the scope in a positive way, leading to more useful retrieved documents. Moreover, the library or framework used in the implementation of a method can refine the scope of the research. For example, many useful documents were retrieved for the methods buildMenu and buildWest of the class MapEditor. In this case, both methods largely used the Swing framework, restricting the search scope. The contrary happens when the topic of a method is too general. Moreover, as in the case of parseFunction of the class Parser, SeaHawk is not able to retrieve relevant documents when the implementation of a method totally relies on code belonging to the project (i.e., no particular standard libraries or well-known frameworks are used). A last scenario in which SeaHawk performs poorly is when it is confronted to badly implemented methods. Tying this back to the program comprehension scenario: Sometimes understanding a method is not easy because the method is badly implemented. This also applies for cases where the signature of a method does not correspond to its implementation.
3.4
Conclusions
We presented an approach completely based on standard information retrieval approaches. We implemented our approach in SeaHawk, a plugin for the Eclipse IDE which let users interact with Stack Overflow documents, to import code snippets and create links between documents and source code by means of language-independent annotations so that developers can use an- notations to take advantage of the versioning system to collaborate and suggest documents to
teammates. SeaHawk automatically generates queries from code entities, by dealing with im- port statements and uncompilable code to extract keywords from Java code entities. Finally, we presented an evaluation of SeaHawk and a discussion of the results we obtained.
Reflections
This chapter presented a reductionist way of treating the information provided by both code and artifacts, reducing them to words, without preserving any information about the contents. The whole pipeline of SeaHawk is designed by ignoring the multi-dimensionality of the information. Indeed, the process takes a code context, projects it to a textual query, and retrieves discussions according to the query by leveraging a standard IR approach based on term frequencies. As shown in the evaluation part, some of the side effects are clearly visible in the evaluation of this approach, where it performs well only with known topic (e.g., fibonacci).
We believe that recommendation systems like SeaHawk have the potential to help developers in program comprehension and software development activities. However, to be effective, the engines of RSSEs like SeaHawk need to go beyond the pure textual representation of the artifacts, and use analyses which consider the different types of information provided by the artifacts (e.g., code information, meta data) as first class entities.
In the next chapter we follow this path, and we abandon the idea that off-the-shelf information retrieval analysis tools might be used to build effective RSSEs. Instead, we focus on building customized tools and models considering several aspects of the information.
4
Turning the IDE into a Self-confident Programming
Assistant
In the previous chapter we discussed an approach for RSSEs using off-the-shelf tools for in- formation retrieval, considering words, in the narrative sense, as the minimal unit to describe information in development artifacts and source code.
In this chapter we tackle two aspects of RSSEs. The first aspect concerns the data analysis, where we aim at analyzing the semantic link between source code in the IDE and a development artifact by weighting different aspects of the information.
The second aspect concerns the interaction between RSSEs and developers, the (almost) total absence of pro-activeness of the current approaches, which leads to manual intervention by the developers.
We describe Prompter, a fully automated RSSE that retrieves and recommends, through push notifications, relevant Stack Overflow discussions to the developer. Prompter makes the IDE a programming prompter that silently observes and analyzes the code context in the IDE, searches for Stack Overflow discussions on the Web, evaluates their relevance by taking into consideration several aspects of the information like code aspects (e.g., code clones, type matching), conceptual aspects (e.g., textual similarity), and Stack Overflow community aspects (e.g., user reputation) to decide when to suggest discussions.
Structure of the chapter
In Section 4.1 we highlight a limitation of the interaction of current RSSEs. In Section 4.2 we present our approach and its implementation in Prompter. In Section 4.3 we present the results of the ranking model evaluation of Prompter (Study I ). In Section 4.4 we present the evaluation of Prompter with developers (Study II ), while the replication of Study I one year later is described in Section 4.5. We discuss threats to validity in Section 4.6, and Section 4.7 concludes the chapter.
4.1
On the pro-activeness of RSSEs
Seminal tools, such as eRose [ZWDZ04], Hipikat [CM03] and DeepIntellisense [HB08], suggest project artifacts in the IDE to provide developers with additional information on specific parts of the system. They come however with a caveat: the developer must proactively invoke them, and, once invoked, they continuously display information, which may defeat their purpose,
as they augment the complexity of what is displayed in the IDE. Ideally, a recommender system should behave like a prompter in a theatre: ready to provide suggestions whenever the actor needs them, and ready to autonomously give suggestions if it feels something is going wrong.
The interaction between the theatre prompter and the actor is similar to the interaction between two developers doing pair programming, working side by side to write code. These developers have different roles, i.e., the driver, who is in charge of writing code, and the observer, who observes the work of the driver [Wil01], tries to understand the context, and, if she has enough confidence, interrupts the driver by giving suggestions. In addition, the driver can consult the observer whenever she needs it, making the observer the programming prompter of the programming actor.
By following a metaphor similar to the interaction between the theatre prompter and the actor, we propose Prompter.