CÉDULA DE PREGUNTAS COLEGIOS:

5.8

Evaluation of the Implementation

Similar to the strategies adopted in the design of PANDArch, various programming techniques employed for implementing the framework also contribute towards meeting the guiding principles. These are summarised in the following table.

Principle Realisation in the Implementation

Pluggable Satisfying the pluggability principle at the implementation level

is primarily achieved by following the design, which provides the necessary foundation to make PANDArch pluggable. As detailed in Section 5.7, the Java implementation of the framework has ensured pluggability by allowing the original (i.e. unmodified) compiled code of the application to be launched through the framework whenever conformance checking is needed. Furthermore, as the required details for executing the subject application is given to PANDArch only at launch time, the framework has no prior knowledge of the application. Once conformance monitoring is complete, the same executable code can be deployed to operate the application as usual.

Non-intrusive This principle, too, is primarily satisfied by following the design. The

approach for implementing runtime probes makes use of the Java agent technology, which allows dynamic instrumentation of compiled Java code. This technology enables non-persistent runtime instrumentation of bytecode which conforms to the non-intrusive principle. Addition- ally, as can be gathered from Section 5.7, PANDArch does not require access to the application source code at any point.

Automated This principle is mostly satisfied by following the design. However,

the choice of architecture modelling notation, which is Grasp, also provides support for complying the framework implementation with this principle. The annotation capability, which is harnessed for embedding mapping information, along with the architecture graph produced by the Grasp compiler enable automated extraction of architecture properties and mappings from a Grasp model.

Specification-driven and

notation-independent

The current implementation of PANDArch, which includes a rules generator for Grasp, is capable of parsing architectures specified in Grasp and generating conformance rules for checking conformance of running Java software. Therefore, the framework implementation can considered specification-driven. Nonetheless, the Java objects implementing conformance rules do not carry any Grasp specific infor- mation, making the framework implementation notation-independent.

Principle Realisation in the Implementation Incompleteness-

tolerant

Tolerance for incomplete architecture specifications is achieved

mainly with theincludeandexcludedirectives that are part of

the mapping scheme developed for Grasp. These two directives respectively instruct PANDArch to monitor only certain portions of the implementation and refrain from monitoring certain portions of the implementation. They provision for incremental conformance checking of a developing software system, or when only a partial specification of the architecture is available. The implemented map- ping configurations and the use of wildcards also permit checking conformance at the level of detail available in the architecture. For instance, if an architecture specifies only the top-level components and not their sub-components, then conformance is checked only for those top-level components.

Self-contained As dictated by the design, PANDArch does not make use of any

external tools for implementing the required capabilities. Although the implementation employs a third-party library for instrumenting Java code, this library is packaged with the rest of the framework and therefore not a prerequisite for deploying the framework.

Performance-centric The framework implements and extends the performance specific

strategies outlined in the design. Most of the performance related techniques are applied in the implementation of monitor com- ponents, and in particular, the Java runtime monitor. These are discussed in detailed in Section 5.3.2.

Extensible The implementation satisfies the extensible principle by complying

with the component-oriented design of the framework. All major components including events, implementation descriptors, confor- mance rules, monitors, evaluators, and conformance rule generators have published interfaces that they expose. Any new component in these categories must also implement their respective interfaces, enabling the framework as a whole to be easily extended to support new event sources, architecture notations, etc.

5.9. CONCLUSIONS 105

5.9

Conclusions

The detailed design and programming strategies adopted to implement the PANDArch framework are presented in this chapter. The chapter also discusses the approaches taken ensure that the implementation conforms to the prescribed architecture. Internal workings of the important components in the framework are discussed along with their extensibility aspects.

A key implementation strategy is the use of Java generics to bind together, at the language level, components that work in a process pipeline. All major components in the framework including monitors, evaluators and rules generators follow this designed this pattern.

The framework implementation pays particular attention to minimising the performance impact on the application being monitored. The techniques employed in the implementation of monitor components, in particular, play a pivotal role in achieving this design objective. The Java runtime monitor component employs both the feature-rich JDI as well as high-performance instrumentation probes to capture runtime events of Java software. The framework gives the user the option to configure the Java monitor to use both these event gathering mechanisms in tandem, striking a balance between performance and comprehensive mining of implementation metadata. The current PANDArch implementation contains a runtime evaluator component that works closely with the Java runtime monitor. Along with the evaluator, the framework also implements

five conformance rules for checking conformance of Java applications at runtime. The

conformance rules themselves are generated from Grasp architecture specifications which are processed using the Grasp rules generator component. The annotations capability in the Grasp ADL provides the means for embedding implementation mappings. The chapter details current mapping configurations used for linking various architecture elements to their code constructs. The chapter concludes with an evaluation of the implementation strategies against the core principles of the framework.

The next chapter presents the evaluation of the currently implemented framework with the help of a test harness as well as three well regarded open source software products.

6

CHAPTER

SIX

EVALUATION

An important aspect in the development of PANDArch is the evaluation of the capabilities of the implemented framework. The chapter describes the overall approach taken for conducting evaluation experiments, along with their limitations and trade-offs. These experiments are performed with the help of a sample Java application and with non-trivial Java software. The results of each experiment are presented in the form of an evaluation report, covering both the detection of conformance violations and decline in performance due to runtime conformance monitoring with PANDArch. An analysis of the observed behaviour follows each report.

6.1

Introduction

The thesis takes a two phased approach for evaluating PANDArch. The first phase entails unit testing conformance rules, to ensure they produce the correct output for a given input. A purpose built sample Grasp architecture and its corresponding Java application are employed to facilitate validating each conformance rule. The second phase involves deploying PANDArch with real- world software implementations and their architectures for evaluating its conformance checking capabilities as well as performance. The open source community offers a large selection of mature Java software from which suitable candidates can be chosen for experimenting with PANDArch. An added advantage with open source software is the unrestricted access to source code, which is crucial for some of the evaluation strategies discussed later in this chapter. The next two sections introduce the approaches for unit testing and evaluating the framework with non-trivial software. They are followed by the evaluation report for unit testing in Section 6.4, while sections 6.5 to 6.7 present results from evaluating PANDArch with each of the chosen open source Java software products. The chapter concludes with a discussion of the effectiveness of PANDArch with respect to this evaluation.