Medidas implementadas para monitorear el desempeño de las unidades de generación

Automating testing tasks is necessary in a product line organization because there will be multiple versions of the tests that will be reapplied in multiple iterations on a single product and across multiple products. Many of the tools used by the development organization will also be used for testing activities. The “Tool Support” practice area in the Framework for Software Product Line Practice developed by the SEI provides a comprehensive discussion [Clements 02b]. In particular, a configuration management tool will be used to manage the multiple versions of each test. This tool or a more specialized one is used to provide trace- ability between the tests, the requirements or specification that they verify and the artifacts to which they are applied.

The benefits of automation are increased when the following basic principles are followed: • Each unit of automation should correspond to a natural unit in the development process

for product lines: for example, group test cases and other assets according to the compo- nent to which they apply.

• Each unit of automation should be sufficiently abstract to be robust in the face of minor changes. For example, automating at the interface level means that a test set can still be applied even when the implementation behind that interface changes.

• Each unit of automation should be designed to be composed of other units. The values returned by a piece of code or the exceptions thrown should follow an overall design standard. For example, have the test-case code return a value of true or false depending on whether the test case passed or failed. Then an aggregating test case can decide whether to proceed with the next step in a larger test case or to abort since a failure has already occurred.

Test-case creation is one of the most time-consuming operations in software development. This effort can be reduced by starting at the product line level, creating general test cases, and then specializing them to a specific product; however, the effort is still considerable. Apply- ing these test cases to the artifact can also be a major resource drain. The automation of test- case creation and execution is accomplished in one of several ways: custom test harnesses, testing tools with scripting languages, or test-case generators.

5.2.1 Custom Test Harnesses

Test environments such as Junit [Junit] provide an execution environment for test cases that have been constructed using a basic programming language. These harnesses are useful at the unit-test level or as a simulation environment for embedded systems. It is easy to create suites of test cases that are structured like the architecture of the software, as discussed in Section 2.3.

This approach takes more initial effort per test case than the other techniques. The tradeoff is increased flexibility and compatibility. Object-oriented programming languages can be used to create flexible environments that easily adapt to changes in the specifications and imple- mentations. Custom programming makes it possible to develop tests that are compatible with other tools being used in the development process.

5.2.2 Test Scripts

Test tools provide an environment for executing test cases and a language for constructing test cases. The languages are becoming more sophisticated so that the line between custom- programming environments, such as Junit, and commercial test tools is blurring. Scripts can be written at a general level and then specialized using inheritance. These tools provide a means of executing scripts, and a few also provide a repository in which test results can be stored and compared from one round of testing to the next.

For the end-user-level testing of systems with GUIs, test scripts can be generated automati- cally using a “capture and playback” mode of a test tool so that, once a human tester creates a test case, it can be repeated without human intervention. These scripts use the names of ob- jects in the interface and are robust with respect to their rearrangements.

These tools are perhaps the easiest to use and only require human resources for the initial round of testing. However, they do not provide any guidance on the semantics of which tests to create or what the tests contain.

5.2.3 Test-Case Generators

Generators use templates to generate test cases with a fixed format and limited variability. These generators require a large amount of effort initially because they require a formal rep- resentation of the specifications that are being tested. Creating the templates also requires effort, which is amortized over the products that use similar types of assets. It is further amor- tized when the development approach also uses a generator approach. Weiss and associates [Ardis 00] have created languages that are specific to product lines and that are used to define products. This same language is the basis for the test-case generator.

Generators do not include an environment for the execution of test cases; however, they do provide more guidance about the content of individual test cases. Generators can lead to more complete testing because as each new test case is generated, different values for inputs can be selected. This results in more complete coverage of the input space.