There exists numerous ways to classify the available test methods. Depending on literature used, they are for example classified by the granularity of the test object, the sight on the test object (black-box vs. white-box) and the property to be demonstrated by test validation. In terms of the test object views, functional and structural methods can be distinguished, whereby hybrid forms or hybrid tests are also possible. However, they can also be classified by temporal aspects. Real-time tests detect temporal properties of the test object, such as the reaction time to an event or the duration of a particular operation. The concurrency test considers the execution of the test object together with other competing applications or components. The load test examines the test object with regard to extreme quantities of certain data, the stress test with regard to extreme quantities of certain events. With UI-testing, the mapping between internal states of the test object and the user interface is checked. Usability tests address the ergonomic adequacy of the test object.
As depicted, there exists a variety of approaches to testing, therefore, to only those with a strong relationship to this work are further addressed in the following subsections.
Testing Methods
Testing of software is well studied and as it would clearly exceed the limits of this thesis, only a brief overview will be provided. Further details on the techniques can be found in [35, 10, 40, 36]. The broad variety of testing techniques, can be categorized by the type of execution and based on the creation of test cases. The most obvious way to differentiate the available testing strategies is by distinguishing between Dynamic Testing and Static Testing (Static Analysis).
Dynamic Testing - Following the definition in [35] dynamic testing is defined as: “This means that we execute the program with specific input values to find failures in its behaviour. In contrast, static techniques (e.g. inspections, walkthroughs, and static analysis tools) do not require execution of the program. One of the big advantages of (dynamic) testing is that we are executing the actual program either in its real environment or in an environment with simulated interfaces as close as possible to the real environment. So we are not only testing that the design and code are correct, but we are also testing the compiler, the libraries, the operating system, and network support and so on.”
In dynamic software testing, executable test objects are executed systematically with the aim of proving the correct implementation of the requirements. This reveals the effects of errors and increases confidence in the test object. At the beginning of the test procedure, selected inputs and applicable outputs are determined based on the specification. After the execution of the test, these are then compared with each other. Depending on the test objective, a distinction is made between functional and non-functional tests. Whereby, non-functional tests check quality criteria such as performance or usability.
For the specification of dynamic tests it is necessary to define both pre-conditions and post- conditions in addition to the actual input. These conditions describe the test object’s state or states of it’s environment. These can also include certain values that are expected in a database or how they have changed afterwards. This way, the tests are reproducible because they can always be carried out under the same conditions. The test cases, especially input and expected output data, are usually derived from the requirement specification.
Static Testing - Static testing or static analysis, in contrast to dynamic testing, is carried out without the execution of the test object. Typical static methods are code reviews or tool- supported formal methods, where the source code of an application is checked against a set of formal rules to identify coding errors. As this type of quality improvement often requires the source code to be available, which is not a prerequisite of the approach proposed in this thesis, it will not be considered any further. However, static approaches can also be applied on other documents, e.g. requirements specifications.
A further distinction is often made in literature (see [10, 41, 32]) between white-box and black- box test procedures.
White-box Testing - White-box testing or logic-driven testing is a testing technique which examines the internal structure of a software under test. The test data is derived from the program logic. One example for white-box testing is a control flow oriented approach. A goal of this technique might be to execute every statement in the program at least once. Often, these goals are defined in terms of coverage criteria. Depending on the chosen criteria, in practice problems arise with loops or through the amount of different conditions.
Black-box Testing - Also referred to as functional testing, data-driven or input-/output- driven testing. Black-box tests, view the software under test as a black-box. The principle of this technique is to neglect the knowledge about the internal structure of the system and find circumstances where the software is not behaving according to the specification. Therefore, the test data used to stimulate the software is derived solely from the specification and not from knowledge about the internals of the program. However, this makes it hard for developers of the software to write the test cases, since they already have a understanding of internal processes. To counter this issue, a principle of Test Driven Development [42] is, that the tests have to be created before the actual development of the software.
Grey-box testing - Grey-box testing or glass-box testing is a combination of both testing techniques explained before. It involves the knowledge about the internal structure of the program to allow a better way of selecting input data. However, the tests are executed like black-box tests. Advantage is, that the tester can choose better test cases since he knows about details of the implementation. This enables to test such things as exception handling, where internal knowledge of the SUT is necessary.
The subject of this work is the (dynamic) test, in which the system to be tested is executed. During test execution, also the SUT is executed and stimulated by inputs, especially through the use of virtual RFID readers, the expected result is then compared through available system interfaces (e.g. accessing a database). Misbehaviour is detected if the observed output does not match the expected output. Further, it uses a black-box approach since the test cases are extracted from the specification, represented by the business processes and additional informa- tion on the physical layout and timing. No specific knowledge of the concrete implementation is needed. The models, which are a result from an analysis of the specification, are transformed to executable tests. However, for the selection of meaningful input data which allows for auto- matic execution of the test cases generated might need some insight of the implemented system in order to evaluate the test result (like a database scheme).
A differentiation with regard to the criterion for selecting test cases is more helpful than the split between the white-box and black-box test: If the selection of test cases is to be based primarily on the required functional characteristics of the system, this represents a function- oriented test. If, on the other hand, it is the goal that the set of test cases, the test suite, is selected in such a way that the internal structure of the test object is covered to a certain degree, it is a structure-oriented test. This thesis focuses on the function-oriented test. It is assumed that the inner structure of the implementation of the test object is not known. With regard to model-based test case generation it should be noted that the structure of the test model is known and this structure is also included in the test case selection. However, this model is an abstract specification of the required functionality of the test object. Furthermore, no connection between the structure of the test model and the inner structure of the test object is assumed. Thus, from the perspective of the test object, it is a function-oriented (or black box) test and not a structure-oriented test.
Test Data Selection - Because of the fact that exhaustive testing is often not feasible in practice, test-data-selection strategies need to be found. This allows to reduce the number of test cases and leads to efficient testing saving costs and time. According to [35] these strategies are called “Input domain-based techniques”.
Different techniques can be used to reduce the amount of test data, which is necessary to gain the maximal revenue out of the test cases. The following list names some of the techniques described in literature [32, 39]:
• equivalence partitioning • pairwise testing
• boundary-value analysis • random testing
In the context of this work, however, test-data-selection techniques receive only little attention. The main focus is on determining test cases by structural properties, in particular by analysing the control flow of activity diagrams. The resulting test cases, thus, already limit the input domain as they contain conditions, which are reflected by the generated test paths.
Testing Level and System Test
Tests for software systems can be carried out at different levels. Each level assesses different artefacts of the system to be tested and they are usually roughly aligned to the phases of software development. However, the characteristic feature is primarily the object to be examined during the test procedure. Test stages are usually defined by their proximity to the source code of the system to be tested or to the respective application level. The range of test levels extends from component tests to integration tests, system tests, acceptance tests and regression tests. A test stage is a set of jointly performed test activities that can be assigned to specific abstraction levels of a software system. However, tests at different test levels are not disjoint. A test level is a characterization of an individual test or an execution suite, not a hierarchical delimitation.
• Module- & Component Test
A module or component is tested against the specification of the module or component. • Integration Test
The modules are combined into composite units and tested against the specification of the technical functional requirements and architecture.
• System Test
The system is tested against the specification of the logical system requirement and archi- tecture.
• Acceptance Test
The system is tested against the user requirements and specifications.
The system test and acceptance test are very similar in their basic structure, but differ in their the observers. The acceptance test is intended to show the customer that the required functions have been implemented. The system test is carried out by the contractor and is meant to demonstrate that the product meets the requirements and is ready for delivery. As a rule, the acceptance test is therefore usually carried out in the actual operating environment. In contrast to this, the system test tries to create a test environment that is as close as possible to the operating environment.
The identification of defects contributes continuously to product quality, which is the reason testing should be carried out at all system levels as early as possible. The system test is therefore embedded between the integration test of a module and the acceptance test. This means that it is a question of viewing at and testing the assembled modules. Accordingly, the system test
aims to secure the application before it is integrated into an overall system for the first time. The focus of the proposed test methodology in this thesis is on the system level. For this reason, system level tests will be discusses in more detail.
The purpose of the system test is to test the integrated system to verify that the specified requirements are met by the product. The reason for this is that many functions and system properties only come from the interdependence of all system components and therefore only become verifiable there. Main objective of the system test is to detect errors before economic or physical damage occurs, build up confidence in the system, prove the functionality of the system for certain boundary conditions (requirements) and achieve a specified test coverage. The basis for system tests are all software requirements, specifications and documents that have the entire system as their topic. As the requirements can only be verified on the basis of externally observable behaviour, the SUT is often considered a black box. Therefore, the system’s inputs and outputs are mainly used to specify the test cases.
In this thesis the test object is the complete system that is executed in a test environment as close as possible to the actual system environment. The system environment therefore consists of hardware (RFID reader, RFID tag), software (RFID middleware, application software) and the network connecting the individual subsystems. Carrying out RFID tests in the real system environment is a major challenge, as the physical and temporal effects play a major role. There- fore, an approach based on virtual readers that are very similar to real readers was chosen in this thesis. The chosen approach does not correspond to the classical system test, but tries to come as close as possible to a real system environment. When checking the system functions, the configuration settings of the individual components are also taken into account. The sys- tem requirements of an RFID application are therefore related to the interaction with reading devices, functional requirements and the states of the data associated with RFID tags. Since RFID systems communicate via error-prone air interfaces, the challenge is to create a suitable test environment.