CONECTORES TEXTUALES

Below, every metric is defined, and we explain how a good and bad score can be distinguished. These definitions and scores are summarised in table 6.1.

Relative metric weight distribution Regarding the weights of the individual met- rics, we have decided to divide them into three groups. The first three metrics are about the setup of the tool (group 1), the next six metrics are about the usage of the tool (group 2), and the final metric is about the continual use of the tool (group 3). The metrics in all groups are important, but regarding the most essential impact on the functionality and efficiency of the tool, we can make a distinction.

The first group only once takes a specific unit of time for a tester, and the second group takes an amount of time for every test or test case. Therefore, the time-usage for the second group has the most substantial impact on the total adoption and us- age time. Next to this, the second group addresses the most essential functional aspects of the tools. That is why these six metrics are regarded as the most impor- tant and used as a tiebreaker when two or more tools would otherwise get the same score.

1. Instructions- the availability of a guideline on how to install and use a tool. Having no instructions results in the minimum score and a complete manual on how to install and use it will yield the maximal score.

2. Installing tool- the complexity and the number of steps required to install a tool. If a tester should follow many complicated steps for the installation, it gets the minimum score. When a tester, however, only needs a download and a single-click installer, it gets a maximal score.

3. Configuring tool- any configuration required after installation and before we can use the tool to test for race conditions. Similar to the installation of the tool, a straightforward process will yield the maximal score, and a complex process involving many steps will result in a minimum score.

4. Control options- the way the tool can be controlled by the tester to perform race condition tests. When any alterations require source code updates, it gets the lowest score. On the other hand, when not only a human (graphical or command line) interface is available, but also an API is available for automatic or remote test integration, it will receive the highest score.

5. Importing requests - the process of getting HTTP requests of interest into the tool for further configuration and using them in a test. When only manual and repeated copy-pasting of raw HTTP packet contents are supported, this is very inconvenient and will yield the worst score. A much better way is the ability to dynamically push requests via multiple kinds of sources like proxies and browsers that were already meant for gathering HTTP requests. These abilities result in good integration and easy usage and will result in the best score.

6. Storing requests - the ability to not only import and use requests but also store them conveniently for later use. If this is not supported at all, the tool will get the lowest score. To the contrary, if both storage and well-designed searching through these requests are supported, the highest score is awarded. 7. Storing configuration- when a tool can be set to work in different ways de-

pending on the type of race condition test, it should support storing this config- uration to avoid repeated work. When not only the storage of settings but also importing and exporting of settings for different tests are supported, we hand out the maximal score. When no such storage is possible, the tool gets the lowest score.

8. Sending requests- the most essential part of a tool is the parallel sending of requests. In order to trigger complex race conditions, the tester often requires multiple parallel requests and delicate timings. If any combination of requests and timings is supported, the tool gets the highest score. If only the bare essential functionality of sending a single request multiple times is supported, we give the minimum score.

9. Aggregation of responses- verification of the occurrence of a race condition is the second-most important functionality of the tool. When the tool prints all HTTP responses to the parallel requests to the standard output, analysing this becomes very cumbersome and therefore yields the lowest score. When ad- vanced and possibly automatic aggregation of responses for quick verification of success is supported, the tool gets the highest score.

10. Future proof - for most research-grade tools, acquiring and particularly using more antiquated tools can be very challenging. An open source and actively maintained tool with updates in the past six months will probably work most conveniently and will be usable for a longer period in the future. Therefore, such a tool will receive the maximal score. On the other hand, a closed source tool that does not seem to be actively maintained hence will receive the lowest score.

Table 6.1:According to the scoring laid out before, this table shows the concrete mapping between metrics and scores that is used when rating the tools.

Metric Worst: - - Bad: - Good: + Best: ++

Instructions None Very limited

instructions

Instructions on, install, config and run

+ elaborate manual on how to

use the tool Installing

tool

Complex and time-consuming: many steps / requirements / bugs

Download, some config and start installation

Configuring

tool Complex and time-consuming

Quick config No config required Control options Via source code changes Via config files Via a GUI /

CLI + via and API Importing requests Manual using copy paste Manual using files Dynamic via API + multiple dynamic methods Storing requests No Yes + automatic / searchable Storing

configuration No Yes + import / export

Sending requests

One request in parallel and no timing config

Different requests / timing configs + any request and timing combination Response- aggregation No

Yes, simple grouping / filtering / highlighting

+ more advanced options

Future-proof Closed source

& abandoned

Closed source & active / open source & abandoned

Open source & active