Factor de conversión

The intention of the benchmarking methodology presented in this section is to evaluate the standard conformance of a range of process engines in an isolated and reproducible fashion. We implemented a benchmarking tool that fully automates this methodology. The tool derives its name, betsy, from BPEL/BPMN Engine Test System and is freely available at https://github.com/uniba-dsg/betsy. Betsy provides a domain model for defining conformance tests and automating the lifecycle of process engines. Furthermore, it executes these conformance tests in a conformance testing workflow. This section sketches the domain model and the testing workflow.

Figure 3.1.: Conceptual Domain Model for the Conformance Benchmark Domain

model The domain model for specifying a benchmark is depicted in Fig. 3.1. This structure applies for benchmarks of BPEL and BPMN engines alike. Every execution of the tool is configured in a TestSuite. The key elements of this suite are Processes and Engines. A Process corresponds to a process model in either

BPEL or BPMN and bundles all engine independent artifacts that encapsulate a single language feature. In the case of BPEL, this is a BPEL process definition and related WSDL, XSD, and XSLT files. In the case of BPMN, it is a single file ending in .bpmn that includes a BPMN Process. A Process in the domain model is the basis for a single conformance test. An Engine represents one of the systems under test and defines methods for managing the lifecycle of a concrete engine instance in a specific version. Furthermore, it provides methods for deploying process models to that engine instance. Specific subclasses of this abstract class are implemented for every engine under test. This is also a major extension point of betsy and the conformance benchmark.

For each Process, a set of TestCases, consisting of several TestSteps that are verified through zero or more TestAssertions can be defined. A TestCase corresponds to a single valid control-flow path through a process model. Hence, multiple TestCases may be necessary to verify if the behavior of the language feature captured in the process model conforms to the standard. TestSteps correspond to steps in the traversal of the control-flow graph. For instance, this can be the creation of a process instance with specific input parameters or the sending of a message to a process instance. A TestAssertion captures expected behavior or output and can be validated after process execution. If all assertions pass, the test case is recorded as successful. If all TestCases linked to a Process are successful, the engine under test is considered to have passed the conformance test. Otherwise, the engine has failed the test.

TestAssertions are evaluated in different ways for BPEL and BPMN engines. For BPEL engines, betsy actively communicates with a running process instance by means of SOAP messages. In this case, TestAssertions are evaluated based on the payload of the messages received from the process instance as a reply. By sending messages at specific points during process instance execution, it is possible to verify if execution takes place as expected. In the case of BPMN, a fully standardized way of specifying message exchanges, working on all tested engines, is lacking, although a RESTful way of interaction is possible for some engines. [38] elaborates on this issue. Instead of actively communicating with a process instance, betsy evaluates execution traces written by the instance. For this reason, there is no communication between betsy and a BPMN process instance during execution.

Process stub The structure of process models used for capturing conformance tests reflects the difference between BPEL and BPMN. In both cases, we use a minimalistic process model, a process stub as the starting point for the implementation of every conformance test. The language elements used in the stub are of most basic nature and are confirmed to work on every engine under test. This ensures that they have no influence on the results of more sophisticated tests. For a conformance test, the process stub is extended with a specific language feature. Conformance tests for BPEL are necessarily based on WSDL. Every BPEL process model implements the same WSDL interface to enable a unified handling of all tests by betsy. To make sure that all engines support the interface, it is relatively simplistic. It contains a single partnerLinkType and several message

definitions. All messages consist of one message part of a primitive type. The portType is made up of one asynchronous operation and two synchronous ones. The difference between the synchronous operations is that one of them defines a fault. The interface binding uses a document/literal style over HTTP, which is the most basic style available [105, Sec. 3]. Additionally, betsy provides a similarly structured web service interface, and Java implementation thereof, during execution. This is needed in the conformance tests for the invoke activity. The process stub for BPEL conformance tests includes a synchronous operation. This is necessary to provide output and assert the correctness of the test execution. As a consequence, every BPEL conformance test requires the usage of variables, assign activities, and reply activities, next to an incoming message activity, either receive or onMessage, for instance creation. List. 3.1 outlines the structure of the process stub. The model starts with an incoming message activity (test prolog). This is followed by the implementation of the test, i. e., a language element in a particular configuration. Finally, the model is concluded by the transmission of the result (test epilog).

Listing 3.1: Outline of the BPEL Test Cases

<process> <partnerLinks /> <variables /> <sequence> <!−−T e s t P r o l o g−−> <receive /> <!−−T e s t i m p l e m e n t a t i o n−−> <!−−T e s t E p i l o g−−> <assign /> <reply /> </sequence> </process>

In the case of BPMN, a process model is not required make use of WSDL. Therefore, a single valid BPMN file is sufficient for implementing conformance tests. As before, a number of basic language elements are required. These are none StartEvents, SequenceFlows, ScriptTasks, and none EndEvents. None Start- and none EndEvents are of the most basic type, which the standard offers. They delineate the start and ending of the process model. As discussed above, active communication with the process instance is not possible in the same fashion as for BPEL conformance tests. Therefore, the tests use ScriptTasks to write execution traces to a log file. In the engine-independent specification of the conformance tests, the ScriptTasks only contain a token that describes the trace that should be written to the log file. The BPMN standard does not require support for a

specific scripting language [26, Section 10.3.3]. Therefore, we inject a script in a language supported by an engine into the body of the ScriptTask on execution of a conformance test. Input data is provided to a process instance through process variables, which are initialized during the creation of the instance. Despite the difference in vocabulary, the resulting process model presented in List. 3.2 is very similar to its counterpart for BPEL conformance tests.

Listing 3.2: Outline of the BPMN Test Cases

<definitions> <process isExecutable=” t r u e ”> <!−−T e s t P r o l o g−−> <startEvent id=” S t a r t E v e n t ”> <outgoing>SequenceFlow_1</outgoing> </startEvent>

<sequenceFlow id=”S e q u e n c e F l o w 1 ” sourceRef=” S t a r t E v e n t ” targetRef=” S t a r t T a s k ” />

<scriptTask id=” S t a r t T a s k ”>

<incoming>SequenceFlow_1</incoming> <outgoing>SequenceFlow_2</outgoing> </scriptTask>

<sequenceFlow id=”S e q u e n c e F l o w 2 ” sourceRef=” S t a r t T a s k ” targetRef=” T e s t I m p l e m e n t a t i o n ” /> <!−−T e s t i m p l e m e n t a t i o n−−>

<!−−T e s t E p i l o g−−>

<sequenceFlow id=”S e q u e n c e F l o w 3 ” sourceRef=” T e s t I m p l e m e n t a t i o n ” targetRef=”EndTask ” /> <scriptTask id=”EndTask ”>

<incoming>SequenceFlow_3</incoming> <outgoing>SequenceFlow_4</outgoing> </scriptTask>

<sequenceFlow id=”S e q u e n c e F l o w 4 ” sourceRef=” EndTask ” targetRef=”EndEvent ” /> <endEvent id=”EndEvent ”> <incoming>SequenceFlow_4</incoming> </endEvent> </process> <definitions> Bench- marking process As discussed, in each conformance test, i. e., in each Process, we insert a specific language feature in a particular configuration as test implementation. On execution, betsy links all Processes with all Engines in a TestSuite and executes all resulting tests sequentially. The execution logic of a benchmark is depicted

Figure 3.2.: Benchmarking Process

in Fig. 3.2 in BPMN notation. This process divides into phases of setup, tear down and test execution. The activity Prepare Test Folders corresponds to the setup phase and the activity Generate Reports to the tear down phase, which are executed exactly once, before and after the test execution phase, respectively. This latter phase corresponds to the execution of all Processes for all Engines in a TestSuite. Test execution takes place sequentially for each process and engine. Sequential execution of the tests implies a longer execution time, but is necessary to avoid side-effects that could occur when testing in parallel. Such side effects manifest for instance in blocked ports and could otherwise compromise the benchmark results.

During setup, the folder structure required for the test execution phase is built. The test execution in turn corresponds to multiple sequential steps. In the first step, a deployable artifact for a specific engine is created from the test files. If needed, this includes modifications to the standard-conformant process models, such as the addition of required engine-specific namespaces. Additionally, engine-specific deployment descriptors are generated. Second, test classes that trigger the execution of a process instance and validate its correctness are created from the TestCases. For BPEL, this involves a SoapUI13 project configuration

and JUnit14 classes. For BPMN, the generation of a JUnit class that evaluates execution traces is sufficient. Third comes the installation of the engine under test. Each engine is installed anew for every test case execution. This is needed for test isolation, since the execution of a previous test case might have broken the engine installation. For instance, we found that a particular test resulted in an infinite loop for one engine, thereby crashing any future process instances executed on this engine. Fourth, the engine is started. Thereafter, the deployable artifact, which was created in the first step of the test execution phase, is deployed to the engine. In the sixth step, the test classes are executed, triggering the creation of a process instance, transmitting messages, and evaluating the result of the instance execution. Finally, the engine is stopped. Afterwards, the next test case is executed. During the tear down phase, all test results are aggregated to reports in CSV and HTML format. Table 3.2 summarizes the differences in the testing of BPMN and BPEL engines described in this section.

Table 3.2.: Differences in Testing BPMN and BPEL Engines

BPMN BPEL

Test Files .bpmn .bpel, .wsdl, .xsd, .xslt

Input Data process variables SOAP message content

Assertion Validation execution traces message exchanges

Test Execution JUnit SoapUI, JUnit