Resinas con relleno a granel

The next crucial ingredient of the conformance benchmark are the test suites of conformance test cases. In total, there are three test suites. These are a BPEL conformance test suite described in the following subsection, a test suite with workflow control-flow pattern implementations in BPEL described in Sect. 3.2.3.2, and a conformance test suite for BPMN described in Sect. 3.2.3.3.

All test cases have undergone quality assurance, to make sure that they correctly reflect the language standards. This includes an XSD validation of the process models, in-group peer review of all process models, as well as review of some process models by external experts. Moreover, an overwhelming majority of the test cases run on at least a single engine. The code used to perform the benchmark of BPEL engines is the v0.3.0 release of betsy and is available at https://github. com/uniba-dsg/betsy/releases/tag/v0.3.0-icsoc2013. The updated code for benchmarking BPMN engines can be found in betsy’s v2.0.0 release, available at https://github.com/uniba-dsg/betsy/releases/tag/2.0.0.

http://www.soapui.org/. We use SoapUI for the testing of BPEL processes to transmit a series of SOAP messages and evaluate the replies.

3.2.3.1. BPEL Conformance Test Suite

The source of all test cases are the normative requirements stated in the BPEL 2.0 standard [27]. A complete listing of all test cases can be found in Table A.1 in the appendix. Normative requirements are specified using common notational conventions [193] (i.e. MUST, MUST NOT, etc.). The test cases are classified into the activity groups defined in the specification and explained in Sect. 2.2.2.4: basic activities [27, pp. 84–97], structured activities [27, pp. 98–114] and scopes [27, pp. 115-147]. Each group also contains test cases for standard faults that are associated with the activities of the group. Table 3.3 provides an overview of the test suite and lists the number of conformance tests per activity.

Table 3.3.: BPEL Conformance Test Suite Basic Activities Structured Activities Scopes

Assign 19 Flow 9 Compensation 5

Empty 1 ForEach 11 CorrelationSets 2

Exit 1 If 5 EventHandlers 8

Invoke 12 Pick 5 FaultHandlers 6 Receive 5 RepeatUntil 2 MessageExchanges 3 Receive & Reply 11 Sequence 1 PartnerLinks 1 Rethrow 3 While 1 Scope-Attributes 3

Throw 5 TerminationHandlers 2 Validate 2 Variables 2 Variables 3 Wait 3 P 65 32 34 Design

decisions Although the BPEL standard is relatively precise, three aspects necessary for executable process models are left open: First, there is no required format for the specification of a partner reference that can be used in the assignment of partnerLinks. This is needed to enable dynamic assignment and runtime selection of services. In the benchmark, we use WS-Addressing EndpointReferences [194] (encapsulated in BPEL’s service-ref container). Second, the URI scheme for identifying XSL stylesheets is not made clear. We use the filename of the stylesheet for its identification. Third, and probably most critical, is the underspecification of the behavior of the engine in case a fault is thrown during execution and not handled by the process instance, while at the same time, request-response operations are still open. Put differently, given a third party waits for a response from a process instance, it is not made clear what the third party will receive in case the instance crashes with a fault. Possible options are a timeout, a default response, or a propagation of the fault that led to the crash. In the test cases, we opt for the latter. This is also the fault propagation mechanism applied by most high-level programming languages and is a prerequisite for distributed fault handling [195].

3.2.3.2. Pattern Test Suite

The conformance test suites puts its focus on single language elements in isolation. Not all of these language elements are equally important in real-world applications. Moreover, it is possible that language elements that conform to the standard in isolation fail to conform when being combined with each other. To gain insights on the implications, which standard-conformance issues have in realistic application scenarios, additional tests are required. Such tests can be found in Workflow Control-Flow Patterns (WCPs) [71]. According to the authors of [71], these patterns capture scenarios that are frequently needed in real-world process-aware applications. Moreover, WCPs have frequently been used as a means for comparing different process languages or process management systems with respect to their expressiveness. Also BPEL and BPMN have been addressed by pattern-based analyses [196–198]. For this reason, we provide a test suite that benchmarks support for WCPs in BPEL engines.

Patterns often require the combination of multiple language constructs for their implementation. The focus of pattern-based analyses lies on how many patterns a language allows to implement and how directly a pattern can be implemented. The lower the amount of constructs needed for an implementation in a particular language, the more direct the support is considered, and hence the more expressive the language is [71]. The original pattern catalog [71] consists of 20 patterns. These are basic control-flow, advanced branching and synchronization, structural, multi-instance, state-based, and cancellation patterns. A discussion of the nature of every pattern is out of scope here, but can be found in [71] or on the homepage of the workflow patterns initiative located at http://www.workflowpatterns.com/. For a detailed description of pattern implementations in BPEL, we refer the interested reader to [196, 199].

Pattern test im- plementa- tion In test suite at hand, we provide implementations of WCPs in BPEL that can be used by the benchmarking tool in the same fashion as conformance tests. The test cases are built by adapting the existing pattern implementations from [196]. The pattern tests use the same process stub and WSDL interface as the standard conformance tests. A pattern is considered to be directly supported (denoted as +), if at least one solution can be found, which uses no more than a single language construct to implement the pattern [71]. If a solution requires a combination of two constructs, it is counted as partial support (denoted as +/−) for the pattern, otherwise, there is no direct support (denoted as −).

BPEL 2.0 provides no direct support for four out of the 20 patterns (Multi-Merge, Discriminator, Arbitrary Cycles and Multiple Instances Without a Priori Runtime Knowledge) [196, 199]. An engine can only improve this degree through the usage of nonstandard extensions. Since the goal of the benchmark is to evaluate standard conformance, we exclude these four patterns from the benchmark. Furthermore, three patterns (Interleaved Parallel Routing, Milestone, and Cancel Activity) are at most partially supported in standard-conformant BPEL. Implementations for these patterns are evaluated in the benchmark. Finally, for four patterns (Multi- Choice, Synchronizing Merge, Multiple Instances Without Synchronization and Multiple Instances With a Priori Design Time Knowledge) alternative solutions

Table 3.4.: Workflow Control-Flow Patterns Test Suite

Patterns BPEL Tests

Basic Control-Flow Patterns

WCP01: Sequence + 1

WCP02: Parallel Split + 1 WCP03: Synchronization + 1 WCP04: Exclusive Choice + 1

WCP05: Simple Merge + 1

Advanced Branching and Synchronization Patterns

WCP06: Multi-Choice + 2 WCP07: Synchronizing Merge + 2 WCP08: Multi-Merge - 0 WCP09: Discriminator - 0 Structural Patterns WCP10: Arbitrary Cycles - 0 WCP11: Implicit Termination + 1 Patterns with Multiple Instances (MI)

WCP12: MI Without Synchronization + 3 WCP13: MI With a Priori Design Time Knowledge + 2 WCP14: MI With a Priori Runtime Knowledge + 1 WCP15: MI Without a Priori Runtime Knowledge - 0 State-based Patterns

WCP16: Deferred Choice + 1 WCP17: Interleaved Parallel Routing +/- 1

WCP18: Milestone +/- 1

Cancellation Patterns

WCP19: Cancel Activity +/- 1

WCP20: Cancel Case + 1

are available in BPEL 2.0, which we cover in different tests. In summary, if an engine passes a pattern test, it provides either direct or partial support for a pattern, depending on the test. Table 3.4 shows the number of tests for the different WCPs in the test suite and the degree of pattern support (direct support, partial support, no direct support) that can be achieved with standard BPEL 2.0. The structure of the tests can be found in Table A.2 in the appendix.

3.2.3.3. BPMN Conformance Test Suite

The third test suite contains tests for assessing the standard conformance of BPMN 2.0 engines. Similar to the test suite for BPEL standard conformance, test cases are derived from the normative parts of the language standard [26]. As discussed in Sect. 2.2.2.3, the focus of this work is exclusively on the executable part of BPMN. Therefore, tests are derived from Chapter 10 of [26] which describes executable process models. The test suite is subdivided into the categories of basics, activities, gateways, events, and errors. The activities category comprises Tasks, SubProcesses, and CallActivities, as specified in [26, Chapter 10.3]. Events can be found in [26, Chapter 10.5] and gateways are defined in [26, Chapter 10.6]. Elements that are part of executable process models, but do not fall into the preceding categories, such as SequenceFlows,

Table 3.5.: BPMN Conformance Test Suite Language Constructs Tests

Basics 6 Lane 1 Participant 1 SequenceFlow 4 Activities 12 CallActivity 2 Multiple Instantiation 5 Looping 3 SubProcess 1 Transaction 1 Gateways 13 ExclusiveGateway 3 InclusiveGateway 2 ParallelGateway 1 ComplexGateway 1 EventBasedGateway 2 GatewayCombinations 4 Events 36 Cancel 1 Compensation 6 Conditional 5 Error 4 Escalation 7 Link 1 Signal 6 Terminate 1 Timer 5 Errors 3 InvalidGatewayCombinations 2 ParallelGateway Conditions 1

belong to the basics category. Finally, the errors category comprises process models that should be rejected by a BPMN engine. These are, for instance, invalid combinations of gateways. The resulting number of tests are listed in Table 3.5. An exhaustive description of all test cases can be found in Table A.3 in the appendix.

Test suite scope As indicated in Sect. 2.2.2.3, the BPMN standard is less precise than the BPEL standard. Implementers of BPMN have a relatively high degree of freedom of interpretation. As a result, it is challenging to write tests for standard conformance. For this reason, the test suite presented here covers only a subset of the executable part of BPMN. More precisely, it excludes specific types of tasks, e.g., UserTask or BusinessRuleTask. Furthermore, the definition of MultipleEvents and inter-process communication is not covered (intra-process communication in terms of communicating SubProcesses is included in the test suite). Since none of the engines we benchmarked provided the required support for WSDL interfaces, message-related events and tasks are omitted. Finally, DataObjects are also out of scope. The conformance evaluation of these elements