The overview of the evaluation results for the complete set of use cases is given in Table 5.1. Two symbols are used to represent the support of the system for a use case: “+” means that the use case is supported, and “-” symbolizes the opposite. For the use cases that consist of several sub use cases (e.g. use case 1) we present the results at the sub use case level. Another thing that needs to be mentioned is that there are two process discovery features in ARIS PPM (Activity sequence and EPC) and Futura Reflect (Mine and Explore). In order to provide a clear overview on the capabilities of the tools, we make the distinction between the two features when presenting the support of the use cases. However, for the situation in which the support of a use case is independent of the two features (e.g. some use cases belonging to the time perspective), the result is given at the system level. Next to the “+” and “-” symbols, we also use the “(+)” symbol to represent the weak support that the EPC representation in ARIS PPM provides for the detecting parallelism, due to the assumption that the concurrent activities need to have identical start and completion timestamps.
The process discovery class of use cases (use cases 1-4) has a rather good support in all the three systems. The exception is use case 3 (Most frequent path), which is only supported by one of the functionalities available in Futura Reflect. The most challenging control-flow patterns for the discovery algorithms turned out to be the parallel activities (supported only by two mining functionalities), OR splits and joins and duplicate activities (both of them not supported at all).
52 CHAPTER 5. EVALUATION RESULTS
Table 5.1: Result Overview of the Use Case-based Evaluation
Use case Acceptance Criteria ARIS PPM Futura Reflect Process A.s. EPC Mine Explore Analyzer
Discover sequence + + + + +
Discover XOR splits and
joins + + + + +
Discover OR splits and joins - - - - - Discover parallel activities - (+) + - - U1. Structure of the
process
Discover structured loop -
Repeat + + + + +
Discover structured loop - Do
while + + + + +
Discover arbitrary loop + + + + + Discover skipped activities + + + + + Discover duplicate activities - - - - - U2. Routing probabili-
ties Assign probabilities to arcs + + - + +
U3. Most frequent path
in the process Show most frequent path - - - + - U4. Distribution of cases
over paths
Show distribution of cases
over paths - + +
U5. Exceptions from the normal path
Show exceptional cases or
paths + - + + -
U6. The degree in which First level - hard coded rules + + - the rules are obeyed Second level - Generic rules + + - U7. Compliance to the Qualitative criterion + - + - explicit process Quantitative criterion - - - U8. Resources per task Show associations resource-
task + + -
U9. Resources involved in a case
Show set of resources in-
volved in a case + + +
U10. Work handovers Show social network + - - + - U11. Central employees Show central employees + - - U12. Throughput time
of cases
Show throughput time per
case + + +
U13. Slowest activities Show duration of activities + - + + U14. Longest waiting
times
Show the waiting times be-
fore activities + - + +
U15. Cycles Show cycles causing time de-
lays - + - + +
U16. Arrival rate of
cases Show arrival rate pattern + - -
U17. Resource utiliza-
tion rate Show resource utilization + + +
U18. Time sequence of events
Show events in a case in a
time sequence + - +
U19. Business rules Show attribute values at
5.1. USE CASES-BASED EVALUATIONS 53
Most of the use cases from the conformance checking category (use cases 5-7) are available in two of the evaluated systems (ARIS PPM and Futura Reflect). The only exception consists of the Quantitative criterionof use case 7 (Compliance to the explicit process), which is not present in any of the tools.
The use cases analyzing the resource (uses cases 8-11) and the time perspective (use cases 12-18) of a process are supported in a great extent. Use case 11 (Central employee) and use case 16 (Arrival rate of cases) constitute the exceptions here, as they are only available in ARIS PPM. The analysis involving the case perspective of the process, represented by use case 19 (Business rules), is supported by two systems (ARIS PPM and Futura Reflect).
Traditionally, ARIS PPM has been focusing on analyzing the performance aspects of the process. This fact is also proved by the good support that the tool provides for the use cases oriented on the resource and the time perspective analysis. At the system level, ARIS PPM offers full support for the organizational (use cases 8-11) and the time (use cases 12-18) analysis. The use cases belonging to the conformance checking are also available in the tool in a large extent (4 out 5 sub use cases). The results obtained for the sub use cases corresponding to the process discovery class differ in one aspect between the two mining functionalities available in the system. The mining approach using the activity sequence representation is only able of detecting basic control-flow patterns (therefore, no OR splits and joins, parallel constructs and duplicate activities), while the approach using the EPC representation enables the discovery of parallelism, provided that the activities executed concurrently have identical start and completion timestamps. The tool provides no information with respect to the frequency of the different paths in the process (use cases 3 and 4). To sum up, ARIS PPM is a very suitable option for analyzing any performance related aspects of a process, given the large number of use cases supported from this category. It has, however, the disadvantage of requiring a complex and rather long set up step.
Futura Reflect supports use cases from the complete process mining spectrum. The algorithm behind the Mine functionality is capable of discovering parallel constructs, besides the basic control-flow pat- terns. The Explore functionality is, however, less powerful in the process discovery part (it supports only the discovery of the basic control-flow patterns), but provides a good insight into the time perspective of the process, by enhancing the discovered model with information about activity durations and waiting times. An important aspect to be noted with respect to the control-flow perspective is that Futura Reflect is the only evaluated system able to highlight the most frequent path at the process level (use case 3). The conformance checking and the organizational analysis classes have both a good support in Futura Reflect (4 out of 5 sub use cases, respectively 3 out of 4 use cases are available). The time performance analysis use cases are either present in the models discovered by the Explore miner, either independent of the two mining functionalities, such as use case 12 (Throughput time of cases) and use case 17 (Resource utilization rate). In conclusion, Futura Reflect provides support for a large number of use cases and sub use cases at the system level. The scores at the functionality level, however, suggest that the Mine miner is intended for discovering complex process models, but provides little information about the additional perspectives of the process. The opposite holds for the Explore functionality, which scores better than Mine, when it comes to analyzing the organizational, the time and the case perspective. Futura Reflect is a tool that covers, thus, the complete spectrum of process mining techniques, and has the advantage of not needing any setting up, due to its web-based nature.
ProcessAnalyzer provides support for discovering the basic control-flow patterns, but implements no conformance checking techniques. The support for the analysis of the organizational perspective, features the limitation brought by the process representation using swimlanes flowcharts, which makes the analysis at resource level impossible. This is also reflected in the weak support for the use cases from this category, only use case 9 (Resources involved in a case) being present. Nevertheless, the spectrum
54 CHAPTER 5. EVALUATION RESULTS
of time perspective analysis techniques has a good support in ProcessAnalyzer (6 out of 7 use cases are available). As a general remark, ProcessAnalyzer is a less powerful system compared with the other two judging by the number of supported use cases. However, it has the advantage of being easy to set up, due to its integration with Microsoft Office Excel, and easy to use. ProcessAnalyzer is a suitable option for analyzing processes consisting of basic constructs and for getting insights into the performance aspects related to the time perspective.
As a last remark with respect to the results of the use case-based evaluation, we would to mention the fact that process mining users should weight the use cases based on their needs and opinions on what is relevant in practice. When deciding what process mining system to use for a particular task, one should first analyze the support for the use cases that will be applied.