This section discusses the evaluation criteria for the identified goals. Further- more, it discusses the rationale for each chosen criterion. Note that the order of this section reflects the overall order of the chapter.
Concurrent Programming Concepts benefit from the OMOP To demon-
strate the benefits of the OMOP, this chapter argues that it simplifies the implementation of supported concurrent programming concepts by solving the problems implementers face when they use common ad hoc approaches. This discussion relies on the problems identified inSec. 3.3. For each of the problems, it illustrates how it is solved using the OMOP and shows the im- provements over existing ad hoc solutions. This comparison shows the ben- efits of the OMOP by showing that it solved the implementation challenges language implementers are facing when they target today’s VMs.
The evaluation discusses how the OMOP addresses these problems as part of three case studies in Sec. 6.2. These case studies are the implementation of Clojure agents, an STM system, and event-loop actors. They are chosen to cover all of the identified problems and all mechanisms provided by the OMOP. Furthermore, the characteristics of the three implemented concurrent programming concepts are sufficiently different from each other to discuss the OMOP and the problems it addresses from different angles. Therefore, the case studies provide the necessary foundation to fully evaluate the OMOP and its benefits.
Fulfillment of Requirements Sec. 3.4 identified concrete requirements as a
foundation for the design of the OMOP. This evaluation discusses these re- quirements as part of the three case studies in Sec. 6.2. For each case study the section evaluates which mechanisms the case study requires and how it maps onto the OMOP. Furthermore, to demonstrate the fulfillment of the re- quirements the discussion includes the evaluation criteria Sec. 3.4 stated for each of the requirements.
To complete the discussion of how the OMOP fulfills its requirements,
Sec. 6.3 discusses the concepts from which the requirements were initially
derived. For each of these concepts, it evaluates the degree to which it is sup- ported by the OMOP. The evaluation differentiates between concepts where the main aspects are covered and concepts where only some aspects are sup- ported by the OMOP to assess the extent to which the concepts benefit. Even if only some of the concept’s aspects are supported, the benefits are substantial, since the OMOP addresses common implementation challenges.
Applicability Sec. 6.4 assesses the applicability of the OMOP by showing
that its use does not increase the implementation size for the supported con- current programming concepts.
Implementation size is often assumed to be an indication for implementa- tion complexity, however, this assessment has an inherently social and subjec- tive component. Therefore, the evaluation here considers only directly mea- surable aspects. It uses implementation size in terms of lines of code (LOC) as a surrogate measure for implementation complexity. As pointed out by a number of studies, size seems to correlate with other metrics to such a de- gree that it is not clear whether there is value in these additional metrics to assess complexity. Jay et al.[2009] found a strong correlation between cy-
clomatic complexity [McCabe,1976] and lines of code. For their experiments,
they go so far as to conclude that cyclomatic complexity has no explanatory power on its own.van der Meulen and Revilla[2007] found that Halstead Vol-
ume [Halstead,1977] and cyclomatic complexity correlate directly with LOC
on small programs. Emam et al. [2001] studied object oriented metrics and
their relation to class size, i. e., LOC. They find that previous evaluations of metrics did not account for class size and that the effects predicted by these metrics can be explained based on size alone. They conclude that the value of these metrics needs to be reevaluated.
Since these indications cast doubt on the value of metrics other than size for assessing complexity, this evaluation uses only size-based metrics as a surrogate for measuring complexity to compare the ad hoc and OMOP-based implementations.
The second aspect of applicability is the ability to vary language guaran- tees.Sec. 6.5.1discusses how the OMOP can be used to vary the semantics of concurrent programming concepts. The goal is to argue that it enables a wide range of variations covering a significant part of the design space spanned by the supported concurrent programming concepts. Based on the agents case study, the section argues that the additional semantic guarantees provided over Clojure agents are the kind of variations that are desirable and a good indication for the variability of the OMOP. The range of supported concepts is an other indication that supports the conclusion that the OMOP provides sufficient flexibility for the definition of language semantics for concurrent programming.
Relevance of supported Concepts To evaluate the relevance, Sec. 6.5.1 ar-
gues that the supported concepts are used today. It gives for each of the supported concepts an example of either recent research in the correspond- ing area, or a programming language that is respected and used in industry. Based on these indications for actual use, it argues that each of the supported
concepts has practical relevance. Consequently, the concepts supported by the OMOP are relevant.
Absence of wide Support in Today’s VMs In order to evaluate the nov-
elty of the OMOP, i. e., to assess whether these concepts are not yet widely supported in VMs,Sec. 6.5.1relies on the VM survey inSec. 3.1. For each con- cept, this analysis assesses whether the concept is supported. Furthermore, the analysis considers the whole set of concepts and its support by a single VM to validate the initial premise that today’s VMs support is insufficient. By showing based on the VM survey that this is the case, it supports the claim that the OMOP can add relevant support to today’s VMs.
Significance In order to evaluate the significance of the support the OMOP
provides, Sec. 6.5.1 argues that the OMOP facilitates the implementation of all concepts that were identified as benefiting from VM support for semantic enforcement. Since the OMOP covers all concepts, there is an indication that the supported set is sufficiently large to warrant interest.
Unifying Substrate Finally, Sec. 6.5.1 discusses the unifying properties of
the OMOP. The goal is to argue that the OMOP makes an abstraction of con- crete programming concepts. The section shows this by relating the concepts to the mechanisms the OMOP exposes and by demonstrating that it is not a one-to-one relationship between mechanisms and concepts. Furthermore, it argues that the OMOP provides a minimal set of elements to fulfill the re- quirements. It demonstrates that the proposed design in its current form is minimal by arguing that none of the parts can be removed without reducing the number of concurrent programming concepts that can benefit from it, and that removing any part would also result in unsatisfied requirements.