LIBRO TERCERO
JUICIO 12 de Julio
Some technologies discussed in this chapter are provided under open source licenses, and it is therefore appropriate to also evaluate the open source quality of these technologies. Two existing frameworks for evaluating open source components are theQualification and Selection of Open Source Software
(QSOS) [265] and theBusiness Readiness Rating(OpenBRR) [263] frameworks.
Deprez and Alexandre [74] evaluated both of these frameworks and found that while QSOS pro- vides more extensive rating criteria and supports versioning, OpenBRR provides scoring with less ambiguities and results can be tailored towards a specific context. At the time of writing, however, the progress of OpenBRR appears to have stalled, with no documentation released since 20052. Never- theless, in this thesis the OpenBRR will be used to form the basis of open source evaluation criteria.
The most recent release of OpenBRR [263] defines 28 metrics categorised into eleven categories, along with arepresentative setof metrics to illustrate the model. The authors argue that these repre- sentative metrics should be tailored towards the particular domain. In this thesis, no such tailoring was performed as it was not necessary; the full range of metric values were present through the evaluations, so no reweighting was necessary.
While every category in the OpenBRR ranking system represents an important part of the overall ranking, five of these categories have been selected as particularly important for the proof-of-concept implementation –quality, support, documentation, adoption and architecture. These categories are those most likely to suggest a simple implementation process, and are described by the OpenBRR standard [263] as follows:
• Quality: “Of what quality are the design, the code, and the tests? How complete and error-free are they?”
1The Open Source Initiative:http://www.opensource.org.
2At the time of writing, the OpenBRR websitehttp://www.openbrr.orgsimply states that “we will soon be updating this site with new content.”
6.2 Metamodelling Environments 161
• Support:“How well is the software component supported?”
• Documentation:“Of what quality is any documentation for the software?”
• Adoption:“How well is the component adopted by community, market, and industry?”
• Architecture: “How well is the software architected? How modular, portable, flexible, extensi- ble, open, and easy to integrate is it?” In particular, a system with a higharchitecturescore will most likely support third-party extensions.
The categories ofusability,performance, security andstability are not too relevant for proof-of- concept implementations, as their impact is reduced within an experimental environment. Addition- ally, thecommunitycategory is very related to thesupportandadoptioncategories, so may be omitted. The values for every OpenBRR metric evaluated against each technology is provided in this thesis in AppendixC.
In the following sections, the overallOpenBRR Ratingof each technology implementation will be calculated, which defines a score from a minimum of 28 (unacceptable) to a maximum of 140 (excel- lent)3. An average score of each of the five important categories discussed above will be provided as a numeric value.
Applying OpenBRR to Closed-Source Projects
At first glance, it seems possible to apply the OpenBRR metrics to closed-source projects. For exam- ple, metrics such as frequency of releases, types of documentation, and activity on mailing lists can all be derived without needing access to open-source artefacts or methodologies. However, other metrics such as number of bugs, the number of security vulnerabilities and the number of unique contribu- tors cannot be derived from a closed-source development methodology. Consequently, closed-source projects are given an OpenBRR ranking ofn/ain the following chapters.
6.2
Metamodelling Environments
A number of metamodelling environments exist in academia and industry to assist the quick develop- ment of model-driven platforms and technologies. It is preferable to reuse an existing metamodelling environment over developing an environment from scratch, as a metamodelling environment repre- sents a significant amount of development effort, and it does not seem like a finished environment would be an improvement over existing environments.
There are many existing metamodelling environments as discussed by Fowler [102, pg. 140]4, such as MetaEdit+ [178], the Meta-Programming System (MPS) by JetBrains [170]5, and the Generic Mod- eling Environment [206]. However, a full evaluation of each of these metamodelling environments is outside the scope of this thesis. In this section, four popular metamodelling environments – selected based on their industry support, integration with other technologies within a model-driven tool stack, academic activity, and source code available through an open source license – will be investigated and evaluated according to a number of evaluation criteria.
3In OpenBRR, each criteria is given an integer ranking between 1 (unacceptable) to 5 (excellent) [263]. A future standard of OpenBRR may benefit in instead changing the scale to use 0–4 to simplify ranking comparisons.
4Fowler uses the term “language workbench” to describe a metamodelling environment.
5The first public release of JetBrains MPS was released in 2009, and as such was not an option when this research was started.
This investigation is particularly important since the metamodel forms the “core” of the resulting development environment, and the choice of metamodelling environment will affect other aspects of the implementation, such as code generation and verification technologies.
Additional Evaluation Criteria
Along with the common comparison criteria discussed earlier in Section6.1.1and Section6.1.2, each metamodelling environment will be evaluated on a number of additional criteria:
• Visual Definition: Can a metamodel be implemented graphically? For a metamodelling envi- ronment that is intended to support the definition of graphical modelling languages, this feature may be essentially provided “for free” by the environment itself.
• Graphical Editor: Does the metamodelling environment provide a graphical interface for in- teracting with a developed model instance, and is this interface the intended means of end-user interaction? Such an environment could be used for both the metamodelling and graphical modelling environments, but it is important that a graphical representation of a model instance should be kept separate from the underlying model instance representation.
• Metamodel Extensibility:If a developer creates an instance of a metamodel, can this metamodel later be extended by a third party? That is, can a third party add additional elements and con- straints to an existing metamodel? Metamodel extensibility allows a third party to extend the modelling environment or adapt it to a particular domain, and is a driving force behind UML Profiles [114].
• Meta-metamodel:What is the underlying meta-metamodel of the developed metamodels? That is, what language or technology is used to represent instances of metamodels used in the mod- elling environment?
• MDA-compliant M3: Is the meta-metamodel compliant to the MDA requirement for a meta- metamodel [185, pg. 88]; that is, has the meta-metamodel been defined in terms of itself? • Metamodel Sources: What kinds of sources may be used to develop a metamodel instance?
Being able to import a metamodel from an existing source may simplify the development of the modelling environment.
• Verification Languages: Which model instance verification languages does the metamodelling environment natively support, if any? In many cases, the metamodelling environment may be extended with additional languages as discussed later in Section3.3.