Although we have not fully analyzed the questionnaire results and interviews yet, we provide a selection of preliminary results that provide an insight into industrial practices and give a glimpse on how our results from the open source projects compare to commercial models.
We received 42 responses by individuals from 16 countries, most of which originating from Germany (24%), USA (12%), Canada (12%), Sweden (7%), Austria (5%), Norway (5%), Brazil (5%), and Spain (5%). The majority of participants had a clear industrial background, with professional experience ranging from <1 to >10 years, and comprising roles such as developer, modeler, team leader, project manager, domain expert, product manager, marketing expert, and researcher. The product lines our respondents were involved with stem from a broad range of domains, for example: automotive, eCommerce, business applications, defense, enterprise resource planning, cyber-physical systems, power industry, telecommunication, and many more.
In general, most of our respondents (>91%) find variability modeling useful10. 76%
use a separate model to describe variability, while 47% annotate existing implementation
artifacts, for example using built-in annotations of the Spring11 [Joh04] component
framework.
7.4.2.1. Benefit of Variability Modeling
While we hypothesize that the primary application of our open source variability modeling languages CDL and Kconfig is product configuration, our survey reveals many more benefits of modeling variability. As shown in Fig. 7.2, many respondents use variability modeling to manage, plan, and document variability, for example, to support developers in keeping an overview on variability, which is also confirmed by our interviews so far. Interestingly, some respondents also use it for marketing purposes, and—as a free-text response—to estimate costs of products.
7.4.2.2. Notations and Tools
As shown in Fig. 7.3, the feature model is the dominant representation of variability among our survey participants. However, many formal (e.g. decision model, DSL, ADL, UML- based representation) and informal (spreadsheet, free-text description) alternatives are commonly used too. Some participants also use the configuration facilities of a component
10
However, there is a significant bias in this question, since we only approached participants that success- fully applied variability modeling. Thoroughly addressing this issue (first question in Appendix C) would require to identify companies that failed in applying variability modeling, which is beyond our scope.
7. Discussion and Outlook 7 8 % 7 5 .6 % 5 6 .1 % 5 3 .7 % 48 .8 % 41 .5 % 3 4.1 % 3 1 .7 % 2 9 .3 % 2 2 % 1 9 .5 % 4.9 % M an agem en t of ex is tin g v ariability P rodu ct con figu ration
R equ irem en ts s pecification D eriv ation of produ cts P lan n in g of v ariability S oftw are deploy m en t D om ain m odelin g
D ocu m en tation Q A/Tes tin g M ark etin g featu re s copin g O th er 0 50 1 00 Design/ Architecture
Figure 7.2.: Benefit of applying variability modeling
framework (e.g. Spring, OSGi, EJB), or describe variability as semi-structured key/value pairs in XML- or text-based property files. Although the latter techniques can hardly be called variability modeling, this multitude of different approaches calls for further research on the benefits and limitations of variability modeling, in particular on indicators that can predict whether the additional effort of explicitly modeling variability pays off in a project. 7 3 .2 % 3 4.1 % 3 1 .7 % 2 9 .3 % 2 6 .8 % 2 4.4% 1 9 .5 % 1 9 .5 % 1 9 .5 % 1 2 .2 % 1 2 .2 % 4.9 % 2 .4% F eatu re m odel U M L- bas ed repres en tation
S preads h eet Key /v alu e pairs D om ain - s pecific lan gu age O th er D ecis ion m odel P rodu ct m atrix F ree-tex t des cription As pect- orien ted lan gu age Arch itectu re des cription lan gu age C on figu ration facilities of a com pon en t fram ew ork G oal m odel 0 50 1 00
Figure 7.3.: Notations used to specify variability
Many of the tools listed in Section 2.2, which we know from literature, are used by our respondents. As shown in Fig. 7.4, the most frequently used commercial tool is pure::variants—perhaps not surprising given that 65% of our respondents are based in Europe, while the second major commercial tool Gears focuses on north american customers.
Our survey also reveals many smaller tools we were not aware yet, such as the decision- model-based tool Tecnalia PLUM [AE07], Hephaestus [BTB09], v.control [MR09], or SPREBA [SG09]. Interestingly, 35% of the respondents use home-grown domain-specific tools, for example, based on Eclipse EMF/xtext, Simulink or IBM Rational Software Architect. Even Microsoft Excel is reported.
7.4. Outlook: Industrial Variability Modeling 3 5 % 3 5 % 2 5 % 2 5 % 2 0 % 1 0 % 5 % 5 % 2 .5 % 2 .5 % P u re::v arian ts H om e- grow n dom ain - s pecific tools O th er open s ou rce tools O th er com m ercial tools
G EAR S F eatu reID E D O P LE R P rodu ct C on figu rator from C am os XF eatu re from P &P S oftw are P rodu ct M odeler from C on figit 0 50 1 00
Figure 7.4.: Variability modeling tools used
7.4.2.3. Scales of Variability Models
Given the unpredictable, heterogeneous modeling languages of our participants, we carefully asked for the number of “units of variability” of their variability models. These units were in most cases features or variation points—both reported by 74%; followed by
configuration options (63%), decisions (29%), and calibration parameters (23%). One
participant also reported deltas. Investigating overlaps between these units among the responses, for example, whether participants treat features and configuration options as the same entitity, is part of our ongoing research. Furthermore, it is not clear whether respondents using pure::variants refer to the feature model or the family model—the latter representing the solution space [Beu04].
Table 7.3 summarizes the percentage of participants reporting their number of models with a specific size. Although it requires further research on the particular “units of variability”, it shows that very large models exist, with almost 22% of participants having models with more than 10,000 units. The majority of models has less than 1000 units, while most participants (additionally or solely, as yet to be analyzed) deal with models that have less than 50 units.
Table 7.3.: Scales of variability models
<50 units 51–100 units 101–1000 units 1001–10000 units >10000 units
0 models 11.9% 19.0% 19.0% 40.5% 38.1% 1 model 35.7% 23.8% 28.5% 14.3% 11.9% 2–5 models 9.5% 14.3% 11.1% 0% 4.8% >5 models 16.7% 7.1% 7.1% 7.1% 4.8% sum (≥1 model) 61,9% 45,2% 46,7% 21,4% 21,5% 7.4.2.4. Complexity Problems
We asked for specific complexity issues that our practitioners faced with variability
7. Discussion and Outlook
evolution, followed by the visualization of variability models. Dependency management and problems with the configuration process, such as resolving conflicts, have only slightly lower frequency. Since our checkbox for dependency management included explosion of dependencies as an example, we strive to expand on dependency management in the interviews. Recall that dependencies in our open source product lines only grew linearly.
5 5 .3 % 5 0 % 5 0 % 5 7 .9 %
3 9 .5 %
1 3 .2 %
Vis u alization of m odels D epen den cy m an agem en t
C on figu ration proces s M odel ev olu tion Traceability O th er 0
50 1 00
Figure 7.5.: Reported complexity problems
Finally, the free-text answers to this question included, among others, modularization for multi product lines [Kru06], tests, model reduction, but also statements such as “getting developers to understand why we do this, and the correct patterns to use”.
47 .4% 6 0 .5 %
2 3 .7 % 3 6 .8 % 2 8 .9 % 2 8 .9 %
47 .4%
1 3 .2 %
D ecom pos ition in to m u ltiple m odels H ierarch ical organ ization of m u ltiple m odels S om e n otion of en caps u lation /in terfaces
betw een m u ltiple m odels Abs traction / s im plification of v ariability Vis u alization of m odels View -bas ed editin g an d v is u alization Au tom ated reas on in g tools O th er 0 50 1 00
Figure 7.6.: Reported strategies to cope with complexity problems
The strategies that our respondents use to tackle these issues are manifold, see Fig. 7.6. 60% organize multiple models in a hierarchy; however, since this proportion is surpris- ingly high, we will investigate whether some accidentally referred to the intra-model organization, such as the feature hierarchy. Two other frequent strategies—decomposition into multiple models and the use of model reasoners—confirm our observations from variability modeling in the systems software domain. Many of the remaining responses in Fig. 7.6 require further investigation in order to draw conclusions. Two interesting free-text answers confirm our Hypothesis 1 (Section 7.1.2), such as the statement “assign configuration / variability dependent tasks to a small selection of people”.
7.4. Outlook: Industrial Variability Modeling
7.4.2.5. Further Observations
Product line introduction strategy. Only 30% of our respondents developed any of
their product lines in a pro-active strategy, that is, scoped, designed, and developed the platform before any product was derived, as is the typical SPLE approach. More frequently—in 50% of the cases—multiple existing products were re-engineered into a product line, and still 45% of the respondents evolved a single initial product into a product line. A combination of any of these three strategies is reported by a quarter of our participants.
This observation confirms, so far, one of the common hypotheses in the product line community that only a minority of real-world product lines follow a pro-active strategy. It is a call for further research on migration support from single products into product lines with systematic variability management.
Broad perspective on variability modeling. One of the most interesting comments on
the survey questionnaire itself retroactively supports our study on variability in non- embedded and non-systems-software projects that use modern dynamic languages. It also confirms our Hypothesis 3 (Section 7.1.2):
Both the field and this study could use a broadening of perspective. My day job is building Java based server side software, which tends to be one of a kind, non product line type software. Java is a rich language and technologies such as spring and maven provide very rich variability tooling. Additionally, using things like puppet for deploying into cloud architectures as well as staging and testing infrastructure means that we have a lot of variability. We deploy in different configurations to different data centers, use feature flags as well as AB testing to test new functionality, etc. My feeling is that the research field still assumes a traditional low tech embedded software perspective where the lack of a lot of things need to be compensated for with variability modeling tooling and cumbersome build systems. So, I don’t model variability, instead I make software that has variation points that are explicitly configurable. The activities of developing and designing when following a continuous deployment model are inseparable.