Versioning refers to creating a second version of a language (or a product) that’s compatible with the previous version. Code and other artifacts from the previous version can either run on the new version or there is a well-defined process for moving from one version to the next.
The development of the second version is a critical point in the lifecycle of any language. At that point, you no longer have a blank slate to draw upon; you need to take into account the existing investments in the previous version as well. The alter-native would be to stop with a single release, or to abandon all the time and money invested in the previous version. These aren’t choices most people would make.
In this chapter, we’ll lay the foundations we need to create the versioning story for our DSLs. After that, we’ll touch on several different approaches to versioning a language.
Many of the approaches that we’ll discuss in this chapter are applicable to existing languages, but these approaches and techniques are most effective if they’re used when you initially design and build the language. I strongly suggest that, even for small languages, you take the time to consider the versioning strat-egy you’ll apply.
In this chapter
DSL versioning considerations
DSL versioning strategies
DSL versioning in the real world
9.1 Starting from a stable origin
The obvious first step in any versioning scenario is to determine where things are right now. In order to make sure that the next version can accommodate the current ver-sion, you need to know what the state of the current version is. For example, if you want your next version to support existing scripts, you need to know what can be done with the language as it is now. This isn’t the obviously redundant step that it may appear to be at first glance.
Trying to reverse engineer language syntax from its implementation is challeng-ing. It’s possible, for sure, but hard. The main hurdle is that a language is usually flexi-ble in its input, and two people will often express the same thing in different ways.
You need some sort of baseline to ensure that you understand how the language is used. In general, I like to have this baseline in two forms:
Executable form—This consists of unit and integration tests that use the language features. These tests allow you to see how various elements of the language are used in context. The tests we discussed in chapter 8 will do nicely here, and they’ll ensure that we also have a regression test suite for the next version.
Documentation—Although tests are my preferred method of diving into a code base, additional documentation will give you the complete picture, so you don’t have to piece it together. This is particularly important when you want a cohe-sive picture of how all elements of the language play together. We’ll discuss how best to approach writing documentation for a DSL in chapter 11.
With both tests and documentation in place, you’ll have a good understanding of how the language is supposed to behave, and you can go on to extend and change it from a well-known position, rather than blindly.
Once you know where you are, you need to figure out where you’re going, what kind of changes you need to make, and what their implications will be on the version-ing story.
The lone language developer
You might think that versioning isn’t an issue for languages that are written by and for a single developer, but this is most certainly not the case. There are always at least two people on any project: the developer who writes the code, and the developer who reads it. Even if they’re the same physical person, they’re looking at the code at different times.
Even if you wrote the code, you can’t count on being able to remember and under-stand all the implications and use cases six months from now. I’ve made that mis-take several times in the past, and it resulted in a lot of delay.
175 Planning a DSL versioning story
9.2 Planning a DSL versioning story
A DSL versioning story defines the overall approach you take to versioning the DSL over time—what versioning strategies you employ and what versioning guarantees you give to DSL users.
In chapters 3 and 5, we defined the parts that a DSL is composed of:
Engine—The code that executes the DSL script and processes the results
Model and API—The data and operations exposed to the DSL
Syntax—The syntax of the language
Environment—The execution environment of the DSL
When we talk about versioning a DSL, we’re talking about making a change to one of those parts (or all of them), while maintaining some desired level of compatibility with the previous version. We’ll examine each of them and look at the implications of each type of change.
9.2.1 Implications of modifying the DSL engine There are several ways to modify the DSL engine.
The best scenario from a versioning perspective is a nonfunctional change, which means the behavior of the engine remains the same in all cases; the change is focused on internal behavior, optimizations, and the like. Any change in observable behavior is a bug, so there can’t be any compatibility issues.
A functional change is when you’ve made some sort of change to the behavior of the engine. In this situation, you need to determine the implications of this change on existing DSL scripts.
In many cases, functional changes to the engine are safe changes to make. The engine is responsible for processing the results of executing the DSL, so the DSL will rarely be affected by this change. By the time the engine is running, the DSL has completed.
But there is one scenario in which a change in the engine can cause compatibility issues for the DSL. Restricting the legal values or operations that a DSL specifies can cause a DSL to fail, because formerly valid values may no longer be valid.
Listing 9.1 shows a valid snippet of a Quote-Generation DSL script.
Versioning means worrying about intended consequences
Although it’s entirely possible for a change to cause unforeseen incompatibilities with previously written DSL scripts and code, that isn’t something that I care about when I am talking about versioning strategies.
If I make a change that has an unforeseen compatibility issue, what I have is a bug, and this should be caught using the regression test suite. Versioning, on the other hand, deals with how one can mitigate the issue of planned incompatibilities.
specification @vacations:
requires @scheduling_work requires @external_connections
Let’s say that you changed the engine to only allow a single requirement for each specified model; this previously valid snippet would become illegal.
We need more information about the reasons for this change to decide how to deal with this issue, so we’ll defer dealing with this for now and move to dealing with changes in the model and the API.
9.2.2 Implications of modifying the DSL API and model
The API and the model exposed to the DSL compose a large part of the syntax that the DSL has. This means your existing investment in building a clear API and a rich model can be used directly in the DSL, which will save you quite a bit of time.
There is a problem with exposing the application’s core API and model to the DSL: any change you make in the core API has to be tested with regard to how it will affect the DSL. In effect, the DSL becomes yet another client of the code, with all the usual implications that has on creating public APIs, worrying about versioning conflicts, and so on.
I don’t expect you to be surprised that I recommend against directly exposing the API and model that are used in the application to the DSL. What I recommend is cre-ating a facade layer between the two—you expose the application concepts that you need to use in the DSL only through a facade. This approach simplifies versioning, because changing the core API will not affect DSL scripts created previously. All the changes stop at the facade layer, which remains static. This allows parallel develop-ment of both the DSL and the model, without worrying about versioning issues between the two. What makes for nice, fluent, readable code doesn’t necessarily make for good DSL syntax, so separating the two is useful even if you haven’t reached the point where you’re considering versioning concerns.
Listing 9.1 A snippet from a Quote-Generation DSL script
Versioning an external DSL
The problem with versioning an internal DSL is that there isn’t usually a formal defi-nition of the syntax. Most of the tools for building external DSLs work by building on a formal syntax definition, which is usually in the form of BNF (Backus-Naur Form, a language for formally defining a language syntax) or syntax derivative.
When you have a formal syntax definition, it’s quite easy to create a diff between ver-sions 1.0 and 2.0 of the language, and see what the syntactic differences between the two are. This is something you need to do yourself when building an internal DSL.
But syntax is a small part of versioning a DSL. The model, API, behavior, and appli-cation integration are all parts of the versioning story, and they all require the version-ing approach that’s outlined in this chapter.
177 Planning a DSL versioning story
WARNING Remember the client audience for your DSL. If you make a change to your DSL, you need to communicate that to the users of the DSL and ensure that they understand what changed, why, and how they should move forward in working with the DSL. This can be a much greater challenge than the mere technical problems you’ll run into.
At the same time, I don’t like having to start with a facade when I have a one-to-one relationship between the facade and the model. I like to use what I call a lazy facade approach: use the real objects until I run into the first situation where the needs of the DSL and the needs of the application diverge, and create the facade at that point.
This approach is often useful when you’re starting development, because it frees you from maintaining a facade when the code and the DSL are both under active development.
Now let’s consider the effect of changing the syntax.
9.2.3 Implications of modifying the DSL syntax
Modifying the syntax is generally what comes to mind when people think about ver-sioning a DSL. This seems to be the most obvious place where incompatibilities may occur, after all.
Indeed, this is a troublesome spot, which requires careful attention to detail (and a robust regression test suite). We’ll look at how to deal with it in detail when we discuss versioning strategies in section 9.4. There are quite a few tools that can help us deal with this issue.
That’s about it from the point of view of changes that can affect the DSL. But we should consider one other thing that may change over time and that can certainly cause incompatibilities down the road—the DSL environment.
9.2.4 Implications of modifying the DSL environment
The DSL environment is as much a part of the DSL as its syntax, but it’s easy to forget that. So easy, in fact, that I forgot to include it in the first draft of this chapter. We talked about the DSL environment in chapter 5. It includes everything from the nam-ing convention, to the script ordernam-ing, to where you execute each DSL script.
Let’s look at an example system where a DSL is used to provide additional business behavior. The DSL scripts are located in the following hierarchy: