Building a domain-specific language (DSL) for structural parts of an application has always been rather easy with Xtext. But structure alone is not sufficient in many cases. When it comes to the behavioral aspects users often fall back to implementing them in Java. The reasons are obvious: expressions and statements are hard to get right and extremely complex and therefore costly to implement.
This document introduces and explains a new API, which allows reusing predefined language constructs such as type references, annotations and fully featured expressions anywhere in your languages. You not only reuse the grammar but the complete implementation including a compiler, interpreter, the whole Java type system and a tight IDE integration. And the best part is, that it is relatively easy to do so. It is actually just two steps:
Using the traditional Xtext grammar language you freely describe the syntax of your language. The specialty for JVM languages is, that you inherit from an abstract grammar org.eclipse.xtext.xbase.Xbase, which predefines the syntax for the reusable parts. You do not need to use all of them directly and you can of course change the syntax or add new concepts, as it seems fit.
Having the grammar defined, you now need to tell Xtext what your language concepts mean in terms of Java constructs. For that purpose you use a so-called model inferrer, a special API that allows you to create any number of Java classes, interfaces or members from your DSL. This hook not only defines how your language is translated to Java, but also defines the scope of the embedded expressions. The expressions from your language 'live' in the context you give them. If you want an expression to have access to certain local variables, just put it into a method with appropriate parameters or use instance fields if they work better.
While in plain Xtext you would usually customize a bunch of further services to fit your needs, Xbase already has JVM model aware implementations almost all of them. For example, a generic code generator generates the Java code directly from the inferred model. The same way, Xbase already integrates your language with JDT to an extend that customizations beyond the JVM model inferrer will only be necessary for very special cases. You can naturally mix Java and DSL code in the same application without any barriers.
The inferred model also reveals your DSL constructs to other JVM languages. The Java type system is used as a common hub to integrate arbitrary languages with each other. You can for instance call templates directly from a script and vice versa. You do not even need to generate the equivalent Java code; everything is based on the Java types you create in the model inferrer.
To illustrate the power and flexibility of these two abstractions, we have built seven example languages using them:
- A simple scripting language
- A Grade-like build DSL
- A DSL for statically-typed MongoDB documents
- A Guice modules DSL
- A Playframework-like HTTP routing language
- A template language
- A Logo-like programming environment for educational purposes
Each language is very simple and focuses on the value a DSL can add to the respective viewpoint. It is meant to give you an idea of what is possible without being a complete practical solution. Yet the languages are flexible and come with powerful core abstractions. We also covered different technical aspects of customizing to the language infrastructure. Some languages have special syntax coloring, others provide customized outline views or content assist. All aspects of a language are still fully customizable when using Xbase.
Please be aware that some of the new API covered in this document is not yet finalized and will likely be improved in future releases in incompatible ways. Usages of such API are marked with a warning.
To run any of the examples, you will need Eclipse 3.6 or better for your platform. In addition, you have to install Xtend and of course Xtext.
If you prefer a simple all-inclusive installation, consider downloading the latest Xtext distribution.
Additional requirements are mentioned in the Running the Example section of each chapter.
You can get the source code for all languages from the github repository at https://github.com/xtext/seven-languages-xtext. The repository contains two folders languages and examples. Download the plug-ins from languages into the root workspace and the examples into the runtime workspace spawned from the root one using Run → Run Configurations... → Eclipse Application → Run (<language>).
Each language consists of several Eclipse projects
org.xtext.<language>– The base infrastructureorg.xtext.<language>.ide-> The generic IDE servicesorg.xtext.<language>.ui– The editor based on Eclipseorg.xtext.<language>.tests– Tests for the languageorg.xtext.<language>.lib– Runtime libraryorg.xtext.<language>.example– Examples for using the language
Some of the languages do not include all of these plug-ins but the general structure is always the same.
Any general code in the examples is implemented in Xtend. Xtend is a more expressive and less verbose way to implement Java applications. It is 100% interoperable with Java APIs and compiles to readable Java code. In addition, it uses the same expressions that we use in our example languages. In fact it is built with the very same API that is described in this document. This should give you a taste of how powerful JVM-languages built with Xtext can actually be.
Xtend is designed to be easy to learn for Java developers. In this section we will shortly describe the most important language features that were used in our examples. For a full description of the Xtend language, please refer to the Xtend documentation.
Just like a Java file, an Xtend file starts with a package declaration and an import section followed by one or more class declarations. Semicolons are optional. All types are public by default. Xtend classes can extend other classes and implement interfaces just like in Java. It does not make any difference whether they were originally declared in Xtend or in Java.
println(foo.bar) // instead of println(foo.getBar())
foo.bar = baz // instead of foo.setBar(baz)
foo.fooBars += foobar // instead of foo.getFooBars().add(foobar)Empty parentheses on method or constructor calls can be skipped.
Methods are introduced with the keyword def or override if they override/implement a super type's method. Methods are public if not specified otherwise. The value of the last expression is returned if no explicit return expression is used and the method's return type is not void.
Variables are declared with the keywords val (final) or var (non-final). Field declarations can use the same syntax as in Java.
Xtend is statically typed, but you do not have to specify the type in a declaration if it can be inferred from the context:
val x = newArrayList('foo', 'bar', 'baz') // x is of type List<String>
def foo() { // equivalent to def int foo()...
1
}// assume the class Foo defines a method foo(Baz)
extension Foo theFoo
def bar(Baz baz) {
baz.foo // calls theFoo.foo(baz)
}Static methods can be put on the extension scope with a static extension import, e.g.
import static extension java.util.Collections.*
...
val foo = 'foo'.singleton // calls Collections.<String>singleton('foo')class Foo {
def foo(Bar it) {
foo // will call it.foo() or if that doesn't exist this.foo()
}
}Xtend provides lambda expressions. These are anonymous functions in square brackets.
[String foo, String bar | foo + bar]
// a function (String foo, String bar) { foo + bar }As this is a bit bulky, there are more rules to make working with lambdas more attractive:
- When a lambda expression is the last argument in a method call, it can be put behind the closing parentheses.
- Lambdas are automatically coerced to interfaces with a single function. Parameter types will be inferred.
- If you skip the declaration of the only parameter, it will be implicitly called
it.
new Thread [ println("Hello concurrent world") ]
// lambda will be coerced to a java.lang.Runnable
val list = #['fooooo', 'fo', 'foo'] // #[] delimits a list literal
list.sortBy[ length ]
// lambda is coerced to a function (String)=>Comparable
// equivalent to list.sortBy[String it | it.length]We most often use this expression in the examples to generate some synthetic Java boilerplate code. Here is an example from the http routing language:
'''
String url = request.getRequestURL().toString();
«FOR route : routes»
{
//java.util.regex.Matcher will be imported in the generated Java file
«Matcher» _matcher = _pattern«route.index».matcher(url);
if (_matcher.find()) {
«FOR variable : route.url.variables»
String «variable.name» = _matcher.group(«variable.index + 1»);
«ENDFOR»
«IF route.condition != null»
if («route.nameOfRouteMethod»Condition(request, response
«FOR v : route.url.variables
BEFORE ", "
SEPARATOR ", "»«v.name»«ENDFOR»))
«ENDIF»
«route.nameOfRouteMethod»(request, response
«FOR v : route.url.variables», «v.name»«ENDFOR»);
}
}
«ENDFOR»
'''