XArchived: Definition Format Enhancement

Definition Format Enhancement

Note: this enhancement was not accepted. It remains here for the record.

Purpose/Goals

1. Integrate project/build.properties into build.sbt
2. Make build.sbt applicable to multiple projects
3. Make it easier to use source dependencies
4. Provide syntax for fundamental concepts
5. Fix name/ID/directory repetition, mismatch issues
6. Allow defining plugins and projects in one file.
7. Try an alternative to blank lines between settings, which is enclose multi-line settings in ``. (Another is to use line continuations.)
8. Allow configuration of the location of the project/ directory
9. Make it possible to define keys in build.sbt
10. Address conceptual mismatch between build file location (project/Build.scala) and base directory (./)
11. Compatibility. How this is accomplished is not specified, but it is assumed existing users will not have to modify their builds.

Example

sbt.version=0.13.0

project x
   `play(SCALA)` // assume play(...) returns a function of type Project => Project that configures defaults for a Play project
   dependsOn a,c
   aggregates a,b,c
   configurations a,b,c
   settings
      demo := 33
      organization := "org.example"
      libraryDependencies += junitDep
      `libraryDependencies += Seq(
         "org.example" % "demo" % "1.0"			
      )`

project a
   dependsOn
      b
      uri("...")
      `junitDep`
      "org.example" % "demo" % "1.0"

project b

settings in x,a,b
   scalaVersion := "2.8.1"

definitions
   val junitDep = "junit" % "junit" % "4.5" % "test"

task demo: Int
   A sample task

project plugins
   settings
      resolvers += "..."
   dependsOn
      "..." % "play" % "..."

Introduction

sbt Version Declaration

A .sbt file starts with an optional declaration of the sbt version to be used. This declaration is special because it is processed by the launcher (before sbt is even loaded). It must be a single line in Java properties file format that defines the property sbt.version. For example:

sbt.version=0.13.0

(Apparently, other valid first lines according to the properties format are sbt.version 0.13.0 and sbt.version:0.13.0. The format could restrict it to just one way if that is desirable.)

When multiple .sbt files are involved, it is an error if the sbt versions are not binary compatible. If they are binary compatible but not exactly the same, the version used is the version in the root of the first project loaded. If the differing sbt versions are in local projects (defined by 'in '), a warning is generated.

Imports

Scala code from sbt or plugins can be made available for use in Scala expressions without a prefix by using imports. Top-level imports have the same form as standard Scala imports, except that they are restricted to a single line and must come immediately after the sbt version declaration (if one exists). These imports do not affect the plugins section, which is processed first and separately from the rest of the file.

Defining a Project

The minimal definition of a project needs to define the project name, as in:

project xyz

Defaults are provided for everything else. A project may only be defined once for a given build. All of a projects sections (dependencies, configurations, settings) must be contained within a single file, but may use Scala code from other files to define a section. For example,

a.sbt

val x = "Demo"

b.sbt

project abc
   name := x

Name validation and normalization

Leading and trailing whitespace are trimmed from the project name. It is an error if the resulting project name is empty or does not contain at least one alphanumeric character (otherwise it would be empty when normalized). The name is then normalized to determine the project ID. Among other uses, this ID is the Scala identifier that may be used to reference the project in Scala code, such as settings. The normalization process removes all characters that are not alphanumeric. If the resulting ID does not start with a letter, a 'p' is prepended to the ID. It converts the character after a sequence of removed characters to uppercase (to obtain camel case). If the resulting ID is not unique, it is made unique by appending a unique integer in the range of 0 to N-1, where N is the number of projects with the same ID

Settings

Settings for the project are defined using Scala. There are single-line settings and multi-line settings, but each is expected to be a Scala expression of type sbt.SettingsDefinition. Both kinds begin on a new, indented line after a project definition or in a settings section. The first non-space character of a multi-line setting is a backtick ` and goes until the final backtick (full syntax described elsewhere). A single line setting goes until the end of the line. For example,

project xyz
   organization := "org.example"
   `libraryDependencies ++= Seq(
      "net.databinder.dispatch" %% "core" % "0.9.0"
   )`

settings
   version := "1.0"

The separate settings section is intended to provide flexibility in structuring the build for readability.

Keywords

The following are keywords, which mainly means they cannot be used as identifiers:

   file, %, uri, build, project, dependsOn, aggregates, configurations, definitions, task, setting, input, extends, include, in, plugins, addSbtPlugin

Plugins

Plugins are declared in a plugins section. This section has the same form as a project section, although typically only the dependsOn and settings sections are used. The project defined by this section is loaded and its full-classpath is available for Scala code in the build definition. Future versions of the .sbt format may allow control over automatic loading and injecting of plugins.

There is one important restrictions on the plugins section, which is that there can only be one plugins section for a self-contained build. The reason for this restriction is that allowing multiple plugins sections conveys isolation and scoping, but this is expensive in time and complexity to implement. Every time a new classpath is involved, there is a new compiler instance and a runtime hit on the order of 1 s. It is still possible to get the isolation and scoping in practice by using source dependencies because they involve separate builds.

Model

The parsing, compilation, and loading model is:

1. The starting set of .sbt files are parsed. Scala expressions are not processed yet.
2. Any .sbt files referenced by includes are parsed, transitively. Include cycles generate an error.
3. Names and identifiers for projects, configurations, and keys are checked for collisions. They all share the same namespace and cannot have the same simple Scala identifier.
4. Verify that there is a single plugins section in all included .sbt files.
5. That single plugins section is built and loaded. The resulting classpath is available for all .sbt files in the build.
6. A single Keys object is defined that contains a val <ID> = *Key[<type>]("<ID>", "<description>") for each declared key (task, setting, or input) in all .sbt files. When evaluating Scala expressions in a .sbt file, key visibility is enforced by explicitly importing keys from this object.
7. A Scala class is defined for each .sbt file that
   1. has the equivalent of a lazy parameter for each project visible from that .sbt file
   2. contains the definition sections of the .sbt file as its body
   3. has top-level imports from <...incomplete...>

Scala Definitions

A definitions section consists of consecutive indented lines that are compiled as a sequence of Scala statements. These statements are available for use in Scala expressions in other sections of the build file. That is, they are typically vals and defs that provide common definitions that may be used throughout the build file.

All vals and defs from all included configuration files are combined into a single compilation in the order of inclusion. These definitions are compiled and loaded before any settings are compiled or loaded. The order of Scala definition sections is preserved for compilation and loading. The definitions from included .sbt files are compiled before the including file and are visible to the definitions.

Settings Section

A settings section consists of a header line followed by an indented sequence of settings. The format of the header varies by section type, but it always starts with a keyword and only consists of a single line. Each setting starts on an indented new line. If the first non-whitespace character is `, then it is a multi-line entry that continues until the next `. The entry may not be empty. It may contain double backticks (``) to represent a single backtick. A multi-line entry consists of the text between that initial and the next, including line endings, which are normalized to \n. A single line entry consists of the text after the indentation until the end of the line, but does not including the line ending. These settings are evaluated with the declarations from the Scala definitions sections visible.

Settings in other scopes

Settings may be declared in other scopes using an in clause. The in clause accepts a list of scopes that will be applied to keys used in the list of settings. For example, this settings section:

project xyz
   settings in Compile
      a := "value"
      b <<= a map { ... }

is equivalent to this sequence of settings:

a in (xyz, Compile) := "value"
b in (xyz, Compile) <<= a in (xyz, Compile) map { ... }

A top-level settings section may be used to define settings in multiple projects at once:

project x
project y

settings in x,y
   a := "value"

This is equivalent to:

project x
   settings
      a := "value"

project y
   settings
      a := "value"

Inline Scala

Most productions have a corresponding Scala type and raw Scala can be used in place of the .sbt syntax. This is intended to keep .sbt useful (manageable) in more advanced cases. Inline Scala code is typed with an expected type according to the associated type of a production. In particular,

1. project ... has type Project
2. each subsection of project has type Project => Project
3. each setting has type SettingsDefinition
4. Individual arguments to dependsOn, aggregates have type ModuleID or (convertible to) ProjectReference
5. The sequence of arguments to dependsOn has type Seq[Either[ProjectReference, ModuleID]]
6. The sequence of arguments to aggregates has type Seq[ProjectReference]
7. configurations have type Configuration
8. names and IDs have type String
9. uri() has type URI, file() has type File

Inline Scala usually references Scala values defined elsewhere, such as in sbt, plugins, full Scala sources, or Scala definitions in the .sbt file. An example of using inline Scala:

definitions
   lazy val x = "Demo"
   lazy val commonConfigs = 
      Seq( config("demo"), config("sample") )

project `x`
   configurations `commonConfigs`

Open issues and known problems/drawbacks

1. configuring sbt plugins v. plain libraries. What works in the current proposal:

   plugins
      settings
         addSbtPlugin("example" % ...)

It would be nice in the simple case to just write:

   plugins
      "example" % ...

but then there would have to be another syntax for normal, non-cross-versioned dependencies and consistency is lost. Consistency is already lost to some extent, because normal projects list dependencies in dependsOn. It is useful for plugins to stay the same as project, though. Using plugins from source comes for free and using full Scala sources for Scala code still works (they are just the source files for the plugins project as they are now). 2. comments. They aren't specified, but it is expected that //, /* */ style comments will be allowed in most places. 3. Requires custom IDE support, custom syntax highlighting, and other disadvantages of a custom syntax. 4. Includes are rough. It could be possible to include the definitions in other files under a prefix, so that include defs.sbt allows referencing a def customTask as defs.customTask. 5. Embedding Scala expressions. Some areas enter Scala mode automatically (like settings). Elsewhere, entering Scala mode requires backticks. Multiple lines of Scala always require explicit backticks. The use of backticks themselves are open for discussion (another suggestion would be ${}). Multiple lines could instead be done with a line continuation character. 6. Increases overall complexity. Perceived complexity may be reduced and/or exposure to complexity may be more incremental than before.

Grammar

sbtFile ::=
   sbtVersion eol
   (import eol)*
   (section eol)+

import ::=
   "import" ws singleLineText

section ::=
   project |
   scalaDefinitions |
   keyDefinitions |
   plugins |
   multiSettings |
   multiConfigurations |
   include

project ::=
   "project" ws Name inDirectory? eol
      (indent projectConfig)*

projectConfig ::=
   projectTransform |
   singleProjectSettings |
   aggregates |
   dependencies |
   configurations |

projectTransform ::=
   scalaContent[Project => Project] eol

singleProjectSettings ::=
   "settings" eol
      (indent scalaContent[SettingsDefinition] eol)+

multiSettings ::=
   "settings" inClause? eol
      (indents scalaContent[SettingsDefinition] eol)+

scalaDefinitions ::= 
   "definitions" eol
      ((indent singleLineText eol) | (ws? eol)) +

aggregates ::=
   "aggregates"
      (ws projectDependency (Comma projectDependency)* eol) |
      eol (indent projectDependency eol)+

inClause ::= ws "in" ws (Name (Comma Name)*)

inDirectory ::= ws "in" ws file

file ::= "file" ws? '(' ws? QuotedString ws? ')'

configurations ::=
   "configurations"
      (ws Name (Comma Name)* eol) |
      eol (indent Name (extendsClause | eol) )+

multiConfigurations ::=
   "configurations" inClause? eol
      (indent Name (extendsClause | eol) )+

extendsClause ::=
   ws "extends"
      (ws Name (Comma Name)* eol) |
      eol (indent Name eol)+

dependencies ::=
   "dependsOn"
      (ws dependency (Comma dependency)* eol) |
      eol (indent dependency eol)+

keyDefinitions ::= keyDefinition*

keyDefinition ::=
   ("task" | ("input" ws "task") | "setting") ws Name ws ':' ws Type eol
      (indent SingleLineText eol)*

plugins ::=
   "plugins"
      (ws plugin (Comma plugin)*)? |
      eol (indent plugin eol)+

plugin ::=
   binaryDependency |
   sourceDependency

dependency ::=
   binaryDependency |
   sourceDependency |
   projectDependency

sourceDependency ::= uri | file

projectDependency ::= Name (Percent QuotedString)?

binaryDependency ::=
   QuotedString (Percent | DPercent) QuotedString Percent QuotedString (Percent QuotedString )?

include ::=
   "include"
   (ws filename (Comma filename)* eol) |
      eol (indent filename eol)+

sbtVersion ::= "sbt.version=" version

version ::= singleLineText

scalaContent[pt] ::=
   multiLineScala |
   singleLineText

multiLineScala ::= "`" ([^`] | "``")+ "`"

singleLineText ::= [^\n\r]+

filename ::=
   <valid filename character, in particular, [^\n\r,] >+

uri ::= "uri" '(' QuotedString + ')'

file ::= "file" '(' QuotedString ')'

eol ::= "\r\n" | '\r' | '\n' | EOF

indent ::= '\t'+ | (' ' ' '+)
   Indentation is relative to the current production's indentation
   Constraint: indentation used must be consistent throughout file

Name ::= 
   <scala identifier> |
   QuotedString

QuotedString ::= DQuote [^\r\n]+ DQuote

DPercent ::= ws "%%" ws
Percent ::= ws '%' ws
DQuote ::= '"'
Comma ::= ws? ',' ws?

ws ::= " "+

Discussion

Enter your alternative proposals and comments in a new section (use ## Your header here).

Discussion of Goals

• No. 7 single- versus multi-lines. The back ticks are horrible and will cause a storm of malice and hatred greater than the current discontent with blank lines. I see no advantage over the blank lines which work fine and do not look ugly.
  • +1 - Backticks used in this way are unfamiliar and therefore confusing. Line continuations, by which I presume we mean shell style \ (backslash) or something similar would at least be familiar though not in the context of Scala code or YAML / properties files---all of which are visually similar to the proposal and likely to provide cues to expectations. How about using indents to indicate line continuations. This would be in keeping with the existing structure which uses indentation to indicate that the lines that follow are all the 'continuation' of a section.
  • Thanks for the feedback. I think it is best to include examples along with alternative proposals. I think blank lines look odd in a structured format like the proposed syntax (or any new syntax) and an example would show this. It doesn't address how Scala expressions could be embedded either, so there would need to either be a syntax for that or disallow that aspect. I'm not attached to backticks, although I think they are visually lightweight and don't conflict with a Scala construct commonly used in builds. I'm surprised line continuations are familiar because of the shell but backticks are not. Backticks, although deprecated now, indicate command substitution in the shell. I considered indentation, but I thought that backticks were very clear and unambiguous, whereas indentation would end up with complicated rules. I'd be happy to see a proposal and examples indicating otherwise. Other container syntax proposals (that don't look like YAML) are fine as well.