sbt 2.0 ideas

[eed3si9n]

This is a discussion thread to get feedback on https://eed3si9n.com/sbt-2.0-ideas.

Please note that sbt is mostly community and Scala Center maintained. I don't work on sbt at work. We are also always happy to hear ideas and issues affecting the open source authors and newcomers, but if you use sbt at work and have issues, feel free to open a new Discussion topic, and please consider contributing the required features or fixes that your work will benefit.

[lolgab]

I'm not sure it's enough to have a global setting for platform.
In rare cases in Scala.js you need to depend on JVM artifacts. If you are writing a macro, you can use a JVM library to be called only at compile time, also Scala.js's scala-library depends on Scala JVM's standard library if I remember correctly.
In those cases maybe people need to fallback to manually adding the _3 suffix manually.
Commenting because I tried fixing the same problem in Mill but without finding a satisfactory enough solution.

[sjrd]

The document says you can override the platform of a dependency with

libraryDependencies += ("com.github.scopt" %% "scopt" % "4.1.0").platform(Platform.jvm)

I believe that addresses that concern.

[eed3si9n]

Yea, what Sébastien said. To go into a bit of the implementation detail, that's why I added platformOpt to ModuleID in addition to tracking platform in ScalaModuleInfo. This allows None state on ModuleID to mean automatic, and we can set Some("jvm") at individual dependency level, which would take precedence over project level platform.

[ckipp01]

Really exciting to read this @eed3si9n! I just wanted to show some strong support of the following:

idea 5: BSP support + persistent workers

I know for the Metals side of things as it pertains to this discussion about defaulting to sbt for BSP having sbt be non-blocking for Metals users would make this choice almost a no-brainer. So I personally think the impact of this change would be huge for us.

Thanks for writing this all up!

[mgladdish]

I don't have concrete suggestions for how to achieve these as I don't know anything like enough about sbt's internals. But I can list a bunch of pain points that have cost me literally months of time working on sbt-based projects and writing a couple of plugins which I'd dearly love to see addressed.

1. Separate classpaths for build plugins. I've spent ages trying to diagnose classpath conflicts across multiple builds when a plugin I've written introduces dependencies whose transitive deps conflict with those from other plugins. Putting all plugins in one classpath soup is really painful.

2. Opaque plugin implementations. Clicking through to what's supposed to be the implementation of a task only to find its relatively useless task definition is really painful. Several times I've just given up trying to find the code that implements a task's behaviour. Although I suspect this one is too baked in to sbt's design to be changed?

3. I don't know how much sbt can do about this one, I guess not a lot, but having the version of Play tied to the version of sbt is also really painful. Stuck on an older version of Play for reasons? Tough, you're also stuck on an old version of sbt.

4. Documentation, documentation, documentation. Sbt's current docs are, er, let's say challenging. Relatively simple everyday build configs are still confusing and it's easier to resort to StackOverflow than it is to find the right bit in sbt's docs. E.g. alternate Test settings, or making one Task dependent on another.

Admittedly some of the above will be down to my shallow understanding of sbt. My counter argument would be I don't want to need a deep understanding of my build tool to do what should be relatively day-to-day changes in my build.
Whilst I appreciate the happy path of simple builds looks nice and simple in a short build.sbt, as soon as I need anything out of the ordinary then it's far from simple, far from discoverable, and takes a lot of head scratching to even know where to begin. And it's great, necessary even, that going fully custom is possible, but I can't help thinking the abstractions of sbt are working against me rather than helping whenever I do.

[eed3si9n]

I am going to focus on the first pain point first:

Separate classpaths for build plugins.

Could you describe what kind of tasks these plugins were performing? Are they mostly around file manipulation?
If so I wonder if we could something like forking off a process, similar to what I've experimented in downloading and running app on the side with sbt-sidedish. iirc this supported Scalafix code generation on sbt 0.13 + Scala 2.10, even though Scalafix was on Scala 2.12. This pattern of creating small app and running it as part of build is a common pattern in Bazel as well.

[mgladdish]

I can't remember the specifics but the plugin does things like POSTing a build artifact to an HTTP endpoint. IIRC adding a dependency on my first-choice http client conflicted horribly with another plugin my project used.

But yes, I guess, pulling the guts out into a standalone app and invoking it from the command line would be a workaround. But surely we can do better than that? The closer we can get sbt to 'regular' scala code the better, surely. And using isolating classloaders is a standard JVM pattern for managing plugins to avoid just these sorts of conflicts.

[ches]

Regarding the docs, as a long-time sbt user and sometimes-plugin-developer I am regularly thankful for the depth and breadth of the docs as they exist today (and they have come a long way).

That said, I do agree it still doesn't feel like an optimal balance has been struck of technical reference and approachable guides. It skews pretty heavily toward the former, and the Howto section that exists—while welcome—is more a potpourri or cookbook than guided paths. The site theme could perhaps stand for a visual freshening, too.

There's an accretion factor, sbt has been around for a good while and its idioms and community best practices have evolved over time, UX has improved, but the docs don't always keep up in every corner, or need decluttering. Many of us have lost the beginner's perspective by this point, too.

Docs tend to be a continual gardening process—the software changes and its users do too, and "good" is subjective. Raising specific feedback for what you're looking for would be helpful, examples of what "should be … day-to-day changes". Discussion threads are likely a good forum for that leading up to doc PRs.

You mentioned task graph wiring, there's a bit of coverage but I agree it could be improved. Another that's been on my mind to try to work on soon is version schemes, they're barely covered in the official docs, mostly only an external blog post and it doesn't have a lot for what app build users need to know as opposed to library publishers.

Just some food for thought, I think we should take specific docs discussion to another thread.

[plokhotnyuk]

Please add built-in support of multi-release JAR to allow assembling of JARs that could be specialized for different Java versions.

That will allow to continue Java 8 support and keep up improving using latest Java versions.

[mdedetrich]

So I have been trying to implement this as part of sbt-mult-release-jar plugin but I came across some fundamental limitations in current SBT 1.9.x (at least for a more ideal implementation that doesn't require having SettingKey for every single JDK version).

Unless I have missed something in that comment, this is adding more weight to just adding proper native support to SBT 2.x.x.

[eed3si9n]

Thanks @plokhotnyuk and @mdedetrich for the comments. I've seen Multi-Release JAR (MR JAR) documented mostly for creating an opt-in usage for JDK 9 and up, but can it also create an opt-in for JDK 8, so by default it's assumed that we have a modern 11/17/21 JDK, but on JDK 8 it does something else?

[mdedetrich]

@eed3si9n I am not entirely sure on this actually. My current understanding is that step up from the lowest JDK you want to support, not down (if that makes sense).

i.e. your codebase is compiled for the lowest JDK that you are meant to support (lets say 8) and then if you have code that is specific for a later JDK thats what you use mult-release jar for (and those compiled classes get put into META-INF/versions/11/ (where 11 stands for JDK 11).

If you run that on JDK 8 it will ignore the classes in META-INF/versions/11/, if you run JDK 11 or later it will load the classes in META-INF/versions/11/ and then the standard JDK 8 ones (which will be located in the standard locations) with the ones in META-INF/versions/11/ taking precedence if there is a duplicate.

https://www.baeldung.com/java-multi-release-jar#mrjar seems to indicate this as well.

[eed3si9n]

ok. JEP 238 does say that:

In a JDK that does support MRJARs, ... it would search for classes and resources first in the Java platform-specific directory corresponding to the currently-running major Java platform release version, then search those for lower versions, and finally the JAR root.

so that's actually pretty good since we can create META-INF/versions/11/ once and it would work for all JDKs 11 and up.

[mdedetrich]

Another thing to note is that the most intuitive way to model this functionality is to extend Compile / Test configuration (since the whole point is to compile classes under a different configuration, i.e. a different JDK) which is another point against the idea 3-A: limit dependency configuration to Compile and Test idea

[Baccata]

idea 8-A: in-source project matrix

This is great. It'd also be great if the internal graph could be optimised to prevent quadratic duplication of settings. Heavy-reliance on project-matrix leads to enormous build graphs in my experience (easily reaching 100K+ settings) which makes the start-up time extremely slow. Not sure what'd be achievable in that regards though.

[eed3si9n]

idea 4: unify settings and tasks

It's hard to make prediction on what would make software fast, but the reason I listed unification of settings and tasks is this. We might still need to evaluate the graph at load time, but if everything is a task, we wouldn't have to initialize them on load until you actually start using them.

[agilesteel]

Would be awesome if the build could be cached/persisted as it is done in mill. In gigantic multi-project builds it can take a minute just to start sbt even though there were no changes to the build file.

This would also solve the issue of injecting env variables into sbt since it would take just milliseconds to start. This currently doesn't work with sbtn.

[eed3si9n]

Splitting this up to #7180.

[nightscape]

It would be great to introduce an abstraction for plugins that is not tied to SBT.

The goal would be to allow sharing most of the core logic of a plugin between build tools like SBT, Mill, Bleep and others, and having thin build-tool specific wrappers around this.

I'm not sure what this abstraction would look like, maybe it is more a convention to put the core logic into ordinary methods inside objects.

Though this is not necessarily tied to SBT 2.0, the timing might be an opportunity window as every plugin author would have to touch the code anyway.

[hamnis]

That is probably just having a core library part which the plugin calls into, moving all settings into parameters of functions (tasks)

[nightscape]

Yes, that may well be sufficient.
Unfortunately, that is not the way most plugins are currently written. Rather, the core logic is mixed with SBT-specific parts and it's really hard to pull out something that can be reused in other build tools.
So maybe an official and clearly visible convention documented in SBT would help already.
A small library (not sure what it would contain though) would make this even more visible and hopefully provide sufficient benefit to plugin authors to default to using it.
Maybe the library could contain some macro- or reflection-magic to turn ordinary methods on objects and their ScalaDocs into SBT TaskKeys with description.

[hamnis]

having a giter8 template which clearly separates the library from the use (plugin) would really help.
Encouragement in documentation would also benefit this.

[hamnis]

Discoverablity of giter8 templates would be really cool.
Maybe sbt could have a registry which it can download, and make available for sbt new,
using github is fine, but having a registry would really help with discoverablity.

[philliptaylorpro]

I have trust issues so it's common for me to run the command rm -rf project/project project/target ./target and then do sbt clean... etc. I'm not sure if it's an unwarrented trust issue and I wonder whether I'm alone in it? If other people relate, perhaps sbt can stop creating project/project and project/target (maybe target/project). That way I only have to delete, or gitignore a single directory.

[bjornregnell]

I also erase target quite often. For example when I switch between scala-cli and sbt, back and forth. Or when I make some major refactoring of packages etc.

[kubukoz]

I just run rm -rf **/target :) unless you work at Target, it should be fine.

that said, this is just a symptom of underlying issues in the compiler/zinc/sbt.

[mbaechler]

Allow to discriminate between implementation dependencies and api dependencies like gradle (https://docs.gradle.org/current/userguide/declaring_dependencies.html).
This allows to avoid exposing a dependency that is an implementation detail.

[rtar]

These ideas look cool. I like them and looking forward to sbt 2.x.

I have a very simple and stupid idea of my own. I wonder if it is worth to explore the change of project definition DSL from the current one, i.e. (project in file("something")) to something more conventional like Project(base = file("something")) as a default way to define a project.

It seems to me that one of the reason sbt creates an impression of being more complicated than it is (I find sbt really simple), is that developers have to learn its DSL, which differs from conventional Scala. I think some DSL is unavoidable, indeed, but making it really minimal would be awesome.

I.e. I remember the sigh of relief when sbt got rid of <+=, <++= and other operators, and another sigh of relief when in operator was replaced by / in most of the cases. May be that simplification trend could be continued also in this case.

[julienrf]

I would like to add a point related to that. I wish it was possible to use sbt without its DSL based on macros. While it is true that the old operators <+=, <++=, etc. were cryptic, at least everything was written in plain, regular, Scala code that I could understand. Today, the semantics of my code is sometimes very hard to predict, and sometimes it fails with very cryptic errors when the macros are expanded, with no possible alternative. In my opinion, an explicit style with the usual¹ operations map, zip, etc. on tasks and settings would be simpler to use than the “direct-style”.

¹ I am expecting the operations map, zip, flatMap, and traverse to be pretty pervasive in Scala codebases, and their meaning to be familiar to many Scala developers.

[julienrf]

I see that the top comment has been upvoted a lot, but the content only discusses the API to define a project (project.in(file("foo")) vs Project(id = "foo", base = file("foo"))). Is this really such a big deal? @rtar, @bjornregnell, and @NicolasRouquette, is there anything else you find confusing in the API?

On my side, I like the builder-like API to create a project. I can conveniently define a function that takes a Project as parameter and calls .enablePlugins or .settings or .dependsOn on it to transform the underlying project definition.

If the problem you mention is just about the fact that the documentation shows (project in file("foo")) instead of project.in(file("foo")), then I think the solution should be to fix the documentation.

By the way, there is already a Project constructor that you can use as follows:

val myProject = Project(id = "my-project", base = file("."))

The fact that nobody uses it means that probably the project thing works well? Note that writing:

val foo = project

Is equivalent to:

val foo = Project(id = "foo", base = file("foo"))

Which does save some boilerplate (the id and base are derived from the val name).

[rtar]

The fact that nobody uses it means that probably the project thing works well?
I think everyone just copy-pastes documentation...

The problem with val foo = project is that it fails to follow the principle of least surprise. We kind of claim is that sbt is kind of simple Scala, and then throwing a complicated macro to a face of the newbie, which (1) takes a name of the project from name of that val, (2) takes name of the directory from the project name, (3) shows a scary error if project is not assigned to val directly.

It is not a problem for seasoned Scala developer, of course, but it is one of the first things junior Scala developer sees in his Scala life, and it really adds an impression that Scala is more complicated than it needs to be.

[julienrf]

Thank you for elaborating. So if I understand well you are saying that the problem is rather that the current API relies on macros to do “magic” things, and shows hard to understand confusing messages in case of errors.

[rtar]

That error is quite clear, it says exactly what needs to be done.

build.sbt:1: error: project must be directly assigned to a val, such as `val x = project`. Alternatively, you can use `sbt.Project.apply`
project
^

It is just too scary. I.e. the developer might get confused of why they are rejected to do the thing, which is, perfectly, normal in Scala.

It is not a huge thing, but it requires an explanation each and every time I teach a person how to use sbt, and there are so many things already to teach, that having one less is always appreciated.

[Rich2]

I think changing "Compile" => "Main" would really help initial comprehension of Sbt.

[SethTisue]

agree. to unpack this a bit:

• Compile / run is a non sequitur
• the confusion between Compile and compile is unnecessary

[laughedelic]

Another reason is consistency:

>  Test/sourceDirectory
[...]/src/test
>  Compile/sourceDirectory
[...]/src/main

src/test is for Test, so src/main should be for Main

[SethTisue]

I've brought this idea up on #7189

[mdedetrich]

So I mentioned this on the equivalent reddit thread (see https://www.reddit.com/r/scala/comments/11q3rm8/comment/jc1dym9/) but I do have a concern with idea 3-B: discourage the use of task scoping.

My main concern with this change is that we seem to be throwing out the baby with the bathwater when it comes to scoping/encapsulation (the whole ethos of using a language like Scala is that we are meant to use abstractions to abstract away commonalities unlike in contrast to languages such as Go). While I understand the task scoping rules are currently confusing I would prefer if we fix that confusion (2.0.x is a breaking release after all so this is an option?) or we come up with something new thats not so confusing for end users.

I also don't see how this would ultimately would be net benefit, such a change would create an parabolic explosion of sbt keys (whenever a sbt plugin auto has to create a new config they need to create a new set of keys) which I would actually argue for a lot of users would be even more confusing than what we have now. At least on the presumption that Scala users are familiar with the concept of abstraction, while we don't have to deal with task scoping we now have to deal with copying over settings. For trivial usecases, sure this can be simpler, but for more complex usecases I fail to see how such a change would be that much better.

[sjrd]

I agree. Task scoping is useful to keep things consistent.

I think a better fix to their complexity is to dramatically simplify task scoping. Scoping is complex because it tries to manage several "orthogonal" axes (project, config, task) while at the same time pretending like they are hierarchical (as emphasized by the usage of /). But they're not hierarchical, since things like (project, Zero, someTask) are valid. And it's those scopes that have weird rules.

I propose that we actually make all scopes strictly hierarchical, by making such scopes (where a Zero is followed by a non-Zero) invalid/unrepresentable. We should be able to live with

• Global
• Global / someProject
• Global / someProject / someConfig
• Global / someProject / someConfig / someTask

Or, if ThisBuild still plays a role, with

• Global
• Global / ThisBuild
• Global / ThisBuild / someProject
• Global / ThisBuild / someProject / someConfig
• Global / ThisBuild / someProject / someConfig / someTask

[mdedetrich]

I propose that we actually make all scopes strictly hierarchical, by making such scopes (where a Zero is followed by a non-Zero) invalid/unrepresentable.

Im all for this, this is also the part of task scoping that I find incredibly confusing. I think if we get rid of this but still keep a basic linear hierarchy (as you describe) we should keep almost all of the main benefit of scoping while getting rid of its most confusing aspect.

[bjornregnell]

Thumbs up for strict hierarchy! But I think it should still be as easy as possible to write "bare settings" so that a top-level scalaVersion := "3.3.3" means Global / ThisBuild / thisProject / scalaVersion := "3.3.3" or similar, so that single-projects builds can be kept really simple.

[eed3si9n]

Generally when people complain about the complexity of sbt, the first thing people mention is the scope delegation rule.
I am not saying we get rid of scoping as a concept, or even task scoping completely, but what I think we should try is to discourage task scoping. When I first learned about task-scoping I thought it would be a useful tool to namespace settings under my main key like scalaxb task or something as scalaxb / contentsSizeLimit, then I learned that it actually doesn't prevent name collision so we need to say scalaxb / scalaxbContentsSizeLimit (or Compile / scalaxb / scalaxbContentsSizeLimit if Compile-scoping had been done), at which point we might as well write scalaxbContentsSizeLimit.

we are meant to use abstractions to abstract away commonalities

Could you share some of the example where task scoping or fallback from task-scoping to something else provide big win?

I propose that we actually make all scopes strictly hierarchical, by making such scopes (where a Zero is followed by a non-Zero) invalid/unrepresentable.

I don't know if this is a good idea either because not all tasks are Compile or Test scoped. For example, update or publish exists across the configurations for example, and we might want to keep:

publish / skip := true

I like this one more because less wiggle room for an unexpected behavior. This one is sort of a border line:

run / fork := true

run is actually Compile / run, but normally you don't call it like that, so run / fork feels normal. Also the danger of these change-the-behavior settings is maybe believing xxx / skip would generally work.

[sjrd]

I don't know if this is a good idea either because not all tasks are Compile or Test scoped. For example, update or publish exists across the configurations for example, and we might want to keep:

publish / skip := true

I like this one more because less wiggle room for an unexpected behavior.

There are a few cases like that, yes. Perhaps one solution to that issue is that tasks/settings could declare at which level they apply. publish could say that it applies at the project level, while compile applies at the configuration level, and skip applies at the task level. It would then be allowed to set or get proj / publish / skip, but not proj / Compile / publish / skip (nor proj / Compile / publish, for that matter).

I'm willing to bet that the overwhelming majority of settings and tasks have exactly one level at which they're designed to be used. That could be turned into a guarantee (checked either at compile time if the types are not too unwieldy, or at graph building time).

This one is sort of a border line:

run / fork := true

run is actually Compile / run, but normally you don't call it like that, so run / fork feels normal.

Except it isn't Compile / run. You're really setting theProject / run / fork, without configuration. Then what gets read is theProject / Compile / run / fork, and that somehow delegates to the other one.

To me this is the prime example of something that's very confusing, and we should force users to write Compile / run / fork.

we are meant to use abstractions to abstract away commonalities

Could you share some of the example where task scoping or fallback from task-scoping to something else provide big win?

The Scala.js per-stage-task keys are one example, I suppose. All of the following tasks and settings exist and are taken into account for both fastLinkJS and fullLinkJS:

• scalaJSLinkerOutputDirectory (user-facing)
• scalaJSLinkerConfig (user-facing)
• scalaJSLinkerBox
• scalaJSLinkerImpl
• scalaJSLinker
• usesScalaJSLinkerTag
• scalaJSModuleInitializersFingerprints
• scalaJSLinkerConfigFingerprint

At least within the plugin, it is extremely valuable to be able to abstract over the target key with scopes. At the user level, it is quite common to redefine parts of scalaJSLinkerConfig at various levels of the config: at the ThisBuild level, per-config level, and definitely per-task level.

[benhutchison]

Since we're in blue-sky 2.0 mode, I'll mention a wish list item raised in discord: I would like a richer language to denote the role of dependencies of a project, to distinguish between:

1. A dependency (project or artifact) that's internal or hidden. The depending project can use it, but downstream modules don't know about it. This is what is typically assumed.
2. A dependency that is intended to be exposed to, and used by, downstream projects. Such a dependency forms part of the "contract" of the depending project, and shouldn't be removed later.

That option 2 is even a thing seems quite controversial. Myself, I find this concept very useful to make dependency graphs closer to minimum spanning trees, rather than hairy thickets full of repeated links.

For example, in a large multi-module one can declare that a widely used library like Cats is exposed by a base module, and then the other modules just depend on base and get the Cat for free.

Presently there's no way afaict to signal this distinction, so I rely on convention, ie docs or comments. But I would like a way in SBT's DSL to denote an exposed dependency.

Although, I'm doubtful if it can be done in general without help from Ivy, which defines the artifact metadata format iiuc. I had a look over the Ivy docs and didn't find anything useful.

[plokhotnyuk]

The current version of sbt has compile-internal and test-internal types of dependencies which are quite helpful for defining of macro dependencies. But other tools like IDEs don't support them properly yet. Please vote for this issue created for IntelliJ IDEA plugin for Scala.

[dragos]

Scala is not particularly well suited for this kind of distinction.

At Databricks we went the Bazel way and enforced only direct compile-time dependencies (what sometimes is called internal). If you wanted to expose it to clients, you'd declare it as "exported". It turns out that an internal dependency that defines a trait, and your code extends that trait, needs to be promoted to exported. Such issues are quite common, and usually impossible to know before you end up in an unfamiliar compiler error message (along the lines of "symbol missong on the classpath", and then you need to guess what library provides it and add it to your classpath).

There are other cases than traits, see this old thread.

While it does not hurt to have the option, I think it may cause some confusion and be less useful than it sounds.

[Atry]

I’m not sure if it is a good idea to consider sbt 2.0 as an opportunity to break source-level compatibility. The Scala community, especially sbt plugin authors, have had a ton of experience in cross build, so breaking binary compatibility in sbt 2.0 might be acceptable and plugin authors could migrate their plugins to sbt 2.0 smoothly. However, breaking source-level compatibility could be a risky idea because it might lose many plugins.

[mdedetrich]

I’m not sure if it is a good idea to consider sbt 2.0 as an opportunity to break source-level compatibility. The Scala community, especially sbt plugin authors, have had a ton of experience in cross build, so breaking binary compatibility in sbt 2.0 might be acceptable and plugin authors could migrate their plugins to sbt 2.0 migrate. However, breaking source-level compatibility could be a risky idea because it might lose many plugins.

We have tools to deal with this now, like scalafix (aka scala migrate) which can work on the AST level to bring proper migrations.

Also at least on surface level it might be possible to do @sjrd suggestions in sbt 1.x with deprecations (i.e. we don't remove the axis concept but if people use/define sbt keys with a Zero axis then they get a definition warning) which will better prepare for sbt 2.x.

[mdedetrich]

Sorry I was typing this on my phone and I didn't see the context of the discussion (I thought it was in response to #7174 (comment)).

The point about scalafix still stands though, and in general depending on the changes it might be possible to do it at source level in sbt 1.x with deprecations.

[Atry]

scalafix is good to migrate deprecated sbt 0.13 features to sbt 1.x, but it is not ideal to remove sbt 1.x features, because plugin authors need to cross build their plugins for 1.x and 2.x. Any behavior changes or removing sbt 1.x features would hurt the ability to cross build, and scalafix does not help on this point.

[keynmol]

One thing I don't think anyone mentioned is JSON support. For custom tasks, caches, and general dealings with JSON from plugins.

May be I'm holding it wrong, but I believe the builtin sjson-new is not nearly as convenient as modern JSON libraries we have.

With SBT 2 being Scala 3, it would be great to have JSON support with both low-level upickle-style API, and automatic derivation out of the box.

An existing library can be used and shaded.

[bjornregnell]

For reference, see the bleep model for a json/yaml discussion: https://bleep.build/docs/model/

[Ichoran]

Is there an explanation for why it's worth all the effort to make a substantially new version with breaking changes?

If basically everyone will find that there are no breaking changes, but SBT2 is more future-proof (e.g. uses Scala 3), great!

But if it's going to require a bunch of manual work to upgrade, we should think seriously about whether SBT is the right core technology to build on. There's a big difference between "just works" and "works except in difficult corner cases" because when you're building someone else's project (including the someone-else of you-a-few-years-ago) of course you don't understand their difficult corner cases. The build is just broken. And there's also a big difference between "just works" and "just works with some easy tweaks" because who really wants to have to do easy tweaks instead of no tweaks?

There are some basic decisions about what is core functionality and what is a plugin, how to organize files, what the syntax of the build file is, how to describe the build both as a static object and as a series of goals, and so on, that have made things considerably less straightforward than, in retrospect, they could have been.

Putting in considerable effort that then asks everyone who uses SBT (i.e. most Scala developers) to also put in significant effort all in order to mostly perpetuate not-quite-ideal choices doesn't seem like an obvious clear win to me, even if there is actually no plausible technical route to a (basically) fully compatible SBT2. It might be a win, but I think it deserves a detailed argument as to why. Otherwise, I think a modernized but fully backward compatible SBT2 is an obvious clear win because it preserves people's existing work for as long as possible.

(Edit: to clarify, the argument would cover things like "why not mill", "why not bazel", "why not something like cargo", "why not a new synthesis of the best ideas from the last 15 years", etc..)

[bjornregnell]

Thea ideal thing would be that if an sbt user bumps from sbt 1.9 to sbt 2.0, then sbt itself rewrites its build definition (after a prompt) so that the build "just works". So even if there are some cases where backwards "build source compatibility" is broken between 1.9 and 2.0, then sbt 2.0 can "mend" itself automatically (after a prompt).

But perhaps such a requirement on progress hinders some of the innovation that @eed3si9n envision for 2.0? It all depends on how smart such an automatic build rewrite can be in relation to how big the changes are... And it speaks for making fewer externally visible changes between 1.9 and 2.0 and postpone major changes to 2.1 in order to ease migration.

[eed3si9n]

Yea. I did discuss this in the Tooling Summit, and added a summary in https://eed3si9n.com/sbt-2.0-rfc-process as well:

continuum of continuity

┌───────────────┐           ┌────────────────────────┐
│               │           │                        │
│ Don't Change  │◄─────────►│ Change Everything      │
│ (stability)   │           │ (perf, UX improvements,│
│               │           │  simplicity etc)       │
└───────────────┘           └────────────────────────┘

Given that sbt has many builds out in the wild, my general philosophy is to avoid making huge jumps conceptually from sbt 1 (see also how long we kept bincompat for sbt 0.13). The iron rule is to bring the ecosystem forward together.

At the same time, since there already is sbt 1 for users who do not want any changes, sbt core devs shouldn’t be too afraid of changes if that could improve sbt’s user experience, performance, or the long-term maintainability of sbt etc.

Similar argument was made during sbt 0.13 era, basically asking for sbt 0.14, which doesn't change anything but release it with Scala 2.12. We ultimately ignored that, and went ahead with modularized sbt 1.x. As @SethTisue observed during the summit, initially people were slow to adopt sbt 1.0, but after there were compelling features such as default Coursier support was added, sbt 1.x became the de-facto standard.

While I think my general instinct is on the keep-compatibility side as well, not taking the plunge to clean things up every few years would also be a mistake given that there is sbt 1.x. sbt 2.x should either out-of-box have compelling features or prepare a way for future features, and people should adopt it based on its merit.

Ideally, I would like sbt 2.x to mostly keep the shape of sbt 1.x-like build.sbt file, but backed by the best ideas from the best ideas from Bazel etc.

[ghostbuster91]

Thanks for sharing your ideas and making a room for that discussion.
Here are my 2 cents:

Get rid of interactive mode

I am a heavy terminal user, and sbt interactive mode was always problematic for me. Sbt reimplements couple of shell features, like reverse search or ctrl+w to delete word. But this doesn't scale. Shells are much powerful these days than that.

Some concrete examples:

1. My shell is integrated with fzf meaning that I can perform reverse search with interactive filtering. Sbt interactive mode prevents using it.
2. ctrl+w is only one motion (probably there are others that sbt reimplemented) but there are many more. Moreover, these motions can be configured and in my terminal ctrl+w deletes only a single word, while in sbt it deletes everything up to the next whitespace character. Besides the so called emacs motions there are people who use vim motions in their shells. Reimplementing all of that doesn't scale.
3. Smarter completion. I use zsh-autosuggestions to have suggestions for any command that I write. There are other plugins that offer similar things. Nowadays, one can even have AI model to serve them suggestions for command line parameters. All of this is not possible when using sbt interactive mode.
4. Composition. Non interactive tools compose easily with other linux command line applications. Once again, being able to pipe results from sbt to fzf and then into a file would be great. Redirecting sbt output to a file should be as easy as with every other command line tool. There shouldn't be a need for filtering any Slf4J statements.

As far as I understand in the past there was a valid reason to have this interactive mode, but it doesn't have to be like that anymore.

(Interactive mode can stay, but there has to be an alternative, non-interactive mode that has feature parity with the interactive one)

Add lock file

Without getting to much into the details, in my previous company we needed to package our scala application using nix. Nix is fully declarative and during the package build it cannot access internet. I know that there is https://github.com/zaninime/sbt-derivation but it didn't work for us because of zaninime/sbt-derivation#8 and even without that it still is kind of a hack. Having a lock file would drastically lower the bar for such integrations.

I am aware of #2989, just bringing it up here.

[laughedelic]

I generally agree about the interactive mode, but since you say

there has to be an alternative, non-interactive mode that has feature parity with the interactive one

Have you tried sbt native client? It's built-in since sbt 1.4. You can run sbtn, sbt --client or set the SBT_NATIVE_CLIENT env var to switch any normal sbt calls to the client mode. It has some autocomplete and doesn't require you maintaining a long-running sbt shell, so it's convenient to use interactively without leaving the terminal. Although it does have some issues, like #6468 and #7012 which make it hard to use for scripting.

[ghostbuster91]

Yes, I tried it once it was released. Some things didn't work back then. I should give it another try, thanks for reminding me about it.

[ghostbuster91]

I tried it once again, seems like it is far from being daily driver. See #6866

Also if I am executing a task in sbt I would expect it to output only the output of that particular task (provided that everything is ok with the build) without all the sbt related information like:

[info] welcome to sbt 1.6.2 (N/A Java 19.0.2)
[info] loading settings for project demos-build-build-build from metals.sbt ...
[info] loading project definition from /home/kghost/dev/demos/project/project/project
[info] loading settings for project demos-build-build from metals.sbt ...
[info] loading project definition from /home/kghost/dev/demos/project/project
[success] Generated .bloop/demos-build-build.json
[success] Total time: 0 s, completed Jun 5, 2023, 11:44:31 AM
[info] loading settings for project demos-build from metals.sbt,plugins.sbt ...
[info] loading project definition from /home/kghost/dev/demos/project
[success] Generated .bloop/demos-build.json
[success] Total time: 0 s, completed Jun 5, 2023, 11:44:32 AM
[info] loading settings for project root from build.sbt ...
[info] set current project to root (in build file:/home/kghost/dev/demos/)
[warn] sbt server could not start because there's another instance of sbt running on this build.
[warn] Running multiple instances is unsupported

[guizmaii]

Hi everyone,

I don't know if it has been asked already, but an annoying limitation of sbt 1.x is that you can only publish to one repository.
There are some sbt plugins more or less maintained that are helping to publish to more than one repo (I found these 2: https://github.com/davidharcombe/sbt-multi-publish and https://github.com/laughedelic/sbt-publish-more).
The issue is that they are more or less maintained.

It'd be nice if sbt was supporting this natively.

Why do you want to publish to more than one repo?
An example could be when you publish private artifacts to private repositories and you need to migrate from one repository to another one (for some Enterprise reasons, for example) so that the transition from the old to the new one is smooth

[sjrd]

In some rare cases, putting functionality in the core makes it substantially easier to maintain for some reason. In all other cases, putting something in core will make it less maintained than when it is in a separate repo, because you are effectively spreading the efforts of the same small core team to more stuff. Asking for something to be in core with the goal for it to be better maintained is nonsensical.

[mdedetrich]

This is something that just came up and would only be feasible/possible in a epic update, but what are the thoughts in treating multi/cross Scala version compilation properly rather than how its currently done? By that I mean currently if you use crossScalaVersions to make your project target multiple scala versions the way that sbt implements this is it just reloads sbt with the different scala version (I think this is done via a Command but I haven't looked into the code). I think historically this implementation is kind of a workaround that hasn't changed, but I could be wrong on that.

The idea would be that you would treat crossScalaVersions as any other standard sbt setting/key, allowing sbt to cross compile Scala versions without even needed to reload. It would likely involve mainlining sbt-projectmatrix with extra code to provide source compatibility with sbt 1.x. In essence it would be done in a similar fashion of how sbt-doge was mainlined into sbt 1.x, this would be the final step in properly supporting multiple scala versions.

[keynmol]

Idea 8-A in the post :) https://eed3si9n.com/sbt-2.0-ideas

[mdedetrich]

Thanks! I am not sure if that idea just involving including sbt-projectmatrix or if it ontop of that also includes changing crossScalaVersions so that it uses sbt-projectmatrix rather than the current implementation.

[ekrich]

We have a ton of projects using - sbt-scalajs-crossproject and sbt-scala-native-crossproject . Are this going to be unified and as easy? What is the potential migration path?

We also have the special build extras powering the Scala.js and Scala Native builds themselves. Honestly, they are even harder to deal with.

[mdedetrich]

We have a ton of projects using - sbt-scalajs-crossproject and sbt-scala-native-crossproject . Are this going to be unified and as easy? What is the potential migration path?

We also have the special build extras powering the Scala.js and Scala Native builds themselves. Honestly, they are even harder to deal with.

This is what I meant earlier by source compatibility, I can't really comment on whether this is possible and if so how far can it be taken.

[mdedetrich]

Also to add onto the discussion, one of the things that is very difficult now is to collect changes that are an output of processing something on multiple versions of scala. One example that I can come up with on the top of my head is scalafix, currently with scalafix you need to specify a scalaBinaryVersion sbt setting because scalafix needs to know what scala version to run against since some formatting/linting options aren't relevant to specific versions of Scala.

If we had proper matrix/row support, scalafix could hypothetically just run against the Scala versions that are supported in your build, collect the formatting/refactoring changes and then apply them as desired.

There are other related issues as well, i.e. for sbt-release to support publishing for all versions of Scala in your project it needs to define the entire release process as a Command, and since a Command essentially just reloads the sbt project behind the scenes. This means that if you need to store state in-between the switching of the Scala versions and/or in between the steps of the release process (which is forced to be a Command because of what I mentioned earlier) you have to do workarounds such as storing the data to some temp file on disk instead of more principled/ideal solutions.

[rssh]

What can be helpful in sbt-2.0: Support forking on Compile state and passing JVM options there.
This will help during the situation when we want to track/change memory consumption during compilation or pass additional properties for the compilation process (such as target JVM and libraries that depend from JVM version)

[kubukoz]

Does setting Test / fork := true as a default sound like a good 2.0 idea?

[eed3si9n]

Not in the fork as it is in sbt 1.x I think. JIT is slow, so for users who want to run testQuick iteratively launching into fork every time adds seconds to JIT scala-library, etc. "idea 5" that I proposed is to run it a separate process called persistent worker so it would be a forked process but kept warm.

[lihaoyi]

FWIW Mill forks the test process by default. Empirically, it is not a problem. Mill supports testLocal in a classloader by default and AFAIK nobody uses it. Most Bazel Scala integrations fork test processes as well

My experiments in other places show about a 0.5-1.0s overhead spawning a new JVM. This isn't nothing, but isn't terrible either. It's up to SBT maintainers to make the call, but my feeling is that all in all shaving off 0.5-1.0s here may not be worth the increase in complexity and decrease in compatibility (not everything can run in classloaders) or hermeticity (long-running JVMs can easily be left in weird states, especially running half-broken code which is what happens when iterating on a test suite)

[mdedetrich]

How about getting rid of globalSettings by adding another parameter/argument to the settingsKey/taskKey functions which forces you to provide a zero like value?

After dabbling/starting to maintain a few sbt community plugins, one pitfall that I see very commonly done is when someone writes a sbt plugin that introduces new SettingKey/TaskKey's, they don't initalize those keys to zero like values in globalSettings which causes a whole host of known problems.

So the idea is that instead of doing this

trait MyPluginKeys {
  lazy val myPluginSetting = settingKey[String]("Some description")
}

You would instead do

trait MyPluginKeys {
  lazy val myPluginSetting = settingKey[String]("Some description", "")
}

Where that argument is a zero like value for that type (in this case its the empty string ""). We could even provide overloaded alternatives such as an implicit typeclass that provides a default zero like value for common types, i.e.

trait Zero[T] {
  val zero: T
}

and then you can provide a list of default zero values, i.e. val stringZero = new Zero[String] { val zero = "" }

The premise here is basically to get rid of globalSettings by forcing default values for keys when you define them.

[mkurz]

IMHO for 2.0 it should be considered to drop the deprecated repo.scala-sbt.org and repo.typesafe.com repos.
Within the last weeks I helped migrating more or less all missing sbt plugins/projects to maven central. I can now build and test Play, Pekko, Akka and even sbt itself with having repo.scala-sbt.org and repo.typesafe.com blocked in my /etc/hosts (and of course having any local artifacts of those repos deleted in my local coursier and even ivy cache).
People can still add the repos themselves if needed, but I doubt when using sbt 2 you will need that repos anymore.
I think we should make clear that you should not use that repos anymore. I think it's better to cut off those repositories sooner rather than later.

sbt/launch/src/main/input_resources/sbt/sbt.boot.properties

Lines 13 to 19 in 2d974be

	[repositories]
	local
	maven-central
	sbt-maven-releases: https://repo.scala-sbt.org/scalasbt/maven-releases/, bootOnly
	sbt-maven-snapshots: https://repo.scala-sbt.org/scalasbt/maven-snapshots/, bootOnly
	typesafe-ivy-releases: https://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
	sbt-ivy-snapshots: https://repo.scala-sbt.org/scalasbt/ivy-snapshots/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly

[sjrd]

That will necessarily be the case by construction, since all plugins will need to be republished, and those repos are now read-only.

[eed3si9n]

As @mkurz linked, the repositories list is a property of sbt launcher, which is independent from sbt 1.x, 2.x, or 0.13.x, so in theory we could consider shipping this as a breaking change in a feature release of sbt 1.10 as well?

[mkurz]

Yeah ship it in 1.10. It's just breaking in a way that the repos wouldn't be there by default anymore, people can still add them back manually. Just make a note in the release notes.

[SethTisue]

Thumbs up to doing this in 1.10. I have also expressed my support on the new ticket Eugene opened (#7484).

[kubukoz]

I was wondering if 2.0 can solve this nicely:

val scala2Settings = Seq(...)
val scala3Settings = Seq(...)

val foo = project.settings(
  if(scalaVersion.value.startsWith("2.")) scala2Settings else scala3Settings
)

I understand it'd require making settings a macro, which isn't ideal

[eed3si9n]

When the expression inside the task is an if expression, sbt 1.x already lifts it up as a Selective functor. I think that's enough hackery that I don't even know ppl have started using so I don't know if we should add more.

[Andrapyre]

What about mixing declarative configuration with sbt's DSL, taking some inspiration from Gradle (https://blog.gradle.org/declarative-gradle)?

Imagine an sbt.toml file where I can set all project settings, especially project version number and dependency version numbers. Any update to settings would be reflected in the sbt.toml file. Tasks, commands, etc. would still of course live in sbt's DSL.

This would make CI/CD easier, as well as IDE support (the Gradle link goes into some of the whys).

[martin-ockajak]

Ideas mentioned here are very good as are others mentioned in this discussion. However the following improvements would have disproportionately positive impact on Scala adoption and growth:

1. Performance - Build tool start time must be below human detection threshold (~100ms) and all its common operations (starting the compilation process, initializing tests, printing a dependency graph, etc.) should be carried out below human boredom threshold (~1s). This would also make SBT shell unnecessary - it's one of the few build tools that have their own shell and I assume most people prefer their own shell anyway. Comprehensive memory and/or file caching and packaging SBT as a native application (scala-native or GraalVM native image) might be good steps here.
2. Standard command-line arguments format - More complex commands like sbt testOnly SomeTest or sbt + test need to work out of the box like in any other build tool without the necessity of surrounding these commands with single or double quotes when being invoked from the shell.
3. Reliable termination mechanics - Pressing Ctrl-C during compilation or test runs has to terminate everything immediately. At the moment terminating complex tests with compounding errors can either take a long time involving several attempts or sometimes cannot be done at all and the SBT process needs to be killed using an external tool.

Scala as a platform and SBT as its default build tool is competing with other modern platforms of which many have excellent ergonomy and user experience. As the technology develops, perceptions of such good ergonomy are shifting from a welcome innovation to a necessary minimum standard everyone intuitively expects.

[eed3si9n]

Performance - Build tool start time must be below human detection threshold (~100ms)

Have you tried sbt --client? Build tools like Bazel and Gradle achieves very high response time basically by running a daemon in the background, and allowing the CLI to get up and send the command over to the server. We have implemented the mechanism as sbt --client also known as sbtn, which is a command-line client written in Scala, and compiled to native image via GraalVM native image. Any start time performance should be predicated on adoption wider adoption of sbtn.

[martin-ockajak]

Yes, I have been using sbt -client but I didn't know it was built using GraalVM native image, thanks for clarifying that. The client does report that it is experimental but perhaps if it is in fact stable enough it could simply be made default (i.e. setting the default value of SBT_NATIVE_CLIENT in the launcher script to true). In case there are outstanding issues or better ways of achieving the same outcome please let me know if/how I can assist.

[bjornregnell]

Any start time performance should be predicated on adoption wider adoption of sbtn.

Perhaps the client should be the default in 2.0 ? (And the shell an opt-in with e.g. sbt --shell)

[kubukoz]

I think this is a non-starter if we don't resolve #6468 first.

[mdedetrich]

I think this is a non-starter if we don't resolve #6468 first.

Agreed

[bjornregnell]

Aha. OK, yes that should be fixed first.

[unkarjedy]

Define SBT API / Better modularisation for SBT internals

Hello!
In IntelliJ Scala Plugin we register special -build modules representing SBT project definition projects.
The classpath of that module includes 83 jar files reported by SBT (via sbt.PluginData#dependencyClasspath)
Those are needed to provide all the ordinary intelligence in IDEA (completion, navigation, etc...)
Right now all of them are accessible from IDEA and SBT (see the comment)

But we started wondering if all of those libraries are supposed to be visible to developers
It feels like many of those jars are implementation details and required only at runtime.
E.g. there are multiple default auto-plugins (sbt.ScriptedPlugin + 8 plugins from sbt.plugins._)
All of them look like an implementation detail, they are automatically applied by SBT, and in most of the cases you don't need to explicitly access them.
Still, they contribute a lot of transitive dependencies.
Those could be extracted to separate modules and hidden by default.
If users still want to use them, they could opt for them "somehow". It can be discussed, but it's doable.

SBT 2.0 looks like a good moment to rethink the internal module structure and define default API entities

[eed3si9n]

This thread was meant to be a feedback for https://eed3si9n.com/sbt-2.0-ideas post, but since then became a general sbt 2.x idea list. I'm going to lock this thread, but feel free to open new topic under https://github.com/sbt/sbt/discussions if anyone else wants to discuss ideas.