Scala 3 publishing plan #2485

keynmol · 2021-09-29T20:37:51Z

I did not find a tracking issue for this, but please point me to it if there is one.

For full transparency, I don't know if it's necessary to publish Scalameta for Scala 3. It would certainly be nice for end users (such as mdoc and its end users) not to propagate the 2.13 versions of certain libraries Scalameta uses.

Straight up publishing for Scala 3 is complicated by the heavy usage of Scala 2 macros, as such a very quick grep shows that at least this many macros/usages of macros will need to be re-written:

And those macros are split into two major categories:

Macro annotations - which are currently not supported at all in Scala 3, for example: https://github.com/scalameta/scalameta/blob/main/scalameta/common/shared/src/main/scala/scala/meta/internal/trees/ast.scala#L11
Faced with the same problem, Cats replaced their macros with Scalafix codegeneration: https://github.com/typelevel/simulacrum-scalafix without breaking any APIs.
Various other macros, whitebox and blackbox. Scala 3's macros are more restricted, but probably most things can be implemented, if may be using slightly different techniques.

Fastparse

Scalameta depends on Fastparse but not for the main parser, rather the auxiliary ones - XML and Scaladoc, e.g. https://github.com/scalameta/scalameta/blob/fb8bee548061d75397b5fea9fb5a3d3e1aabecd0/scalameta/parsers/shared/src/main/scala/scala/meta/internal/parsers/ScaladocParser.scala

Fastparse is a heavily macro-based project, and while some preparation work has started for Scala 3 port: com-lihaoyi/fastparse#252 I'm not sure it will be easy or will progress quickly.

Fast parser options are thin on the ground for Scala 3 atm, cats-parse being the only one published and comparable in performance (afaik). Perhaps a hand-rolled parser will be easier or a macro-less version of Fastparse for Scala 3...

Consideration: Scalameta is an active project, powering several very important tools in the ecosystem, and the macro work above is a multi-week/month invasive undertaking.

To make it possible, some preliminary work needs to happen:

Inline sourcecode dependency in the https://github.com/scalameta/fastparse fork.
- Should be inlined in scala.meta.internal.sourcecode package and dependency removed from the build
- Inline sourcecode and geny fastparse#3
Inline geny dependency in the https://github.com/scalameta/fastparse fork.
- Should be inlined in scala.meta.internal.geny package and dependency removed from the build
- Inline sourcecode and geny fastparse#3
Modify build to add Scala 3 versions where it is necessary (i.e. ignore Native, ignore Scalac plugin, etc.)
Move macro definition and usage files into scala-2 folders in the common module
- This allows the Scala 2 version to compile and operate without changes
- The work on porting the macros can continue in parallel in separate sources, scala-3 folder
Keep as much code as possible in the src/main/scala, if it can be cross-compiled (I've done this exercise, there's not that many files, sadly)
Disable full cross-compilation on CI, set Scala 3 sources compilation/testing as a separate workflow which doesn't fail the build
Gradually build out the meta machinery needed to compile Scalameta's code.

The text was updated successfully, but these errors were encountered:

olafurpg · 2021-09-30T05:52:52Z

This looks like a great plan! Thank you for initiating this discussion.

What 2.13 dependencies cause most problems for downstream users? I’m curious if we can replace them somehow so scalameta_2.13 will be dependency free or only depend on Scala 3 libraries (using the 2.13 tasty reader).

keynmol · 2021-09-30T08:11:15Z

└─ org.scalameta:scalameta_2.13:latest.stable -> 4.4.28 (possible incompatibility)
   ├─ org.scala-lang:scala-library:2.13.6
   ├─ org.scala-lang:scalap:2.13.6
   │  └─ org.scala-lang:scala-compiler:2.13.6
   │     ├─ net.java.dev.jna:jna:5.3.1
   │     ├─ org.jline:jline:3.19.0
   │     ├─ org.scala-lang:scala-library:2.13.6
   │     └─ org.scala-lang:scala-reflect:2.13.6
   │        └─ org.scala-lang:scala-library:2.13.6
   └─ org.scalameta:parsers_2.13:4.4.28
      ├─ org.scala-lang:scala-library:2.13.6
      └─ org.scalameta:trees_2.13:4.4.28
         ├─ org.scala-lang:scala-library:2.13.6
         ├─ org.scalameta:common_2.13:4.4.28
         │  ├─ com.lihaoyi:sourcecode_2.13:0.2.7
         │  ├─ com.thesamet.scalapb:scalapb-runtime_2.13:0.11.4
         │  │  ├─ com.google.protobuf:protobuf-java:3.15.8
         │  │  ├─ com.thesamet.scalapb:lenses_2.13:0.11.4
         │  │  │  ├─ org.scala-lang:scala-library:2.13.6
         │  │  │  └─ org.scala-lang.modules:scala-collection-compat_2.13:2.4.4
         │  │  │     └─ org.scala-lang:scala-library:2.13.5 -> 2.13.6
         │  │  ├─ org.scala-lang:scala-library:2.13.6
         │  │  └─ org.scala-lang.modules:scala-collection-compat_2.13:2.4.4
         │  │     └─ org.scala-lang:scala-library:2.13.5 -> 2.13.6
         │  └─ org.scala-lang:scala-library:2.13.6
         └─ org.scalameta:fastparse-v2_2.13:2.3.1
            ├─ com.lihaoyi:geny_2.13:0.6.5
            └─ com.lihaoyi:sourcecode_2.13:0.2.3 -> 0.2.7

The common annoyances are:

geny (coming from shadowed fastparse)
sourcecode (^ same)
scala-collection-compat
scala protobuf stuff

keynmol · 2021-09-30T08:13:17Z

Given that fastparse is already shadowed, we can do the same with geny and sourcecode

olafurpg · 2021-09-30T09:06:55Z

Is it possible to use the 2.13 tasty reader to upgrade those dependencies to Scala 3?

keynmol · 2021-09-30T09:09:57Z

So you mean having fastparse_2.13 depend on geny_3 and sourcecode_3 instead?

I think this will lead to opposite problem in 2.13 artifacts downstream (i.e. mdoc_2.13), as they will now have _3 artifacts resolved and SBT will barf again.

Is this what you meant?

olafurpg · 2021-09-30T09:36:04Z

You are right, that will cause problems for 2.13 artifacts. Can we inline those dependencies instead (excluding protobuf) to make them disappear? Both geny and sourcecode are tiny, and I suspect we're not using scala-collection-compat in many places.

olafurpg · 2021-09-30T09:37:56Z

It would of course be ideal if we can build Scalameta against Scala 3, but the checklist above shows that it will require a lot of effort. There shouldn't be any technical problems for Scala 3 users to consume scalameta_2.13 if it doesn't bring transitive 2.13 dependencies.

keynmol · 2021-09-30T10:13:34Z

Yep, I'm all for inlining at first, to remedy the current pain points.

I'm still eager to at least start on some of the macros, but may be in a separate sbt module. Will need a bit of rejigging to have at least some tests run against those, but I want to minimise the churn for the contributors to the main parts of scalameta.

keynmol · 2021-10-04T08:53:04Z

Modified the plan to inline both sourcecode and geny first, as it's best risk/reward move at this stage.

halfmatthalfcat · 2023-03-28T15:12:04Z

@keynmol what's the status (beyond the comments in here) on this effort? I'm sure it's a big effort but it's forcing me into 2.13.10 vs 3.x since I need quasiquote support.

tgodzik · 2023-03-28T15:28:45Z

@keynmol what's the status (beyond the comments in here) on this effort? I'm sure it's a big effort but it's forcing me into 2.13.10 vs 3.x since I need quasiquote support.

We discussed it recently during the Scala tooling summit and this is non-trivial migration. We might at some point pursue a simpler macro system for Scala 3, but we haven't decided on anything yet.

smarter · 2023-04-11T18:22:44Z

Here are my notes from the tooling summit, note that this is focused on porting all of scalameta which is why it mentions the core macro annotation, but we might instead prioritize porting the quasiquotes macros to make scalameta more usable from Scala 3, we didn't investigate those during the summit.

During our in-person discussion we mostly focused on the @ast macro annotation since this is the core macro annotation used to define the scalameta ast.

The annotation is defined in https://github.com/scalameta/scalameta/blob/main/scalameta/common/shared/src/main/scala/scala/meta/internal/trees/ast.scala, it is rather long but there's comments along the way describing each step of the form // step X: ...
https://github.com/scalameta/scalameta/blob/main/scalameta/common/shared/src/main/scala/scala/meta/internal/trees/ast.md shows an example expansion of the ast, although it isn't completely up-to-date anymore.
Several of the steps in the macro annotation do not require adding new public members and could be implemented using the new experimental Scala 3 macro annotation support.
... but there's also quite a lot of public definitions added. These could be expanded using a scalafix rule.

A lot of the complexity of the macro is related to its efficient lazy copying mechanism (see

scalameta/scalameta/common/shared/src/main/scala/scala/meta/internal/trees/ast.scala

Lines 143 to 152 in 8b47a88

    
           // step 6: implement the unimplemented methods in InternalTree (part 1) 
        
           // The purpose of privateCopy is to provide extremely cheap cloning 
        
           // in the case when a tree changes its parent (because that happens often in our framework, 
        
           // e.g. when we create a quasiquote and then insert it into a bigger quasiquote, 
        
           // or when we parse something and build the trees from the ground up). 
        
           // In such a situation, we copy all private state verbatim (tokens, denotations, etc) 
        
           // and create lazy initializers that will take care of recursively copying the children. 
        
           // Compare this with the `copy` method (described below), which additionally flushes the private state. 
        
           // This method is private[meta] because the state that it's managing is not supposed to be touched 
        
           // by the users of the framework.

). An initial version of a Scala 3 implementation could perhaps skip those.

nafg · 2023-07-12T19:49:38Z

Any guess when this might happen?

tgodzik · 2023-07-13T08:29:47Z

Not currently no, but you can use the Scalameta 2.13.x version in Scala 3 currently.

henricook · 2023-12-09T16:15:18Z

Would you mind giving me a dummy's explanation of how to use Scalameta 2.13.x in my Scala 3 project please?

mkurz · 2023-12-09T21:28:43Z

@henricook

libraryDependencies += ("org.scalameta" %% "scalameta" % "4.8.14").cross(CrossVersion.for3Use2_13),

We use parser like this in twirl: https://github.com/playframework/twirl/blob/382474d3ebc3761b37cb1af7138926fcb4364159/build.sbt#L119

keynmol mentioned this issue Oct 4, 2021

Inline sourcecode and geny scalameta/fastparse#3

Open

bjaglin mentioned this issue Feb 14, 2022

Separate versioning of sbt-scalafix and scalafix-core scalacenter/scalafix#1146

Open

bjaglin mentioned this issue Apr 10, 2022

Scala 3 support scalacenter/scalafix#1316

Closed

5 tasks

bjaglin mentioned this issue Jun 15, 2022

Add scalafixProject helper to sbt-scalafix? scalacenter/scalafix#1619

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scala 3 publishing plan #2485

Scala 3 publishing plan #2485

keynmol commented Sep 29, 2021 •

edited

olafurpg commented Sep 30, 2021

keynmol commented Sep 30, 2021

keynmol commented Sep 30, 2021

olafurpg commented Sep 30, 2021

keynmol commented Sep 30, 2021

olafurpg commented Sep 30, 2021

olafurpg commented Sep 30, 2021

keynmol commented Sep 30, 2021

keynmol commented Oct 4, 2021

halfmatthalfcat commented Mar 28, 2023

tgodzik commented Mar 28, 2023

smarter commented Apr 11, 2023 •

edited

nafg commented Jul 12, 2023

tgodzik commented Jul 13, 2023

henricook commented Dec 9, 2023

mkurz commented Dec 9, 2023

Scala 3 publishing plan #2485

Scala 3 publishing plan #2485

Comments

keynmol commented Sep 29, 2021 • edited

Fastparse

olafurpg commented Sep 30, 2021

keynmol commented Sep 30, 2021

keynmol commented Sep 30, 2021

olafurpg commented Sep 30, 2021

keynmol commented Sep 30, 2021

olafurpg commented Sep 30, 2021

olafurpg commented Sep 30, 2021

keynmol commented Sep 30, 2021

keynmol commented Oct 4, 2021

halfmatthalfcat commented Mar 28, 2023

tgodzik commented Mar 28, 2023

smarter commented Apr 11, 2023 • edited

nafg commented Jul 12, 2023

tgodzik commented Jul 13, 2023

henricook commented Dec 9, 2023

mkurz commented Dec 9, 2023

keynmol commented Sep 29, 2021 •

edited

smarter commented Apr 11, 2023 •

edited