Logistics of splitting the repository #4140
Replies: 11 comments 41 replies
-
|
http4s-bench could, like, scatter. Projects could have their own mdoc sites. "Integration" docs are tricky, if we want to show an end-to-end example of a server backend, some middlewares, and a JSON codec. The docs project currently depends on the kitchen sink. Examples projects have the same conundrum. Scalafix is, um, split into the fixed modules? |
Beta Was this translation helpful? Give feedback.
-
|
Some of these I care about and some of these, I, well, don't. Rather than pre-emptively establishing each repository, we should consider a call for a maintainer of each. |
Beta Was this translation helpful? Give feedback.
-
|
Some historical precedent on satellite repositories: argonaut used to break faster than we did. We would support one version of argonaut in the main repo, and we had another http4s-argonaut-6-1 or whatever for the disfavored version. The satellite versions were poorly maintained. Reasons we might do better this time:
Maintainership is key for this to be a success, but I'm also less worried: if absolutely nobody cares enough to maintain a module, why should the core maintainers keep dragging it along? |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
I think yes, each satellite repo would need a parallel branch structure. (I think most projects should have this: production branch(es) and development.) Multiple versions of an integration complicate things. If we chose to support Servlet 3 and Servlet 5 concurrently, would it be 2x branches? I don't want to do that. But in this proposal, we could upgrade to Servlet 5 "someday" without a corresponding breaking change to http4s-core.
I think to start with, 0.22 -> 0.22 and 1.0 -> 1.0. But since breaking changes can be made in a module without breaking the entire ecosystem, this correspondence is not guaranteed to last. (Thought exercise: what happens if circe-0.15 falls out of the sky the week after we release http4s-core-0.22.0?) |
Beta Was this translation helpful? Give feedback.
-
|
Edit: September-November 2020, Scala 2.12 and 2.13:
Analysis to follow. |
Beta Was this translation helpful? Give feedback.
-
|
Am I the only one who doesn't hate XML? |
Beta Was this translation helpful? Give feedback.
-
|
I’m ok with xml for data, not a fun for builds |
Beta Was this translation helpful? Give feedback.
-
|
In retrospect, I'm not sure I still think that circe, jawn, or dsl belong here, though I think dsl would be a monster to remove due to tests. Nothing says the schism needs to complete in one fell swoop, and what was laid out here is about to be materialized before 0.23.12. |
Beta Was this translation helpful? Give feedback.
-
|
I bet @bplommer can scalafix the DSL away :P |
Beta Was this translation helpful? Give feedback.
-
Btw, nobody says we have to remove the DSL from the core repo after we schism it. We just have to stop publishing it (and possibly merge it into the testing support project). From there it could eventually rot out of existence. |
Beta Was this translation helpful? Give feedback.
-
|
Ember has a volunteer if that was not obvious. |
Beta Was this translation helpful? Give feedback.
-
|
Do you prefer it is a side repo or is built-in? Blaze is where it is "by default," and Ember is a more modern default, but you've also been bummed by compatibility constraints in the past. I'm gathering a few more numbers and about to post another idea that keeps Ember here. |
Beta Was this translation helpful? Give feedback.
-
|
I think we could either make them all independent, or keep blaze/ember together as a default pair. |
Beta Was this translation helpful? Give feedback.
-
|
Oh, do you think evicting blaze is too aggressive? It puts a spotlight on Ember, I think for better in the long term and for worse in the short term. |
Beta Was this translation helpful? Give feedback.
-
|
I'm happy to evict blaze and leave ember in the spotlight. Does put a bit more pressure on, but I think we're approaching a strong place. The question is things like Date,etc where parsing is currently our bottleneck. |
Beta Was this translation helpful? Give feedback.
-
|
👍 to this. Now that ember is also the server/client for Node.js (and blaze will never cross). |
Beta Was this translation helpful? Give feedback.
-
|
I don't know whether this has already been discussed, but what about publishing the abstract testing stuff which is currently |
Beta Was this translation helpful? Give feedback.
-
|
They're not really "laws", but maybe we could rebrand them as such. We already publish those, but they don't depend on client. What we really want is more of a "test compatibility kit" -- an http4s backend behaves like ____. I think in one form or another, it's a good idea and increasingly necessary. |
Beta Was this translation helpful? Give feedback.
-
|
FTR I will try to put a PR together for this when CE3/Dotty are (more or less) settled. |
Beta Was this translation helpful? Give feedback.
-
|
I expect our first Dotty milestone to be ready within the next few days, and I don't expect Twirl or Scalatags to have a release. We're going to need to figure out what to do about those real soon. |
Beta Was this translation helpful? Give feedback.
-
|
(I do still expect Twirl will make the jump, but the time frame is still "hopefully in the next couple months"-ish) |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, @SethTisue! For now it requires a gross hack, but we can keep it compiling and just not published for Dotty. As long as there is good intent and no obvious alternative, I'm inclined to try to keep modules alive. I think both conditions are met here. libraryDependencies.value.map {
case module if module.name == "twirl-api" =>
module.withDottyCompat(scalaVersion.value)
case module => module
} |
Beta Was this translation helpful? Give feedback.
-
|
I'm curious how the new JS-only modules will fit into this. Namely:
Why they should stay in the main repo:
Why they should leave:
|
Beta Was this translation helpful? Give feedback.
-
|
Well, I have no idea on Scala.JS. I was thinking about breaking versions of scala-library, but this is not the time of day to allow myself to get riled up. |
Beta Was this translation helpful? Give feedback.
-
|
Hmm, good point about the Scala.js compiler. That's now built-in to dotty, and I doubt they'll make a breaking change to the compiler plugin for Scala 2.x. They just released Scala.js 1.7 with no known bugs. |
Beta Was this translation helpful? Give feedback.
-
|
Update on this: the dom modules are moved out, ultimately due to the build/test environment complexities they require. That only leaves |
Beta Was this translation helpful? Give feedback.
-
|
I'm going to need to spend some time understanding the new dependencies. Any time we use one of our backends to test the others, we have to be careful of circular dependencies (which the build should catch), or coupling that makes things harder to test later. That's why some of the JVM servers are implemented on the raw Sun server. (Previously on raw Jetty, but even that caused problems as we started thinking about Servlet 5!) I think we're in at least a good interim place. |
Beta Was this translation helpful? Give feedback.
-
|
Yup, I'm mindful of this :) to avoid precisely this problem the node-serverless module no longer has tests (it is tested implicitly when ember-client itself tests against it). It uses Node's built-in server so is equivalent to the raw Sun server. |
Beta Was this translation helpful? Give feedback.
-
|
For http4s-servlet, I ran: $ git filter-repo --force --path servlet --path testing --path examples/src --path examples/war --replace-message ../expressions.txtTo rewrite PR references, I used this #==>http4s/http4s#With the result on remote branch $ git merge --allow-unrelated-histories core/filteredAnd then I copied relevant parts of the build. |
Beta Was this translation helpful? Give feedback.
-
|
I wish I had done this when we split http4s-dom, but it didn't have much history at the time anyway. |
Beta Was this translation helpful? Give feedback.
-
|
It doesn't produce a buildable history, but that's still in core, and it preserves proper credit. |
Beta Was this translation helpful? Give feedback.
-
|
Ok, I just gave this a try in https://github.com/http4s/http4s-scala-xml which has just published a snapshot :) A couple notes from me:
Otherwise Ross's instructions are good and it was easy. |
Beta Was this translation helpful? Give feedback.
-
|
Right, I neglected to mention I'm deleting the tags that come into the new repo. |
Beta Was this translation helpful? Give feedback.
-
|
One thing we missed was changes from 0.23 -> 1.0 in the spun off repos. Those changes are lost and not currently in a cherry-pickable form. Blaze is going to be the worst of these. I'll try to figure out a pattern and post it here. |
Beta Was this translation helpful? Give feedback.
-
|
Crap. I reckon it was something like: But I was also unable to replicate it 🤔 |
Beta Was this translation helpful? Give feedback.
-
|
The If you don't have it in your shell history, I can figure something else out for this one. |
Beta Was this translation helpful? Give feedback.
-
|
I do all my OSS in ephemeral cloud machines, so that history is long gone 😕 I don't think I put |
Beta Was this translation helpful? Give feedback.
-
|
If I run your command from b30ad20, it's reproducible. If I run it from main, it's not. I can try to cherry-pick them, but I fear I'll be reliving a lot of merge conflicts. Worst case, I can merge an alternate view of the history. Oof. |
Beta Was this translation helpful? Give feedback.
-
|
Sorted in http4s/blaze#683 |
Beta Was this translation helpful? Give feedback.
-
|
Several of these ancillary repos are starting to get an 0.23 and a 1.0 branch. I know we've done work to versioned docs, but I am not sure on the details. We need to figure out whether to set that up, or just pick a documented branch. |
Beta Was this translation helpful? Give feedback.
-
|
The jdk-http-client is the current best example of how to set this up. |
Beta Was this translation helpful? Give feedback.
-
|
I have a new proposal for how to do versioned documentation, that is easier to set up. See: |
Beta Was this translation helpful? Give feedback.
-
|
I'm releasing http4s-scala-xml off main before releasing off series/0.23. This defeats our "linearized releases" strategy to dedupe the changelog. Maybe that's okay. |
Beta Was this translation helpful? Give feedback.
-
We've long discussed spinning off integrations into their own repositories. We have precedents in jdk-http-client, netty, finagle, and armeria, each of which began life separately.
We are facing new pressure to split off modules already here:
A problem for users is that most http4s modules currently have version, and now patch versions will vary. (imagine http4s-circe-0.22.3 on http4s-core-0.22.5). This hasn't been a problem for http4s-jdk-http-client users. Doing this in 0.21 would be traumatic, but starting in 0.22 and 1.0, we might be able to document our way out of it.
A hassle for us is that series/0.21, for as long as it's maintained, will stop merging cleanly to series/0.22 and beyond.
A huge benefit for us is that integrations can sink or swim on contributor interest. Does anybody still care about http4s-json4s? Help merge Scala Steward requests! Does nobody care? Let it stagnate!
I propose this points to splitting shortly before the v0.22.0-RC cycle, and a major curtailing of maintenance activity on series/0.21 at the same time. And integrations that don't support Dotty when everything else does may go first.
Beta Was this translation helpful? Give feedback.
All reactions