Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GitHub Packages doesn't seem to support sbt (yet) #1

Closed
djspiewak opened this issue Aug 1, 2019 · 23 comments
Closed

GitHub Packages doesn't seem to support sbt (yet) #1

djspiewak opened this issue Aug 1, 2019 · 23 comments
Labels
bug Something isn't working

Comments

@djspiewak
Copy link
Owner

djspiewak commented Aug 1, 2019

Sbt's publication model just doesn't appear to work with GitHub Packages, for reasons that aren't entirely clear. This tweet seems to imply that this is a known issue, or at the very least something that isn't entirely unexpected. Once GitHub adds support for sbt, or at least stops breaking it, we'll close this issue and do a proper release of the working plugin.

For the record, the error looks like this:

[info] [info] Set current project to sbt-github-packages-tests-publish (in build file:/private/var/folders/vm/h_lhw5wn573cw70qd6l0ljt80000gn/T/sbt_78381fbc/publish/)
[info] [info] Packaging /private/var/folders/vm/h_lhw5wn573cw70qd6l0ljt80000gn/T/sbt_78381fbc/publish/target/scala-2.12/sbt-github-packages-tests-publish_2.12-0.1-SNAPSHOT-sources.jar ...
[info] [info] Updating ...
[info] [info] Done packaging.
[info] [info] Wrote /private/var/folders/vm/h_lhw5wn573cw70qd6l0ljt80000gn/T/sbt_78381fbc/publish/target/scala-2.12/sbt-github-packages-tests-publish_2.12-0.1-SNAPSHOT.pom
[info] [info] Done updating.
[info] [info] Packaging /private/var/folders/vm/h_lhw5wn573cw70qd6l0ljt80000gn/T/sbt_78381fbc/publish/target/scala-2.12/sbt-github-packages-tests-publish_2.12-0.1-SNAPSHOT-javadoc.jar ...
[info] [info] Done packaging.
[info] [info] Packaging /private/var/folders/vm/h_lhw5wn573cw70qd6l0ljt80000gn/T/sbt_78381fbc/publish/target/scala-2.12/sbt-github-packages-tests-publish_2.12-0.1-SNAPSHOT.jar ...
[info] [info] Done packaging.
[info] [info] 	published sbt-github-packages-tests-publish_2.12 to https://maven.pkg.github.com/djspiewak/sbt-github-packages/com/codecommit/sbt-github-packages-tests-publish_2.12/0.1-SNAPSHOT/sbt-github-packages-tests-publish_2.12-0.1-SNAPSHOT.pom
[info] [error] java.io.IOException: Access to URL https://maven.pkg.github.com/djspiewak/sbt-github-packages/com/codecommit/sbt-github-packages-tests-publish_2.12/0.1-SNAPSHOT/sbt-github-packages-tests-publish_2.12-0.1-SNAPSHOT.jar was refused by the server: Forbidden
[info] [error] 	at org.apache.ivy.util.url.AbstractURLHandler.validatePutStatusCode(AbstractURLHandler.java:79)
[info] [error] 	at sbt.internal.librarymanagement.ivyint.GigahorseUrlHandler.upload(GigahorseUrlHandler.scala:191)
[info] [error] 	at org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
[info] [error] 	at org.apache.ivy.util.FileUtil.copy(FileUtil.java:150)
[info] [error] 	at org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:84)
[info] [error] 	at sbt.internal.librarymanagement.ConvertResolver$LocalIfFileRepo.put(ConvertResolver.scala:366)
[info] [error] 	at org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
[info] [error] 	at sbt.internal.librarymanagement.ConvertResolver$ChecksumFriendlyURLResolver.put(ConvertResolver.scala:118)
[info] [error] 	at sbt.internal.librarymanagement.ConvertResolver$ChecksumFriendlyURLResolver.put$(ConvertResolver.scala:105)
[info] [error] 	at sbt.internal.librarymanagement.ConvertResolver$$anonfun$defaultConvert$lzycompute$1$PluginCapableResolver$1.put(ConvertResolver.scala:165)
[info] [error] 	at org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:216)
[info] [error] 	at sbt.internal.librarymanagement.IvyActions$.$anonfun$publish$5(IvyActions.scala:497)
[info] [error] 	at sbt.internal.librarymanagement.IvyActions$.$anonfun$publish$5$adapted(IvyActions.scala:496)
[info] [error] 	at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:788)
[info] [error] 	at scala.collection.Iterator.foreach(Iterator.scala:937)
[info] [error] 	at scala.collection.Iterator.foreach$(Iterator.scala:937)
[info] [error] 	at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
[info] [error] 	at scala.collection.IterableLike.foreach(IterableLike.scala:70)
[info] [error] 	at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
[info] [error] 	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
[info] [error] 	at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:787)
[info] [error] 	at sbt.internal.librarymanagement.IvyActions$.publish(IvyActions.scala:496)
[info] [error] 	at sbt.internal.librarymanagement.IvyActions$.$anonfun$publish$3(IvyActions.scala:144)
[info] [error] 	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[info] [error] 	at sbt.internal.librarymanagement.IvyActions$.withChecksums(IvyActions.scala:157)
[info] [error] 	at sbt.internal.librarymanagement.IvyActions$.withChecksums(IvyActions.scala:151)
[info] [error] 	at sbt.internal.librarymanagement.IvyActions$.$anonfun$publish$1(IvyActions.scala:144)
[info] [error] 	at sbt.internal.librarymanagement.IvyActions$.$anonfun$publish$1$adapted(IvyActions.scala:134)
[info] [error] 	at sbt.internal.librarymanagement.IvySbt$Module.$anonfun$withModule$1(Ivy.scala:239)
[info] [error] 	at sbt.internal.librarymanagement.IvySbt.$anonfun$withIvy$1(Ivy.scala:204)
[info] [error] 	at sbt.internal.librarymanagement.IvySbt.sbt$internal$librarymanagement$IvySbt$$action$1(Ivy.scala:70)
[info] [error] 	at sbt.internal.librarymanagement.IvySbt$$anon$3.call(Ivy.scala:77)
[info] [error] 	at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:95)
[info] [error] 	at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:80)
[info] [error] 	at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:99)
[info] [error] 	at xsbt.boot.Using$.withResource(Using.scala:10)
[info] [error] 	at xsbt.boot.Using$.apply(Using.scala:9)
[info] [error] 	at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:60)
[info] [error] 	at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:50)
[info] [error] 	at xsbt.boot.Locks$.apply0(Locks.scala:31)
[info] [error] 	at xsbt.boot.Locks$.apply(Locks.scala:28)
[info] [error] 	at sbt.internal.librarymanagement.IvySbt.withDefaultLogger(Ivy.scala:77)
[info] [error] 	at sbt.internal.librarymanagement.IvySbt.withIvy(Ivy.scala:199)
[info] [error] 	at sbt.internal.librarymanagement.IvySbt.withIvy(Ivy.scala:196)
[info] [error] 	at sbt.internal.librarymanagement.IvySbt$Module.withModule(Ivy.scala:238)
[info] [error] 	at sbt.internal.librarymanagement.IvyActions$.publish(IvyActions.scala:134)
[info] [error] 	at sbt.Classpaths$.$anonfun$publishTask$4(Defaults.scala:2416)
[info] [error] 	at sbt.Classpaths$.$anonfun$publishTask$4$adapted(Defaults.scala:2416)
[info] [error] 	at scala.Function1.$anonfun$compose$1(Function1.scala:44)
[info] [error] 	at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:40)
[info] [error] 	at sbt.std.Transform$$anon$4.work(System.scala:67)
[info] [error] 	at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
[info] [error] 	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
[info] [error] 	at sbt.Execute.work(Execute.scala:278)
[info] [error] 	at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
[info] [error] 	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
[info] [error] 	at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
[info] [error] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info] [error] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[info] [error] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info] [error] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info] [error] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info] [error] 	at java.lang.Thread.run(Thread.java:748)

If you look towards the top, you'll see that the pom successfully published, but the first jar file was rejected with Forbidden. @alexarchambault reports that the ordering of the publishes is irrelevant, and jars are rejected regardless.

@djspiewak djspiewak added the bug Something isn't working label Aug 1, 2019
@lokkju
Copy link

lokkju commented Sep 13, 2019

As far as I can tell, -sources, -javadoc, -tests are not supported

using sbt-aether-deploy, I have successfully published to github packages from sbt

project/plugins.sbt

addSbtPlugin("no.arktekk.sbt" % "aether-deploy" % "0.23.0")
addSbtPlugin("com.dwijnand" % "sbt-dynver" % "4.0.0")

build.sbt

externalResolvers += "GitHub lokkju Apache Maven Packages" at "https://maven.pkg.github.com/lokkju/github-action-sbt"
publishTo := Some("GitHub lokkju Apache Maven Packages" at "https://maven.pkg.github.com/lokkju/github-action-sbt")
credentials += Credentials("GitHub Package Registry", "maven.pkg.github.com", "lokkju", "<GITHUB_TOKEN>")

// GitHub package repo isn't supporting javadoc and sources
publishArtifact in (Compile, packageDoc) := false
publishArtifact in (Compile, packageSrc) := false

sbt aetherDeploy

@lokkju
Copy link

lokkju commented Sep 13, 2019

An additional aspect; if I use the Charles Web Debugging Proxy, everything works fine. Without it, I occasionally get errors about the server failing to response.

@djspiewak
Copy link
Owner Author

@lokkju Hmm, what about multi-module builds?

@lokkju
Copy link

lokkju commented Sep 13, 2019

I haven't tried them; key here seems to be using aether-deploy, though it doesn't entirely fix it. I'm suspecting it's an issue with the SSL support in the client versions used by each tool.

I'm going to try a few different java http libraries, and see if I can replacate a simple working and not working configuration. I'll keep you updated.

@lokkju
Copy link

lokkju commented Sep 15, 2019

So, after more exploration, I've found some interesting symptoms.

Just using curl to try to upload packages, I've found;

  • The first file you PUT is treated as the main artifact, regardless of the filename or extension
  • you can then upload as many (valid) POM files as you like, as long as they have unique names
  • you can't upload any other non-POM files
  • Files with JAR and POM file extensions are parsed by github, and invalid formats will report an error as the message of the HTTP 4xx error that is returned.

The key takeaway currently is that you must upload the JAR artifact first, then publish the POM.

I have an email into support, we'll see if they provide more information.

@djspiewak
Copy link
Owner Author

Hmm, so no artifact signing either, then?

@lokkju
Copy link

lokkju commented Sep 15, 2019

It auto-generates the *.md5 and *.sha1 for any files uploaded.

Also, found this: https://github.com/sbt/librarymanagement/blob/d09f9f81b664baaac15054730fbcb51e1b240de2/ivy/src/main/scala/sbt/internal/librarymanagement/IvyActions.scala#L123

Artifactory deals with the publishing (and republishing) of SNAPSHOTs using a strict rule on the order of publishing. The ;build.timestamp=... suffix is the alternative.

The strict rule is:

publish the main artefact (which has no classifier)
publish the POM/ivy.xml file
publish additional artefacts which have a classifier

I can't seem to get classifiers deployed at all so far, which is bothersome, and part of what I have a question into github about.

I'll probably create a pure maven/java project and see if it works then

@francisdb
Copy link

francisdb commented Oct 4, 2019

Just tried deploying a snapshot using the default publish and got this:

[error] stack trace is suppressed; run last publish for the full output
[error] (publish) java.io.IOException: PUT operation to URL https://maven.pkg.github.com/company/foo/bar/utils/3.24.0-SNAPSHOT/utils-3.24.0-SNAPSHOT.pom failed with status code 400: Bad Request

No mention of snapshots in the docs

In fact I have the same for normal packages:

java.io.IOException: PUT operation to URL https://maven.pkg.github.com/company/foo/bar/utils/utils/0.0.1/utils-0.0.1.pom failed with status code 400: Bad Request
[error] java.io.IOException: PUT operation to URL https://maven.pkg.github.com/company/foo/bar/utils/utils/0.0.1/utils-0.0.1.pom failed with status code 400: Bad Request
[error] 	at org.apache.ivy.util.url.AbstractURLHandler.validatePutStatusCode(AbstractURLHandler.java:82)
[error] 	at org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:264)
[error] 	at org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
[error] 	at org.apache.ivy.util.FileUtil.copy(FileUtil.java:150)

I wonder what the body of that 400 looks like

@tovbinm
Copy link

tovbinm commented Oct 11, 2019

Are there any updates on this?

@djspiewak
Copy link
Owner Author

No updates yet! @lokkju's experiments are the latest, I think. I want to play around with it some more, but honestly it sounds like GitHub Packages still doesn't support hosting files which are part and parcel to a standard Maven distributable (such as signatures, documentation, and classifiers). It's hard to do anything practical with it as an artifact host until they correct these deficiencies.

@francisdb
Copy link

francisdb commented Oct 14, 2019

Some more info, testing with plain sbt 1.3.2 (no extra plugins)

You need to publish to the repository of your project:

publishTo := Some("GitHub Package Registry" at "https://maven.pkg.github.com/[username]/[project]")

credentails should be defined like this

credentials += Credentials("GitHub Package Registry","maven.pkg.github.com","[username]","[token]")

Your token needs the following scopes: read:packages, repo and write:packages

but once you publish you end up with these proposed package coordinates (where organization="io.test" and name="test"):

<dependency>
  <groupId>com.github.[username]/[project]</groupId>
  <artifactId>io.test.test_2.13</artifactId>
  <version>0.1</version>
</dependency> 

Snapshots can be uploaded but they probably have no special handling as you can't upload them twice

Pom upload works, but the rest fails:

[info] published test_2.13 to https://maven.pkg.github.com/[username]/[project]/io/test/test_2.13/0.0.3/test_2.13-0.0.3.pom
[error] java.io.IOException: Access to URL https://maven.pkg.github.com/[username]/[project]/io/test/test_2.13/0.0.3/test_2.13-0.0.3.jar was refused by the server: Forbidden response=Error: "test_2.13-0.0.3.jar" in version 0.0.3 of "io.test.test_2.13" has already been published.

@lokkju
Copy link

lokkju commented Oct 14, 2019

@francisdb JAR upload fails because SBT uploads the files out of order; from what I can remember, it's a long standing bug in SBT (IvyActions.scala) where the publish function uses a map type instead of seq type. With the move to the Coursier dependency management system, this could self resolve.

My suggestion would be to override the upload process itself within this plugin, making sure to upload the pom first. Of course, this still doesn't help with classifiers, which I need support for.

@francisdb
Copy link

looks like on sbt 1.3.2 Coursier is not in use for uploading artifacts, only for downloading

@lokkju
Copy link

lokkju commented Oct 14, 2019

Also, I haven't had any response from GitHub on this issue

@mkurz
Copy link
Contributor

mkurz commented Oct 14, 2019

Hmmm... I just had a very quick look. sbt 1.2.8 is still used here everyhwere:

ThisBuild / sbtVersion := "1.2.8"


(Maybe missed some however)

I guess these need upgrades?

@mkurz
Copy link
Contributor

mkurz commented Oct 14, 2019

Pull request: #2

@mkurz
Copy link
Contributor

mkurz commented Oct 14, 2019

Coursier is only used by default starting with sbt 1.3.0

@francisdb
Copy link

my testing was with 1.3.2

@mkurz
Copy link
Contributor

mkurz commented Oct 14, 2019

@francisdb Got it.

@francisdb
Copy link

francisdb commented Oct 18, 2019

Looks like Github has changed some things, was able to do a full upload today. (vanilla sbt 1.3.3)

[info] 	published test_2.13 to https://maven.pkg.github.com/francisdb/foo/com/example/test_2.13/0.0.9/test_2.13-0.0.9.pom
[info] 	published test_2.13 to https://maven.pkg.github.com/francisdb/foo/com/example/test_2.13/0.0.9/test_2.13-0.0.9.jar
[info] 	published test_2.13 to https://maven.pkg.github.com/francisdb/foo/com/example/test_2.13/0.0.9/test_2.13-0.0.9-sources.jar
[info] 	published test_2.13 to https://maven.pkg.github.com/francisdb/foo/com/example/test_2.13/0.0.9/test_2.13-0.0.9-javadoc.jar

The web ui is still confused about the sbt suffix

<dependency>
  <groupId>com.example.test_2</groupId>
  <artifactId>13</artifactId>
  <version>0.0.9</version>
</dependency> 

@djspiewak
Copy link
Owner Author

This is exciting! I'm going to do some testing today and see if I can close this out. Very very exciting.

@djspiewak
Copy link
Owner Author

I can confirm that, with a few tweaks to the plugin to use @francisdb's experimental results, things are now working! The webui issues with cross-build suffixes do not appear to affect actual resolution, just the UI. So publication and resolution now work with the plugin. I haven't yet tested things with a private repository; my guess is that they aren't quite working yet, but things are good enough that I've published 0.1.0.

Thank you, everyone, for all your hard work and experimentation! The plugin is remarkably trivial thanks to all of you.

@mkurz
Copy link
Contributor

mkurz commented Jan 22, 2020

@francisdb

Snapshots can be uploaded but they probably have no special handling as you can't upload them twice

You can upload them twice: You can delete a snapshot in the GitHub UI and then again upload a new snapshot (meaning same snapshot version). That worked for me. A bit cumbersome, but it works.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants