"Invalid tree info" - Fileless directories

[blast-hardcheese]

Following along with sbt-microsite: Publish using GitHub4s, I've created a token with the repo scope, and have Enable SSO'd (which is required by my environment).

I was then able to use the Github API to perform some simple tests:

• List commits for my repo
• List entries for a given tree

Despite this, when attempting to use publishMicrosite, I get:

[error] Error committing filesGitHub returned an error: Failed invoking with status : 422 body :
 {"message":"Invalid tree info","documentation_url":"https://developer.github.com/v3/git/trees/#create-a-tree"}

        at sbtorgpolicies.github.syntax$GHResponseOps.execE(syntax.scala:53)
        at sbtorgpolicies.github.syntax$EitherTOps.execE(syntax.scala:43)
        at sbtorgpolicies.github.GitHubOps.commitDir(GitHubOps.scala:172)
        at sbtorgpolicies.github.GitHubOps.commitDir(GitHubOps.scala:148)
        at microsites.MicrositeAutoImportSettings.$anonfun$micrositeTasksSettings$13(MicrositeKeys.scala:305)
        at scala.Function1.$anonfun$compose$1(Function1.scala:44)
        at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:40)
        at sbt.std.Transform$$anon$4.work(System.scala:67)
        at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
        at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
        at sbt.Execute.work(Execute.scala:278)
        at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
        at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: github4s.GithubResponses$UnsuccessfulHttpRequest: Failed invoking with status : 422 body :
 {"message":"Invalid tree info","documentation_url":"https://developer.github.com/v3/git/trees/#create-a-tree"}
        at github4s.HttpRequestBuilderExtensionJVM.toEntity(HttpRequestBuilderExtensionJVM.scala:84)
        at github4s.HttpRequestBuilderExtensionJVM.toEntity$(HttpRequestBuilderExtensionJVM.scala:74)
        at github4s.jvm.Implicits$.toEntity(Implicits.scala:21)
        at github4s.HttpRequestBuilderExtensionJVM$$anon$1.$anonfun$runMap$3(HttpRequestBuilderExtensionJVM.scala:67)
        at scala.util.Try$.apply(Try.scala:209)
        at sbtorgpolicies.github.instances$$anon$2.capture(instances.scala:57)
        at sbtorgpolicies.github.instances$$anon$2.capture(instances.scala:56)
        at github4s.HttpRequestBuilderExtensionJVM$$anon$1.runMap(HttpRequestBuilderExtensionJVM.scala:61)
        at github4s.HttpRequestBuilderExtensionJVM$$anon$1.run(HttpRequestBuilderExtensionJVM.scala:34)
        at github4s.HttpClient.post(HttpClient.scala:159)
        at github4s.api.GitData.createTree(GitData.scala:251)
        at github4s.free.interpreters.Interpreters$$anon$9.$anonfun$apply$7(Interpreters.scala:295)
        at cats.data.Kleisli.$anonfun$map$1(Kleisli.scala:19)
        at cats.data.KleisliFlatMap.$anonfun$tailRecM$2(Kleisli.scala:534)
        at cats.instances.TryInstances$$anon$1.tailRecM(try.scala:57)
        at cats.instances.TryInstances$$anon$1.tailRecM(try.scala:15)
        at cats.data.KleisliFlatMap.$anonfun$tailRecM$1(Kleisli.scala:534)
        at cats.data.Kleisli.$anonfun$map$1(Kleisli.scala:19)
        at cats.data.KleisliFlatMap.$anonfun$tailRecM$2(Kleisli.scala:534)
        at cats.instances.TryInstances$$anon$1.tailRecM(try.scala:57)
        at cats.instances.TryInstances$$anon$1.tailRecM(try.scala:15)
        at cats.data.KleisliFlatMap.$anonfun$tailRecM$1(Kleisli.scala:534)
        at github4s.Github$GithubIOSyntaxEither.exec(Github.scala:62)
        at sbtorgpolicies.github.syntax$GHResponseOps.execE(syntax.scala:51)
        ... 19 more

I found 47degrees/sbt-org-policies#388 (comment), which was largely unhelpful other than to suggest that perhaps despite supplying the token, it doesn't work as advertised (possibly in my SSO environment).

Any assistance would be appreciated!

[bilki]

Hi @blast-hardcheese! Is your microsite repo initialized? Also double check that your sbt values micrositeGithubOwner and micrositeGithubRepo are correct.

[blast-hardcheese]

@bilki I presume so. It would be easier to just link where I am at this point.

• PR: Microsite revamp guardrail-dev/guardrail#347
• build.sbt: https://github.com/twilio/guardrail/pull/347/files#diff-f96947b443ddd5823c201c553e5dad4cR11
  -gh-pages: https://github.com/twilio/guardrail/tree/gh-pages

I was successfully able to publish to a non-SSO-protected repo with the API token, fwiw.

[bilki]

Hey, @blast-hardcheese would you mind (if possible) trying to publish the microsite to another clean empty branch, just to be sure it's not the content of gh-pages that is provoking this bug?

[blast-hardcheese]

@bilki OK, so after digging quite a bit, I discovered that the root problem here is I'm getting NewTreeRequest(Some(...), Nil), as I have a tree containing only trees.

I've narrowed down what I suspect is the culprit: def getAllFiles: List[File] = Option(dirToCommit.listFiles()).toList.flatten.filter(_.isFile).

It seems as though we're doing top-down to create all the directories, but I suspect github really wants us to navigate depth first, creating the trees as the full hierarchy.

I'm wrapping up for the night, but this is where I got. If you've got some advice on how to proceed, (is flipping processing from top-down to bottom-up acceptable to you?) I'd appreciate it.

Thanks so far.

[bilki]

Hey @blast-hardcheese, could you check if your site folder contains any empty dir (in the machine from where you are trying to publish)? I get the exact same error Invalid tree info when I try to commitDir a folder that contains empty dirs. As soon as I remove them, it works like a charm.

[blast-hardcheese]

@bilki I have a directory that is empty, save for other directories, which contain files. The problem, as I've seen it, is that we create trees only containing blobs, then go back and create the next deepest tree with those blobs, with the parent being another tree.

My proposal would be to do a depth-first approach, committing the deepest tree, then collecting that tree's hash as another sibling in the next highest up tree to generate.

What do you think?

[stanleydunne]

I am seeing the same error "Invalid tree info" when trying to use the API to add individual files to my repo. I don't have empty folders.
Has a solution been found?
Edit : I found other issue in my own code that was corrupting the data I was passing in. Ignore my question.

[bilki]

Sorry about the delay @blast-hardcheese. That sounds like a possible solution, have you already tried it? I honestly don't know in what order github expects dir creations to happen. Anyway, you could open an issue at https://github.com/47deg/sbt-org-policies/issues, which is where the order on how commit dirs are updated is decided, and it's where your suggestion will apply.

[calvellido]

I've been using github4s through sbt-microsites too, and after messing a bit with this, I can confirm that the error will happen with a project structure like this. Source directory (src/main):

• resources▼
  • microsite▼
    • data▼
      • menu.yml
• scala▼
  • HelloWorld.scala
• tut▼
  • docs▼
    • index.md
    • section-1-1.md
    • section-1-2.md
    • section-1.md
  • index.md

Then, processed through sbt-microsites, this is the actual content that will be committed with github4s:

• css►
• docs►
• highlight►
• img►
• js►
• microsite▼
  • data▼
    • menu.yml
• _config.yml
• index.html

I found that github4s will struggle to commit this structure (422), with the culprit being the fileless microsite directory. As a workaround I added a .gitkeep file near the data directory there and then github4s can proceed.

Somehow, a thing that indirectly solve this, is to use sbt-microsites in a different module, instead of the root one 🤷‍♂️ This solves it because using sbt-microsites that way, the microsite/data/menu.yml won't be generated (hence not committed), and the case is that it shouldn't (47degrees/sbt-microsites#335).

---

But, summarizing, and getting back to the bug, after some other tests, we can conclude that github4s will fail to commit a tree that include some kind of fileless directory like this (I don't think other considerations on the tree shape will matter really for this case):

• static▼
  • subdir▼
    • data.yml

Being subdir the only child of the static directory.

[artemkorsakov]

I have the same problem.
While I had such a structure the publication was successful:

• dir1
  • dir2
    • index.md
    • file1.md
    • file2.md
  • index.md
  • file1.md
  • file2.md
• index.md
• file1.md
• file2.md

When I got such a structure that the publication began to fail with an error "Invalid tree info":

• dir1
  • dir2
    • index.md
    • file1.md
    • file2.md
  • dir3
    • dir4
      • index.md
      • file1.md
      • file2.md
  • index.md
  • file1.md
  • file2.md
• index.md
• file1.md
• file2.md

I got directory dir3 with no files in it - just another directory (dir4)
The error "Invalid tree info" went away only after I added the file to the directory dir3:

• dir1
  • dir2
    • index.md
    • file1.md
    • file2.md
  • dir3
    • dir4
      • index.md
      • file1.md
      • file2.md
    • index.md
  • index.md
  • file1.md
  • file2.md
• index.md
• file1.md
• file2.md