Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Selecting pollen package results in infinite recursion #301

Closed
MazeChaZer opened this issue May 24, 2020 · 14 comments · Fixed by #303
Closed

Selecting pollen package results in infinite recursion #301

MazeChaZer opened this issue May 24, 2020 · 14 comments · Fixed by #303

Comments

@MazeChaZer
Copy link

Now that Pollen builds correctly from source (#269) I would like to integrate it into my deployment setup. That means, I need a Pollen derivation I can reference in the derivation that builds my website. I'm assuming that the way to obtain the pollen derivation would be this:

{ pkgs ? import <nixpkgs> {}
, racket2nix ? pkgs.fetchFromGitHub {
      owner = "fractalide"; repo = "racket2nix";      
      rev = "fe09e5cf77103d5b4c3eac094cec2c0444c4573a";
      sha256 = "0cpjp908zzbq6igav1m562q3d9942s2wxp04jla5dwhghlqvxb5q";
  }
}:
import racket2nix { package = "pollen"; }

But this gives me (running nix-build --show-trace):

error: while evaluating the attribute 'buildInputs' of the derivation 'racket-minimal-7.7-pollen' at /nix/store/abxqsa3k14xmias7nj1m3givyp00napc-source/racket-packages.nix:161:3:
while evaluating the attribute 'buildInputs' of the derivation 'racket-minimal-7.7-alexis-util+auto-syntax-e+scope-operations+scribble-math+stxp...' at /nix/store/abxqsa3k14xmias7nj1m3givyp00napc-source/racket-packages.nix:161:3:
while evaluating the attribute 'buildInputs' of the derivation 'racket-minimal-7.7-backport-template-pr1514' at /nix/store/abxqsa3k14xmias7nj1m3givyp00napc-source/racket-packages.nix:161:3:
infinite recursion encountered, at undefined position

😢

Sidenote: I tried another way to obtain a pollen derivation by fetching the Pollen source from GitHub and putting that into buildRacketPackage, but that didn't work. I can describe what I found out trying to get that to work, but because the previous approach seems to be the right one, I'll omit that for now.

@MazeChaZer
Copy link
Author

I know this software isn't maintained anymore, but could you confirm I'm going for the right approach here? That would help me trying to find a solution.

@MazeChaZer
Copy link
Author

MazeChaZer commented May 25, 2020

So I went through all depdencies of Pollen to see if any of them to see if any of them are the cause of the infinite recursion. It seems that rackjure is the problem:

  • base (probably wrong version tested) ✔️
  • txexpr (probably wrong version tested) ✔️
  • sugar (probably wrong version tested) ❌ build failed, no infinte recursion
  • markdown (probably wrong version tested) ✔️
  • htdp ✔️
  • at-exp-lib ✔️
  • html-lib ✔️
  • rackjure ⚠️ infinte recursion!
    • base (probably wrong version tested) ✔️
    • rackunit-lib ✔️
    • threading-lib (probably wrong version tested) ✔️
  • web-server-lib ✔️
  • scribble-lib ✔️
  • scribble-text-lib ✔️
  • rackunit-lib ✔️
  • gui-lib ✔️
  • string-constants-lib ✔️
  • net-lib ✔️

I tested these libraries by replacing pollen in the above code snipped with the library name. I only included deps in this list, not build-deps.

@clacke
Copy link
Member

clacke commented May 27, 2020

I'm delighted to see someone making use of racket2nix, and I will try to make this work for you. :-)

Thank you for the thorough report!

You're going about it the right way, and the problem is the homegrown circular dependency detection that racket2nix uses.

I've been thinking of implementing Tarjan's algorithm in pure Nix, so that it could benefit others in the Nix ecosystem, and also be able to dynamically detect new cycles (or removed cycles) in packages brought in or modified by overlays.

But if you are running into issues right now, the quicker way would be to improve the cycle detection in racket2nix by using graph-lib.

@clacke
Copy link
Member

clacke commented May 27, 2020

The quicker-quicker way would be to identify the precise issue with rackjure and adjust it manually with an override.

I'll look at it this weekend, but if you want to get ahead faster, you could look at build-racket-default-overlay.nix. It's not exactly documented, but it's not entirely cryptic either, if you're a bit familiar with nixpkgs.

Here's what happens when racket2nix encounters a cycle, have a look at alexis-utils around line 779 in racket-packages.nix (search for circular.*alexis):

  • It removes the offending packages from the racketBuildInputs, so that Nix doesn't experience a recursion.
  • It puts the names of the packages involved in the cycle into circularBuildInputs. This is for human consumption, all it does for the build is that a non-empty circularBuildInputes makes it not really build anything and instead print the intentionally left blank notice.
  • It synthesizes a big derivation for all the packages involved in the cycle, with all their sources in extraSrcs.
  • It adds the big cycle derivation to the racketBuildInputs of the original derivation.

If you can identify what racket2nix is doing wrong in this specific case, then you can add overrides to the overlay for doing it right.

@clacke
Copy link
Member

clacke commented May 27, 2020

It seems the issue is around the very example I picked:

$ nix-build --no-out-link --argstr package rackjure --show-trace
error: while evaluating the attribute 'buildInputs' of the derivation 'racket-minimal-7.7-rackjure' at /Users/cw/src/git/fractalide/racket2nix/racket-packages.nix:161:3:
while evaluating the attribute 'buildInputs' of the derivation 'racket-minimal-7.7-alexis-util+auto-syntax-e+scope-operations+scribble-math+stxp...' at /Users/cw/src/git/fractalide/racket2nix/racket-packages.nix:161:3:
while evaluating the attribute 'buildInputs' of the derivation 'racket-minimal-7.7-backport-template-pr1514' at /Users/cw/src/git/fractalide/racket2nix/racket-packages.nix:161:3:
infinite recursion encountered, at undefined position

So something depended on by alexis-util and friends refers back to the cycle (is it backport-template-pr1514?), but racket2nix doesn't detect it.

@clacke
Copy link
Member

clacke commented May 27, 2020

I looked briefly at backport-template-pr1514 and it does indeed have a cycle with the alexis-util cycle, so I removed it from the deps, gave it a non-empty circular dependency list and added it to the cycle sources, but that just revealed debug-scopes as the next issue. I won't go deeper on this at the moment, gotta tend to my paid work. :-)

@MazeChaZer
Copy link
Author

Thanks for digging into this! That are some good hints, if I have the time I will look into this too and try to understand what's going on this weekend.

@MazeChaZer
Copy link
Author

MazeChaZer commented Jun 1, 2020

Ok I think I made some progress here. After wrangling around in racket-packages.nix for a few hours, modifying the dependencies like you described, I was finally able to get rid of the infinite recursion error. I had to add ~10 libraries into the big cycle and had to remove many dependencies from racketBuildInputs that were already provided by cyclicDependencies. It is super hacky and I don't think this can be merged back, especially because I edited racket-packages.nix directly, which is a generated file. You can see my commit here, I'm also attaching it to this comment for future reference. Sadly, when I run nix-build --no-out-link --argstr package pollen --show-trace now, I get another error:

these derivations will be built:
  /nix/store/wnqrqzf1q20rjy9qknx6lcin5kizhf0f-racket-minimal-7.7-alexis-util+auto-syntax-e+scope-operations+scribble-math+stxp....drv
  /nix/store/2wdjplv3bi2s0b3x6rv6r7yxj6cjf96f-racket-minimal-7.7-backport-template-pr1514.drv
  /nix/store/917czc7rqx5jaj8b05qgnkzhm3j0glaq-racket-minimal-7.7-hyper-literate.drv
  /nix/store/2yy8a35c6f3xfy8vkqrysj092p90djxx-racket-minimal-7.7-rackjure.drv
  /nix/store/098l8mh2hkhd8vc7vy4jlx9qh7f343wl-racket-minimal-7.7-debug-scopes.drv
  /nix/store/h4y4la3dx11sbdlzmby52qwjyrcn306r-racket-minimal-7.7-kw-utils.drv
  /nix/store/0h6dbfsnx14dafzqd5shx28pz0rqdkcv-racket-minimal-7.7-mutable-match-lambda.drv
  /nix/store/0ib3wfd0wmwnlj9rfcdqd8w9b47bnv6a-racket-minimal-7.7-alexis-util.drv
  /nix/store/kyild1qil63pzk1rp5znh6acs3vdhm90-racket-minimal-7.7-sugar.drv
  /nix/store/2k0nlfcx9ksaz70ja1raq16b57vs6a66-racket-minimal-7.7-txexpr.drv
  /nix/store/8i22rabradbhjwclq4n5mcz9asdc8k7f-racket-minimal-7.7-slideshow-doc.drv
  /nix/store/dibnhqbg17csmglzys6rx35wgawcf9jv-racket-minimal-7.7-phc-toolkit.drv
  /nix/store/yvvqfglq24qygq9vn5q04z326f1xibkm-racket-minimal-7.7-scribble-enhanced.drv
  /nix/store/a76bz72qp69jf2px10jdwbfv7sychvjx-racket-minimal-7.7-threading-doc.drv
  /nix/store/d3fzn4yssc240aa4ak2yihffg0hgahbl-racket-minimal-7.7-plot-doc.drv
  /nix/store/fkrrixij7s5jws58f7kb3lkc5awp2w7i-racket-minimal-7.7-type-expander.drv
  /nix/store/xz2zpcmhk1aq3ag1h7gnwxfiasrq6w5j-racket-minimal-7.7-pollen.drv
building '/nix/store/wnqrqzf1q20rjy9qknx6lcin5kizhf0f-racket-minimal-7.7-alexis-util+auto-syntax-e+scope-operations+scribble-math+stxp....drv'...
unpacking sources
unpacking source archive /nix/store/yx3ldzwqvar9bbd8yh57hy77mxgk76pj-version-case
/build
unpacking source archive /nix/store/pnivr3v9903fkmhxv3hccgadzjgn5d6g-threading-doc
/build
unpacking source archive /nix/store/c9vg1l8rcy55ybgnaw1vmjs5xv4v95k2-type-expander
/build
unpacking source archive /nix/store/p67wk8ljiv6q6qsl6v6abhr051kwzkvd-scribble-enhanced
/build
unpacking source archive /nix/store/n7gz92ih9v7j1ljp30q59ccs9wisqdj7-rackjure
/build
unpacking source archive /nix/store/adwf378m4di6f43jb9ddhha8xbcwranw-phc-toolkit
/build
unpacking source archive /nix/store/b5rn1lmzhjpp42g6dmyzwsqnpi838p8x-mutable-match-lambda
/build
unpacking source archive /nix/store/2pfszwdsdcq0acgv9sxlwg87c6dykmvl-kw-utils
/build
unpacking source archive /nix/store/mp8jyk6bx3gq0cb7fmh9dq8ndzz1wy5z-hyper-literate
/build
unpacking source archive /nix/store/4s7qfgbqbqhw8arn53xz7znc7zac9vlw-debug-scopes
/build
unpacking source archive /nix/store/virzjz9lpsq05cgxqm23hnjyxgmqhi52-backport-template-pr1514
/build
unpacking source archive /nix/store/mjcinjc8wy85f3rl254ngf81207r7wz7-alexis-util
/build
unpacking source archive /nix/store/9nj5qsbrd7d5fc9yxkn5rh08vxhbydh8-auto-syntax-e
/build
unpacking source archive /nix/store/8cz6lq2152i8yk5iayxgy3yxapbn516q-scope-operations
/build
unpacking source archive /nix/store/f3r7sih6wqbbqwa978ckqpxxfr99v743-scribble-math
/build
unpacking source archive /nix/store/dvvyzvhv444mry50vd3iylxnk7wzzj6x-stxparse-info
/build
unpacking source archive /nix/store/4a5qxj5ca4pqnbxgd33kcja4x4lhzzk8-subtemplate
/build
unpacking source archive /nix/store/gxaj6hz7zdwm5lqsp2vklpgg8c1w2qyp-threading
/build
mkdir: cannot create directory 'threading-doc': File exists
builder for '/nix/store/wnqrqzf1q20rjy9qknx6lcin5kizhf0f-racket-minimal-7.7-alexis-util+auto-syntax-e+scope-operations+scribble-math+stxp....drv' failed with exit code 1
cannot build derivation '/nix/store/xz2zpcmhk1aq3ag1h7gnwxfiasrq6w5j-racket-minimal-7.7-pollen.drv': 1 dependencies couldn't be built
error: build of '/nix/store/xz2zpcmhk1aq3ag1h7gnwxfiasrq6w5j-racket-minimal-7.7-pollen.drv' failed

Here is the patch just in case I will remove my repository in the future: patch.zip
I am working on fe09e5c

@clacke
Copy link
Member

clacke commented Jun 7, 2020

Hi! Thanks for your hard work. I'll use this to better understand what kind of cycle racket2nix isn't seeing. I started making some diagrams, but they would need some more love before they become useful, haha.

catalog

@MazeChaZer
Copy link
Author

Interesting 😄
Looks good, thanks for your help!

@clacke
Copy link
Member

clacke commented Jun 8, 2020

The problem you're seeing there is that you have the self."threading-doc".src twice. Remove one of them and try again. I did that, but all the dependencies are taking forever to build, so I don't know yet if there's anything more.

@clacke
Copy link
Member

clacke commented Jun 8, 2020

Turns out, if I remove the big racket-doc cycle, these are actually the only cycles left. The alexis-util cycle doesn't exist, it's a false positive.

PR incoming with proper cycle detection using graph-lib, ripping out my home-grown monstrosity.

catalogg

@clacke
Copy link
Member

clacke commented Jun 8, 2020

$ cat depgraph.rkt 
#lang racket

(define catalog (read))

(printf "digraph {\n")

(for ([(name package) (in-hash catalog)])
  (for ([dependency (hash-ref package 'dependency-names)])
    (printf "\"~a\" -> \"~a\"\n" name dependency)))

(printf "}\n")
$ racket depgraph.rkt < catalog.rktd > catalog.dot
$ grep -vFf <(grep reverseCircular.*compatibility racket-packages.nix | tr '[[:blank:]]' '\n' | sed -ne 's/"\([^"]*\)"/\1/p') < catalog.dot > catalog0.dot
$ old_i=0; old_len=$(wc -l < catalog$old_i.dot); for i in {1..9} {a..z}; do grep -Ff <(awk '/->/ { print "-> " $1 } END { print "{"; print "}" }' catalog$old_i.dot) catalog$old_i.dot | grep -Ff <(awk '/->/ { print $3 " ->" } END { print "{"; print "}" }' catalog$old_i.dot) > catalog$i.dot; len=$(wc -l < catalog$i.dot); wc -l catalog$i.dot; (( old_len == len )) && break; old_len=$len; old_i=$i; done
$ dot -Tpng < catalogg.dot > catalogg.png  # and that's the above graph

@MazeChaZer
Copy link
Author

Thank you @clacke!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants