rash build result varies from parallelism #52

bmwiedemann · 2019-04-09T11:13:54Z

While working on reproducible builds for openSUSE, I found that
our rash package varies when comparing builds from 1-core-VM and a 4-core-VM.

Two builds from 1-core-VMs are identical though, so there is some race going on during the build.

Here is an extract from the lengthy total diff:

/usr/share/racket/pkgs/shell-pipeline/shell/demo/compiled/more-pipeline-operators_rkt.zo differs at offset '41' (data)
--- a 2019-04-08 20:45:02.853104872 +0000
+++ b 2019-04-08 20:45:02.853104872 +0000
@@ -1,33 +1,33 @@
 00000000  23 7e 03 37 2e 32 06 72  61 63 6b 65 74 44 02 00  |#~.7.2.racketD..|
 00000010  00 00 12 00 00 00 11 63  6f 6e 66 69 67 75 72 65  |.......configure|
-00000020  2d 72 75 6e 74 69 6d 65  0f 89 00 00 9a 02 00 00  |-runtime........|
+00000020  2d 72 75 6e 74 69 6d 65  27 89 00 00 9a 02 00 00  |-runtime'.......|
 00000030  38 00 00 00 00 00 00 00  00 00 00 00 4c 00 00 00  |8...........L...|
-00000040  c3 88 00 00 00 00 00 00  00 00 00 00 23 7e 03 37  |............#~.7|
-00000050  2e 32 06 72 61 63 6b 65  74 42 ae 89 99 2b 9f 9c  |.2.racketB...+..|
-00000060  e5 3a 9d 55 7d ed 50 8a  32 5e 75 4f da 18 f2 00  |.:.U}.P.2^uO....|
+00000040  db 88 00 00 00 00 00 00  00 00 00 00 23 7e 03 37  |............#~.7|
+00000050  2e 32 06 72 61 63 6b 65  74 42 6a 31 80 57 72 da  |.2.racketBj1.Wr.|
+00000060  ae 16 e0 34 bc 8e f6 ef  f9 08 48 78 7f 33 f2 00  |...4......Hx.3..|
 00000070  00 00 01 00 00 07 00 17  00 37 00 3c 00 42 00 4f  |.........7.<.B.O|
...

/usr/share/racket/pkgs/rash/rash/demo/compiled/rc17-demo-modbeg_rkt.zo differs at offset '41' (data)
@@ -1,33 +1,33 @@
 00000000  23 7e 03 37 2e 32 06 72  61 63 6b 65 74 44 02 00  |#~.7.2.racketD..|
 00000010  00 00 12 00 00 00 11 63  6f 6e 66 69 67 75 72 65  |.......configure|
-00000020  2d 72 75 6e 74 69 6d 65  08 20 03 00 8b 02 00 00  |-runtime. ......|
+00000020  2d 72 75 6e 74 69 6d 65  0a 20 03 00 8b 02 00 00  |-runtime. ......|
 00000030  38 00 00 00 00 00 00 00  00 00 00 00 4c 00 00 00  |8...........L...|
-00000040  bc 1f 03 00 00 00 00 00  00 00 00 00 23 7e 03 37  |............#~.7|
-00000050  2e 32 06 72 61 63 6b 65  74 42 22 58 4a f2 09 a4  |.2.racketB"XJ...|
-00000060  40 6f 08 3e ee 1d aa ae  dd d0 52 72 b7 24 a5 07  |@o.>......Rr.$..|
-00000070  00 00 01 00 00 07 00 27  00 40 00 4e 00 51 00 55  |.......'.@.N.Q.U|
-00000080  00 6e 00 72 00 76 00 8c  00 a8 00 c0 00 c9 00 d1  |.n.r.v..........|
-00000090  00 df 00 e3 00 ed 00 f2  00 f8 00 04 01 11 01 26  |...............&|
-000000a0  01 34 01 43 01 48 01 4f  01 5e 01 67 01 77 01 82  |.4.C.H.O.^.g.w..|
-000000b0  01 8e 01 99 01 ba 01 da  01 f4 01 ff 01 0b 02 10  |................|
-000000c0  02 16 02 1b 02 24 02 2c  02 32 02 3c 02 4c 02 54  |.....$.,.2.<.L.T|
-000000d0  02 5c 02 73 02 7d 02 8a  02 98 02 b3 02 b9 02 c6  |.\.s.}..........|
-000000e0  02 d4 02 e6 02 f1 02 00  03 0c 03 1b 03 22 03 38  |.............".8|
-000000f0  03 51 03 5e 03 68 03 72  03 84 03 8a 03 9c 03 ae  |.Q.^.h.r........|
-00000100  03 c4 03 c7 03 d3 03 d6  03 d9 03 dd 03 e1 03 e5  |................|
-00000110  03 ea 03 ef 03 f3 03 f9  03 07 04 0c 04 18 04 26  |...............&|
-00000120  04 2e 04 30 04 32 04 34  04 38 04 3a 04 41 04 43  |...0.2.4.8.:.A.C|
-00000130  04 46 04 48 04 4b 04 6b  04 85 04 9a 04 ac 04 bd  |.F.H.K.k........|
-00000140  04 c6 04 db 04 f2 04 fd  04 0a 05 11 05 2d 05 31  |.............-.1|
-00000150  05 3f 05 64 05 80 05 9a  05 ae 05 b3 05 b8 05 cc  |.?.d............|
-00000160  05 e7 05 eb 05 f7 05 13  06 15 06 18 06 1a 06 35  |...............5|
-00000170  06 37 06 53 06 57 06 66  06 6b 06 70 06 7a 06 8a  |.7.S.W.f.k.p.z..|
-00000180  06 97 06 a2 06 a6 06 ad  06 b5 06 bb 06 c2 06 c8  |................|
-00000190  06 d2 06 e3 06 f2 06 08  07 18 07 28 07 30 07 35  |...........(.0.5|
-000001a0  07 3a 07 40 07 45 07 4a  07 4f 07 56 07 67 07 6e  |.:.@.E.J.O.V.g.n|
-000001b0  07 7a 07 8c 07 9d 07 a9  07 b5 07 c1 07 ca 07 de  |.z..............|
-000001c0  07 e2 07 eb 07 f9 07 fe  07 0c 08 25 08 33 08 40  |...........%.3.@|
-000001d0  08 4b 08 56 08 72 08 7f  08 8c 08 99 08 a7 08 b3  |.K.V.r..........|
-000001e0  08 bf 08 cc 08 d2 08 d8  08 f0 08 fc 08 08 09 1d  |................|
-000001f0  09 34 09 4a 09 5f 09 6c  09 83 09 91 09 a3 09 b4  |.4.J._.l........|
+00000040  be 1f 03 00 00 00 00 00  00 00 00 00 23 7e 03 37  |............#~.7|
+00000050  2e 32 06 72 61 63 6b 65  74 42 52 c2 82 78 e7 99  |.2.racketBR..x..|
+00000060  f6 f6 7f e2 4e ea 55 9d  34 88 49 0f ce a6 a5 07  |....N.U.4.I.....|
+00000070  00 00 01 00 00 07 00 28  00 41 00 4f 00 52 00 56  |.......(.A.O.R.V|
+00000080  00 6f 00 73 00 77 00 8d  00 a9 00 c1 00 ca 00 d2  |.o.s.w..........|
+00000090  00 e0 00 e4 00 ee 00 f3  00 f9 00 05 01 12 01 27  |...............'|
+000000a0  01 35 01 44 01 49 01 50  01 5f 01 68 01 78 01 83  |.5.D.I.P._.h.x..|
+000000b0  01 8f 01 9a 01 bb 01 db  01 f5 01 00 02 0c 02 11  |................|
+000000c0  02 17 02 1c 02 25 02 2d  02 33 02 3d 02 4d 02 55  |.....%.-.3.=.M.U|
+000000d0  02 5d 02 74 02 7e 02 8b  02 99 02 b4 02 ba 02 c7  |.].t.~..........|
+000000e0  02 d5 02 e7 02 f2 02 01  03 0d 03 1c 03 23 03 39  |.............#.9|
+000000f0  03 52 03 5f 03 69 03 73  03 85 03 8b 03 9d 03 af  |.R._.i.s........|
+00000100  03 c5 03 c8 03 d4 03 d7  03 da 03 de 03 e2 03 e6  |................|
+00000110  03 eb 03 f0 03 f4 03 fa  03 08 04 0d 04 19 04 27  |...............'|
+00000120  04 2f 04 31 04 33 04 35  04 39 04 3b 04 42 04 44  |./.1.3.5.9.;.B.D|
+00000130  04 47 04 49 04 4c 04 6c  04 86 04 9b 04 ad 04 be  |.G.I.L.l........|
+00000140  04 c7 04 dc 04 f3 04 fe  04 0b 05 12 05 2e 05 32  |...............2|
+00000150  05 40 05 65 05 81 05 9b  05 af 05 b4 05 b9 05 cd  |.@.e............|
+00000160  05 e8 05 ec 05 f8 05 14  06 16 06 19 06 1b 06 36  |...............6|
+00000170  06 38 06 54 06 58 06 67  06 6c 06 71 06 7b 06 8b  |.8.T.X.g.l.q.{..|
+00000180  06 98 06 a3 06 a7 06 ae  06 b6 06 bc 06 c3 06 c9  |................|
+00000190  06 d3 06 e4 06 f3 06 09  07 19 07 29 07 31 07 36  |...........).1.6|
+000001a0  07 3b 07 41 07 46 07 4b  07 50 07 57 07 68 07 6f  |.;.A.F.K.P.W.h.o|
+000001b0  07 7b 07 8d 07 9e 07 aa  07 b6 07 c2 07 cb 07 df  |.{..............|
+000001c0  07 e3 07 ec 07 fa 07 ff  07 0d 08 26 08 34 08 41  |...........&.4.A|
+000001d0  08 4c 08 57 08 73 08 80  08 8d 08 9a 08 a8 08 b4  |.L.W.s..........|
+000001e0  08 c0 08 cd 08 d3 08 d9  08 f1 08 fd 08 09 09 1e  |................|
+000001f0  09 35 09 4b 09 60 09 6d  09 84 09 92 09 a4 09 b5  |.5.K.`.m........|
 00000200

+++ new//usr/share/racket/pkgs/rash/rash/demo/compiled/rc17_rkt.dep     2019-04-05 12:00:00.000000000 +0000
@@ -1 +1 @@
-("7.2" racket ("efe0dc5499c61488b11498a84f78962ffa64b116" . "408d55c1c3a6e91ef58aa3821ba4cc2704944c0e") (collects #"linea" #"read.rkt") (collects #"racket" #"base.rkt") (collects #"racket" #"runtime-config.rkt") (collects #"rash" #"private" #"lang-funcs.rkt") (collects #"syntax" #"module-reader.rkt"))
+("7.2" racket ("efe0dc5499c61488b11498a84f78962ffa64b116" . "b86f4cd74942f160d7cdb7d9b2fc87f18423b045") (collects #"linea" #"read.rkt") (collects #"racket" #"base.rkt") (collects #"racket" #"runtime-config.rkt") (collects #"rash" #"private" #"lang-funcs.rkt") (collects #"syntax" #"module-reader.rkt"))
/usr/share/racket/pkgs/rash/rash/demo/compiled/setup_rkt.dep differs (ASCII text, with very long lines)

Unfortunately I know too little about racket or rash to debug this without some help.

The text was updated successfully, but these errors were encountered:

bmwiedemann · 2019-04-09T11:58:59Z

longer diff: http://rb.zq1.de/compare.factory-20190405/rash-compare.out (showing first 200 lines of each file's diff)

willghatch · 2019-04-09T17:41:44Z

Thanks! This looks like a problem more generally with Racket's build system, so I've added an issue to the Racket repository as well: racket/racket#2601

willghatch · 2019-04-09T17:48:16Z

Or perhaps I'm wrong! I'll have to look through the implementation and see if/where I'm using some non-deterministic hash-table traversal or gensym...

willghatch · 2019-04-09T18:11:40Z

I've removed some code that was likely the culprit (gensyms in macros that I was using for some reason). So if this were re-done using the master branch it would perhaps be reproducible now. I can't think of any other macros that could be doing anything non-reproducible.

samth · 2019-04-09T18:36:30Z

@willghatch usually you can test this by compiling to zo on a few different machines (or with -j options to raco make) and seeing if you get the same contents.

willghatch · 2019-04-09T19:36:08Z

@samth Thanks. After trying to think of a more convenient way and failing, I've been doing exactly that.

Which leads me to the bad news that it's still not reproducible. I think I've isolated it to a single file (git-info.rkt). But I haven't figured out what causes it yet. The only macro used there should be deterministic. It uses gensym at runtime to get a singleton value, but that shouldn't affect reproducible compilation (and removing it doesn't fix it). Anyway, I'll look at this some more after I eat some lunch.

willghatch · 2019-04-09T22:06:51Z

So my last statement of isolation was way off, that was just an artifact of the way I first started re-kicking builds. At this point it seems that: Rash's main.rkt and its dependencies are reproducible. But any file using main.rkt via #lang rash that also uses begin-for-syntax, define-syntax, or otherwise directly has code at phase 1 is not reproducible (though using pre-defined Rash macros causes no problem). I'm confused as to how this is happening. My best guess is that it has something to do with how my #%module-begin macro is set up. But generally I'm mystified.

willghatch · 2019-04-09T22:17:58Z

@samth Thanks for your help on this, by the way. Aside from gensym and hash-table traversals, are there any other features that come to mind? As far as I know, raco setup gets parallelism by using places, not by spawning new processes. If it were spawning new processes, I wouldn't expect gensym or hash-table traversals to cause non-reproducibility, but I'm not really sure how those things work in places where some things are shared. Even with places I find this slightly confusing, since each place has its own garbage collector and everything. But I've never really used places and I'm not sure what is shared besides the ability to send things over channels.

samth · 2019-04-12T13:52:38Z

Both gensym and hash traversal order are affected by what order things happen in globally (for example, memory addresses or how many gensyms have been created). So lots of things can affect them.

samth · 2019-04-12T14:00:55Z

Looking at the expansion of a simple rash module, here are a few things that might be an issue.

It looks like there's an options hash that gets put into the module -- is the order that those keys appear deterministic?
It looks like there are some gensyms in something about current-paramter-environment

I also noticed that a trivial #lang rash program expands to 360+ lines of code. Could some of that be moved into functions where you expand to a reference to that? That makes it less likely that the expansion could change non-deterministically while also making zo sizes smaller.

bmwiedemann · 2019-04-12T14:39:10Z

When having non-deterministic hashes in other languages, we usually sort when iterating over them.
https://github.com/bmwiedemann/theunreproduciblepackage/tree/master/hash has 2 examples.

willghatch · 2019-04-13T01:41:37Z

1. It looks like there's an options hash that gets put into the module -- is the order that those keys appear deterministic?

That's actually a run-time hash. But I've removed it anyway because it was stupid.

2. It looks like there are some gensyms in something about `current-paramter-environment`

That's it. Today I traced it down to my use of splicing-syntax-parameterize. This program does not build reproducibly: ``` #lang racket/base (require racket/stxparam racket/splicing (for-syntax racket/base)) (define-syntax-parameter param #'something) (splicing-syntax-parameterize ([param (make-rename-transformer #'some-identifier)]) (begin-for-syntax (void))) ``` Looking into the implementation of `splicing-syntax-parameterize`, it uses a `do-local` function that uses `gensym`. I'll look into that more tomorrow or early next week.

I also noticed that a trivial `#lang rash` program expands to 360+ lines of code. Could some of that be moved into functions where you expand to a reference to that? That makes it less likely that the expansion could change non-deterministically while also making zo sizes smaller.

I did a little bit of that today while poking around it. I should certainly do some more.

willghatch · 2019-04-13T01:42:38Z

@samth I really appreciate your help on this. Thanks!

bmwiedemann · 2019-08-22T11:39:34Z

Hi, were there patches done that I can test?

bmwiedemann · 2024-02-06T15:54:31Z

This is still a problem with
racket-8.11.1
rash-0.2

+++ new//usr/share/racket/pkgs/rash/rash/compiled/experimental_rkt.dep  2022-12-27 00:00:00.000000000 +0000
@@ -1 +1 @@
-("8.11.1" ta6le ("c766a8edc6824eb84469005591ebecfd6a74207e" . "010b0d6788990d0e951970461b12bdfa90d34464") (collects #"racket" #"base.rkt") (collects #"racket" #"runtime-config.rkt") (collects #"rash" #"private" #"escapable-template.rkt") (collects #"rash" #"private" #"lang-funcs.rkt"))
+("8.11.1" ta6le ("c766a8edc6824eb84469005591ebecfd6a74207e" . "0d7d0978458e495d931d91cc1a50fb926f7dd956") (collects #"racket" #"base.rkt") (collects #"racket" #"runtime-config.rkt") (collects #"rash" #"private" #"escapable-template.rkt") (collects #"rash" #"private" #"lang-funcs.rkt"))

willghatch mentioned this issue Apr 9, 2019

raco reproducible builds racket/racket#2601

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rash build result varies from parallelism #52

rash build result varies from parallelism #52

bmwiedemann commented Apr 9, 2019

bmwiedemann commented Apr 9, 2019

willghatch commented Apr 9, 2019

willghatch commented Apr 9, 2019

willghatch commented Apr 9, 2019

samth commented Apr 9, 2019

willghatch commented Apr 9, 2019

willghatch commented Apr 9, 2019

willghatch commented Apr 9, 2019

samth commented Apr 12, 2019

samth commented Apr 12, 2019

bmwiedemann commented Apr 12, 2019

willghatch commented Apr 13, 2019 via email

willghatch commented Apr 13, 2019

bmwiedemann commented Aug 22, 2019

bmwiedemann commented Feb 6, 2024

rash build result varies from parallelism #52

rash build result varies from parallelism #52

Comments

bmwiedemann commented Apr 9, 2019

bmwiedemann commented Apr 9, 2019

willghatch commented Apr 9, 2019

willghatch commented Apr 9, 2019

willghatch commented Apr 9, 2019

samth commented Apr 9, 2019

willghatch commented Apr 9, 2019

willghatch commented Apr 9, 2019

willghatch commented Apr 9, 2019

samth commented Apr 12, 2019

samth commented Apr 12, 2019

bmwiedemann commented Apr 12, 2019

willghatch commented Apr 13, 2019 via email

willghatch commented Apr 13, 2019

bmwiedemann commented Aug 22, 2019

bmwiedemann commented Feb 6, 2024