AFNI build fails with /bin/mv: cannot stat 'Dimon1': No such file or directory

[TheChymera]

This being the full build.log.
The version is 20.1.16. This worked before, so perhaps this is a dependency issue... Any ideas?

[afni-rickr]

Hi @TheChymera. Strange. The error seems to be due to a missing selenium function in the /usr/local/libmri.a file. Was libmri.a compiled at the same time as the current Dimon1, or is it from some previous build? What Makefile are you using for this? Or is it via cmake?

In any case, note that not only are we no longer distributing Dimon1, we may disable selenium in general (see -DSELENIUM_READY). But still, this error is puzzling.

[TheChymera]

@afni-rickr /usr/local/libmri.a is not provided by any other package. Is it then provided via the AFNI build system? The full build command is located at the beginning of the log:

make all 'CC=x86_64-pc-linux-gnu-gcc -march=native -O3 -pipe -fPIC -DREAD_WRITE_64 -DLINUX2 -Wcomment -Wformat -DUSE_TRACING -DHAVE_XDBE  -DDONT_USE_XTDESTROY' ; /bin/mv -f *.o .. ; )

Any idea what I could try to do? maybe not patch out the -DSELENIUM_READY out of the Makefile along with the Python bits, as discussed earlier? #213
Patch file seen here: https://gitweb.gentoo.org/proj/sci.git/commit/?id=83fd8b496d322f3f6130ff251f0567486f35a14a

[TheChymera]

@afni-rickr

Tried not patching -DSELENIUM_READY out, that reverted to the previous Python error.

I also tried with the newest tag, AFNI_20.3.03 and it fails in the same fashion (build.log) :( Any ideas what else I can try?

[neurolabusc]

@TheChymera

1. For AFNI, you typically do make vastness not make all if you want to re-build the supporting libraries.
2. looks like you are building on a Gentoo distribution using the makefile for linux_fedora_19_64. I do not have Gentoo, but that may explain some issues.

[afni-rickr]

@TheChymera
To be sure, please run
ls -l /usr/lib64/libmri.a
and be sure that is being compiled along with the rest of this. That is an AFNI library that should be local to the build, not stored at the system level (since it tends to change with the new code).

But it is still confusing, because without -DSELENIUM_READY, selenium_init() should never be called, and the variable is not in the build log, and the function does not exist in /usr/lib64/libmri.a.
Can you verify that the build is being run from a clean tree? To be positive, please run the build from a completely fresh afni_src (or afni/src) directory, without any old .{a,o,so} files.

To see where the problem might be coming from, what is the output from:
grep -rl Py_Initialize .

[afni-rickr]

Oh, I am thinking that actually /usr/lib/libmri.a (which does not seem to be compiled in this particular build) was compiled with -DSELENIUM_READY, and so the build is failing because it is missing the python library (via -lpython2.7). For example, try:
grep Py_Initialize /usr/lib/libmri.a

Does it show a match?

[TheChymera]

grep Py_Initialize /usr/lib64/libmri.a (the path being where the file is actually installed on my system) returns nothing. The file looks like a binary, so I'm unsure whether grepping it would likely return a lot of meaningful strings. To quote part of it:

�<��#��/�L6�D=�<D�K�<^�Tl�����<�L�D�<��<�T����$�<+�;�K��W��d��s���L��D��<���T��<�2�<;�LB�L���<�D%�,�<8�TB��g�t�<}�����L���<�D��T����<�<	���L��#�<*�D1�:�TD��i�p�<}�<���L�D�<��T���h�<�<���L�D	�<��T'��5�f<�<I�<U��a�Lh�Do�<v��T���c�<�<���L�D�<��T����<�<!��-�L4��;�<B�DI�R�T\���p�<�<���L���<�D��T����<�<!��-�L4�D;�<B�<R�T`��g�~��u�<�L�D�<��<�T���u�<�����L���<%�D,�3�<C�TM���x�<��T���<�L�D�<��<�T���x�<�-�T9��E�<T��`��l�Ls�Dz�<��<�T���x�<�L��D��<���Tf�s�<�����L��D��<���T3@<LLSDZ<ah<tT�<��L�D�<��<�T�^<L#D*<18<DTR�`xm<v��LD<<T���w�<�L�D<T"�0v=<F�R�^LeDl<sz<T�u<�"�.L5D<<CJ<ZTh�vs<LD<<T���q�<����LD
                                     <<*T8�FaS<\LcDj<qx<T�<����L�D�<��<�#<,L3D:<AH<TTb�p}<��LD<<�T����<LD
<<$T2�@M<V�b�nLuD|<<T��<�L�D�<��<�T�<&�2�>LEDL<SZ<jTx�<LD<<�T����<��LD<#*<:TH�Vbc<lLsDz<<T�<�����L�D�<�<
�<=LDDK<RY<lTz�����m�����g<���,=gEhmo�unj%mH(H�X@�0�` P`p���4�P0t��P�p$�P@��� D@Xl0P���` !!,@"@p"T#h%&�'((<)P@*+�-�p-�`.0.H.`.x�/ /@/
                                                                                                                                     Pa.symtab.strtab.shstrtab.rela.text.data.bss.rodata.str1.1.rela.data.rel.local.rodata.rodata.cst8.rodata.cst32.rodata.cst16.comment.note.GNU-stack.rela.eh_frame @Hk&� �Xq 12��E�� @@`U�� j�  x`+ 0+#++@�2+L       x^��/971            1609252531  0     0     100644  12824     `
ELF>/@@

Interestingly, the error only seems to occur if I try to downgrade from 20.3.03 to 20.1.16. Upgrading seems to work fine, as well as installing either without another version present. Grepping grep Py_Initialize /usr/lib64/libmri.a returns nothing for either version.

[afni-rickr]

@TheChymera For me, a binary file works fine, showing "Binary file matches" (assuming there is something to match). So maybe that symbol is not there, as you say. You could try: strings /usr/lib64/libmri.a | grep Py_Init, just to make sure.
I had not realized this, but when using libmri.a, SELENIUM_READY can be set (in the top-level Makefile, say). However when creating libmri.so (as the newer linux Makefiles do), that is forcibly turned off in Makefile.INCLUDE.
The question of installing with another version present is the concern. It is very likely the case that one of them is building with it, and one is not. I would be happy to look at both complete build logs to see. In any case, the libmri.* library should always correspond with the current version.

[TheChymera]

@afni-rickr thanks for offering to look into this. In any case, strings /usr/lib64/libmri.a | grep Py_Init returns nothing for any of the versions.

Here are the full build logs:

• 20.1.16 de novo: https://chymera.eu/debug/afni/update-errors/afni-20.1.16
• 20.3.03 de novo: https://chymera.eu/debug/afni/update-errors/afni-20.3.03
• 20.3.03 with 20.1.16 pre-installed: https://chymera.eu/debug/afni/update-errors/afni-20.1.16-to-20.3.03
• 20.1.16 with 20.3.03 pre-installed (FAILS): https://chymera.eu/debug/afni/update-errors/afni-20.3.03-to-20.1.16

[afni-rickr]

@TheChymera These do not have any mention of SELENIUM, and the error is now regarding mcw_realloc.
In general, there should never be a pre-install (of a different libmri.a, if that is what you mean). Those are the functions that change the most often. That library should always be built along with the binaries. If the only failure is when doing a pre-install, then maybe there is no problem.

What OS, version and Makefile are you using? Maybe we could set up a build machine and Makefile that will work for you.

[TheChymera]

@afni-rickr

there should never be a pre-install
But there will always be one when you're updating to a new version. This all worked for a few years now.

The distribution is Gentoo Linux. I am using the linux_fedora_19_64 Makefile. A few years ago, when I first made build instructions for AFNI available via Gentoo Science, I went through the makefiles and this one seemed to work best. We also had a jab at setting up cmake for AFNI (a student of mine submitted a gigantic PR at some point), but that seems to have died out.

[afni-rickr]

@TheChymera Do you mean that you first "make" the libraries, then the executables later? That seems fine. I just mean that the libraries (like libmri) should always be updated with the rest of the programs (programs should be linked to currently built library).
But then I may not understand what you mean by "20.1.16 with 20.3.03 pre-installed".

@leej3 did set up a cmake system, which he has been using. That might require a conda env (see tests/environment.yml). However this is not currently working for me.
There is a web page, in case that is helpful: Cmake for AFNI

[leej3]

@TheChymera, I'd be delighted to help you transition to the cmake build if you are interested. I believe I used the student's work (Tharvik right?) along with the sustained efforts from the folks at neurodebian. It's been working very nicely for me for a little over a year (minus the R stuff which I never really figured out a stable set of dependencies for).

There's various options that you can tweak for your purposes (mostly listed in this module).

The development dependencies for AFNI should not be different from the make build. With that said, there are various packages that one can use for runtime binaries or libraries for dynamic linking instead of directly building from the source in the afni repository... such as f2c, gts, qhull, jpeg, glw (but see this), and hopefully at some point nifti and gifti.
You might find the base dockerfile useful for determining dependencies albeit on a debian system, though I have included a lot of extra programs there for convenience during development.

See the dockerfile for the cmake build for an example command to drive the build

[TheChymera]

@afni-rickr

there should never be a pre-install
But there will always be one when you're updating to a new version. This all worked for a few years now.

@leej3
Yes, that sounds great! Where would we start?

[afni-rickr]

@TheChymera
If you always do a pre-install, would you please clarify what is meant by that? If it means first building libmri.so or .a, then building the rest to link to that library, it is good.
But if it means, for example, having the 20.3.03 package executables link to libmri.a from 20.1.16, that is not so good.

[TheChymera]

@afni-rickr

If you always do a pre-install

I don't do a pre-install, but whenever I install a different version, obviously there will be the previous version on the system.
The problem seems to be that AFNI somehow is influenced by these files which exist outside the sandbox, which is really strange.

To further clarify, going back to #232 (comment) — before I do “20.1.16 with 20.3.03 pre-installed” that means that the files installed at the end of the “20.3.03 de novo” log are already on the system. That is what those logs show, that depending on what version the system already has, the new build somehow fails. Again, these aren't files pre-existing in the sandbox. The sandbox is always empty to begin with, these files exist only in the filesystem hierarchy from where the user can use them.

[afni-rickr]

@TheChymera
To be sure, why is libmri.a being installed into a system directory (/usr/lib64) here? That is not needed for the binaries that are being compiled. And as we see, the current build linking to an old libmri is not even expected to work.

Or is it being used for someone else writing and compiling new code? If not, there might be no reason to put that on the system. If you do want to install it though, the build process should be:

1. make libmri.a
2. mv libmri.a /usr/lib64
3. make totality

But to back up, is there any reason to install that under /usr/lib64?