Aretomo/warp compatibility flags possibly wrong in job submission

[jmdobbs]

Hi,
With the update from yesterday, aretomo is now running fine. However, warp is unable to find the .xf file. I think this may because the wrong --outimod flag is passed during the job submission. The job passes outimod 1, whereas the usual flag for compatibility with warp is --outimod 2
--outxf is also set to 0 by default (should possibly be 1?)

024-05-01 07:28:48.217 Executing AreTomo2 in /struct/mahamid/jdobbs/warp2_testing/data_gseries/processing/tiltstack/gTS_003 with arguments: -InMrc gTS_003.st -AngFile gTS_003.rawtlt -VolZ 0 -OutBin 0 -TiltAxis 85.2 -1 -AlignZ 100 -TiltCor 1 -OutImod 1 -DarkTol 0 -OutMrc gTS_003_aligned.mrc -Gpu 0

[alisterburt]

Hi @jmdobbs ! What version of AreTomo are you using?

For info, we've been working with 1.3.4 and the flags used and subsequent import work for us with that version

We have plans to test AreTomo2 and have heard mumblings about AreTomo3 but haven't got to testing/integrating them yet

[jmdobbs]

Hi! Odd, it's AreTomo2 here but the flags should be the same. The AreTomo --help for 1.3, like for 2.0, says that --outImod 1 should only generate imod files for relion4, not warp. But I guess I am assuming that you still use the same inputs. What is the directory structure I should expect? I get .aln files, and a subfolder called gTS_003_Imod which contains a _st.xf file (and newst.com, tilt.com, .tlt, .xtilt files)

[alisterburt]

@jmdobbs I haven't looked at this carefully for a while but I know AreTomo behaviour has been known to change between versions

The way AreTomo tries to output compatible metadata for different pipelines is a little strange, Warp and RELION should both take xf/tlt files in the usual IMOD format but there are some ways to run AreTomo which produce an "aligned" tilt series and provide this as input to RELION - this adds an unnecessary extra interpolation but allows for some correction of sample deformations

In any case, when we run AreTomo 1.3.4 with the arguments here the tilt series alignments are imported correctly and tomograms are generated according to those alignments

[alisterburt]

Could you attempt with 1.3.4 and report back? If I don't hear back I'll close this in a few days as it appears Warp is working as expected here

[dtegunov]

The flags were working as described in the docs until some version, then I updated to 1.3.4 and had to change them to get anything useful. It makes sense that they fixed this bug in a later version. That's what we get by relying on buggy behavior.

However, I also saw another WarpTools bug when testing this yesterday where it complained about not finding the .xf when I specified --output_processing. Are you possibly running into this one? I'll look into it today.

[jmdobbs]

That makes sense, I used 1.3.0 (I think?) and 2.0 and got the same results. We don't have 1.3.4 installed.

The exact command I ran was
WarpTools ts_aretomo --settings TS_settings.settings --angpix 20 --alignz 2000 --min_fov 0.5

[jmdobbs]

from the testing @ThomasHoffmann77 and I have done: even with aretomo1.3.4 the warp flags produce the incorrect result, our aretomo installation (for Cuda118) appears to behave according to the docs

[jmdobbs]

Thomas was able to patch our module to use --outimod 2, and this didn't generate the relion4 STA files (newst.com, etc), but warp still couldn't find the xf. However, this allowed us to recognize that it was the formatting of the generated files that was wrong. Running aretomo with these parameters (above) generated gTS_003_st.xf, where warp needs gTS_003.xf. Changing the name allows warp to properly import it.

[alisterburt]

@jmdobbs I'm going to update our wrapper to work with AreTomo2 which should fix this - please hold 🙂

[ThomasHoffmann77]

Thomas was able to patch our module to use --outimod 2, and this didn't generate the relion4 STA files (newst.com, etc), but warp still couldn't find the xf. However, this allowed us to recognize that it was the formatting of the generated files that was wrong. Running aretomo with these parameters (above) generated gTS_003_st.xf, where warp needs gTS_003.xf. Changing the name allows warp to properly import it.

I added the patch to the draft in easybuilders/easybuild-easyconfigs#20419

[alisterburt]

closed by #57