Gromacs restart flawed

[ehhartmann]

Gromacs simulation continuation relies on the cpt file which (I think) we can't modify to restart after a reaction. This is why we supply coordinates, velocities and other variables with the trr, edr and maybe other files. From the Gromacs side, this doesn't seem to be properly maintained.
In my opinion we should ask some Gromacs developers how to restart reliably.

[ehhartmann]

Example of gromacs internal float problem with times in the trr and edr file.

  kimmdy_full) [hartmaec@cascade-007 7_equilibrium]$ gmx grompp -p /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/5_apply_recipe/pep_out_d4.top -c /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/6_place_atom/coord_mod.gro -f /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/md_0_1ns.mdp -n /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/index.ndx -o equilibrium.tpr -maxwarn 5 -t /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/6_place_atom/coord_mod.trr -e /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/2_equilibrium/equilibrium.edr
               :-) GROMACS - gmx grompp, 2021.7-plumed-2.9.0 (-:

                            GROMACS is written by:
     Andrey Alekseenko              Emile Apol              Rossen Apostolov     
         Paul Bauer           Herman J.C. Berendsen           Par Bjelkmar       
       Christian Blau           Viacheslav Bolnykh             Kevin Boyd        
     Aldert van Buuren           Rudi van Drunen             Anton Feenstra      
    Gilles Gouaillardet             Alan Gray               Gerrit Groenhof      
       Anca Hamuraru            Vincent Hindriksen          M. Eric Irrgang      
      Aleksei Iupinov           Christoph Junghans             Joe Jordan        
    Dimitrios Karkoulis            Peter Kasson                Jiri Kraus        
      Carsten Kutzner              Per Larsson              Justin A. Lemkul     
       Viveca Lindahl            Magnus Lundborg             Erik Marklund       
        Pascal Merz             Pieter Meulenhoff            Teemu Murtola       
        Szilard Pall               Sander Pronk              Roland Schulz       
       Michael Shirts            Alexey Shvetsov             Alfons Sijbers      
       Peter Tieleman              Jon Vincent              Teemu Virolainen     
     Christian Wennberg            Maarten Wolf              Artem Zhmurov       
                           and the project leaders:
        Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel

Copyright (c) 1991-2000, University of Groningen, The Netherlands.
Copyright (c) 2001-2022, The GROMACS development team at
Uppsala University, Stockholm University and
the Royal Institute of Technology, Sweden.
check out http://www.gromacs.org for more information.

GROMACS is free software; you can redistribute it and/or modify it
under the terms of the GNU Lesser General Public License
as published by the Free Software Foundation; either version 2.1
of the License, or (at your option) any later version.

GROMACS:      gmx grompp, version 2021.7-plumed-2.9.0
Executable:   /hits/fast/mbm/hartmaec/sw/gromacs-2021.7_cascade/run/bin/gmx
Data prefix:  /hits/fast/mbm/hartmaec/sw/gromacs-2021.7_cascade/run
Working dir:  /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/7_equilibrium
Command line:
  gmx grompp -p /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/5_apply_recipe/pep_out_d4.top -c /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/6_place_atom/coord_mod.gro -f /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/md_0_1ns.mdp -n /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/index.ndx -o equilibrium.tpr -maxwarn 5 -t /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/6_place_atom/coord_mod.trr -e /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/2_equilibrium/equilibrium.edr

Ignoring obsolete mdp entry 'title'
Ignoring obsolete mdp entry 'ns_type'
Setting the LD random seed to -9048354

Generated 2211 of the 2211 non-bonded parameter combinations
Generating 1-4 interactions: fudge = 0.5

Generated 2211 of the 2211 1-4 parameter combinations

Excluding 3 bonded neighbours molecule type 'Reactive'

turning H bonds into constraints...

Excluding 2 bonded neighbours molecule type 'SOL'

turning H bonds into constraints...

Excluding 1 bonded neighbours molecule type 'NA'

turning H bonds into constraints...

Excluding 1 bonded neighbours molecule type 'CL'

turning H bonds into constraints...

atom name 7 in /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/5_apply_recipe/pep_out_d4.top and /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/6_place_atom/coord_mod.gro does not match (HH33 - H)
atom name 7 in /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/5_apply_recipe/pep_out_d4.top and /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/6_place_atom/coord_mod.gro does not match (HH33 - H)

WARNING 1 [file pep_out_d4.top, line 1462]:
  1 non-matching atom name
  atom names from
  /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/5_apply_recipe/pep_out_d4.top will be used
  atom names from
  /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/6_place_atom/coord_mod.gro will be ignored



NOTE 1 [file pep_out_d4.top, line 1462]:
  The bond in molecule-type Reactive between atoms 26 C and 27 O has an
  estimated oscillational period of 1.9e-02 ps, which is less than 10 times
  the time step of 2.0e-03 ps.
  Maybe you forgot to change the constraints mdp option.

Number of degrees of freedom in T-Coupling group Protein is 83.94
Number of degrees of freedom in T-Coupling group non-Protein is 4035.06

Determining Verlet buffer for a tolerance of 0.005 kJ/mol/ps at 300 K

Calculated rlist for 1x1 atom pair-list as 1.035 nm, buffer size 0.035 nm

Set rlist, assuming 4x4 atom pair-list, to 1.000 nm, buffer size 0.000 nm

Note that mdrun will redetermine rlist based on the actual pair-list setup

Reading Coordinates, Velocities and Box size from old trajectory

Will read whole trajectory
trr version: GMX_trn_file (single precision)
Last frame          0 time   25.200   

Using frame at t = 25.2 ps

Starting time for run is 0 ps
Opened /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/2_equilibrium/equilibrium.edr as single precision energy file
Last energy frame read 126 time   25.200          

-------------------------------------------------------
Program:     gmx grompp, version 2021.7-plumed-2.9.0
Source file: src/gromacs/fileio/enxio.cpp (line 1186)

Fatal error:
Could not find frame with time 25.200001 in
'/hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/2_equilibrium/equilibrium.edr'

For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------

[jmbuhr]

Do we have a minimal way of reproducing this with just gromacs?

[ehhartmann]

This is reproduced with just gromacs.

gmx grompp -p /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/5_apply_recipe/pep_out_d4.top -c /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/6_place_atom/coord_mod.gro -f /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/md_0_1ns.mdp -n /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/index.ndx -o equilibrium.tpr -maxwarn 5 -t /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/6_place_atom/coord_mod.trr -e /hits/fast/mbm/hartmaec/workdir/dopa_kimmdy/kimmdy01ns_006/2_equilibrium/equilibrium.edr

I added a line to run_gmx so that the exact command and cwd are printed in the logfile if gromacs fails.

[jmbuhr]

I meant from the start without any kimmdy involvement. Like a simulation that runs fine with gromacs and then doesn't continue with gromacs due to the issue. I assume this would just be the simulation kimmdy starts in the step before?

[ehhartmann]

Yeah, one could check whether this issue already occurs without modifying the tpr and edr and if it doesn't modify them by hand and check again.