redmine
The following list summarizes feature requirements established during development of https://github.com/kassonlab/gmxapi
- MPI environment sharing
- Binding external objects with interfaces for external code to retain access to launched tasks (such as objects instantiated during thread-MPI launch)
- Accessible simulation stop signal
- Launch MD runner through library calls
- this issue and a cluster of Redmine issues with subtasks should be good
- documentation and visual aids like the attached progress chart should probably be in the repository somewhere
- Gmxapi interfaces should continue functioning with unchanged semantics for other GROMACS changes, or API level needs to be incremented according to semantic versioning.
- External projects need to be tested outside of the gromacs build tree to validate external interfaces of installation. Suggested external projects: Python package, sample_restraint, yet-to-be-written integration test suite.
- Tests should be clear about the API version they are testing, and we should test all versions that aren’t unsupported (though we need a policy in this regard) and we can note whether new API updates are backwards compatible.
- Forward-compatibility testing: we should at least know whether we are breaking old client code and include release notes, regardless of policy
- ABI compatibility testing? (should we test mismatched compilers and such?)
- Example code in documentation should be tested, if possible.
- mdrun CLI program is an API client
Relates to #2229
- ForceProviders obtained after tMPI threads have spawned.
- MDModules list extended at runtime during simulation launch.
- External code may be provided to the runner to instantiate or get a handle to a module.
- Expanded Context class can broker object binding by registering and holding factory functions for modules, as well as other resources that may be implemented differently in different environments.
- Somewhere in here, MDModules either need access to the integral timestep number or the ability to register call-backs or signals on a schedule.
Relates to #2590, #2574, #1972
Do MDModules live in a scope of tight association with an integrator? Do we need other concepts, like RunnerModules? Or subdivisions like MDForceModule, MDObserverModule, MDControlModule?
Ultimately tied to gmx5 and gmx24, but we can start stabilizing the external interfaces now. The external interfaces are for (a) user interface / workflow management code, and (b) MD extension code. We define a simple message-passing C structure along with PyCapsule name and semantics. An MD extension object can provide a factory method with which the MD Runner can get an IMDModules interface at simulation launch. The object pointed to may exist before and/or after the lifetime of the simulation. It must be understood that the IMDModule handle will be obtained on every rank. Design should consider future infrastructure and needs, but does not need to implement now. (expressing data dependencies and locality, negotiating parallelism, expressing periodicity) Short-term implementation may require workarounds for some of these, but the workaround can mostly be segregated from this issue’s resolution.
Relates to #2590
Relates to #1972, #2590, https://github.com/kassonlab/sample_restraint
Relates to #2224
- Libgmxapi requests resources from libgromacs from the current node
- CUDA environment can be manipulated but we shouldn’t have to deal with that for a while
- Evolving task scheduling interfaces, expressing data locality
- Concepts of time and timestep
Christian Blau actively working on this from mid-July
Log "file" artifacts are produced through API, allowing extensibility and abstraction from filesystem dependence. Progress has already been made in this direction, but the logging resource could be more clearly owned by the client code (or a Context object owned or managed on behalf of the client code) rather than created and destroyed in, say, the Mdrunner.
Also relates to #2570
currently the gmx binary has a commandline runner thing that catches the exceptions, reports an error and exits, but the API can and should do something else, because it plays the same role as the commandline runner
- Status type defines the interface for discovering operation success or failure, plus details.
- Consistent status object interface is portable across Python, C++, and C
- Status object can be used to ferry information across API boundaries from exceptions thrown. Exceptions could be chained / status nested.
Questions:
- What are concerns and solutions for memory allocation for status objects? Should objects own one or generate one on function return?
- Should the API (or Context) keep a Status singleton? A Status stack? Or should operations create ephemeral Status objects, or objects implementing a Status interface?
- Should the status object contain strings, reference strings mapped by enum, or defer textual messages to messaging and logging facilities?
(for better testing) - GlobalTopology class and IGlobalTopologyUser interface underway will help here, so that client changes to the global topology can ripple through to the modules because the ones that care have registered themselves at setup time
- break up mdrun program into clearly defined layers and phases
- CLI program parses various inputs in order to launch an Mdrunner object that is CLI-agnostic
- launching tMPI threads and other significant changes of state establish a sequence or hierarchy of invariants through RAII and/or State pattern.
- Sebastian Wingbermuhle working now on aspects of this for hybrid MC/MD (ref #2375, …)
- Structure, topology
- Microstate
- Simulation state
- Simulation parameters
- Runtime parameters / execution environment
- Anything else?
Event hooks or signals for
- checkpoint
- time step number or delta / trajectory advancement
- input configuration
- input topology
- input state
- simulation parameters
- output data streams
Generally, replace std::exit (gmx_fatal)with exceptions
- Root out gmx_fatal, clearly define regular exit points and exception throwers
- API firewall should catch exceptions from gmx and convert to status objects for ABI compatibility. (gmx17)
- Clearly document regular and irregular shutdown behavior under MPI, tMPI, and generally, specifying responsibilities
- Create issue tickets for discovered missing exception safety, memory leaks, opportunities for RAII refactoring, and complicated protocols that should either be better documented or replaced with a clearer hierarchy (or sequence) of invariants
Abstraction for status messages, such as are currently printed to stdout or stderr
Short term: mdrun CLI-like functionality to override other input is sufficient
Long term: sufficient API to update parameters between phases of simulation work
Implementation roadmap is probably
- Inject argv fields
- Write to input_rec or other structures
- Interact with MDOptions framework
- Generate runnable input from user input
- United implementation for workflow API and utility functions (e.g. possibility of deferred execution / data transfer)
- Ultimately should not require writing output to (tpr) file
- File inputs ultimately should be generalized to API objects
- United implementation for workflow API and utility functions (e.g. possibility of deferred execution / data transfer)
- Utility API should be sufficient to reimplement CLI tools
- I/O should ultimately be separate from algorithm; filesystem interaction optional
- Consider feature requirements of other projects such as MDAnalysis.
Establish policies and layout for external (installed) API documentation, extension API for plug-in developers, and developer documentation for API and library implementation levels. Integrate with previous webpage-sphinx and doxygen targets and output.
https://redmine.gromacs.org/issues/701 Add symbol visibility macros
https://redmine.gromacs.org/issues/988 Definition of "public API"
https://redmine.gromacs.org/issues/2045 API design and language bindings
https://redmine.gromacs.org/issues/2139 More precise/explicit documentation conventions
https://redmine.gromacs.org/issues/2375 Clarify execution phases for MD simulation
https://redmine.gromacs.org/issues/2585 Infrastructure supporting external API
https://redmine.gromacs.org/issues/2587 Provide Context (e.g. to runner code) to manage client and runtime environment
https://redmine.gromacs.org/issues/2698 gmxapi documentation integration
https://redmine.gromacs.org/issues/2756 gmxapi integration testing
https://redmine.gromacs.org/issues/2763 GMXAPI layout / grouping in development docs "modules" page
https://redmine.gromacs.org/issues/2764 gmxapi version updates for post release-2019
https://redmine.gromacs.org/issues/2896 Python packaging
https://redmine.gromacs.org/issues/2919 C++ style guidelines for namespace use
https://redmine.gromacs.org/issues/2961 How should Python package find GROMACS resources under various circumstances?
https://redmine.gromacs.org/issues/2985 Python package documentation
https://redmine.gromacs.org/issues/2993 Scalar and structured type expression and definitions for API
https://redmine.gromacs.org/issues/2994 Data flow topology in gmxapi 2020
https://redmine.gromacs.org/issues/2996 gmxapi execution model
https://redmine.gromacs.org/issues/2998 Update Python detection
https://redmine.gromacs.org/issues/3014 gmxapi example Python scripts
https://redmine.gromacs.org/issues/3038 Improvements to MD plugin development environment
https://redmine.gromacs.org/issues/3040 Refactor Restraint module
https://redmine.gromacs.org/issues/3130 Interim handling of gmxapi data references.
https://redmine.gromacs.org/issues/3132 gmxapi testing support tools
https://redmine.gromacs.org/issues/3133 Cookiecutter for sample_restraint
https://redmine.gromacs.org/issues/3134 gmxapi Python exception names may need improvement
https://redmine.gromacs.org/issues/3136 gmxapi.operation data flow topology unclear or incomplete
https://redmine.gromacs.org/issues/3138 Improve ensemble support in Context specification.
https://redmine.gromacs.org/issues/3139 gmxapi Futures should be subscribable
https://redmine.gromacs.org/issues/3140 Allow explicit input definition for gmxapi.operation function wrapper
https://redmine.gromacs.org/issues/3141 gmxapi File placeholders missing from beta release
https://redmine.gromacs.org/issues/3145 gmxapi.mdrun needs improved (and public) interface for attaching an MD extension.
https://redmine.gromacs.org/issues/3146 update gmxapi serialization scheme
https://redmine.gromacs.org/issues/3147 gmxapi workflow checkpointing
https://redmine.gromacs.org/issues/3148 Roadmap for gmxapi filesystem interactions.
https://redmine.gromacs.org/issues/3149 Python user interface for obtaining simulation artifacts as files.
https://redmine.gromacs.org/issues/3150 gmxapi data type annotations are confusing and inadequate
https://redmine.gromacs.org/issues/3152 Infrastructure and patterns for expressing public interfaces
https://redmine.gromacs.org/issues/3153 Let CMake process module directories earlier to support more modern CMake idioms.
https://redmine.gromacs.org/issues/3154 Update use of Python subprocess for >=3.5
https://redmine.gromacs.org/issues/3174 Proposed gmxapi syntax update: "input" and "output" key word arguments -> "inputs" and "outputs"
https://redmine.gromacs.org/issues/3177 Spack package management support
https://redmine.gromacs.org/issues/3179 Clarify access to parallel data outputs
https://redmine.gromacs.org/issues/3192 gmxapi MD cannot distinguish between coupled and uncoupled simulations
https://redmine.gromacs.org/issues/3227 solvate does not check return status of gmx_file_rename
https://redmine.gromacs.org/issues/3228 gmxapi.commandline_operation should gracefully handle input/output file arguments
https://redmine.gromacs.org/issues/3271 Adopt PEP-518 for Python package build system configuration.
https://redmine.gromacs.org/issues/3273 gmxapi setup.py complains too much
https://redmine.gromacs.org/issues/3279 Provide more advice for building client software