Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support a fddatautilities umbrella package #361

Open
jcfreeman2 opened this issue Apr 5, 2024 · 1 comment
Open

Support a fddatautilities umbrella package #361

jcfreeman2 opened this issue Apr 5, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@jcfreeman2
Copy link
Contributor

...which will depend directly only on hdf5libs and rawdatautils, and can be used for data analysis but isn't a full blown DAQ system the way fddaq and nddaq are.

A major part of this Issue is that we want to have an image available to people who want to use fddatautilities which is much slimmer than the fddaq/nddaq analogue, ghcr.io/dune-daq/nightly-release-alma9:development_v5 and the image which it derives from, ghcr.io/dune-daq/alma9-slim-externals:v2.1, both north of 10 GB.

A secondary effect of this Issue is it should give us a real-world sense of what extending our framework to support diverse types of umbrella package entails; this topic is covered in more detail in Issue #338

@jcfreeman2 jcfreeman2 self-assigned this Apr 5, 2024
@jcfreeman2 jcfreeman2 added the enhancement New feature or request label Apr 5, 2024
jcfreeman2 pushed a commit that referenced this issue Apr 9, 2024
…ocally on daq.fnal.gov

fddatautilities is an umbrella package which depends on a couple of
far-detector-focused packages as well as the core set of packages
(currently named "dunedaq" as it has for a couple of years, but soon
to be renamed "coredaq"). dunedaq has been given a build variant
called "subset" which when it assumes the value of "datautilities"
will only depend on the subset of core packages which fddatautilities
needs.

Note that it may make sense to have the variant value match up
perfectly with the name of the target package it's intended to
support.

In order to not try and solve everything at once, the package.py files
for the umbrella packages needed for this build are located in a
temporary subdirectory called issue361_dev_files. With this
subdirectory I can sidestep worrying about the make-release-repo.py
algorithm whenever I want to modify and test the relevant package.py
files for the umbrella packages during the development process.

I've successfully build fddatautilities using the files in this commit
via calling build-release.sh manually inside of a container on daq.fnal.gov
which is volume-mounted to a v2.1 externals area.
jcfreeman2 pushed a commit that referenced this issue Apr 10, 2024
…reate an image with only externals needed for fdreadoututilities
jcfreeman2 pushed a commit that referenced this issue Apr 10, 2024
…CHE-dness, force the layers to be redone in the image by sticking a command before them
jcfreeman2 pushed a commit that referenced this issue Apr 10, 2024
… try to get the cache to be ignored so I can use a different staging location on daq.fnal.gov than normal
jcfreeman2 pushed a commit that referenced this issue Apr 12, 2024
@jcfreeman2
Copy link
Contributor Author

Status update:

What we have so far:

  • ghcr.io/dune-daq/alma9-slim-externals:fddatautilities, an image which contains only the externals needed by fddatautilities and which is 6.8 GB as opposed to 9.1 GB for ghcr.io/dune-daq/alma9-slim-externals:development_v5
  • ghcr.io/dune-daq/nightly-release-alma9:fddatautilities, which contains the fddatautilities umbrella package

To get fddatatutilities inside a container based on the above image, you can just do the following:

 . /cvmfs/dunedaq-development.opensciencegrid.org/nightly/NFDDU_DEV_240408_A9/spack-installation/share/spack/setup-env.sh 
spack load fddatautilities

and then run spack spec ..., etc., on it.

As you can see, NFDDU_DEV_240408_A9 is a new repository, separate from the core DUNE DAQ repository or the externals repository. In this sense it's analogous to NFD_DEV_240408_A9. While it is the case that the fddetdataformats and rawdatautils packages in this repository are also found in NFD_DEV_240408_A9, this won't in general be the case for different collections of packages (e.g., run control packages) in the future; thus I made the decision to use a separate repo rather than inserting an fddatautilities umbrella package into the far detector repo.

fddatautilities depends on a variant of dunedaq (*), dunedaq subset=datautilities. Moving forward I believe it makes sense that whenever we have a new umbrella package, we add a new possible value to subset to specify which core dunedaq packages are needed. To extend that point, we likely should do the same thing for the externals repo - currently it's the case that the externals repo in the image is just the usual ext-v2.1 externals repo but with all unnecessary externals deleted out of it. Instead we should have externals subset=datautilities, etc.

One question that will be persistent as we move forward is, how do we decide which packages should go into the core DUNE DAQ repository, and which should go into the umbrella-package-specific repository? There are tradeoffs involved: the choice I made here means that fddetdataformats and rawdatautils are duplicated between the fddatautilities and fddaq repositories, but on the other hand they don't take up a ton of space and it seems like potential overkill to throw any package which might ever be used by more than one umbrella package into the core DUNE DAQ repository.

Another question: I imagine that we want to treat these new umbrella packages as exactly analogous to fddaq and nddaq, correct? In other words, whenever a new umbrella package like fddatautilities gets added it should be possible to pass it to dbt-setup-release and dbt-create?

(*) Work on this Issue began before dunedaq was renamed coredaq a few days ago

jcfreeman2 pushed a commit that referenced this issue Apr 25, 2024
…ronment variables expected by wf-setup-tools.sh
jcfreeman2 pushed a commit that referenced this issue Apr 25, 2024
…evel between fddaq/nddaq vs. fddatautilities during this Issue's development process
jcfreeman2 pushed a commit that referenced this issue Apr 26, 2024
jcfreeman2 pushed a commit that referenced this issue Apr 26, 2024
jcfreeman2 pushed a commit that referenced this issue Apr 27, 2024
…uccessful fddatautilities build in a container
jcfreeman2 pushed a commit that referenced this issue Apr 29, 2024
jcfreeman2 pushed a commit that referenced this issue Apr 29, 2024
jcfreeman2 pushed a commit that referenced this issue Apr 30, 2024
…ck name is deduced from the nightly tag acronym to support adding a letter to denote a test build (e.g., FDT instead of FD)
jcfreeman2 pushed a commit that referenced this issue Apr 30, 2024
jcfreeman2 pushed a commit that referenced this issue May 1, 2024
…es image has access to the name of the nightly tag
jcfreeman2 pushed a commit that referenced this issue May 1, 2024
…image as apparently it's basically impossible to copy a local absolute directory into an image via a Dockerfile
jcfreeman2 pushed a commit that referenced this issue May 1, 2024
…utilities external image to create the lightest-weight fddatautilities external image possible
jcfreeman2 pushed a commit that referenced this issue May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant