Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Start discussing how to make the creation of the JSON metadata file easier/faster #25

Open
bieryAtFnal opened this issue Jun 14, 2022 · 0 comments
Labels
feature New code or additional features

Comments

@bieryAtFnal
Copy link
Collaborator

It has been noticed that the creation of the JSON metadata files on np04-srv-001 and -002 often takes a non-trivial amount of time per file.
This is because we need to open up the raw data file to fetch information that we put in the JSON metadata file. The worst case is building up the list of TriggerRecord numbers in the file (~5 seconds when there are no "write" operations going on), but even just opening the file to read an attribute can take a second or two if the file is not cached.

One option is to talk with our offline colleagues and see if they really need all of the JSON metadata elements that we are now providing, and if now, drop them.

Another option might be to have the HDF5-file-writing code create a small file that accompanies each raw data file, and that companion file has the information that is needed by the JSON metadata file creator and whatever else may want to know about the contents of the raw data file.

This latter option isn't a totally new idea. We have talked about companion files in the past that tell us the full list of files that contain a particular trigger, when a trigger spans multiple files, and what I'm suggestion above might be a natural part of that.

@bieryAtFnal bieryAtFnal added the feature New code or additional features label Jun 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New code or additional features
Projects
None yet
Development

No branches or pull requests

1 participant