Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support new IDAaaS data cache index files #37013

Closed
4 of 5 tasks
thomashampson opened this issue Mar 13, 2024 · 1 comment · Fixed by #37187
Closed
4 of 5 tasks

Support new IDAaaS data cache index files #37013

thomashampson opened this issue Mar 13, 2024 · 1 comment · Fixed by #37187
Assignees
Labels
IDAaaS Only Only on IDAaaS ISIS Team: Core Issue and pull requests managed by the Core subteam at ISIS
Milestone

Comments

@thomashampson
Copy link
Contributor

thomashampson commented Mar 13, 2024

Is your feature request related to a problem? Please describe.
When using the ISIS experiment data archive, there is a quick way of looking up files for a given instrument name and run number (see here).
ISIS users (who are not staff) do not have access to the data archive, and instead must rely on the instrument data cache. Currently there is a workaround where each of the experiment directories associated with a user is automatically added to their list of mantid user directories, but for some people with access to many experiments this results in a very slow search.

Describe the solution you'd like
The IDAaaS team will be adding a new index file named INSTRUMENT_index.txt to the top level of each instrument folder, e.g.

/data/instrument/ALF/ALF_index.txt
/data/instrument/ARGUS/ARGUS_index.txt
/data/instrument/CHRONUS/CHRONUS_index.txt

The file is a text file with two columns: run number and relative path to the data location, e.g.

ALF_index.txt
89123 2024/RB2220540-3
89124 2024/RB2220540-3
89125 2024/RB2220540-3
…
89138 2024/unknown/RB2030018
89139 2024/unknown/RB2030018
…
…

The run numbers in a given file will be unique and in ascending order.

Our task is to make use of these new index files to efficiently find data files based on the instrument name and run number in an analogous way to the archive search service.

We should consider the precedence of where to search first - archive, local files, data cache?

We may want the user to be able to override the default precedence.

Additional context
The original issue created for discussion is here and may provide more context:
#36923

List of things to do

  • Read a given JSON file and return the file path for a given run number
  • Handle the case where the file to load contains instrument name, e.g. MAR11060
  • Handle the case where there is no instrument name provided, use the default instrument
  • Consider precedence of where to look for files first (local files, archive, instrument data cache)
  • UI changes - more options in "manage user directories"?
@thomashampson thomashampson added ISIS Team: Core Issue and pull requests managed by the Core subteam at ISIS IDAaaS Only Only on IDAaaS labels Mar 13, 2024
@thomashampson thomashampson added this to the Release 6.10 milestone Mar 13, 2024
@GuiMacielPereira GuiMacielPereira linked a pull request Apr 19, 2024 that will close this issue
Copy link

Closed by #37187.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
IDAaaS Only Only on IDAaaS ISIS Team: Core Issue and pull requests managed by the Core subteam at ISIS
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants