Exclude datasets on Tape or Disk from CLI query by default

[mmarchegiani]

The DataDiscoveryCLI is saving in the fileset_available.json.gz file also files that are not readable since they are stored on Tape sites.
As an example, this record is stored in the available fileset when querying the dataset /DYJetsToLL_M-50_TuneCP5_13TeV-amcatnloFXFX-pythia8/RunIISummer20UL18NanoAODv9-106X*/NANOAODSIM:

{'root://lyogrid06.in2p3.fr:1094//dpm/in2p3.fr/home/cms/data//store/mc/RunIISummer20UL18NanoAODv9/DYJetsToLL_M-50_TuneCP5_13TeV-amcatnloFXFX-pythia8/NANOAODSIM/106X_upgrade2018_realistic_v16_L1v1-v2/230000/00EA9563-5449-D24E-9566-98AE8E2A61AE.root': {'object_path': 'Events',
  'steps': None,
  'uuid': None}}

which has also the 'steps' field as None.

By direct query to DAS with dasgoclient we can find the site where the file is stored:

$ dasgoclient -query "site file=/store/mc/RunIISummer20UL18NanoAODv9/DYJetsToLL_M-50_TuneCP5_13TeV-amcatnloFXFX-pythia8/NANOAODSIM/106X_upgrade2018_realistic_v16_L1v1-v2/230000/00EA9563-5449-D24E-9566-98AE8E2A61AE.root"

T1_FR_CCIN2P3_Tape
[...]

It would be desirable to exclude the Tape and Disk sites by default, since these files are not readable through xrootd.

At the moment the only way out is to exclude the offending site by blacklist or via regular expression.

FYI @iasonkrom @valsdav

[lgray]

Good catch - thanks!

[ikrommyd]

@valsdav, Could you please take care of this when you get the time?

[andrzejnovak]

Hold up, why is Disk not reachable over xrd? Also this shouldn't happen quietly ever, preferably the user should get an error with a message of how to opt-in into skipping files

[lgray]

Yes - agreed - an error and opt-in would be a much better outcome than silently shifting your endpoints around.

[ikrommyd]

You can select file sites one by one if I remember correctly but there's too many of them to do over whole datasets.
When I created some fileset jsons last week that had Tape sites inside them I would get a TLS error during reading