Inference Submission: name of <system_desc_id>_<implementation_id>_<scenario>.json

[bitfort]

The rules look for a file of: "<system_desc_id><implementation_id>.json" then submission checker looks for a file of "<system_desc_id>_<implementation_id>.json":

impl = system_file[len(system_desc) + 1:-5]

[psyhtest]

The rules actually mention <system_desc_id>_<implementation_id>_<scenario>.json in the title, as <system_desc_id>_<implementation_id>.json in the text of that section. It seems that _<scenario> is superfluous here?

Furthermore, no examples are provided ([ TODO David Kanter to add ]). But surely "Starting weights filename?" is benchmark-specific? Should <implementation_id be actually benchmark_id? Or are we missing a whole level here?

[guschmue]

fixed with this PR: mlcommons/inference#496

We now allow what the doc wants <system_desc_id>_<implementation_id>_<scenario>.json but we also allow <system_desc_id>_<implementation_id>.json

[psyhtest]

Thanks @guschmue. I'm still not clear what implementation_id is. For the code/ subdir, I had to set it to task+'_'+scenario, where task is image-classification or object-detection, to make the checker happy, but I'm not sure if it's right.

[guschmue]

this says
code/benchmark/implementation_id
and in my case I used reference as implementation_id. It is undefined what the structure below that. I did not want to have the full inference tree there so I created a README that says git clone https://github.com/mlperf/inference

[psyhtest]

Thanks @guschmue, I think I get it now.

Do you have structure like this then:

code/
  mobilenet/
    reference/
      README.md
  resnet/
    reference/
      README.md
  ...

where each README.md is identical? It's even worse in my case where for one implementation (image-classification-tflite) I have dozens of models (mobilenet*) and hence duplicate leaf README.mds.

It seems it would be better to swap the levels:

code/
  reference/
    mobilenet/
      README.md
    resnet/
      README.md

[psyhtest]

(although this doesn't solve the problem of duplicate README.mds...)

[guschmue]

Yes, I agree. That also would enable us to use code/implementation_id for everything as default that does not explicitly list a model. v0.7 I guess @petermattson .