Late submissions restricted to only extraordinary circumstances

[arjunsuresh]

Post-submission extra grace period to avoid late submissions due to submission packaging related activities

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[arjunsuresh]

@DilipSequeira @psyhtest Please review this change

[DilipSequeira]

@arjunsuresh I think the extra grace period is misconceived - it adds complexity with very little benefit, compared to simply extending the grace period to two hours without explanation or penalty.

[arjunsuresh]

@DilipSequeira Completely agree that it adds an extra complexity which is good to avoid. I'm trying to ensure that there won't be any packaging related delays which can warrant a voting in the review meetings -- which have already happened in multiple rounds. If with a 2 hours grace period this is guaranteed to be avoided, we can simply change the 1 hour grace period to 2 hours.

[DilipSequeira]

IMO packaging delays are no different from other kinds of engineering delays and shouldn't be treated specially. It's up to submitters to manage overall submission risk, and we should not incentivize them to optimize up to the deadline and leave packaging to the grace period (and then the review period.)

[arjunsuresh]

@DilipSequeira Though this is only applicable for first time submitters, packaging MLCommons inference results can take a few minutes to about an hour or so and the submitters may not foresee this.

During the review meetings we have seen arguments like all results were collected before the deadline and so the result must be allowed - in a way saying that "packaging delay" was allowed but not some other engineering delay which might have delayed the result generation. Can we have a rule here which can avoid such situations (voting in review meetings due to packaging delays) in future rounds?

[johntran-nv]

I agree with Dilip's comments here. Submitters should try to adhere to the deadline, and not wait until the last minute. We shouldn't incentivize submitters to wait until the last minute, which extending the grace period would do.

@erichan1 , any thoughts?

[erichan1]

I agree with @johntran-nv. In the stated rules I'd prefer that we just have a simple grace period. Whether that's one or two hours is up for debate, although I think one is fine. In the last round of training I don't believe there were any submitters who submitted after the grace period.

[arjunsuresh]

Thank you @johntran-nv and @erichan1 for confirming no issues in Training submissions. We had this issue in inference submissions which went for voting and my PR is intended to ensure that no submitter takes this voting as a precedent and asks for the same in future rounds. I'll remove the extended grace period remark once we have an agreement on what exactly should go there to ensure a smooth submission process.

[tjablin]

I am concerned about the final-most deadline moving later and later into the day. If we add a second post-submission grace period, could we please move the initial submission deadline one hour earlier?

[arjunsuresh]

Sure @tjablin. We'll ensure that the final-most deadline is not shifted from the current one.

[DilipSequeira]

During the review meetings we have seen arguments like all results were collected before the deadline and so the result must be allowed - in a way saying that "packaging delay" was allowed but not some other engineering delay which might have delayed the result generation

Let's please not do "Some people say..." - if members of the working group want to make the argument that they should be able to optimize right up to the deadline and then do packaging afterwards, or that packaging delays are different from other kinds of delays, they should make that argument, and justify the benefits in terms of the costs.

[arjunsuresh]

@DilipSequeira As told earlier this PR is intended only to ensure no such voting scenario come in future review meetings. So either way I propose other submitter (potential voters in review meetings) to give their opinions. On our side we don't really need this "extra delay" to "optimize right up to the deadline" but it is proposed to "save everyone's time" during the review meetings. I'm fine with removing the extra grace period and to change the normal grace period to 2 hours or even keep at current 1 hour. But WG must discuss this and all submitter agree to stick to it from next rounds.

[DilipSequeira]

@arjunsuresh just out of curiosity... how long have you been a WG member? What you are proposing is exactly what happened in 2.0: one submitter was late with their submission, the review was extremely contentious, the WGs debated what to do, and that's how the current rules got written. You seem to be implying that it wasn't discussed, or that not all submitters agreed at the time that that those were the rules. In fact, the submitters who made late submissions this time were all working group members at that time and all had an opportunity to participate in the debate.

Normally, I would be averse to writing "this will be strictly enforced" into the rules, because a submitter inclined to do so might argue that their understanding was that rules that omit this wording will not be strictly enforced. But it's apparent that some submitters were of the opinion that the deadline would not be strictly enforced, so perhaps here being explicit does more harm than good.

[arjunsuresh]

@DilipSequeira yes, I have been attending the WG meetings I believe exactly since the time this rule was written. (But the late submission happened in 1.1 and not 2.0 and there was no late submission for 2.0 probably because of this rule :) ). But as Nvidia might already be aware - WG meetings are not attended by all submitter (we have more attendance for review meetings) and so this rule or the grace period was clearly not known to majority of the submitter- I think none of the submitter for r2.1 used this 1 hour grace period. I had clearly seen the PR for late submission being discussed in WG but even then could not see it in the rules as I was checking the "Inference Policies" repository where as this rule is in "Policies" repository.

So are you implying that we should leave the rule as such and again have a voting in 3.0 or 3.1 round? If for "submitter A" review committee did a voting, isn't it unfair that they don't do it for "submitter B" in next round? That's why I'm proposing a re-discussion possibly both in the WG as well as in the review meeting to get a consensus of all submitter to take a decision on grace period and avoid voting in future rounds.

[arjunsuresh]

We have this in the rules saying that review committee decisions do not create precedents. But I'm not sure if that will matter in next round of submissions.

[tjablin]

The last submission through the web interface was August 5, 2022, 12:52:56 PM. The penultimate submission was eleven seconds earlier.

[DilipSequeira]

@arjunsuresh If we all agree that we need rules, I think we must assume that everyone must read the rules document, and no-one can claim they are due an exemption due to not being aware of the rules. (MLPerf is customarily very generous to new submitters, and I see no reason why we would change that, but this time it was not new submitters who had problems.)

I would fix this by clarifying that 5.14 is a valid form of submission only if there is a defect in the submission checker, that a valid submission tarball must be received by the hard deadline, and that the burden is on the submitter to demonstrate that the tarball in question could not be uploaded via the GUI because the submission checker was broken.

[arjunsuresh]

Thank you @DilipSequeira. I suppose 5.14 should be completely removed to avoid confusion. In the Late Submission section in 5.3.1 we can add your proposal like below.

In case the submitter cannot upload the valid submission tarball by the hard deadline due to a defect in the submission checker, he should send a hash of the valid submission tarball to the chairs before the deadline. The burden is on the submitter to demonstrate that the tarball in question could not be uploaded via the GUI because the submission checker was broken, and the tarball must have the complete results including all required compliance tests and must pass the submission checker once the required fix is done.

"and no-one can claim they are due an exemption due to not being aware of the rules"

This is true. But when a decision goes to voting, most people won't vote for a rule which they themselves are not aware of.

[DilipSequeira]

@arjunsuresh that sounds reasonable. Maybe you could redraft the PR (without gendered pronouns would also be preferable I think.)

[arjunsuresh]

Thanks @DilipSequeira. I have redrafted the PR. Does it sound fine now?

[DilipSequeira]

lgtm, minor wordsmithing suggestions.

[DilipSequeira]

Thanks Arjun

I have no opinion on "a" vs "some" here.

The hash means the chair does not immediately need to download and preserve the tarball.

On reflection, I think the email should be sent to the submitters' mailing list, not the chair, to remove the chair from the critical path to making the submission available for review. It can be sent during the grace period (after submissions are supposed to be submitted, in principle) and thus the submitter is not disadvantaged.

What do you think?

[arjunsuresh]

"Some" is just to be more legally safe as there can be more than "one" possible issue with the submission checker.

I agree with your proposal to send the email to the submitters' mailing list as then any submitter can verify the content of the tarball and removes the burden from the chair.

[rnaidu02]

Arjun to make changes to Pr to provide examples of what are not allowed for this sentence “No other changes to the tarball other than those directly related to the submission checker fix will be allowed.”

[rnaidu02]

@johntran-nv @erichan1 Can you look at the final draft and approve this PR?