Proposal: Guidelines v5.3

[blocksbarcelona]

Hey Guilds, please review the major changes for the V5.3 evaluation criteria and provide some feedback here on github

https://wax-oig.notion.site/v5-3-Office-of-Inspector-General-Guidelines-PROPOSED-0caf1d3ce9924fea8fd4a8e1eb305124?pvs=4

Major Changes in v5.3

• min. AA API version added ([v1.3.21](https://github.com/pinknetworkx/eosio-contract-api/releases/tag/v1.3.21))
• Grace period for incident ⇒ 100/21d == +5% for uptime (generously for timezones and workload involved)
• Delphi Price Oracle uptime requirement clarified to be 95%
• hyperion v3.3.5 as a req.
• Decoupled Tech Evaluation from Product Evaluation
  • tech evaluation every 14 days
  • tech evals done until 17.00 on that day ⇒ day of roundtable
  • no fixed appeal period
  • 2 weeks (14d) rolling period
  • starting with v5.3 guidelines (24th October

---

Minor Changes

• Wording changes for clarity and to adjust to the separation of infrastructure and contribution evaluations
• Replaced the term SLA with SLR, added paragraphs for term definition
• Added the definition of Grace Periods to account for network wide incidents
• CPU Benchmark ⇒ limit defined against upper quartile
• Moved from % to fixed missed rounds for point requirements
• Adjusted the uptime percentages in minimum and secondary requirements to align with the new schedule

Thank you!

[coachbjork]

Question on frequency of API testing

• how often is that checked?

The frequency of checks can have a big impact on the total reported downtime, which might provide an unfair perspective.
e.g. if the API is down during a check, but works again after a few seconds.
While having a short downtime between checks would show as 100% uptime.
How is this handled?

Question on the Uptime and score balancing
The 95% uptime for Oracle price feed, API's etc. Means you can be down for about 16.8 hours per 14 days ((14days x 24h)*0.05).

The 70% on Partial history means a downtime of 100 hours every 2 weeks, and get half the points of someone providing 100% uptime.
That doesn't seem as reasonable as the difference for Full History Uptime, where a good service is 4x the points.
Good service on partial = 2 points
Bad service on partial = 1 point
Good service on Full History = 4 points
Bad service on full = 1 point

The goal should be to provide a better service for the community, yes?

On Full History Vs Partial
With disk faliures, or other crashes, it's rather quick and easy to repair and fix a partial history, while fixing issues with Full History can require a lot of time and work. So the scoring for Partial / Full, and the uptime requirement, doesn't seem to be that balanced for this either.

With Hardware or Datacenter Failiures, downtime for Full History is more expected than for Partial History, even with proper backups.
Should that be reflected in the way points are handled?

Impact of the increased tech evals
With the increased frequency of Tech evals, will the major voters adapt their voting accordingly to any changes?
Or how does that impact guilds that are close to the border?
If they fall below the threshold, and still continue to produce, they keep getting those rewards.
What's the plan for this?

Chain API Criteria
Is there a reason why we don't set a criteria with the new endpoints when this is being specified and updated?
e.g. send_transaction2 and get_transaction_status

[Rossco99]

In regards to the v5.3 OIG Proposed Guidelines, I have the following comments:

Subjective Guild Reviews

It is important to incentivise the correct Guild performance behaviour with SLR’s, however demonstratively effective Guilds shouldn’t be egregiously punished especially for situations out of their control.

For example the minimum requirement of 98% Round and Block availability. Even if the Guild unregisters immediately or fails over to a backup node they may still fail this SLR as the nodeos code takes about 4-5 rounds to make the chain change. This is compounded when the issue occurs a couple times, even though the Guild is very active and trying to remediate.

We believe Guilds should be viewed with a degree of subjectivity by the OIG when it can be argued that a Guild is clearly engaged and remediating an issue out of their control. Especially in regards to an instant minimum requirements failure.

The code is always bleeding edge, things happen and engaged Guilds are constantly trying to improve.

History-API Conditions

Full history is very expensive to run and occationally Hyperion being soo bleeding edge may experience bugs.

We think the first low tier (70 - 95% / 1 point) Full History SLR should be changed to (80 - 90% / 2points)

Backups

We believe the following backup SLR's are sufficent:

• Chain 0.5 1/day 30d ret
• AA 0.5 1/month 3m ret (Maybe a question for Wombat .. what woudl help them?)
• SH 0.5 1/6month 0 ret (Not as important as the blocks.log , but it's 8TB)
• blocks.log 0.5 1/month 0 ret
• ES 1.0 1/week 30d ret (Points for Full Backup)

[kaefergeneral]

I think there is a slight misunderstanding here.
To fail the 98% Round and Block availability minimum requirement as an active producer, you would have to miss a total of 2300 blocks or 192 rounds within a 14 day window.

[Rossco99]

Thanks for correcting me, yes I misunderstood the penalty.

I still do feel that some subjectivity could be used by the OIG especially when the block and round bonus points are for such a narrow margin .. which barely covers a failover.