Redesign to support other scoring methods

[eldering]

In the past there have been various requests to support other scoring methods than pass/fail with penalty time, some have been listed in scoring

Before we try to support any of these, we should consider what a common (internal) interface/data structures would look like to generically support these. This would include making a trade-off where the more weird scoring methods we'd support, the broader and less well-defined this interface would become. This ticket is to discuss that interface.

We should then also refactor our code so that all scoring related code is hidden behind implementations of this interface. Currently there is plenty of code scattered around the code base.

[eldering]

Some comments/questions for this:

1. I think we should not allow a scoring method to take arbitrary data from the system to determine the score (like: the current CPU load on the server, to name something silly). So what should be the set of allowed input to a score for a single submission? For team team on a problem? For a team in the whole contest?
2. Related, what is the data format for the score for a submission, problem, etc? A float, or can it be any more complex structure like, {"points": 123, "time": 6534.061, "penalty": 20, "foobar": [1, 2, 3]}?

[meisterT]

Some thoughts here:

• We probably want to try to implement the current score in seconds and runtime as a tie breaker using whatever new mechanism we design as a litmus test.
• We need to check what the implications on score cache and rank cache are.
• Partial scoring seems to be then the easiest to add.
• As data format I'm thinking of a list of (named) floats, it should be doable to represent any scoring function with that. Then this list of floats can be used for sorting/ranking teams (that implies it's maximized and you need to negate one of the floats if you want to minimize). In addition, we probably want to add a text representation (e.g. for the scoreboard).
• How do the new scoring methods affect the event feed?