Negative test cases

[tobbee]

As discussed with Daniel Silhavy, dash.js has a need for negative test cases.
One way to produce some negative test cases would be to use the "Chaos Streaming Proxy" (CSP) https://github.com/Eyevinn/chaos-stream-proxy
but it would be nice to have some functionality built in for more accessibility.

The CSP uses a syntax like:

?statusCode=[{rsq:3, code:404}, {req:4, code:409}]&timeout=[{rsq:5, rate:1000000}]

to describe responses to segment requests relative to a live point. This uses a stateful
representation at the server (at least for relative sequence numbers), and is therefore not ideal for a common resource.
For livesim2 to be stateless, one can instead us a cycle that repeats relative to the start time.
A possible URL part could look like

/statusCode_[{cycle:30, rsq: 0, code: 404, rep: video1000}]

cycle = cycle in seconds
rsq = relative sequence number from cycle start, 0 meaning the first segment
code = HTTP response code
rep = comma-separated list of representation IDs, if not present, all representation will be targeted

With this scheme, we could make a segment request result in a 404 Not Found response every 30s.

In addition, we have the livesim1 signalling of two different baseURLs that can switch

 baseurl_u40_d20/baseurl_d40_u20

gives an MPD with two baseURLs which alternate in which one is up and which one in terms of seconds. (u = up, d = down). In this example, the first one is up the first 40s of every minute, while the second one is down the first 40s of every minute.

This could be done as a separate feature.

Other features may be response times, or problems in updating the manifest, but we need a limit, so the suggestion for v 1.0 is to only do the segment response scheme above.

@dsilhavy What is your view?

[Murmur]

Stateless built-in-chaos in LiveSim would be nice, either a simple queryparam syntax or more complex scenario config file in a manifest+segment folder. Config file could be activated with an url param.

Typical use-case is live streaming where player's edgelive timestamp is near realtime, CDN loadbalancer nodes are a bit off-sync to serve the latest segment file.

Second case is a slow network to download larger video segments but small audio segment was still fine but this probably is the player's buffer handler to adjust a resolution ladder and make longer presentation delay.

[dsilhavy]

@tobbee I like the idea and also the syntax of the segment response scheme. This is a very useful feature to test the player robustness.

Another thing is creating gaps by unaligned segments or even segments for which the summed up trun duration does not match the PTS of the next segment. However, this can also be tested with VoD content, I got some testcases for this in our functional test suite. Thats why I consider the 404s more important.

I suggest to go ahead as you proposed and implement the segment response scheme.

[tobbee]

Thanks for the feedback. I'll go ahead and implement the proposal above as part of 1.0.

The opportunity to load more advanced settings from a file and have the file name part of URL is a nice one,
but I think it is more interesting for local servers than for the common one.
A more dynamic approach would be that you could upload some configuration, and that it would expire
after a couple of hours, so that you can keep it for some time, but not forever.
In any case, I'll not implement that now.