[CORE-2256] Fix hint return and display

[robert-uhl]

Reworked paging and hints. Added a plethora of tests against a testing Loki instance. Worked with @pleeko to test expected vs. actual results.

[codecov]

Codecov Report

Attention: Patch coverage is 75.34247% with 18 lines in your changes are missing coverage. Please review.

Project coverage is 58.30%. Comparing base (3599156) to head (d32ab66).
Report is 2 commits behind head on master.

Files	Patch %	Lines
src/server/logs/cmds/cmds.go	16.66%	10 Missing ⚠️
src/server/logs/logs.go	82.14%	5 Missing ⚠️
src/server/logs/query.go	86.95%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9968      +/-   ##
==========================================
+ Coverage   58.28%   58.30%   +0.02%     
==========================================
  Files         614      614              
  Lines       75593    75610      +17     
==========================================
+ Hits        44060    44088      +28     
+ Misses      30980    30964      -16     
- Partials      553      558       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jrockway]

As explained on Slack, I don't think we need this feature. The chances of having a lot of lines with the same timestamp is pretty much zero, and even then, you still can't get duplicate-timestamped lines that exceed the loki server's limit. I'd just let it be.

You should keep the changes that let you ingest old samples. I'd expect some existing tests to be failing if they can't, though.

[pleeko]

TY for implementing these fixes!

[robert-uhl]

The chances of having a lot of lines with the same timestamp is pretty much zero, and even then, you still can't get duplicate-timestamped lines that exceed the loki server's limit.

With just promtail, you’re correct that the odds of it happening by chance are zero or close to it, but anyone in the cluster is able to write logs with any timestamp (within the bounds Loki permits). I think that it’s necessary to be at least aware enough of the possibility to not crash or loop infinitely when encountering runs of logs with the same timestamp, and the complexity involved even in handling that is enough that it gets one pretty close to just handling duplicate timestamps generally.

even then, you still can't get duplicate-timestamped lines that exceed the loki server's limit

This is also true; it’s a limitation of Loki which cannot be worked around (I’m pretty sure).

I share your concern regarding the amount of complexity this adds, but I think that it’s necessary, and hope that the extensive test coverage provides some level of protection from implementation bugs. It’d be great if Loki guaranteed unique timestamps, or if we didn’t rely on it, but I believe that this is the least bad solution.