Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CORE-2256] Fix hint return and display #9968

Merged
merged 46 commits into from
Apr 30, 2024
Merged
Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
99d9efd
Fix hint return and display
robert-uhl Apr 22, 2024
54cca02
Don’t double-print from/to
robert-uhl Apr 22, 2024
a2a50c6
Use verbatim timestamp
robert-uhl Apr 22, 2024
837b1a9
Add offset to time range filter
robert-uhl Apr 23, 2024
d50282e
Merge branch 'master' into rau/core-2256-debug-improve-hints
robert-uhl Apr 23, 2024
d18537e
Attempt to avoid off-by-one error in pages
robert-uhl Apr 23, 2024
d833bba
Merge branch 'rau/core-2256-debug-improve-hints' of github.com:pachyd…
robert-uhl Apr 23, 2024
53824dd
Correct command hint
robert-uhl Apr 23, 2024
1c061f6
Merge remote-tracking branch 'origin/master' into rau/core-2256-debug…
robert-uhl Apr 23, 2024
fa6a0d1
Add first time range test case
robert-uhl Apr 24, 2024
f42ba09
Add test for expected log
robert-uhl Apr 24, 2024
97d1f86
Check for presence of hint
robert-uhl Apr 24, 2024
a4090c9
Check for expected older & newer hints
robert-uhl Apr 24, 2024
0061e89
Fix test
robert-uhl Apr 24, 2024
4f76888
Remove debug line and improve defaulting
robert-uhl Apr 24, 2024
911fb05
Add options when creating a testing Loki instance
robert-uhl Apr 24, 2024
a7d554e
Add a test for getting logs with no from or until
robert-uhl Apr 24, 2024
8da2348
Merge remote-tracking branch 'origin/master' into rau/core-2256-debug…
robert-uhl Apr 24, 2024
29ec4d0
Pre-merge commit
robert-uhl Apr 25, 2024
e3efe31
Merge remote-tracking branch 'origin/master' into rau/core-2256-debug…
robert-uhl Apr 25, 2024
3f2e8b0
Support limit and hint
robert-uhl Apr 25, 2024
d1e07ad
Remove unused errPublisher
robert-uhl Apr 25, 2024
7f3ff9f
Properly handle offset and limit
robert-uhl Apr 25, 2024
273670e
Add another offset-related test
robert-uhl Apr 25, 2024
bdb2659
Merge remote-tracking branch 'origin/master' into rau/core-2256-debug…
robert-uhl Apr 25, 2024
5381ad4
Test that older hint also work
robert-uhl Apr 25, 2024
0360588
Remove commented-out lines
robert-uhl Apr 25, 2024
9cdeb85
Add some backwards-running tests
robert-uhl Apr 26, 2024
d4c7c67
Merge remote-tracking branch 'origin/master' into rau/core-2256-debug…
robert-uhl Apr 26, 2024
1456559
Merge remote-tracking branch 'origin/master' into rau/core-2256-debug…
robert-uhl Apr 26, 2024
99d3cf0
Correct test & results
robert-uhl Apr 26, 2024
0737712
Correct test
robert-uhl Apr 26, 2024
d219581
Merge branch 'master' into rau/core-2256-debug-improve-hints
robert-uhl Apr 26, 2024
95f0c3f
Add debug logs for console team
robert-uhl Apr 29, 2024
b85e7e2
Merge branch 'rau/core-2256-debug-improve-hints' of github.com:pachyd…
robert-uhl Apr 29, 2024
8654f30
Another debugging log for console
robert-uhl Apr 29, 2024
5fa4568
Another debugging log
robert-uhl Apr 29, 2024
1301174
More debugging
robert-uhl Apr 29, 2024
ee257a2
Try a fix
robert-uhl Apr 29, 2024
0754524
|| ≠ &&
robert-uhl Apr 29, 2024
db728cd
Use getter
robert-uhl Apr 29, 2024
66bb4a4
Fix tests
robert-uhl Apr 29, 2024
c9dafc8
Remove missing test
robert-uhl Apr 29, 2024
bd67878
Merge branch 'master' of github.com:pachyderm/pachyderm into rau/core…
robert-uhl Apr 30, 2024
e7cb564
Merge branch 'master' into rau/core-2256-debug-improve-hints
robert-uhl Apr 30, 2024
d32ab66
Merge branch 'master' into rau/core-2256-debug-improve-hints
robert-uhl Apr 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
12 changes: 12 additions & 0 deletions proto-docs.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions proto-docs.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions python-sdk/pachyderm_sdk/api/logs/__init__.py

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions src/internal/jsonschema/logs/GetLogsRequest.schema.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions src/internal/jsonschema/logs/GetLogsResponse.schema.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions src/internal/jsonschema/logs/LogFilter.schema.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions src/internal/jsonschema/logs/PagingHint.schema.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions src/internal/jsonschema/logs/TimeRangeLogFilter.schema.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

21 changes: 18 additions & 3 deletions src/internal/lokiutil/testloki/testloki.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,13 +32,25 @@ type TestLoki struct {
lokiErrCh <-chan error
}

type Option func(config *map[string]any)

func WithoutOldSampleRejection(config *(map[string]any)) {
(*config)["limits_config"].(map[string]any)["reject_old_samples"] = false
}

func WithCreationGracePeriod(d time.Duration) func(config *(map[string]any)) {
return func(config *(map[string]any)) {
(*config)["limits_config"].(map[string]any)["creation_grace_period"] = d
}
}

robert-uhl marked this conversation as resolved.
Show resolved Hide resolved
// New starts a new Loki instance on the local machine.
func New(ctx context.Context, tmp string) (*TestLoki, error) {
func New(ctx context.Context, tmp string, opts ...Option) (*TestLoki, error) {
var errs error
attempts := 5
for i := 0; i < attempts; i++ {
log.Debug(ctx, "attempting to start loki", log.RetryAttempt(i, attempts))
l, err := buildAndStart(ctx, tmp)
l, err := buildAndStart(ctx, tmp, opts...)
if err != nil {
errors.JoinInto(&errs, errors.Wrapf(err, "startup attempt %d", i))
continue
Expand All @@ -50,7 +62,7 @@ func New(ctx context.Context, tmp string) (*TestLoki, error) {
return nil, errors.Wrapf(errs, "loki failed to start after %d attempts", attempts)
}

func buildAndStart(ctx context.Context, tmp string) (*TestLoki, error) {
func buildAndStart(ctx context.Context, tmp string, opts ...Option) (*TestLoki, error) {
bin, ok := bazel.FindBinary("//tools/loki", "loki")
if !ok {
log.Debug(ctx, "can't find //tools/loki via bazel, using loki in $PATH")
Expand Down Expand Up @@ -106,6 +118,9 @@ func buildAndStart(ctx context.Context, tmp string) (*TestLoki, error) {
config["server"].(map[string]any)["http_listen_port"] = port
config["server"].(map[string]any)["grpc_listen_port"] = port + 1
config["schema_config"].(map[string]any)["configs"].([]any)[0].(map[string]any)["from"] = "2020-10-24"
for _, opt := range opts {
opt(&config)
}

configBytes, err := yaml.Marshal(config)
if err != nil {
Expand Down
146 changes: 78 additions & 68 deletions src/logs/logs.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions src/logs/logs.pb.validate.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions src/logs/logs.pb.zap.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.