Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support testing timestamping that is applied using an eval during indexing #402

Open
ibilling-splunk opened this issue Sep 13, 2021 · 2 comments

Comments

@ibilling-splunk
Copy link

It seems to us that the framework is not really testing timestamp extraction as the _time test will always pass regardless of the sample and props&transforms.

Our sample:
{"RoleLocation":"South Central US","time":"##Timestamp##"}

Our transform:

INGEST_EVAL = _time=coalesce(strptime(spath(_raw, "time"), "%Y-%m-%dT%H:%M:%S.%QZ"),strptime(spath(_raw, "time"), "%m/%d/%Y %l:%M:%S %p"),_time)

Our test(token.0.replacement was intentionally set to a bad value to prove the test will always pass):

[azure_activity_log.sample]
host_type = plugin

input_type = modinput
index = main
sourcetype = azure:activity
sourcetype_to_search = azure:activity
sample_count = 1
expected_event_count = 1

timestamp_type = event

token.0.token = ##Timestamp##
token.0.replacementType = timestamp
token.0.replacement = a
token.0.field = _time

earliest = -30m
latest = -30m

See picture for Ingested event with correct timestamp(now-30min) specified by test but neither is in sample nor ingest time.
image
No props&transforms were added when testing.
All tests passed

@rfaircloth-splunk 's suggestion is to have a new timestamp_type that:

this would need an option for "none" or "random" so that the even level ts is either not sent or sent invalid to confirm the eval itself works

@ryanfaircloth
Copy link
Contributor

Thanks for the report we will schedule this shortly

@sxiesplunk
Copy link

Another similar case is "when you set the auto_extract_timestamp argument to true in the /event URL", TIME_PREFIX in props will take effect. @rfaircloth-splunk do you think we need another issue for this or will the fix deal with it too?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants