Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[receiver/splunkhecreceiver] use ACK extension in HEC receiver #31767

Merged
merged 74 commits into from Apr 18, 2024
Merged
Show file tree
Hide file tree
Changes from 71 commits
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
48ad9aa
[receiver/splunkhecreceiver] use ACK extension in HEC receiver
zpzhuSplunk Mar 21, 2024
2a84069
add go mod
zpzhuSplunk Mar 21, 2024
38a8f47
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Mar 21, 2024
9a0c22f
add go mod
zpzhuSplunk Mar 21, 2024
f9d1f83
fix test
zpzhuSplunk Mar 22, 2024
2418764
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Mar 22, 2024
f8de4f9
fix stage failures
zpzhuSplunk Mar 22, 2024
213b3eb
address failures
zpzhuSplunk Mar 25, 2024
aad9a1e
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Mar 25, 2024
5d9cfdb
address test failure
zpzhuSplunk Mar 25, 2024
aaa367b
test
zpzhuSplunk Mar 25, 2024
9bf53c0
test
zpzhuSplunk Mar 25, 2024
9b592fb
test
zpzhuSplunk Mar 25, 2024
8870816
test
zpzhuSplunk Mar 25, 2024
219c82d
test
zpzhuSplunk Mar 25, 2024
1950a55
test
zpzhuSplunk Mar 25, 2024
6dcf281
test
zpzhuSplunk Mar 25, 2024
a6e34cb
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Mar 25, 2024
3edf903
test
zpzhuSplunk Mar 25, 2024
9264f7a
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Mar 25, 2024
71de004
address MR comments
zpzhuSplunk Mar 27, 2024
3a88455
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Mar 27, 2024
59005d9
test
zpzhuSplunk Mar 27, 2024
8c311a1
test
zpzhuSplunk Mar 27, 2024
129d108
test
zpzhuSplunk Mar 27, 2024
686c871
test
zpzhuSplunk Mar 27, 2024
2ddbf68
test
zpzhuSplunk Mar 27, 2024
cf85bb9
fix lint
zpzhuSplunk Mar 27, 2024
81cc6ec
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Mar 27, 2024
e423917
test
zpzhuSplunk Mar 27, 2024
ba51276
test
zpzhuSplunk Mar 27, 2024
e4a3959
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Mar 27, 2024
510673f
test
zpzhuSplunk Mar 27, 2024
9f91b67
test
zpzhuSplunk Mar 27, 2024
e9b9fb3
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Mar 27, 2024
18dbe87
address PR comments
zpzhuSplunk Mar 28, 2024
43be1f3
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Mar 28, 2024
6d0b621
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Apr 2, 2024
e8035b9
address MR comments
zpzhuSplunk Apr 2, 2024
f0e3f20
fix lint
zpzhuSplunk Apr 2, 2024
883b778
make backward compatible
zpzhuSplunk Apr 3, 2024
5749407
address PR comment
zpzhuSplunk Apr 3, 2024
594229b
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Apr 3, 2024
b3571a5
fix CI
zpzhuSplunk Apr 3, 2024
4df1c3f
address PR comments
zpzhuSplunk Apr 10, 2024
8a96d70
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Apr 10, 2024
504626a
rebase
zpzhuSplunk Apr 10, 2024
bd58cdb
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Apr 11, 2024
f350656
test
zpzhuSplunk Apr 11, 2024
ddb578e
test
zpzhuSplunk Apr 11, 2024
634c0d5
test
zpzhuSplunk Apr 11, 2024
48cba1f
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Apr 11, 2024
a6e89cc
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Apr 11, 2024
df7d667
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Apr 15, 2024
5eb0892
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Apr 15, 2024
821f5c3
address PR comments
zpzhuSplunk Apr 15, 2024
ef7f747
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Apr 16, 2024
d37dd11
fix lint
zpzhuSplunk Apr 16, 2024
943dd41
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Apr 17, 2024
8ea06a4
address comments
zpzhuSplunk Apr 17, 2024
c8c27d5
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Apr 17, 2024
dac6cb3
address PR comments pt2
zpzhuSplunk Apr 17, 2024
ca2cae5
Update receiver/splunkhecreceiver/receiver.go
atoulme Apr 17, 2024
1d387da
address PR comments
zpzhuSplunk Apr 17, 2024
0bdab19
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Apr 18, 2024
3621ea1
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Apr 18, 2024
abb1afe
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Apr 18, 2024
2b54dae
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Apr 18, 2024
230fea4
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Apr 18, 2024
e661567
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Apr 18, 2024
d1787eb
Merge branch 'main' into hec_receiver_ack
zpzhuSplunk Apr 18, 2024
4f77fe0
Update receiver/splunkhecreceiver/README.md
zpzhuSplunk Apr 18, 2024
1597db0
Update receiver/splunkhecreceiver/README.md
zpzhuSplunk Apr 18, 2024
5560287
Update receiver/splunkhecreceiver/README.md
zpzhuSplunk Apr 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
27 changes: 27 additions & 0 deletions .chloggen/hec_receiver_ack.yaml
@@ -0,0 +1,27 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: 'enhancement'

# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
component: 'splunkhecreceiver'

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: "adding support for ack in the splunkhecreceiver"

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [26376]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext:

# If your change doesn't affect end users or the exported elements of any package,
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
# Optional: The change log or logs in which this entry should be included.
# e.g. '[user]' or '[user, api]'
# Include 'user' if the change is relevant to end users.
# Include 'api' if there is a change to a library API.
# Default: '[user]'
change_logs: [user]
3 changes: 3 additions & 0 deletions cmd/configschema/go.mod
Expand Up @@ -483,6 +483,7 @@ require (
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/opencensusexporter v0.98.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/opensearchexporter v0.98.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter v0.98.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/extension/ackextension v0.98.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/extension/asapauthextension v0.98.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/extension/awsproxy v0.98.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/extension/basicauthextension v0.98.0 // indirect
Expand Down Expand Up @@ -1150,6 +1151,8 @@ replace github.com/open-telemetry/opentelemetry-collector-contrib/extension/enco

replace github.com/open-telemetry/opentelemetry-collector-contrib/extension/encoding => ../../extension/encoding

replace github.com/open-telemetry/opentelemetry-collector-contrib/extension/ackextension => ../../extension/ackextension

replace github.com/open-telemetry/opentelemetry-collector-contrib/connector/grafanacloudconnector => ../../connector/grafanacloudconnector

replace github.com/open-telemetry/opentelemetry-collector-contrib/extension/sumologicextension => ../../extension/sumologicextension
1 change: 1 addition & 0 deletions cmd/oteltestbedcol/builder-config.yaml
Expand Up @@ -99,3 +99,4 @@ replaces:
- github.com/open-telemetry/opentelemetry-collector-contrib/pkg/pdatautil => ../../pkg/pdatautil
- github.com/openshift/api v3.9.0+incompatible => github.com/openshift/api v0.0.0-20180801171038-322a19404e37
- github.com/outcaste-io/ristretto v0.2.0 => github.com/outcaste-io/ristretto v0.2.1
- github.com/open-telemetry/opentelemetry-collector-contrib/extension/ackextension => ../../extension/ackextension
3 changes: 3 additions & 0 deletions cmd/oteltestbedcol/go.mod
Expand Up @@ -166,6 +166,7 @@ require (
github.com/mostynb/go-grpc-compression v1.2.2 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/extension/ackextension v0.98.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/internal/common v0.98.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal v0.98.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/internal/filter v0.98.0 // indirect
Expand Down Expand Up @@ -391,3 +392,5 @@ replace github.com/open-telemetry/opentelemetry-collector-contrib/pkg/pdatautil
replace github.com/openshift/api v3.9.0+incompatible => github.com/openshift/api v0.0.0-20180801171038-322a19404e37

replace github.com/outcaste-io/ristretto v0.2.0 => github.com/outcaste-io/ristretto v0.2.1

replace github.com/open-telemetry/opentelemetry-collector-contrib/extension/ackextension => ../../extension/ackextension
2 changes: 2 additions & 0 deletions exporter/elasticsearchexporter/integrationtest/go.mod
Expand Up @@ -255,3 +255,5 @@ replace github.com/open-telemetry/opentelemetry-collector-contrib/receiver/sapmr
replace github.com/open-telemetry/opentelemetry-collector-contrib/pkg/resourcetotelemetry => ../../../pkg/resourcetotelemetry

replace github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter => ../../prometheusremotewriteexporter

replace github.com/open-telemetry/opentelemetry-collector-contrib/extension/ackextension => ../../../extension/ackextension
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't figure why this change is there. Is it added by make multimod?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or make crosslink ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it looks like make crosslink is adding this line

3 changes: 3 additions & 0 deletions go.mod
Expand Up @@ -489,6 +489,7 @@ require (
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f // indirect
github.com/nginxinc/nginx-prometheus-exporter v0.11.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/extension/ackextension v0.98.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/extension/observer v0.98.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/extension/sumologicextension v0.98.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/awsutil v0.98.0 // indirect
Expand Down Expand Up @@ -1153,3 +1154,5 @@ replace github.com/open-telemetry/opentelemetry-collector-contrib/internal/sqlqu
replace github.com/open-telemetry/opentelemetry-collector-contrib/extension/encoding => ./extension/encoding

replace github.com/open-telemetry/opentelemetry-collector-contrib/extension/encoding/otlpencodingextension => ./extension/encoding/otlpencodingextension

replace github.com/open-telemetry/opentelemetry-collector-contrib/extension/ackextension => ./extension/ackextension
9 changes: 8 additions & 1 deletion internal/splunk/common.go
Expand Up @@ -23,11 +23,14 @@ const (
DefaultSeverityTextLabel = "otel.log.severity.text"
DefaultSeverityNumberLabel = "otel.log.severity.number"
HECTokenHeader = "Splunk"
HecTokenLabel = "com.splunk.hec.access_token" // #nosec
HTTPSplunkChannelHeader = "X-Splunk-Request-Channel"
dmitryax marked this conversation as resolved.
Show resolved Hide resolved

HecTokenLabel = "com.splunk.hec.access_token" // #nosec
crobert-1 marked this conversation as resolved.
Show resolved Hide resolved
// HecEventMetricType is the type of HEC event. Set to metric, as per https://docs.splunk.com/Documentation/Splunk/8.0.3/Metrics/GetMetricsInOther.
HecEventMetricType = "metric"
DefaultRawPath = "/services/collector/raw"
DefaultHealthPath = "/services/collector/health"
DefaultAckPath = "/services/collector/ack"
)

// AccessTokenPassthroughConfig configures passing through access tokens.
Expand Down Expand Up @@ -112,3 +115,7 @@ type HecToOtelAttrs struct {
// Host indicates the mapping of the host field to a specific unified model attribute.
Host string `mapstructure:"host"`
}

type AckRequest struct {
Acks []uint64 `json:"acks"`
}
6 changes: 6 additions & 0 deletions receiver/splunkhecreceiver/README.md
Expand Up @@ -51,6 +51,10 @@ The following settings are optional:
* `hec_metadata_to_otel_attrs/sourcetype` (default = 'com.splunk.sourcetype'): Specifies the mapping of the sourcetype field to a specific unified model attribute.
* `hec_metadata_to_otel_attrs/index` (default = 'com.splunk.index'): Specifies the mapping of the index field to a specific unified model attribute.
* `hec_metadata_to_otel_attrs/host` (default = 'host.name'): Specifies the mapping of the host field to a specific unified model attribute.
* `ack` (no default): defines the ackextension to use for acking of events
zpzhuSplunk marked this conversation as resolved.
Show resolved Hide resolved
* `extension` (no default): Specifies the ack extension ID the receiver should use. If left blank, ack is turned off.
zpzhuSplunk marked this conversation as resolved.
Show resolved Hide resolved
* `path` (default = '/services/collector/ack'): The path ack extension will listen on for ack requests, if the extension is enabled.
zpzhuSplunk marked this conversation as resolved.
Show resolved Hide resolved

Example:

```yaml
Expand All @@ -67,6 +71,8 @@ receivers:
sourcetype: "mysourcetype"
index: "myindex"
host: "myhost"
ack:
extension: ack/in_memory
```

The full list of settings exposed for this receiver are documented [here](./config.go)
Expand Down
12 changes: 12 additions & 0 deletions receiver/splunkhecreceiver/config.go
Expand Up @@ -4,6 +4,7 @@
package splunkhecreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/splunkhecreceiver"

import (
"go.opentelemetry.io/collector/component"
"go.opentelemetry.io/collector/config/confighttp"

"github.com/open-telemetry/opentelemetry-collector-contrib/internal/splunk"
Expand All @@ -21,6 +22,9 @@ type Config struct {
confighttp.ServerConfig `mapstructure:",squash"` // squash ensures fields are correctly decoded in embedded struct

splunk.AccessTokenPassthroughConfig `mapstructure:",squash"`

Ack `mapstructure:"ack"`

// RawPath for raw data collection, default is '/services/collector/raw'
RawPath string `mapstructure:"raw_path"`
// Splitting defines the splitting strategy used by the receiver when ingesting raw events. Can be set to "line" or "none". Default is "line".
Expand All @@ -30,3 +34,11 @@ type Config struct {
// HecToOtelAttrs creates a mapping from HEC metadata to attributes.
HecToOtelAttrs splunk.HecToOtelAttrs `mapstructure:"hec_metadata_to_otel_attrs"`
}

// Ack defines configuration for the ACK functionality of the HEC receiver
type Ack struct {
// Extension defines the extension to use for acking of events. Without specifying an extension, the ACK endpoint won't be exposed
Extension *component.ID `mapstructure:"extension"`
// Path for Ack API, default is '/services/collector/ack'. Ignored if Extension is not provided.
Path string `mapstructure:"path"`
}
2 changes: 2 additions & 0 deletions receiver/splunkhecreceiver/config_test.go
Expand Up @@ -44,6 +44,7 @@ func TestLoadConfig(t *testing.T) {
RawPath: "/foo",
Splitting: SplittingStrategyLine,
HealthPath: "/bar",
Ack: Ack{Path: "/services/collector/ack"},
HecToOtelAttrs: splunk.HecToOtelAttrs{
Source: "file.name",
SourceType: "foobar",
Expand All @@ -70,6 +71,7 @@ func TestLoadConfig(t *testing.T) {
RawPath: "/services/collector/raw",
Splitting: SplittingStrategyLine,
HealthPath: "/services/collector/health",
Ack: Ack{Path: "/services/collector/ack"},
HecToOtelAttrs: splunk.HecToOtelAttrs{
Source: "com.splunk.source",
SourceType: "com.splunk.sourcetype",
Expand Down
6 changes: 5 additions & 1 deletion receiver/splunkhecreceiver/factory.go
Expand Up @@ -49,7 +49,11 @@ func createDefaultConfig() component.Config {
},
RawPath: splunk.DefaultRawPath,
HealthPath: splunk.DefaultHealthPath,
Splitting: SplittingStrategyLine,
Ack: Ack{
Extension: nil,
Path: splunk.DefaultAckPath,
},
Splitting: SplittingStrategyLine,
}
}

Expand Down
6 changes: 5 additions & 1 deletion receiver/splunkhecreceiver/go.mod
Expand Up @@ -3,9 +3,11 @@ module github.com/open-telemetry/opentelemetry-collector-contrib/receiver/splunk
go 1.21

require (
github.com/google/uuid v1.6.0
github.com/gorilla/mux v1.8.1
github.com/json-iterator/go v1.1.12
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter v0.98.0
github.com/open-telemetry/opentelemetry-collector-contrib/extension/ackextension v0.98.0
github.com/open-telemetry/opentelemetry-collector-contrib/internal/common v0.98.0
github.com/open-telemetry/opentelemetry-collector-contrib/internal/sharedcomponent v0.98.0
github.com/open-telemetry/opentelemetry-collector-contrib/internal/splunk v0.98.0
Expand Down Expand Up @@ -38,8 +40,8 @@ require (
github.com/go-viper/mapstructure/v2 v2.0.0-alpha.1 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/hashicorp/go-version v1.6.0 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/klauspost/compress v1.17.8 // indirect
github.com/knadh/koanf/maps v0.1.1 // indirect
github.com/knadh/koanf/providers/confmap v0.1.0 // indirect
Expand Down Expand Up @@ -106,3 +108,5 @@ retract (
replace github.com/open-telemetry/opentelemetry-collector-contrib/internal/sharedcomponent => ../../internal/sharedcomponent

replace github.com/open-telemetry/opentelemetry-collector-contrib/pkg/golden => ../../pkg/golden

replace github.com/open-telemetry/opentelemetry-collector-contrib/extension/ackextension => ../../extension/ackextension
2 changes: 2 additions & 0 deletions receiver/splunkhecreceiver/go.sum

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.