Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem to configure sampling override #3511

Open
pasomaal opened this issue Jan 23, 2024 · 6 comments
Open

Problem to configure sampling override #3511

pasomaal opened this issue Jan 23, 2024 · 6 comments
Assignees

Comments

@pasomaal
Copy link

Is your feature request related to a problem? Please describe.
How do I configure sampling to e.g. 30% and keep all underlying process exceptions logged.
I have a running process where requests are logged and sampled at 30%.
A background process is generating exceptions and these seem also to be sampled at 30%.
How do I keep all exceptions logged?

Describe the solution you would like
Keep all exceptions logged using a sampling override.

Additional context
I tried the below configuration, but doesn't seem to work.

{
  "connectionString": "someconnectionstring",
  "role": {
    "name": "SomeRoleName"
  },
  "instrumentation": {
	"logging": {
	  "level": "WARN"
	}
  },
  "sampling": {
	"percentage": 30
  },
  "preview": {
	"sampling": {
	  "overrides": [
	    {
		  "telemetryType": "exception",
		  "attributes": [
		    {
			  "key": "exception.type",
			  "value": ".*",
			  "matchType": "regexp"
			}
		  ],
		  "percentage": 100
		}
	  ]
	}
  }
}
```
@heyams
Copy link
Contributor

heyams commented Jan 23, 2024

@pasomaal it should work if there is an attribute called "exception.type".

can you double check log at the debug level?
image

@heyams heyams self-assigned this Jan 23, 2024
@heyams
Copy link
Contributor

heyams commented Jan 23, 2024

@pasomaal can you try this instead:

{
  "connectionString": "someconnectionstring",
  "role": {
    "name": "SomeRoleName"
  },
  "instrumentation": {
	"logging": {
	  "level": "WARN"
	}
  },
  "sampling": {
	"percentage": 30
  },
  "preview": {
	"sampling": {
	  "overrides": [
	    {
		  "telemetryType": "exception",
		  "percentage": 100 // **this will keep Log exception to 100%. pls note that span exception is not supported because exception is present as a span event.**
		}
	  ]
	}
  }
}

please let me know if works for you.

@pasomaal
Copy link
Author

pasomaal commented Jan 30, 2024

@heyams

Thanks for your quick answer.
In one application where the exception is thrown by a background task, I'm not able to override of type telemetryType "exception" without the attributes section as in your last answer. The same configuration seems to work in another application where the exceptions occur within a HTTP call.

All exceptions are logged in the second configured application.
There are 2 questions that I try to accomplish without success:

  1. Scenario 1: sample at 30% and have complete context logged for failing operations.
    Is it possible to set the sampling e.g. to 30% and have the complete context (request, dependencies, traces and exceptions) logged for failing operations?
  2. Scenario 2: remove SQL query statements in dependencies only for successful operations.
    Is it possible to configure SQL dependencies only having the SQL queries logged on failing operations.

Kind Regards,

Patrick

@heyams
Copy link
Contributor

heyams commented Feb 7, 2024

@pasomaal
sorry for the delay.

  1. scenario 1: it's not possible currently using our java agent. it requires tail-based sampling. However, we use head-based sampling, and it doesn't know whether a request is successful or failed at the beginning of the span.
  2. scenario 2 is possible using OpenTelemetry Java Extension. We're going to release 3.5 GA, which will support Extension. When it becomes available, i can come up with a sample app and demonstrate how it is implemented using the extension.

Hope this helps. Thank you for your patience.

@pasomaal
Copy link
Author

pasomaal commented Feb 20, 2024

Helen,

Thanks for your answer!

  1. Scenario 1: any plans to add tail-base sampling?
  2. Scenario 2: hope to see 3.5 GA soon

Kind Regards,

Patrick

@heyams
Copy link
Contributor

heyams commented Feb 29, 2024

@pasomaal 3.5.0 is live today.
I have passed on your # 1 to our PM. No confirmation yet. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants