Unable to use debuginfod server to fetch executable/debuginfo/sources for C++ binary crash

[aallrd]

Self-Hosted Version

24.2.0

CPU Architecture

x86_64

Docker Version

20.10.23

Docker Compose Version

2.15.1

Steps to Reproduce

I have a demo C++ program sentry_crash that I am using to try the Sentry native crash reporting feature.
It uses the Sentry Native SDK 0.7.0, and is compiled from the source file sentry_crash.cpp:

#include <cstring>
#include <iostream>

#include <sentry.h>

static void *invalid_mem = (void *)1;
static void trigger_crash() {
    memset((char *)invalid_mem, 1, 100);
}

int main(int argc, char **argv) {
  sentry_options_t *options = sentry_options_new();
  sentry_options_set_dsn(options, "https://1a13fe9bf23f4599bff0a3b097446969@sentry.company.com/3");
  sentry_options_set_database_path(options, ".sentry-native");
  sentry_options_set_debug(options, 0);
  sentry_options_set_release(options, "hello@1.0.0");
  sentry_init(options);
  std::cout << "Hello !\n";
  trigger_crash();
  sentry_close();
  std::cout << "Bye bye !\n";
  return 0;
}

$ g++ -m64 -std=c++20 -g [...] sentry_crash.cpp -o sentry_crash

I have a processing script that creates a debuginfo archive that I upload on our internal debuginfod server at https://debuginfod.company.com:

$ readelf -n sentry_crash | grep -oP 'Build ID: \K.+'
38764668023bff0de3f58ede7686fe91a1a9a79e
$ ./process.sh sentry_crash
Created debuginfo archive: sentry_crash.38764668023bff0de3f58ede7686fe91a1a9a79e.tar.zst
$ zstd -dqc sentry_crash.38764668023bff0de3f58ede7686fe91a1a9a79e.tar.zst | tar -tf -
[...]
./home/jenkins/sentry-demo/sentry_crash
./home/jenkins/sentry-demo/sentry_crash.cpp
./home/jenkins/sentry-demo/sentry_crash.debug

Once this archive is uploaded and processed on the debuginfod server, I can query it successfully using debuginfod-find:

# Fetching the debuginfo
$ debuginfod-find -v debuginfo sentry_crash
debuginfod_find_debuginfo 38764668023bff0de3f58ede7686fe91a1a9a79e
server urls "https://debuginfod.company.com"
checking build-id
checking cache dir /home/jenkins/.cache/debuginfod_client
using timeout 90
init server 0 https://debuginfod.company.com
url 0 https://debuginfod.company.com/buildid/38764668023bff0de3f58ede7686fe91a1a9a79e/debuginfo
query 1 urls in parallel
Progress 1 / 0
committed to url 0
server response No error
got file from server
found /home/jenkins/.cache/debuginfod_client/38764668023bff0de3f58ede7686fe91a1a9a79e/debuginfo (fd=4)
Downloaded from https://debuginfod.company.com/buildid/38764668023bff0de3f58ede7686fe91a1a9a79e/debuginfo
/home/jenkins/.cache/debuginfod_client/38764668023bff0de3f58ede7686fe91a1a9a79e/debuginfo

# Fetching the executable and making sure the hash is the same
$ debuginfod-find executable 38764668023bff0de3f58ede7686fe91a1a9a79e
/home/jenkins/.cache/debuginfod_client/38764668023bff0de3f58ede7686fe91a1a9a79e/executable
$ md5sum sentry_crash /home/jenkins/.cache/debuginfod_client/38764668023bff0de3f58ede7686fe91a1a9a79e/executable
6ea1d85023c26cf81b6a4d239a652abb  sentry_crash
6ea1d85023c26cf81b6a4d239a652abb  /home/jenkins/.cache/debuginfod_client/38764668023bff0de3f58ede7686fe91a1a9a79e/executable

# Fetching source file
$ debuginfod-find source sentry_crash /home/jenkins/sentry-demo/sentry_crash.cpp
/home/jenkins/.cache/debuginfod_client/38764668023bff0de3f58ede7686fe91a1a9a79e/source##home##jenkins##sentry-demo##sentry_crash.cpp

I have configured this debuginfod instance as SymbolServer Repository on my Sentry project:

Expected Result

After the sentry_crash binary is executed, I expect the native crash issue reported in Sentry to be symbolicated and the source context to be available.

Actual Result

It seems that Sentry is not able to pull data from the debuginfod server:

I manually uploaded the sentry_crash binary using the Sentry CLI:

$ sentry-cli dif upload --project demo sentry_crash
> Found 1 debug information file
> Prepared debug information file for upload
> Nothing to upload, all files are on the server

It was still not able to report a symbolicated issue with source context.
In the Debug Files Candidates view I found that Sentry seemed to contact the debuginfod server with the correct build ID but failed with an error: "download failed: destination is restricted":

I disabled SSL on the debuginfod server with the same result.

I tried to serve the files with a plain HTTP server after reading in the doc that "The files need to reside under buildid/ on the server." but it did not change anything:

Manually downloading the file was possible from the server hosting the Sentry stack:

$ curl -X GET --head http://debuginfod.company.com/buildid/38764668023bff0de3f58ede7686fe91a1a9a79e/debuginfo
HTTP/1.1 200 OK
Date: Wed, 20 Mar 2024 15:34:21 GMT
Content-Type: application/octet-stream
Content-Length: 51944

I only managed to get a symbolicated stack with source context after manually uploading the files with the Sentry CLI:

$ sentry-cli dif upload --project demo sentry_crash.debug
> Found 1 debug information file
> Prepared debug information file for upload
> Uploading completed in 0.042s
> Uploaded 1 missing debug information file
> File upload complete:

  UPLOADED 68467638-3b02-0dff-e3f5-8ede7686fe91 (sentry_crash.debug; x86_64 debug companion)

$ sentry-cli dif bundle-sources sentry_crash.debug
/home/jenkins/sentry-demo/sentry_crash.src.zip

$ sentry-cli dif upload --project demo sentry_crash.src.zip
> Found 1 debug information file (1 with embedded sources)
> Prepared debug information file for upload
> Uploading completed in 0.06s
> Uploaded 1 missing debug information file
> File upload complete:

  UPLOADED 68467638-3b02-0dff-e3f5-8ede7686fe91 (sentry_crash.src.zip; x86_64 sources)

Is there anything obvious I missed regarding the debuginfod support by Sentry ?
I followed this documentation: Symbol Servers (debuginfod)
I found later that Symbolicator does not support fetching sources from debuginfod: symbolicator: symbol-server-compatibility
Related issue for Symbolicator: #445

Thank you.

Event ID

No response

[hubertdeng123]

Just reading through, this does not seem like a specific issue that pertains towards self-hosted Sentry. I will transfer this over to symbolicator

[Swatinem]

I believe your DNS resolves to any one of these ranges, which is then resulting in such errors:

symbolicator/crates/symbolicator-service/src/utils/http.rs

Lines 13 to 28 in 63555b8

	// https://en.wikipedia.org/wiki/Reserved_IP_addresses#IPv4
	"0.0.0.0/8",
	"10.0.0.0/8",
	"100.64.0.0/10",
	"127.0.0.0/8",
	"169.254.0.0/16",
	"172.16.0.0/12",
	"192.0.0.0/29",
	"192.0.2.0/24",
	"192.88.99.0/24",
	"192.168.0.0/16",
	"198.18.0.0/15",
	"198.51.100.0/24",
	"224.0.0.0/4",
	"240.0.0.0/4",
	"255.255.255.255/32",

You should be able to set the connect_to_reserved_ips to true in order to circumvent this. @hubertdeng123 can help you find symbolicators config file, as I’m unsure how exactly one can modify that in self-hosted.

I think enabling this setting should be reasonable in self-hosted, as you control the whole infra and events, so there shouldn’t be any possibility of leaking internal requests via things like source context, etc.

[aallrd]

Hello,
Thank you for looking at this issue.
I confirm that our network is configured in one of these ranges.

[aallrd]

I added the configuration in the symbolicator/config.yml file in the Sentry self-hosted source bundle:

echo "connect_to_reserved_ips: true" >> symbolicator/config.yml

After restarting the docker-compose stack the "download failed: destination is restricted" error was gone.

Side questions if I may:

• I had to disable SSL on the debuginfod server since I use self-signed certificates, any configuration on Sentry's side to either provide a certificate bundle / skip the certificate verification ?
• I had to configure the download URL of the debuginfod server has http://debuginfod.company.com/buildid for Sentry to successfully download files, but I expected the URL to simply be http://debuginfod.company.com since it works for the debuginfod-find client, do you know why ?

[Swatinem]

I assume you are using the Sentry-provided Symbolicator Docker image?
So it might not be that straight forward to just add your custom CA to the system certificate store.

Otherwise, we have recently added a per-source configuration that skips certificate validation, like so:
https://github.com/getsentry/sentry/blob/f3b740199ef0b8057c2c68a377fd3b4d42346deb/src/sentry/conf/server.py#L3324-L3330
Though this I believe is not exposed as a global setting (@loewenheim ?), and its also not exposed in the symbol source configuration UI. It was just added recently as a workaround for a "misbehaving" public symbol server.

Similarly, the /buildid/ path has to be part of the url by convention (and for backwards compatibility):
https://github.com/getsentry/sentry/blob/f3b740199ef0b8057c2c68a377fd3b4d42346deb/src/sentry/conf/server.py#L3387-L3394

---

Glad to hear things are working out otherwise, I will close this issue then.

[loewenheim]

@Swatinem Correct, I'm not sure we want to expose this setting to users.