Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved Symbolizer #4500

Open
brancz opened this issue Apr 9, 2024 · 1 comment
Open

Improved Symbolizer #4500

brancz opened this issue Apr 9, 2024 · 1 comment

Comments

@brancz
Copy link
Member

brancz commented Apr 9, 2024

Parca's memory usage is very unpredictable, mainly because of symbolization.

A memory profile from https://demo.parca.dev/: https://pprof.me/4e9382bfefe0002d4b786fe7524eed4e

Around 80% of the total memory usage can be attributed to symbolization-related things:

  1. metastore: stores all the stacktraces, and tracks the ones that haven't been symbolized yet (using badgerdb)
  2. symbolizer: the different symbolizer implementations (DWARF, gopclntab, symtab) all load entire sections into memory, as well as caches parts of the processing

The metastore additionally causes headaches as it can't just be rebuilt from scratch whenever it's wiped.

I think both of these can be significantly improved by changing our symbolization strategy a bit:

  1. store stack traces in the columnar database (unsymbolized if they are not symbolized at write time)
  2. perform read-time symbolization: instead of performing symbolization for absolutely everything in time asynchronously, only symbolize and cache those things that are actually being queried when they're being queried
  3. improve symbolizer implementations to not require loading the whole section into memory (this will require forking the stdlib DWARF, gopclntab and symtab implementations but shouldn't be too difficult since just all accesses straight to a byte slice just need to be replaced with an io.ReaderAt)

This should remove most of the excessive memory usage.

@zdyj3170101136
Copy link
Contributor

yeah~.

my ingester machine use 32 core 128 GB machine do symbolization for thousands of machine in real time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants