Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cardinality explosion in flow_traffic metrics #94

Open
slrtbtfs opened this issue Mar 22, 2021 · 1 comment · May be fixed by #95
Open

Cardinality explosion in flow_traffic metrics #94

slrtbtfs opened this issue Mar 22, 2021 · 1 comment · May be fixed by #95

Comments

@slrtbtfs
Copy link

Goflows Prometheus metrics flow_traffic_(bytes|packets|summary_size_bytes) include labels for remote_ip and remote_port.

Some services that send flows (e.g. pmacctd) use a new outgoing port each time they send flows to goflow, which results in the creation of several new time series. For a single pmacctd client with almost no network traffic, this creates thousands of time series each day, which, as profiling reveals, consume several hundred MBs of heap memory.

This leads goflow to eventually run out of memory.

One solution to this would be omitting the remote_port label from these metrics.

slrtbtfs added a commit to slrtbtfs/goflow that referenced this issue Mar 22, 2021
Fixes cloudflare#94.

However it theoretically has the potential to break some existing usecases, where the
distiction between remote ports is relevant.

Signed-off-by: Tobias Guggenmos <tobias.guggenmos@uni-ulm.de>
@slrtbtfs slrtbtfs linked a pull request Mar 22, 2021 that will close this issue
@SuperQ
Copy link
Contributor

SuperQ commented Mar 27, 2021

Randomizing on every flow seems a bit broken, IMO. But I agree, the remote_port, which should be an ephemeral port, doesn't seem like a good identifier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants