You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There is enough mem and CPU on the machine where the prometheus is running.
This is the stats from prometheus
Number of Series | Number of Chunks | Number of Label Pairs | Current Min Time | Current Max Time
-- | -- | -- | -- | --
39772186 | 170739354 | 712874 | 2024-04-11T07:00:00.000Z (1712818800000) | 2024-04-29T15:24:44.917Z (1714404284917)
I am unable to find a right configuration for Prometheus to stop it from growing WAL forever. When I deleted the WAL folder it had files as old 20 days.
Is there any confiuration that I have wrong?
The current WAL size in 4 hours is 140GB.
Upgrade is planned, but would it be the solution for the problem here?
Thanks
What did you expect to see?
The WAL is only retained for 2 hours or so.
What did you see instead? Under which circumstances?
The WAL folder not cleaned up.
System information
Linux 6.1.58+ x86_64
Prometheus version
Build Information
Version 2.46.0
Revision cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0
Branch HEAD
BuildUser root@42454fc0f41e
BuildDate 20230725-12:31:24
GoVersion go1.20.6
Prometheus configuration file
No response
Alertmanager version
No response
Alertmanager configuration file
No response
Logs
No response
The text was updated successfully, but these errors were encountered:
What did you do?
I have prometheus version 2.46 installed with the follwoing configuration..
I deleted the WAL folder as it took more than 7 hours for a replay to be compelted
ts=2024-04-26T11:40:10.556Z caller=head.go:792 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=4m27.932583409s wal_replay_duration=7h15m17.527240298s wbl_replay_duration=205ns total_replay_duration=7h23m9.09340072s
There is enough mem and CPU on the machine where the prometheus is running.
This is the stats from prometheus
I am unable to find a right configuration for Prometheus to stop it from growing WAL forever. When I deleted the WAL folder it had files as old 20 days.
Is there any confiuration that I have wrong?
The current WAL size in 4 hours is 140GB.
Upgrade is planned, but would it be the solution for the problem here?
Thanks
What did you expect to see?
The WAL is only retained for 2 hours or so.
What did you see instead? Under which circumstances?
The WAL folder not cleaned up.
System information
Linux 6.1.58+ x86_64
Prometheus version
Prometheus configuration file
No response
Alertmanager version
No response
Alertmanager configuration file
No response
Logs
No response
The text was updated successfully, but these errors were encountered: