unit-state.db grows too large on some deployments #736
Comments
mustafakemalgilor
added a commit
to mustafakemalgilor/charm-helpers
that referenced
this issue
Sep 21, 2022
The unitdata code keeps environment variable revisions in kv_revisions table, and all of the environment variables serialized into JSON and pushed into the table on each hook invocation. There is a check to see if current env variable set is same with the previous one, but some of the environment variables (e.g. JUJU_CONTEXT_ID) is always changing, which means whole env variable list is serialized and pushed to the DB on every single hook invocation. That has become a problem in deployments that has very large (e.g. ~70 KiB) environment variable lists. Given that update-status is running periodically (every 5 mins by default) and having many charms in the same environment, the disk space of the host running the charms runs out over time and needs manual intervention to clear up kv_revisions table. This fix aims to avoid that by keeping only the delta of environment variable list as a revision. Note that this is a breaking change and will break any charms that are relying on `kv_revisions` env values Closes-Bug: LP#1930173
mustafakemalgilor
added a commit
to mustafakemalgilor/charm-helpers
that referenced
this issue
Sep 21, 2022
The unitdata code keeps environment variable revisions in kv_revisions table, and all of the environment variables serialized into JSON and pushed into the table on each hook invocation. There is a check to see if current env variable set is same with the previous one, but some of the environment variables (e.g. JUJU_CONTEXT_ID) is always changing, which means whole env variable list is serialized and pushed to the DB on every single hook invocation. That has become a problem in deployments that has very large (e.g. ~70 KiB) environment variable lists. Given that update-status is running periodically (every 5 mins by default) and having many charms in the same environment, the disk space of the host running the charms runs out over time and needs manual intervention to clear up kv_revisions table. This fix aims to avoid that by keeping only the delta of environment variable list as a revision. Note that this is a breaking change and will break any charms that are relying on `env` value's layout on returned from `gethistory()` function. Closes-Bug: LP#1930173
mustafakemalgilor
added a commit
to mustafakemalgilor/charm-helpers
that referenced
this issue
Sep 21, 2022
The unitdata code keeps environment variable revisions in kv_revisions table, and all of the environment variables serialized into JSON and pushed into the table on each hook invocation. There is a check to see if current env variable set is same with the previous one, but some of the environment variables (e.g. JUJU_CONTEXT_ID) is always changing, which means whole env variable list is serialized and pushed to the DB on every single hook invocation. That has become a problem in deployments that has very large (e.g. ~70 KiB) environment variable lists. Given that update-status is running periodically (every 5 mins by default) and having many charms in the same environment, the disk space of the host running the charms runs out over time and needs manual intervention to clear up kv_revisions table. This fix aims to avoid that by keeping only the delta of environment variable list as a revision. Note that this is a breaking change and will break any charms that are relying on `env` value's layout on returned from `gethistory()` function. Closes-Bug: LP#1930173
mustafakemalgilor
added a commit
to mustafakemalgilor/charm-helpers
that referenced
this issue
Sep 21, 2022
The unitdata code keeps variable revisions in kv_revisions table, and as a part of that all of the environment variables serialized into JSON and pushed into the table if current set of environment variables differ from the previous ones. But the code considers environment variable list as a whole and even if a single environment variable changes, whole list gets pushed as a revision. Some of the environment variables (e.g. JUJU_CONTEXT_ID) is always changing, which means whole env variable list is serialized and pushed to the DB on every single hook invocation. This has become a problem in deployments with very large (e.g. ~70 KiB) environment variable lists. Given that update-status is running periodically (every 5 mins by default) and many charms running in the same environment, the disk space of the host running the charms runs out over time, raising a need of manual intervention to clear up the kv_revisions table. This fix aims to avoid that by keeping only the delta of environment variable list as a revision. As a result, only the changed environment variables will be logged into the kv_revisions table. Note that this is a breaking change and will break any charms that are relying on `env` value's layout on returned from `gethistory()` function. Closes-Bug: LP#1930173
|
This is an interesting bug report, and I didn't even realise that the unit db kept this data. I wonder why? I can understand keeping data for the main hooks, but update-status seems like the odd one out. I wonder if the solution is to not (by default) log env for update-status? |
That would also work, but the bug will resurface if other hooks are called often enough. I have no idea why this data is being kept or whether changing the current behavior would break any downstream charms or not. I tried to locate any charms that rely on It would be great if anybody familiar with this particular part of the code base chime in and enlightens us about the rationale and use cases. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The size of
unit-state.dbis growing too large (>=10GiB) on some deployments, causing out-of-space issues.I've prepared a script to generate a summary of the database, which can be found in the attachments. The script summarizes the
kv_revisionsandhookstables and runssqlite3_analyzerover the database.py-sqlite-analyzer.zip
I ran it over a large DB (13GiB), the output is as follows:
To summarize the output above; the
kv_revisionstable is taking %99.8 of the space, and theenvvariable revisions account for nearly all the rows in thekv_revisionstable. In this specific environment, the JSON-serialized list of environment variables is nearly ~70KiB in size, and on each hook invocation, this 70 KiB of data is being pushed into the kv_revisions table. So even if the charm is standing idle, theupdate-statushook will be invoked every 5 minutes, and this solely will produce 7 GiB of data in a year, and this is for asinglecharm. This is obviously bad and causes out-of-disk-space issues for some deployments.I've reviewed the code that produces the revisions, and it is not blindly pushing all environment variables to the database. It checks whether the value has changed before pushing it as a revision. But in the environment variables scenario, some variables like
JUJU_CONTEXT_IDconstantly change on each hook invocation. So,envgets pushed as a revision no matter what.To eliminate this issue, I have several ideas at hand:
1-) Implement a policy to keep last N revisions only
2-) Limit hooks & kv_revisions table max row count to N rows at most
3-) Exclude update-status from hooks & kv_revisions
4-) Rotate the database periodically (e.g. daily/weekly/monthly), compress & store the old ones
5-) Store only the delta for env
6-) Do not keep revisions for env at all?
7-) ... any other ideas?
Thanks in advance.
The text was updated successfully, but these errors were encountered: