Improve the performance of the postgres insertion. #4149
Conversation
- The first version commited 7c5ce9f was working ok but as the tables grew the postgres worker for that connection hit constantly 100% cpu usage on my small machine and also data was lost. So the functions needed to be optimized. - The function collectd_insert() was split into 2 parts. One part just collecting the data, create the instances and realign the timestamps. Second part: move_data_to_table() that must be called periodically. It also creates the plugin tables and the asociated indices.
|
|
||
| The current structure creates 3 identifiers and 3 lines for each entry. The identifiers get reused. It reports "192.168.myping.ip" as type. | ||
| The current structure creates 3 identifiers and 3 lines for each entry. The identifiers get reused. It reports "192.168.200.123" as type. |
There was a problem hiding this comment.
Where the IP value comes from?
There was a problem hiding this comment.
I just choose random IP.... because it seemed better.
There was a problem hiding this comment.
From RFC5737:
The blocks 192.0.2.0/24 (TEST-NET-1), 198.51.100.0/24 (TEST-NET-2), and 203.0.113.0/24 (TEST-NET-3) are provided for use in documentation.
There was a problem hiding this comment.
Suggestion: spell out what is being reported here and provide an example. The quotes make it look like this is the literal IP address that is being reported each time.
“The ping plugin reports an IP address in the type instance field, e.g. "192.0.2.42".”
| */1 * * * * root /opt/collectd/etc/move-data.sh | ||
| ``` | ||
|
|
||
| The procedure collectd_cleanup() is the maintenance function. It has as an argument the number of days where to keep the data. It can be called by pgagent or a similar mechanism like "CALL collectd_cleanup(180)". This delete all data that is older than 180 days. |
There was a problem hiding this comment.
| The procedure collectd_cleanup() is the maintenance function. It has as an argument the number of days where to keep the data. It can be called by pgagent or a similar mechanism like "CALL collectd_cleanup(180)". This delete all data that is older than 180 days. | |
| The procedure collectd_cleanup() is the maintenance function. It has as an argument the number of days to keep the data. It can be called by pgagent or a similar mechanism like "CALL collectd_cleanup(180)". This deletes all data that is older than 180 days. |
|
This is good to merge with @eero-t 's suggestions. Thank you to the two of you! |
|
Hi @schorsch1976, I'd love to merge this. Can you apply the suggested changes? |
octo
added
the
Waiting for response - 1st time
Co-authored-by: Eero Tamminen <eero.t.tamminen@intel.com>
Co-authored-by: Eero Tamminen <eero.t.tamminen@intel.com>
Co-authored-by: Eero Tamminen <eero.t.tamminen@intel.com>
Co-authored-by: Eero Tamminen <eero.t.tamminen@intel.com>
Co-authored-by: Eero Tamminen <eero.t.tamminen@intel.com>
Co-authored-by: Eero Tamminen <eero.t.tamminen@intel.com>
Co-authored-by: Eero Tamminen <eero.t.tamminen@intel.com>
Co-authored-by: Eero Tamminen <eero.t.tamminen@intel.com>
Commited :) |
octo
added
Maintenance
The first version commited 7c5ce9f was working ok but as the tables grew the postgres worker for that connection hit constantly 100% cpu usage on my small machine and also data was lost. So the functions needed to be optimized.
The function collectd_insert() was split into 2 parts. One part just collecting the data, create the instances and realign the timestamps. Second part: move_data_to_table() that must be called periodically. It also creates the plugin tables and the associated indices.
This was the CPU Load development with my first version. As you can see, it increased steadily.
This was one day with the high load. The postgres process of that connection hit 100% of the core usage.
This is the updated version with improves the performance drastically.
