New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mechanism for incremental config updates #5
Comments
#40 should to the job for the prototype, a later version will need a more sophisticated mechanism though. |
We just discussed this issue again. Since its emerge, the current periodic synchronization was introduced by @julianbrost, commented above. It runs on a secondly basis, fetching the whole configuration every time. Even as it is questionable how big end user's configuration might become, this approach does not scale and introduces unnecessary load. Regarding the current Go code, a variant of the second suggestion would be the easiest to implement. All configuration tables would require two new columns - Another thought we discussed was an audit log or journal for the configuration. As the config lives within the database, both (some) users and developers might want to track its changes. The third option, or a variant thereof, would allow creating a configuration audit log. However, such an audit log would require a full diff to be useful as otherwise it is a half-baked feature and users will complain. Regardless of which way we are going to choose, a garbage collection will become necessary. The second option is going to pile up @nilmerg: What do you think? I am currently in favor of the second option, as outlined above. This should also be easier to implement in Icinga Notifications Web. |
These rows still serve the purpose of providing additional information for old history entries. So I wouldn't say that garbage collection is immediately necessary with this variant. But that probably also depends a bit on the individual tables and whether these are actually referenced from the history. One question will then arise: do we want that for every config table or just for the main ones? Like for example, if a user is added to a group, there would be two options: track each membership individually with |
While the OP mentioned temporal tables or system tables, which might look promising, those are not available in PostgreSQL. The SQL:2011 revision specifies system-versioned tables as implemented in MariaDB, but PostgreSQL currently only has external extensions therefore. |
I don't really care. Or rather, Web doesn't. Whether we'll need to update two additional columns or insert a new row in a new table, isn't a big difference. |
Then we're aiming at a variant of the second version with a icinga-notifications/internal/config/runtime.go Lines 57 to 66 in 8ca9efd
|
Required for incremental configurations in icinga-notifications as it introduces time comparisons: Icinga/icinga-notifications#5 References #753.
Enable incremental configuration updates by introducing two new columns - changed_at and deleted - for all tables directly referenced in the ConfigSet. The other relationship tables are requiring a changed_at update in their relative parent table. As a limitation, deleted rows within the database cannot be detected as the deletion logic now completely relies on the deleted column. Thus, deletions must be performed by setting both changed_at and deleted. Closes #5.
Enable incremental configuration updates by introducing two new columns - changed_at and deleted - for all tables directly referenced in the ConfigSet. The other relationship tables are requiring a changed_at update in their relative parent table. As a limitation, deleted rows within the database cannot be detected as the deletion logic now completely relies on the deleted column. Thus, deletions must be performed by setting both changed_at and deleted. Closes #5.
Enable incremental configuration updates by introducing two new columns - changed_at and deleted - for all tables directly referenced in the ConfigSet. The other relationship tables are requiring a changed_at update in their relative parent table. As a limitation, deleted rows within the database cannot be detected as the deletion logic now completely relies on the deleted column. Thus, deletions must be performed by setting both changed_at and deleted. Closes #5.
Enable incremental configuration updates by introducing two new columns - changed_at and deleted - for all tables directly referenced in the ConfigSet. The other relationship tables are requiring a changed_at update in their relative parent table. As a limitation, deleted rows within the database cannot be detected as the deletion logic now completely relies on the deleted column. Thus, deletions must be performed by setting both changed_at and deleted. Closes #5.
Periodically fetching the full configuration from the database and searching it for changes would become infeasible quickly. Therefore, we will require a mechanism for getting notified about individual configuration changes in the database.
The exact details of the implementation are up to discussion, some proposals follow (I currently favor option 3).
Option 1: Web notifies the daemon
The backend daemon already needs a listener for events from sources, it could just add a new endpoint that Web could ping every time it updates something.
Pros:
Cons:
Option 2: Update information in affected tables
Add columns like
changed_at
/deleted_at
to affected tables similar to system-versioned tables and regularly poll for rows with a higher value than the last seen one.Pro:
Cons:
now()
, monotonic values aren't guaranteed. On the other hand using a sequence number would be annoying with MySQL as there seems to be no explicit sequence type and only one auto-increment column seems to be supported.Option 3: Change queue table in the database
Create a table like
config_change (id bigserial, table text, foreign_pk bigint)
. Each time Web creates/updates/deletes a row, it inserts a row into this table referring the affected row.Pros:
time
,username
,diff
and we would have a config change/audit log.id
column would provide reliable sequencing of changes.Cons:
The text was updated successfully, but these errors were encountered: