Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scheduled Downtime won't be changed/deleted after reload #10042

Open
Mordecaine opened this issue Apr 16, 2024 · 5 comments · May be fixed by #10062
Open

Scheduled Downtime won't be changed/deleted after reload #10042

Mordecaine opened this issue Apr 16, 2024 · 5 comments · May be fixed by #10062
Assignees
@yhabteab
Labels
area/runtime Downtimes, comments, dependencies, events bug Something isn't working ref/IP
Milestone
2.15.0

Comments

@Mordecaine
Copy link

Mordecaine commented Apr 16, 2024 •
edited

Describe the bug

If I create a Scheduled Downtime in a global Zone via DSL, it is not possible to delete/change it afterwards

To Reproduce

  • Create a scheduled Downtime /etc/icinga2/zones.d/global-satellites/downtimes.conf:
	apply ScheduledDowntime "test123" to Host {
	  author = "Lasse"
	  comment = "test 123"

	  fixed = false
	  duration = 60m

	  ranges = {
	    "sunday 3" = "20:00-22:00"
	  }

	  assign where host.name == "msd-lwtest01"
	}
  • Reload
  • Check Downtime:
    icinga2 object list --type ScheduledDowntime --name 'msd-lwtest01!test123'
  • Change Downtime:
	apply ScheduledDowntime "test123" to Host {
	  author = "Lasse"
	  comment = "test 123"

	  fixed = false
	  duration = 60m

	  ranges = {
	    "sunday 1" = "20:00-22:00"
	  }

	  assign where host.vars.cloud_object_name == "vm-msd-lwtest01"
	}
  • Reload
  • Check Downtime:
    icinga2 object list --type ScheduledDowntime --name 'msd-lwtest01!test123'

Result: The old Downtime will not be changed and is still active
In the Log you can find the following lines:

[2024-04-16 13:41:24 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "msd-lwtest01!test123"
[2024-04-16 13:41:24 +0200] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/37356bcb-7b59-441c-82fc-f384fae69a60/conf.d/downtimes/msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965.conf
[2024-04-16 13:41:24 +0200] information/ConfigObjectUtility: Created and activated object 'msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965' of type 'Downtime'.
[2024-04-16 13:41:24 +0200] information/Downtime: Added downtime 'msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965' between '2024-04-21 20:00:00' and '2024-04-21 22:00:00', author: 'Lasse', flexible with 3600s duration
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!service-rsyslog' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!pending_mcms_updates' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!pending_updates' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!sentinelone-agent-status' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!disk' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!service-sssd' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!service-sentinelone' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!swap' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!interface-status' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!pending_updates_security-only' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!uptime' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!disk-io' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!service-chronyd' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!ntp-status' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!memory' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!about-me' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!dmesg' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!mountpoints-status' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!service-vmtoolsd' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!selinux-mode' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!cpu' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!service-crond' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:57 +0200] notice/ApiListener: Received config update for object: {"config":"object Downtime \"916b75be-2881-40f4-ab62-de302e93c965\" ignore_on_error {\n\tauthor = \"Lasse\"\n\tauthoritative_zone = \"master\"\n\tcomment = \"test 123\"\n\tconfig_owner = \"msd-lwtest01!test123\"\n\tconfig_owner_hash = \"7d7c18a918e1b2e7397f941b7080a5b0a8c1b65905d86d123838f21f69e1e7e3\"\n\tduration = 3600.000000\n\tend_time = 1713729600.000000\n\tentry_time = 1713267684.576673\n\tfixed = false\n\thost_name = \"msd-lwtest01\"\n\tparent = \"\"\n\tscheduled_by = \"msd-lwtest01!test123\"\n\tstart_time = 1713722400.000000\n\ttriggered_by = \"\"\n\tversion = 1713267684.576721\n\tzone = \"msd-private_vcloud\"\n}\n","modified_attributes":{},"name":"msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965","original_attributes":[],"type":"Downtime","version":1713267684.576721,"zone":"msd-private_vcloud"}
[2024-04-16 13:46:01 +0200] notice/ApiListener: Received config update for object: {"config":"object Downtime \"916b75be-2881-40f4-ab62-de302e93c965\" ignore_on_error {\n\tauthor = \"Lasse\"\n\tauthoritative_zone = \"master\"\n\tcomment = \"test 123\"\n\tconfig_owner = \"msd-lwtest01!test123\"\n\tconfig_owner_hash = \"7d7c18a918e1b2e7397f941b7080a5b0a8c1b65905d86d123838f21f69e1e7e3\"\n\tduration = 3600.000000\n\tend_time = 1713729600.000000\n\tentry_time = 1713267684.576673\n\tfixed = false\n\thost_name = \"msd-lwtest01\"\n\tparent = \"\"\n\tscheduled_by = \"msd-lwtest01!test123\"\n\tstart_time = 1713722400.000000\n\ttriggered_by = \"\"\n\tversion = 1713267684.576721\n\tzone = \"msd-private_vcloud\"\n}\n","modified_attributes":{},"name":"msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965","original_attributes":[],"type":"Downtime","version":1713267684.576721,"zone":"msd-private_vcloud"}
[2024-04-16 13:46:05 +0200] notice/ApiListener: Received config update for object: {"config":"object Downtime \"916b75be-2881-40f4-ab62-de302e93c965\" ignore_on_error {\n\tauthor = \"Lasse\"\n\tauthoritative_zone = \"master\"\n\tcomment = \"test 123\"\n\tconfig_owner = \"msd-lwtest01!test123\"\n\tconfig_owner_hash = \"7d7c18a918e1b2e7397f941b7080a5b0a8c1b65905d86d123838f21f69e1e7e3\"\n\tduration = 3600.000000\n\tend_time = 1713729600.000000\n\tentry_time = 1713267684.576673\n\tfixed = false\n\thost_name = \"msd-lwtest01\"\n\tparent = \"\"\n\tscheduled_by = \"msd-lwtest01!test123\"\n\tstart_time = 1713722400.000000\n\ttriggered_by = \"\"\n\tversion = 1713267684.576721\n\tzone = \"msd-private_vcloud\"\n}\n","modified_attributes":{},"name":"msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965","original_attributes":[],"type":"Downtime","version":1713267684.576721,"zone":"msd-private_vcloud"}
[2024-04-16 13:46:08 +0200] notice/ApiListener: Received config update for object: {"config":"object Downtime \"916b75be-2881-40f4-ab62-de302e93c965\" ignore_on_error {\n\tauthor = \"Lasse\"\n\tauthoritative_zone = \"master\"\n\tcomment = \"test 123\"\n\tconfig_owner = \"msd-lwtest01!test123\"\n\tconfig_owner_hash = \"7d7c18a918e1b2e7397f941b7080a5b0a8c1b65905d86d123838f21f69e1e7e3\"\n\tduration = 3600.000000\n\tend_time = 1713729600.000000\n\tentry_time = 1713267684.576673\n\tfixed = false\n\thost_name = \"msd-lwtest01\"\n\tparent = \"\"\n\tscheduled_by = \"msd-lwtest01!test123\"\n\tstart_time = 1713722400.000000\n\ttriggered_by = \"\"\n\tversion = 1713267684.576721\n\tzone = \"msd-private_vcloud\"\n}\n","modified_attributes":{},"name":"msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965","original_attributes":[],"type":"Downtime","version":1713267684.576721,"zone":"msd-private_vcloud"}
[2024-04-16 13:46:56 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "msd-lwtest01!test123"
[2024-04-16 13:46:56 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 'msd-lwtest01!test123': Error: Cannot remove downtime 'msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965'. It is owned by scheduled downtime object 'msd-lwtest01!test123'
[2024-04-16 13:47:56 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "msd-lwtest01!test123"
[2024-04-16 13:47:56 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 'msd-lwtest01!test123': Error: Cannot remove downtime 'msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965'. It is owned by scheduled downtime object 'msd-lwtest01!test123'
[2024-04-16 13:48:56 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "msd-lwtest01!test123"
[2024-04-16 13:48:56 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 'msd-lwtest01!test123': Error: Cannot remove downtime 'msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965'. It is owned by scheduled downtime object 'msd-lwtest01!test123'
[2024-04-16 13:49:56 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "msd-lwtest01!test123"
[2024-04-16 13:49:56 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 'msd-lwtest01!test123': Error: Cannot remove downtime 'msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965'. It is owned by scheduled downtime object 'msd-lwtest01!test123'
[2024-04-16 13:50:56 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "msd-lwtest01!test123"
[2024-04-16 13:50:56 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 'msd-lwtest01!test123': Error: Cannot remove downtime 'msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965'. It is owned by scheduled downtime object 'msd-lwtest01!test123'
[2024-04-16 13:51:56 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "msd-lwtest01!test123"
[2024-04-16 13:51:56 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 'msd-lwtest01!test123': Error: Cannot remove downtime 'msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965'. It is owned by scheduled downtime object 'msd-lwtest01!test123'
[2024-04-16 13:52:56 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "msd-lwtest01!test123"
[2024-04-16 13:52:56 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 'msd-lwtest01!test123': Error: Cannot remove downtime 'msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965'. It is owned by scheduled downtime object 'msd-lwtest01!test123'

Expected behavior

The Downtime should be updated or deleted. Furthermore, in the logfile is written, 'Checkable msd-lwtest01!cpu' already in a NOT-OK state. Triggering downtime now, but the downtime should be started in the future and not at the reload time.

Your Environment

Include as many relevant details about the environment you experienced the problem in

  • Version used (icinga2 --version): 2.14.1
  • Operating System and version: RedHat 8.9
  • Enabled features (icinga2 feature list): api checker debuglog icingadb influxdb2 mainlog notification syslog
  • Icinga Web 2 version and modules (System - About): 2.12.1
  • Config validation (icinga2 daemon -C): No Errors
  • If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.
@log1-c
Copy link
Contributor

log1-c commented Apr 18, 2024 •
edited

Tested this on a different system and can confirm, but also with IcingaDB installed.

apply ScheduledDowntime "test123" to Host {
	  author = "Test"
	  comment = "test 123"

	  fixed = false
	  duration = 60m

	  ranges = {
	    "sunday 3" = "20:00-22:00"
	  }

	  assign where host.name == "g-ici-mon01"
	}

Object 'g-ici-mon01!test123' of type 'ScheduledDowntime':
  % declared in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40
  * __name = "g-ici-mon01!test123"
  * author = "Test"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 2:4-2:18
  * child_options = "DowntimeNoChildren"
  * comment = "test 123"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 3:4-3:23
  * duration = 3600
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 6:4-6:17
  * fixed = false
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 5:4-5:16
  * host_name = "g-ici-mon01"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40
  * name = "test123"
  * package = "_etc"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40
  * ranges
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 8:4-10:4
    * sunday 3 = "20:00-22:00"
  * service_name = ""
  * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 40
    * last_line = 1
    * path = "/etc/icinga2/zones.d/master/downtime.conf"
  * templates = [ "test123" ]
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40
  * type = "ScheduledDowntime"
  * vars = null
  * zone = "master"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40

[2024-04-18 09:21:31 +0200] information/ConfigObjectUtility: Created and activated object 'g-ici-mon01!8ee97360-f554-4896-bf93-2a188859ad42' of type 'Downtime'.
[2024-04-18 09:21:31 +0200] information/Downtime: Added downtime 'g-ici-mon01!8ee97360-f554-4896-bf93-2a188859ad42' between '2024-04-21 20:00:00' and '2024-04-21 22:00:00', author: 'Test', flexible with 3600s duration

Change downtime to

apply ScheduledDowntime "test123" to Host {
          author = "Test"
          comment = "test 123"

          fixed = false
          duration = 60m

          ranges = {
            "sunday 4" = "20:00-22:00"
          }

          assign where host.vars.is_icinga2_master
        }

systemctl reload icinga2 

Object 'g-ici-mon01!test123' of type 'ScheduledDowntime':
  % declared in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40
  * __name = "g-ici-mon01!test123"
  * author = "Test"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 2:4-2:18
  * child_options = "DowntimeNoChildren"
  * comment = "test 123"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 3:4-3:23
  * duration = 3600
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 6:4-6:17
  * fixed = false
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 5:4-5:16
  * host_name = "g-ici-mon01"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40
  * name = "test123"
  * package = "_etc"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40
  * ranges
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 8:4-10:4
    * sunday 4 = "20:00-22:00"
  * service_name = ""
  * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 40
    * last_line = 1
    * path = "/etc/icinga2/zones.d/master/downtime.conf"
  * templates = [ "test123" ]
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40
  * type = "ScheduledDowntime"
  * vars = null
  * zone = "master"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40

[2024-04-18 09:26:55 +0200] information/Downtime: Added downtime 'g-ici-mon01.!498f381c-7246-42a9-815d-8d6a100f79d7' between '2024-04-28 20:00:00' and '2024-04-28 22:00:00', author: 'Test', flexible with 3600s duration
[2024-04-18 09:26:55 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 'g-ici-mon01!test123': Error: Cannot remove downtime 'g-ici-mon01!8ee97360-f554-4896-bf93-2a188859ad42'. It is owned by scheduled downtime object 'g-ici-mon01!test123'

Removing the config file then yields no results via icinga2 object list but both downtimes remain visible in the webinterface.
And no log entries any more like the one above.
image

Icinga2 2.14.2
Icinga Web 2 Version 2.12.1
icinga/icinga-php-library 0.13.1
icinga/icinga-php-thirdparty 0.12.1
director 1.11.1
icingadb 1.1.1
incubator 0.22.0

edit: Can only get rid of the downtimes in the webinterface when delteing, reloading and then recreating the host object

@log1-c
Copy link
Contributor

log1-c commented Apr 18, 2024

Just did another test, this time on a IDO based system.

Same/Similar behavior, this time the downtime was configured via the Icinga Director and I only modified the range, without touching the assignment condition.
image

[2024-04-18 10:20:25 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 's049wl04!Wartungsfenster 20.04. EDI/ERP/BI': Error: Cannot remove downtime 's049wl04!358770b1-def1-4a09-8a7a-f2b126f5036a'. It is owned by scheduled downtime object 's049wl04!Wartungsfenster 20.04. EDI/ERP/BI'
[2024-04-18 10:20:25 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 's049as43!Wartungsfenster 20.04. EDI/ERP/BI': Error: Cannot remove downtime 's049as43!acdbc208-3e43-403d-876a-1cff6e2f5102'. It is owned by scheduled downtime object 's049as43!Wartungsfenster 20.04. EDI/ERP/BI'
[2024-04-18 10:20:25 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 'c049vm115!Wartungsfenster 20.04. EDI/ERP/BI': Error: Cannot remove downtime 'c049vm115!8befcb1e-bb2b-4d3b-b58c-787cb0418fca'. It is owned by scheduled downtime object 'c049vm115!Wartungsfenster 20.04. EDI/ERP/BI'

"Both" downtimes visible in the webinterface:
image

@Mordecaine
Copy link
Author

Mordecaine commented Apr 18, 2024 •
edited

I think all scheduled downtimes are affected:
Fixed Scheduled Downtimes: yes
Flexible Scheduled Downtimes: yes
In a global Zone: yes
In a non global Zone: yes
IDO: yes
IcingaDB: yes

@Nayakum
Copy link

Nayakum commented May 6, 2024

Not entirely the same behaviour, but I had a similar issue a couple years back where downtimes were "copied" by changing aspects of them, e.g. adding comments: #9142 At that time, it was considered as working as intended.

@yhabteab yhabteab added bug Something isn't working ref/IP area/runtime Downtimes, comments, dependencies, events labels May 15, 2024
@yhabteab
Copy link
Member

ref/IP/53497

@yhabteab yhabteab added this to the 2.15.0 milestone May 15, 2024
@yhabteab yhabteab self-assigned this May 15, 2024
@yhabteab yhabteab linked a pull request May 15, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yhabteab yhabteab

Labels
area/runtime Downtimes, comments, dependencies, events bug Something isn't working ref/IP
Projects
None yet
Milestone
2.15.0
Development

Successfully merging a pull request may close this issue.

Don't fail to remove obsolete downtimes Icinga/icinga2
4 participants
@log1-c @Mordecaine @yhabteab @Nayakum