Icinga2 Service fails when a service is created via the API for a host that doesn't exist

[kidzeivo]

Describe the bug

Adding a service to a host that does not exist in icinga2 via the api adds the service and causes the icinga2 to fail.

To Reproduce

Provide a link to a live example, or an unambiguous set of steps to reproduce this bug. Include configuration, logs, etc. to reproduce, if relevant.

1. Send a put request to icinga2 api to add a service on a host that does not exist

curl -k -u user:password -X PUT 'https://localhost:5665/v1/objects/services/hostname.com!ssh'  -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{"templates":["service-template"],"attrs":{"check_command":"check_ssh","display_name":"SSH","vars":{"port":"22","notification_type":"email"}}}'

Expected behavior

Icinga2 should error and not add the service causing icinga2 to fail.

{"results":[{"code":500.0,"errors":["Error: Validation failed for object 'hostname.com!ssh' of type 'Service'; Attribute 'host_name': Object 'hostname.com' of type 'Host' does not exist.\nLocation: in /var/lib/icinga2/api/packages/_api/localhost-1495214738-1/conf.d/services/hostname.com!ssh.conf: 6:2-6:37"],"status":"Object could not be created."}]

Your Environment

• Version used (icinga2 --version):
  *icinga2 - The Icinga 2 network monitoring daemon (version: r2.14.2-1)

Copyright (c) 2012-2024 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later https://gnu.org/licenses/gpl2.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
Platform: Ubuntu
Platform version: 22.04.4 LTS (Jammy Jellyfish)
Kernel: Linux
Kernel version: 5.15.0-100-generic
Architecture: x86_64

Build information:
Compiler: GNU 11.4.0
Build host: runner-hh8q3bz2-project-575-concurrent-0
OpenSSL version: OpenSSL 3.0.2 15 Mar 2022

Application information:

General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2

Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var

Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid

• Operating System and version:
  Distributor ID: Ubuntu
  Description: Ubuntu 22.04.4 LTS
  Release: 22.04
  Codename: jammy

• Enabled features (icinga2 feature list):
  Disabled features: command compatlog debuglog elasticsearch gelf graphite influxdb influxdb2 journald livestatus notification opentsdb perfdata syslog
  Enabled features: api checker icingadb ido-mysql mainlog

• Icinga Web 2 version and modules (System - About):
  Icinga Web 2 Version 2.12.1
  Git commit cd2daeb2cb8537c633d343a29eb76c54cd2ebbf2
  PHP Version 8.3.4
  Git commit date 2023-11-15
  Get Icinga Support
  Icinga Community
  Report a bug
  Icinga Documentation
  Loaded Libraries
  icinga/icinga-php-library 0.13.1
  icinga/icinga-php-thirdparty 0.12.1
  Loaded Modules
  director 1.11.1 Configure
  icingadb 1.1.1 Configure
  incubator 0.22.0 Configure
  monitoring 2.12.1 Configure
  setup 2.12.1 Configure

• Config validation (icinga2 daemon -C):
  [2024-03-22 13:37:53 -0400] information/cli: Icinga application loader (version: r2.14.2-1)
  [2024-03-22 13:37:53 -0400] information/cli: Loading configuration file(s).
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Committing config item(s).
  [2024-03-22 13:37:53 -0400] warning/ApiListener: Attribute 'key_path' for object 'api' of type 'ApiListener' is deprecated and should not be used.
  [2024-03-22 13:37:53 -0400] warning/ApiListener: Attribute 'ca_path' for object 'api' of type 'ApiListener' is deprecated and should not be used.
  [2024-03-22 13:37:53 -0400] warning/ApiListener: Attribute 'cert_path' for object 'api' of type 'ApiListener' is deprecated and should not be used.
  [2024-03-22 13:37:53 -0400] warning/ApiListener: Please read the upgrading documentation for v2.8: https://icinga.com/docs/icinga2/latest/doc/16-upgrading-icinga-2/
  [2024-03-22 13:37:53 -0400] information/ApiListener: My API identity: localhost
  [2024-03-22 13:37:53 -0400] warning/ApplyRule: Apply rule 'email-notification' (in /etc/icinga2/conf.d/notifications.conf: 77:1-77:54) for type 'Notification' does not match anywhere!
  [2024-03-22 13:37:53 -0400] warning/ApplyRule: Apply rule 'apt' (in /etc/icinga2/conf.d/apt.conf: 1:0-1:18) for type 'Service' does not match anywhere!
  [2024-03-22 13:37:53 -0400] warning/ApplyRule: Apply rule 'ping6' (in /etc/icinga2/conf.d/services.conf: 34:1-34:21) for type 'Service' does not match anywhere!
  [2024-03-22 13:37:53 -0400] warning/ApplyRule: Apply rule '' (in /etc/icinga2/conf.d/services.conf: 57:1-57:65) for type 'Service' does not match anywhere!
  [2024-03-22 13:37:53 -0400] warning/ApplyRule: Apply rule '' (in /etc/icinga2/conf.d/services.conf: 65:1-65:53) for type 'Service' does not match anywhere!
  [2024-03-22 13:37:53 -0400] warning/ApplyRule: Apply rule 'icinga' (in /etc/icinga2/conf.d/services.conf: 73:1-73:22) for type 'Service' does not match anywhere!
  [2024-03-22 13:37:53 -0400] warning/ApplyRule: Apply rule 'load' (in /etc/icinga2/conf.d/services.conf: 81:1-81:20) for type 'Service' does not match anywhere!
  [2024-03-22 13:37:53 -0400] warning/ApplyRule: Apply rule 'procs' (in /etc/icinga2/conf.d/services.conf: 92:1-92:21) for type 'Service' does not match anywhere!
  [2024-03-22 13:37:53 -0400] warning/ApplyRule: Apply rule 'swap' (in /etc/icinga2/conf.d/services.conf: 100:1-100:20) for type 'Service' does not match anywhere!
  [2024-03-22 13:37:53 -0400] warning/ApplyRule: Apply rule 'users' (in /etc/icinga2/conf.d/services.conf: 108:1-108:21) for type 'Service' does not match anywhere!
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Instantiated 4 NotificationCommands.
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Instantiated 13 Notifications.
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Instantiated 1 IcingaApplication.
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Instantiated 3 Hosts.
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Instantiated 16 EventCommands.
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Instantiated 1 IcingaDB.
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Instantiated 1 FileLogger.
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Instantiated 3 Zones.
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Instantiated 1 CheckerComponent.
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Instantiated 1 User.
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Instantiated 1 Endpoint.
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Instantiated 3 ApiUsers.
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Instantiated 1 ApiListener.
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Instantiated 313 CheckCommands.
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Instantiated 1 UserGroup.
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Instantiated 3 TimePeriods.
  [2024-03-22 13:37:53 -0400] information/ConfigItem: Instantiated 18 Services.
  [2024-03-22 13:37:53 -0400] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
  [2024-03-22 13:37:53 -0400] information/cli: Finished validating the configuration file(s).

[julianbrost]

Icinga2 should error and not add the service causing icinga2 to fail.

What exactly do you mean by fail?

{"results":[{"code":500.0,"errors":["Error: Validation failed for object 'hostname.com!ssh' of type 'Service'; Attribute 'host_name': Object 'hostname.com' of type 'Host' does not exist.\nLocation: in /var/lib/icinga2/api/packages/_api/localhost-1495214738-1/conf.d/services/hostname.com!ssh.conf: 6:2-6:37"],"status":"Object could not be created."}]

That looks like an error you'll receive as an API response when attempting to create a service on a non-existing host. What behavior would you expect instead?

[kidzeivo]

That is the error I received on an older version of icinga2 (icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.5-1)). I expected the new version to behave the same.

[julianbrost]

So what exactly is the behavior with 2.14.2? I'm missing a description of that behavior, that's probably why I overlooked that you wrote this under "expected behavior".

[kidzeivo]

You should be able to recreate the bug if you run the following 2 commands:

1. Create a host via the API to a hostgroup that does not exist
   curl -k -u user:password -X PUT 'https://localhost:5665/v1/objects/hosts/hostname1' -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{"attrs":{"groups":["NoGroup"],"display_name":"hostname1","vars":{"host_name":"hostname1"}}, "templates":["generic-host"]}'

{"results":[{"code":500,"errors":["Error: Validation failed for object 'hostname1' of type 'Host'; Attribute 'groups': Object 'NoGroup' of type 'HostGroup' does not exist.\nLocation: in /var/lib/icinga2/api/packages/_api/e55a066a-19ab-4f1a-aa62-b2304fb5d858/conf.d/hosts/hostname1.conf: 5:2-5:23"],"status":"Object could not be

2. Create a service on the host above hostname1 even though it is not in icinga2
   curl -k -u user:password -X PUT 'https://localhost:5665/v1/objects/services/hostname1!ssh' -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{"templates":["generic-service"],"attrs":{"check_command":"ssh","display_name":"SSH"}}'
   curl: (56) OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0

The service file is created in icinga2 even though host does not exist. This causes the icinga2 service to fail.

[development] /var/lib/icinga2/api/packages/_api/e55a066a-19ab-4f1a-aa62-b2304fb5d858/conf.d/services # ls
'hostname1!ssh.conf'

[yhabteab]

The service file is created in icinga2 even though host does not exist. This causes the icinga2 service to fail.

[development] /var/lib/icinga2/api/packages/_api/e55a066a-19ab-4f1a-aa62-b2304fb5d858/conf.d/services # ls 'hostname1!ssh.conf

The config in that file should have something like object Service "ssh" ignore_on_error { which tells the Icinga 2 service not to fail/crash if the object could not be loaded/created successfully, and that file will be removed the next time the Icinga 2 service is reloaded/restarted. So, if your Icinga 2 service fails to start, it is definitely not due to this misconfigured config/file and it would also be helpful if you would share the errors you received when starting Icinga 2 instead of just saying This causes the Icinga 2 service to fail.