custom_query perfdata multi-row output improvements

[swanson8r]

Allow custom_query checks with multiple rows of output to populate the second returned column as the perfdata variable name. Pass the result value as the perfdata value. Helpful for count(*) GROUP BY queries.

Example: alert when the count of any fruit is below 5. Output performance data for each fruit.

check_postgres.pl -H localhost --action custom_query --query="SELECT count(*) as result, name FROM fruit GROUP BY name" --valtype=integer --warn='5' --reverse

Current output without this change is not very useful: repeats the name of the 2nd column as the perfdata variable name, provides the name of each fruit as the perfdata value, but not the count. Repeats the threshold as the value.

POSTGRES_CUSTOM_QUERY WARNING: 5 | time=0.01s name=apple;5 name=orange;5; name=banana;5

With the change, trends in fruit counts can now be tracked over time in the performance data:

POSTGRES_CUSTOM_QUERY WARNING: 5 | time=0.01s apple=4;5 orange=12;5; banana;10;5

[swanson8r]

Added comments to explain exactly how the query results are processed into performance data

[swanson8r]

check the number of rows returned by the custom query

[swanson8r]

If more than one row is returned, store the perfdata to a temporary variable and check if it has already been appended to the output.

[swanson8r]

Build test are failing for PGVERSION=9.3 and older, possibly an issue with the tests and not with this PR?

Test Summary Report
-------------------
t/02_replication_slots.t (Wstat: 65024 Tests: 20 Failed: 0)
  Non-zero exit status: 254
  Parse errors: No plan found in TAP output
Files=55, Tests=928, 138 wallclock secs ( 0.39 usr  0.12 sys + 52.63 cusr 14.84 csys = 67.98 CPU)
Result: FAIL
Failed 1/55 test programs. 0/928 subtests failed.
make: *** [test_dynamic] Error 255
The command "PGBINDIR=/usr/lib/postgresql/$PGVERSION/bin make test TEST_VERBOSE=1" exited with 2.
after_script
0.01s$ tail -n 200 test_database_check_postgres*/pg.log
==> test_database_check_postgres2/pg.log <==
LOG:  database system was shut down at 2017-09-25 19:26:34 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
FATAL:  role "check_postgres_testing" does not exist
ERROR:  cannot drop language plpgsql because extension plpgsql requires it
HINT:  You can drop extension plpgsql instead.
STATEMENT:  DROP LANGUAGE IF EXISTS plpgsql
LOG:  received smart shutdown request
LOG:  shutting down
LOG:  database system is shut down
==> test_database_check_postgres3/pg.log <==
LOG:  database system was shut down at 2017-09-25 19:26:36 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
FATAL:  role "check_postgres_testing" does not exist
ERROR:  cannot drop language plpgsql because extension plpgsql requires it
HINT:  You can drop extension plpgsql instead.
STATEMENT:  DROP LANGUAGE IF EXISTS plpgsql
LOG:  received smart shutdown request
LOG:  shutting down
LOG:  database system is shut down
==> test_database_check_postgres/pg.log <==
LOG:  database system was shut down at 2017-09-25 19:25:23 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
FATAL:  role "check_postgres_testing" does not exist
FATAL:  sorry, too many clients already
ERROR:  canceling statement due to statement timeout
STATEMENT:  BEGIN;SET statement_timeout=1000;COMMIT;SELECT version() AS v
ERROR:  syntax error at or near "THIS" at character 42
STATEMENT:  BEGIN;SET statement_timeout=30000;COMMIT;THIS IS NOT A QUERY
ERROR:  relation "pg_listener" does not exist at character 68
STATEMENT:  BEGIN;SET statement_timeout=30000;COMMIT;SELECT count(*) AS c FROM pg_listener WHERE relname = 'foo'
FATAL:  database "foo" does not exist
ERROR:  syntax error at or near "check_postgres_logfile_error_202400242604" at character 42
STATEMENT:  BEGIN;SET statement_timeout=30000;COMMIT;check_postgres_logfile_error_202400242604 check_postgres.pl DB=postgres PID=7303 Time=Mon Sep 25 19:25:58 2017
ERROR:  syntax error at or near "check_postgres_logfile_error_844899224043" at character 42
STATEMENT:  BEGIN;SET statement_timeout=30000;COMMIT;check_postgres_logfile_error_844899224043 check_postgres.pl DB=postgres PID=7333 Time=Mon Sep 25 19:26:00 2017
ERROR:  cannot drop language plpgsql because extension plpgsql requires it
HINT:  You can drop extension plpgsql instead.
STATEMENT:  DROP LANGUAGE IF EXISTS plpgsql
ERROR:  relation "foobar.sl_status" does not exist at character 234
STATEMENT:  BEGIN;SET statement_timeout=30000;COMMIT;SELECT
	 ROUND(EXTRACT(epoch FROM st_lag_time)) AS lagtime,
	 st_origin,
	 st_received,
	 current_database() AS cd,
	 COALESCE(n1.no_comment, '') AS com1,
	 COALESCE(n2.no_comment, '') AS com2
	FROM foobar.sl_status
	JOIN foobar.sl_node n1 ON (n1.no_id=st_origin)
	JOIN foobar.sl_node n2 ON (n2.no_id=st_received)
	ORDER BY 1 DESC
ERROR:  view "sl_status" does not exist
STATEMENT:  DROP VIEW slony_testing.sl_status
ERROR:  table "sl_node" does not exist
STATEMENT:  DROP TABLE slony_testing.sl_node
ERROR:  schema "slony_testing" does not exist
STATEMENT:  DROP SCHEMA slony_testing
ERROR:  canceling statement due to statement timeout
STATEMENT:  BEGIN;SET statement_timeout=1000;COMMIT;SELECT pg_sleep(2)
LOG:  received smart shutdown request
LOG:  shutting down
LOG:  database system is shut down
Done. Your build exited with 1.

[swanson8r]

Initialize perfdata variables before they are compared

[swanson8r]

Get the returned custom query row count

[keithf4]

Any chance someone can review this and see about merging it? Without it, custom queries are pretty much useless if you need to use perfdata.

[drapiti]

Please merge this fix as we require these changes also.