Nice metrics for postfix_exporter

[deajan]

Hello @sergeymakinen

So far, I'm running two different postfix exporters, of which only yours seems to be maintained.
In order to diagnose postfix, I'd love to see the following metrics added to your postfix exporter:

• postfix queue average mail age
• number of enqueued messages

I'd like to have some more values, but I get alot of NaN values, so I guess I need to open another issue first before asking you things that the exporter already does perhaps.

PS: I updated your grafana dashboard to use vectors, and added some metrics. I'll post them if needed.

[deajan]

Adding some metrics I'd love to see (would even sponsor dev).

• postfix_smtpd_messages_rejected_total{code="",extended_code="", reason=""}

Where code is 450,530,550 and so on and where extended code would be "4.5.1", "5.7.0" etc.
This would help identify blacklist rejections.

The ultimate must have for me would be the reason variable.
I'd add regexes in the postfix_exporter yaml conf file, which will fill the reason depending on a regex:

Reason	Regex
blacklist	reputation|blacklist
conf	spf|dkim|dns
remote_problem	recipient address|temporary local problem|internal resource|mailbox unavailable
remote_timeout	connexion timed out|unreachable

[sergeymakinen]

Thanks @deajan. For any new metric it's crucial to have anonymized logs as I only host a single mail server for myself and even though I use to dig through the Postfix source code to understand how log entries are being composed, it would greatly increase a chance to add any metric I'm not sure about/not reproducible.

[deajan]

Hi @sergeymakinen ;)
I indeed thought you'd need logs ;)
It's a pain to do so, but I managed to anonymize two log files from different servers (postfix v3.4.24 for the big file, postfix 3.7.6 for the small file).
I choose to make two different log files since usually different server profiles may produce various logs.

Hope this helps ;)

mail_anonymized_srvm.zip
mail_anonymized_srvp.zip

[sergeymakinen]

Hey @deajan,

Great, thank you! :) I'll dig into the logs and hopefully get what I need. 🤞

[deajan]

Hello Sergey,

Sorry for being late for the party ;)
I've upgraded your postfix_exporter on one of my servers, and so far I wasn't able to get it to run properly with the new additions.

I've used /usr/local/bin/postfix_exporter --config.file=/etc/postfix_exporter.yml, where the config file contains the following:

host_replies:
  - regexp: out of storage|recipient address|temporary local problem|internal resource|mailbox unavailable
    text: remote_problem
  - regexp: connection timed out|unreachable
    text: remote_timeout
  - regexp: spf|dkim|dns
    text: SPF / DKIM / DNS local server config error
  - regexp: reputation|blaklist
    text: Our source IP has bad reputation
  - type: queue_status
    regexp: (.+)
    text: $1
  - type: other
    regexp: (?i)gr(a|e)ylist
    text: Remote host has graylisted us
noqueue_reject_replies:
  - regexp: '(Client host rejected: cannot find your hostname|Recipient address rejected: Rejected by SPF)'
    text: $1
  - regexp: (.+)
    text: $1

Using --config.check, the exporter tells me that the file is valid.
But when querying the metrics, I don't get any postfix_noqueue_reject_replies_total nor postfix_smtp_status_replies_total entries.
Anything I missed perhaps ?

[sergeymakinen]

Hi @deajan,

Did a quick test using a test data with your config and it seems to work for me. Probably one thing to take care about: Go regexes are case sensitive by default, so you might want test your expected log lines on https://regex101.com (or similar) against regexes and share with me if they match but it still doesn't work for you.

[deajan]

You're right, exporter works as advertised. Just didn't let it run enough time, and on 26th there isn't a big mail flow, so that's what got me.
I'll continue evaluating the new feature ;)
Wish you nice holidays.

[deajan]

Well, it seems that I still got a problem nevertheless.
While the noqueue metrics show up after receiving some shady mailserver connections, the postfix_smtp_status_replies_total doesn't seem to show up.

I've checked my regexes via regex101 (with go flavor), and I get no results.
Below is a log I checked in regex101 with regex (?i)dsn=2\.0\.0 that matches, but produces no result in postfix_exporter

Dec 26 18:45:59 mail postfix/submission/smtpd[11925]: connect from unknown[231.231.231.256]
Dec 26 18:45:59 mail postfix/submission/smtpd[11925]: B6CFE52606B3: client=unknown[231.231.231.256], sasl_method=LOGIN, sasl_username=me@example.tld
Dec 26 18:45:59 mail postfix/cleanup[12649]: B6CFE52606B3: message-id=<000701da3823$63051820$290f4860$@them.tld>
Dec 26 18:45:59 mail postfix/qmgr[19515]: B6CFE52606B3: from=<me@example.tld>, size=2795, nrcpt=1 (queue active)
Dec 26 18:45:59 mail postfix/smtp[12651]: warning: DNSSEC validation may be unavailable
Dec 26 18:45:59 mail postfix/smtp[12651]: warning: reason: dnssec_probe 'ns:.' received a response that is not DNSSEC validated
Dec 26 18:46:01 mail postfix/smtp[12651]: B6CFE52606B3: to=<someoneelse@them.tld>, relay=asuperserver.tld[1.2.3.4]:25, delay=2, delays=0.24/0.04/0.35/1.4, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 4T02Hr4S5hz9vf6k)
Dec 26 18:46:01 mail postfix/qmgr[19515]: B6CFE52606B3: removed

Here's a full copy of my postfix_exporter.yml file:

host_replies:
  - regexp: (?i)out of storage|recipient address|temporary (local )?problem|internal resource|mailbox (is )?unavailable
    text: remote_problem
  - regexp: (?i)recipient address
    text: remote_user_not_existing
  - regexp: (?i)connection time(d)? out|timeout|unreachable
    text: remote_not_reachable
  - regexp: (?i)spf|dkim|dns
    text: local_conf_problem
  - regexp: (?i)reputation|blacklist
    text: bad_reputation
  - regexp: (?i)dsn=2\.0\.0
    text: sent
  - regexp: (?i)dsn=2\.6\.0
    text: sent_in_spam_folder
  - type: other
    regexp: (?i)gr(a|e)ylist
    text: greylisting
  - type: queue_status
    regexp: (.+)
    text: $1
noqueue_reject_replies:
  - regexp: (?i)Client host rejected|cannot find your hostname|Rejected by SPF
    text: spam
  - regexp: (?i)Recipient address rejected|Recipient address not found|User unknown
    text: user_not_existing
  - regexp: (.+)
    text: $1

Also, I've tried to understand the documentation, but I don't understand, what is type: other|queue_status for ? Perhaps I did something bad in my config with this setting ?

[deajan]

Hmm.. Finally got some 550 error message in postfix_smtp_status_replies_total metric

postfix_smtp_status_replies_total{code="550",enhanced_code="5.5.0",status="bounced",text="remote_problem"} 1

I just wonder why dsn=2\.0\.0 doesn't catch any output.

[sergeymakinen]

Per the configuration the exporter is only able to capture host replies on erroneous queue statuses (hence queue_status) or other host replies when they are out of context. And as specified in the configuration, the regex is only being matched against reply texts, not codes/enhanced codes or whole lines.

I think the following changes are necessary to make it more configurable (while keeping sane):

1. Remove type from host_replies and make it only handle those host ... said ... are out of context (aka type: other)
2. Create an additional configuration for status replies and add parsing successful replies as well:

   Dec 26 18:46:01 mail postfix/smtp[12651]: B6CFE52606B3: to=<someoneelse@them.tld>, relay=asuperserver.tld[1.2.3.4]:25, delay=2, delays=0.24/0.04/0.35/1.4, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 4T02Hr4S5hz9vf6k)
   

3. Add an ability to match against code/enhanced code/text as needed.

Hopefully these changes should help, I think that's how the configuration file would look like:

status_replies:
  - regexp: (?i)out of storage|recipient address|temporary (local )?problem|internal resource|mailbox (is )?unavailable
    text: remote_problem
  - regexp: (?i)recipient address
    text: remote_user_not_existing
  - regexp: (?i)connection time(d)? out|timeout|unreachable
    text: remote_not_reachable
  - regexp: (?i)spf|dkim|dns
    text: local_conf_problem
  - regexp: (?i)reputation|blacklist
    text: bad_reputation
  - regexp: 2\.0\.0
    match: enhanced_code # or `code` or `text` which is default
    text: sent
  - regexp: 2\.6\.0
    match: enhanced_code
    text: sent_in_spam_folder
  - regexp: (?i)gr(a|e)ylist
    text: greylisting
  - regexp: (.+)
    text: $1
noqueue_reject_replies:
  - regexp: (?i)Client host rejected|cannot find your hostname|Rejected by SPF
    text: spam
  - regexp: (?i)Recipient address rejected|Recipient address not found|User unknown
    text: user_not_existing
  - regexp: (.+)
    text: $1

[sergeymakinen]

You might want to grab v2.0.0-beta.2, take a look at changes in the CONFIGURATION.md and give it a try. :) Happy New Year!

[deajan]

Thank you, will definitly do... next year ;)
Happy New Year too!