Plugin Tail Config
This page contains some example configurations for the Tail plugin. This page is meant as a cookbook, so if you have a configuration for an aspect not handled here or a daemon not present, please feel free to add anything that's useful for you.
Exim is a mail transfer agent (MTA) which is the default mail server in Debian.
The example config collects the rate of incoming and outgoing messages, bounces, deferred messages, IP connections and messages handled by a specific “router”.
<File "/var/log/exim4/mainlog">
Instance "exim"
<Match>
Regex "<="
DSType "CounterInc"
Type "email_type"
Instance "incoming"
</Match>
<Match>
Regex "=>"
DSType "CounterInc"
Type "email_type"
Instance "outgoing"
</Match>
<Match>
Regex "=="
DSType "CounterInc"
Type "email_type"
Instance "defer"
</Match>
<Match>
Regex "\\*\\*"
DSType "CounterInc"
Type "email_type"
Instance "bounce"
</Match>
<Match>
Regex "IP connection count = ([1-9][0-9]*)"
DSType "GaugeAverage"
Type "gauge"
Instance "connection_count"
</Match>
<Match>
Regex "Spam_checked Spam-Score: ([5-9]|[1-9][0-9])"
DSType "CounterInc"
Type "email_type"
Instance "saspam"
</Match>
<Match>
Regex "R=virtdomain"
DSType "CounterInc"
Type "email_type"
Instance "router-virtdomain"
</Match>
<Match>
Regex "R=smarthost"
DSType "CounterInc"
Type "email_type"
Instance "router-smarthost"
</Match>
<Match>
Regex "R=dnslookup"
DSType "CounterInc"
Type "email_type"
Instance "router-dnslookup"
</Match>
<Match>
Regex "R=system_aliases"
DSType "CounterInc"
Type "email_type"
Instance "router-system_aliases"
</Match>
<Match>
Regex "R=fail"
DSType "CounterInc"
Type "email_type"
Instance "router-fail"
</Match>
</File>
Postfix is a mail transfer agent (MTA) which is widely used (e.g., used in large parts of the Debian infrastructure).
The example config collects the number of connections, rejected messages, various status codes, messages size, and queue delay information.
mail_counter is defined as value:DERIVE:0:U (see types.db(5) for details) and is used instead of counter to prevent counter overflows when restarting collectd.
<File "/var/log/mail.log">
# or: <File "/srv/rsyslog/mail.log">
Instance "postfix"
#Since 5.8, Collectd supports new options:
#Plugin "postfix"
#Instance "main"
# number of connections
# (incoming)
<Match>
Regex "\\<postfix\\/smtpd\\[[0-9]+\\]: connect from\\>"
DSType "DeriveInc"
Type "mail_counter"
Instance "connection-in-open"
</Match>
<Match>
Regex "\\<postfix\\/smtpd\\[[0-9]+\\]: disconnect from\\>"
DSType "DeriveInc"
Type "mail_counter"
Instance "connection-in-close"
</Match>
<Match>
Regex "\\<postfix\\/smtpd\\[[0-9]+\\]: lost connection after .* from\\>"
DSType "DeriveInc"
Type "mail_counter"
Instance "connection-in-lost"
</Match>
<Match>
Regex "\\<postfix\\/smtpd\\[[0-9]+\\]: timeout after .* from\\>"
DSType "DeriveInc"
Type "mail_counter"
Instance "connection-in-timeout"
</Match>
<Match>
Regex "\\<postfix\\/smtpd\\[[0-9]+\\]: setting up TLS connection from\\>"
DSType "DeriveInc"
Type "mail_counter"
Instance "connection-in-TLS-setup"
</Match>
<Match>
Regex "\\<postfix\\/smtpd\\[[0-9]+\\]: [A-Za-z]+ TLS connection established from\\>"
DSType "DeriveInc"
Type "mail_counter"
Instance "connection-in-TLS-established"
</Match>
# (outgoing)
<Match>
Regex "\\<postfix\\/smtp\\[[0-9]+\\]: setting up TLS connection to\\>"
DSType "DeriveInc"
Type "mail_counter"
Instance "connection-out-TLS-setup"
</Match>
<Match>
Regex "\\<postfix\\/smtp\\[[0-9]+\\]: [A-Za-z]+ TLS connection established to\\>"
DSType "DeriveInc"
Type "mail_counter"
Instance "connection-out-TLS-established"
</Match>
# rejects for incoming E-mails
<Match>
Regex "\\<554 5\\.7\\.1\\>"
DSType "DeriveInc"
Type "mail_counter"
Instance "rejected"
</Match>
<Match>
Regex "\\<450 4\\.7\\.1\\>.*Helo command rejected: Host not found\\>"
DSType "DeriveInc"
Type "mail_counter"
Instance "rejected-host_not_found"
</Match>
<Match>
Regex "\\<450 4\\.7\\.1\\>.*Client host rejected: No DNS entries for your MTA, HELO and Domain\\>"
DSType "DeriveInc"
Type "mail_counter"
Instance "rejected-no_dns_entry"
</Match>
<Match>
Regex "\\<450 4\\.7\\.1\\>.*Client host rejected: Mail appeared to be SPAM or forged\\>"
DSType "DeriveInc"
Type "mail_counter"
Instance "rejected-spam_or_forged"
</Match>
# status codes
<Match>
Regex "status=deferred"
DSType "DeriveInc"
Type "mail_counter"
Instance "status-deferred"
</Match>
<Match>
Regex "status=forwarded"
DSType "DeriveInc"
Type "mail_counter"
Instance "status-forwarded"
</Match>
<Match>
Regex "status=reject"
DSType "DeriveInc"
Type "mail_counter"
Instance "status-reject"
</Match>
<Match>
Regex "status=sent"
DSType "DeriveInc"
Type "mail_counter"
Instance "status-sent"
</Match>
<Match>
Regex "status=bounced"
DSType "DeriveInc"
Type "mail_counter"
Instance "status-bounced"
</Match>
<Match>
Regex "status=SOFTBOUNCE"
DSType "DeriveInc"
Type "mail_counter"
Instance "status-softbounce"
</Match>
# message size
<Match>
Regex "size=([0-9]*)"
DSType "DeriveAdd"
Type "ipt_bytes"
Instance "size"
</Match>
# delays (see [http://serverfault.com/questions/24121/understanding-a-postfix-log-file-entry] for details)
# total time spent in the Postfix queue
<Match>
Regex "delay=([\.0-9]*)"
DSType "GaugeAverage"
Type "gauge"
Instance "delay"
</Match>
# time spent before the queue manager, including message transmission
<Match>
Regex "delays=([\.0-9]*)/[\.0-9]*/[\.0-9]*/[\.0-9]*"
DSType "GaugeAverage"
Type "gauge"
Instance "delay-before_queue_mgr"
</Match>
# time spent in the queue manager
<Match>
Regex "delays=[\.0-9]*/([\.0-9]*)/[\.0-9]*/[\.0-9]*"
DSType "GaugeAverage"
Type "gauge"
Instance "delay-in_queue_mgr"
</Match>
# connection setup time including DNS, HELO and TLS
<Match>
Regex "delays=[\.0-9]*/[\.0-9]*/([\.0-9]*)/[\.0-9]*"
DSType "GaugeAverage"
Type "gauge"
Instance "delay-setup_time"
</Match>
# message transmission time
<Match>
Regex "delays=[\.0-9]*/[\.0-9]*/[\.0-9]*/([\.0-9]*)"
DSType "GaugeAverage"
Type "gauge"
Instance "delay-trans_time"
</Match>
</File>
policyd-weight is a policy daemon for the Postfix MTA.
The example config collects the number of messages rated unknown, accepted, deferred, and rejected.
<File "/var/log/mail.log">
Instance "policyd_weight"
<Match>
Regex "policyd-weight.*action=DUNNO\\>"
DSType "CounterInc"
Type "mail_counter"
Instance "unknown"
</Match>
<Match>
Regex "policyd-weight.*action=PREPEND\\>"
DSType "CounterInc"
Type "mail_counter"
Instance "accepted"
</Match>
<Match>
Regex "policyd-weight.*action=450\\>"
DSType "CounterInc"
Type "mail_counter"
Instance "deferred"
</Match>
<Match>
Regex "policyd-weight.*action=550\\>"
DSType "CounterInc"
Type "mail_counter"
Instance "rejected"
</Match>
</File>
Postgrey is a Postfix policy server.
The sample config collects the number of messages that were greylisted, accepted (by the greylist), and accepted by whitelist.
<File "/var/log/mail.log">
Instance "postgrey"
<Match>
Regex "\\<action=greylist, reason=new\\>"
DSType "CounterInc"
Type "mail_counter"
Instance "greylisted"
</Match>
<Match>
Regex "\\<action=pass, reason=triplet found\\>"
DSType "CounterInc"
Type "mail_counter"
Instance "accepted"
</Match>
<Match>
Regex "\\<action=pass, reason=client whitelist\\>"
DSType "CounterInc"
Type "mail_counter"
Instance "client_whitelist"
</Match>
</File>
This sample collects records of HTTP errors 502 (Bad Gateway) and 504 (Gateway Timeout).
<File "/var/log/nginx/nginx-error.log">
Instance "nginx"
<Match>
Regex "\\(61: Connection refused\\)"
DSType "DeriveInc"
Type "derive"
Instance "err_502"
</Match>
<Match>
Regex "\\(60: Operation timed out\\)"
DSType "DeriveInc"
Type "derive"
Instance "err_504"
</Match>
</File>
Type "derive" is used to avoid peaks when collectd is restarted.
This example shows how collectd can be used to collect request processing stats from your backend, such as response time and responses count. To make this example work on your system, you should add new log format to the nginx configuration and use that format to log requests to your backend (e.g. Apache and php-fpm).
<Plugin "tail">
<File "/var/log/nginx/apache-backend1.log">
Instance backend1
<Match>
Regex ".*"
DSType "DeriveInc"
Type "nginx_requests"
#Instance "requests"
</Match>
<Match>
Regex "^\\S+ \"([0-9.]+)\""
DSType "GaugeAverage"
Type "response_time"
Instance "AvgRespTime"
</Match>
<Match>
Regex "^\\S+ \"([0-9.]+)\""
DSType "GaugeMin"
Type "response_time"
Instance "MinRespTime"
</Match>
<Match>
Regex "^\\S+ \"([0-9.]+)\""
DSType "GaugeMax"
Type "response_time"
Instance "MaxRespTime"
</Match>
##Since Collectd-5.7 there is Distribution DSType supported.
<Match>
Regex "^\\S+ \"([0-9.]+)\""
Type "response_time"
<DSType "Distribution">
Percentile 10
Percentile 30
Percentile 50
Percentile 70
Percentile 90
Percentile 95
BucketType "http_rate"
Bucket 0.000 0.100
Bucket 0.100 0.200
Bucket 0.200 0.500
Bucket 0.500 1.000
Bucket 1.000 2.000
Bucket 2.000 5.000
Bucket 5.000 0
</DSType>
</Match>
</File>
</Plugin>
Nginx config:
http {
... other directives ...
log_format main '[$host] "$upstream_response_time" '
'$remote_addr - $remote_user [$time_local] $status '
'"$request" $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$upstream_addr"';
... other directives ...
server {
...
location / {
...
access_log /var/log/nginx/apache-backend1.log main;
access_log /var/log/nginx/site.example.com-access.log;
proxy_pass/fastcgi_pass ...
...
}
location ~* \.(gif|jpg|jpeg|ico|js|css| ... )$ {
#access_log off;
access_log /var/log/nginx/site.example.com-access.log;
try_files $uri $uri/ @backend;
}
location @backend {
access_log /var/log/nginx/apache-backend1.log main;
access_log /var/log/nginx/site.example.com-access.log;
proxy_pass/fastcgi_pass ...
...
}
... other locations and directives ...
}
... other servers ...
}
The log_format setting changes formatting to add $upstream_response_time as a second field to the logged line. This value is then picked up by the regex at in the Tail plugin configuration. In the nginx config above, requests to backend separated from requests to static files, so the regex ".*" at instance "requests" is acceptable. Two access_log directives are used to get a complete access log (with static and dynamic requests both).
Unfortunately, this example does not support requests which processed by several backends (step by step, by using nginx internal redirects, error handlers, etc). In that case, $upstream_response_time has several values, separated by commas and colons, which is not supported by the regular expression in this example.
You can tune this example for your needs, to get charts for $upstream_response_length, $request_length, $request_time, etc. Refer to ngx_http_upstream_module#variables and ngx_http_log_module.html#log_format for a list of available variables.