View source for Plugin:Tail/Config

{{Example config|Tail}}

== Exim ==

''[http://www.exim.org/ Exim]'' is a ''mail transfer agent'' (MTA) which is the default mail server in [http://www.debian.org/ Debian].

The example config collects the rate of incoming and outgoing messages, bounces, deferred messages, IP connections and messages handled by a specific “router”.

 <File "/var/log/exim4/mainlog">
   Instance "exim"
   <Match>
     Regex "<="
     DSType "CounterInc"
     Type "email_type"
     Instance "incoming"
   </Match>
   <Match>
     Regex "=>"
     DSType "CounterInc"
     Type "email_type"
     Instance "outgoing"
   </Match>
   <Match>
     Regex "=="
     DSType "CounterInc"
     Type "email_type"
     Instance "defer"
   </Match>
   <Match>
     Regex "\\*\\*"
     DSType "CounterInc"
     Type "email_type"
     Instance "bounce"
   </Match>
   <Match>
     Regex "IP connection count = ([1-9][0-9]*)"
     DSType "GaugeAverage"
     Type "gauge"
     Instance "connection_count"
   </Match>
   <Match>
     Regex "Spam_checked  Spam-Score: ([5-9]|[1-9][0-9])"
     DSType "CounterInc"
     Type "email_type"
     Instance "saspam"
   </Match>
   <Match>
     Regex "R=virtdomain"
     DSType "CounterInc"
     Type "email_type"
     Instance "router-virtdomain"
   </Match>
   <Match>
     Regex "R=smarthost"
     DSType "CounterInc"
     Type "email_type"
     Instance "router-smarthost"
   </Match>
   <Match>
     Regex "R=dnslookup"
     DSType "CounterInc"
     Type "email_type"
     Instance "router-dnslookup"
   </Match>
   <Match>
     Regex "R=system_aliases"
     DSType "CounterInc"
     Type "email_type"
     Instance "router-system_aliases"
   </Match>
   <Match>
     Regex "R=fail"
     DSType "CounterInc"
     Type "email_type"
     Instance "router-fail"
   </Match>
 </File>

== Postfix ==

''[http://www.postfix.org/ Postfix]'' is a ''mail transfer agent'' (MTA) which is widely used (e.g., used in large parts of the [http://www.debian.org Debian] infrastructure).

The example config collects the number of connections, rejected messages, various status codes, messages size, and queue delay information.

<!-- XXX: use the same type in all examples? -->
<code>mail_counter</code> is defined as <code>value:DERIVE:0:U</code> (see {{Manpage|types.db|5}} for details) and is used instead of <code>counter</code> to prevent counter overflows when restarting collectd.

 <File "/var/log/mail.log">
 # or: <File "/srv/rsyslog/mail.log">
   Instance "postfix"
    
   #Since 5.8, Collectd supports new options:
   #Plugin "postfix"
   #Instance "main"
     
    # number of connections
    # (incoming)
    <Match>
      Regex "\\<postfix\\/smtpd\\[[0-9]+\\]: connect from\\>"
      DSType "DeriveInc"
      Type "mail_counter"
      Instance "connection-in-open"
    </Match>
    <Match>
      Regex "\\<postfix\\/smtpd\\[[0-9]+\\]: disconnect from\\>"
      DSType "DeriveInc"
      Type "mail_counter"
      Instance "connection-in-close"
    </Match>
    <Match>
      Regex "\\<postfix\\/smtpd\\[[0-9]+\\]: lost connection after .* from\\>"
      DSType "DeriveInc"
      Type "mail_counter"
      Instance "connection-in-lost"
    </Match>
    <Match>
      Regex "\\<postfix\\/smtpd\\[[0-9]+\\]: timeout after .* from\\>"
      DSType "DeriveInc"
      Type "mail_counter"
      Instance "connection-in-timeout"
    </Match>
    <Match>
      Regex "\\<postfix\\/smtpd\\[[0-9]+\\]: setting up TLS connection from\\>"
      DSType "DeriveInc"
      Type "mail_counter"
      Instance "connection-in-TLS-setup"
    </Match>
    <Match>
      Regex "\\<postfix\\/smtpd\\[[0-9]+\\]: [A-Za-z]+ TLS connection established from\\>"
      DSType "DeriveInc"
      Type "mail_counter"
      Instance "connection-in-TLS-established"
    </Match>
    # (outgoing)
    <Match>
      Regex "\\<postfix\\/smtp\\[[0-9]+\\]: setting up TLS connection to\\>"
      DSType "DeriveInc"
      Type "mail_counter"
      Instance "connection-out-TLS-setup"
    </Match>
    <Match>
      Regex "\\<postfix\\/smtp\\[[0-9]+\\]: [A-Za-z]+ TLS connection established to\\>"
      DSType "DeriveInc"
      Type "mail_counter"
      Instance "connection-out-TLS-established"
    </Match>
 
   # rejects for incoming E-mails
   <Match>
     Regex "\\<554 5\\.7\\.1\\>"
     DSType "DeriveInc"
     Type "mail_counter"
     Instance "rejected"
   </Match>
   <Match>
     Regex "\\<450 4\\.7\\.1\\>.*Helo command rejected: Host not found\\>"
     DSType "DeriveInc"
     Type "mail_counter"
     Instance "rejected-host_not_found"
   </Match>
   <Match>
     Regex "\\<450 4\\.7\\.1\\>.*Client host rejected: No DNS entries for your MTA, HELO and Domain\\>"
     DSType "DeriveInc"
     Type "mail_counter"
     Instance "rejected-no_dns_entry"
   </Match>
    <Match>
      Regex "\\<450 4\\.7\\.1\\>.*Client host rejected: Mail appeared to be SPAM or forged\\>"
      DSType "DeriveInc"
      Type "mail_counter"
      Instance "rejected-spam_or_forged"
    </Match>
 
   # status codes
   <Match>
     Regex "status=deferred"
     DSType "DeriveInc"
     Type "mail_counter"
     Instance "status-deferred"
   </Match>
   <Match>
     Regex "status=forwarded"
     DSType "DeriveInc"
     Type "mail_counter"
     Instance "status-forwarded"
   </Match>
   <Match>
     Regex "status=reject"
     DSType "DeriveInc"
     Type "mail_counter"
     Instance "status-reject"
   </Match>
   <Match>
     Regex "status=sent"
     DSType "DeriveInc"
     Type "mail_counter"
     Instance "status-sent"
   </Match>
   <Match>
     Regex "status=bounced"
     DSType "DeriveInc"
     Type "mail_counter"
     Instance "status-bounced"
   </Match>
   <Match>
     Regex "status=SOFTBOUNCE"
     DSType "DeriveInc"
     Type "mail_counter"
     Instance "status-softbounce"
   </Match>
 
   # message size
   <Match>
     Regex "size=([0-9]*)"
     DSType "DeriveAdd"
     Type "ipt_bytes"
     Instance "size"
   </Match>
 
   # delays (see [http://serverfault.com/questions/24121/understanding-a-postfix-log-file-entry] for details)
   # total time spent in the Postfix queue
   <Match>
     Regex "delay=([\.0-9]*)"
     DSType "GaugeAverage"
     Type "gauge"
     Instance "delay"
   </Match>
   # time spent before the queue manager, including message transmission
   <Match>
     Regex "delays=([\.0-9]*)/[\.0-9]*/[\.0-9]*/[\.0-9]*"
     DSType "GaugeAverage"
     Type "gauge"
     Instance "delay-before_queue_mgr"
   </Match>
   # time spent in the queue manager
   <Match>
     Regex "delays=[\.0-9]*/([\.0-9]*)/[\.0-9]*/[\.0-9]*"
     DSType "GaugeAverage"
     Type "gauge"
     Instance "delay-in_queue_mgr"
   </Match>
   # connection setup time including DNS, HELO and TLS
   <Match>
     Regex "delays=[\.0-9]*/[\.0-9]*/([\.0-9]*)/[\.0-9]*"
     DSType "GaugeAverage"
     Type "gauge"
     Instance "delay-setup_time"
   </Match>
   # message transmission time
   <Match>
     Regex "delays=[\.0-9]*/[\.0-9]*/[\.0-9]*/([\.0-9]*)"
     DSType "GaugeAverage"
     Type "gauge"
     Instance "delay-trans_time"
   </Match>
 </File>

== policyd-weight ==

''[http://www.policyd-weight.org/ policyd-weight]'' is a policy daemon for the Postfix MTA.

The example config collects the number of messages rated ''unknown'', ''accepted'', ''deferred'', and ''rejected''.

 <File "/var/log/mail.log">
   Instance "policyd_weight"
   <Match>
     Regex "policyd-weight.*action=DUNNO\\>"
     DSType "CounterInc"  
     Type "mail_counter"  
     Instance "unknown"
   </Match>
 
   <Match>
     Regex "policyd-weight.*action=PREPEND\\>"
     DSType "CounterInc"  
     Type "mail_counter"  
     Instance "accepted"
   </Match>
 
   <Match>
     Regex "policyd-weight.*action=450\\>"
     DSType "CounterInc"  
     Type "mail_counter"  
     Instance "deferred"
   </Match>
 
   <Match>
     Regex "policyd-weight.*action=550\\>"
     DSType "CounterInc"  
     Type "mail_counter"  
     Instance "rejected"
   </Match>
 </File>

== Postgrey ==

''[http://postgrey.schweikert.ch/ Postgrey]'' is a Postfix policy server.

The sample config collects the number of messages that were ''greylisted'', ''accepted'' (by the greylist), and ''accepted by whitelist''.

 <File "/var/log/mail.log">
   Instance "postgrey"
   <Match>
     Regex "\\<action=greylist, reason=new\\>"
     DSType "CounterInc"
     Type "mail_counter"
     Instance "greylisted"
   </Match>
  
   <Match>
     Regex "\\<action=pass, reason=triplet found\\>"
     DSType "CounterInc"
     Type "mail_counter"
     Instance "accepted"
   </Match>
  
   <Match>
     Regex "\\<action=pass, reason=client whitelist\\>"
     DSType "CounterInc"
     Type "mail_counter"
     Instance "client_whitelist"
   </Match>
 </File>

== Nginx ==

This sample collects records of HTTP errors 502 (Bad Gateway) and 504 (Gateway Timeout).

 <File "/var/log/nginx/nginx-error.log">
   Instance "nginx"
   <Match>
     Regex "\\(61: Connection refused\\)"
     DSType "DeriveInc"
     Type "derive"
     Instance "err_502"
   </Match>
   <Match>
     Regex "\\(60: Operation timed out\\)"
     DSType "DeriveInc"
     Type "derive"
     Instance "err_504"
   </Match>
 </File>

Type "[[derive]]" is used to avoid peaks when ''collectd'' is restarted.

=== Backend stats ===

This example shows how ''collectd'' can be used to collect request processing stats from your backend, such as response time and responses count. To make this example work on your system, you should add new log format to the ''nginx'' configuration and use that format to log requests to your backend (e.g. Apache and php-fpm).

  <Plugin "tail">
    <File "/var/log/nginx/apache-backend1.log">
      Instance backend1
      <Match>
        Regex ".*"
        DSType "DeriveInc"
        Type "nginx_requests"
        #Instance "requests"
      </Match>
      <Match>
        Regex "^\\S+ \"([0-9.]+)\""
        DSType "GaugeAverage"
        Type "response_time"
        Instance "AvgRespTime"
      </Match>
      <Match>
        Regex "^\\S+ \"([0-9.]+)\""
        DSType "GaugeMin"
        Type "response_time"
        Instance "MinRespTime"
      </Match>
      <Match>
        Regex "^\\S+ \"([0-9.]+)\""
        DSType "GaugeMax"
        Type "response_time"
        Instance "MaxRespTime"
      </Match>
      ##Since Collectd-5.7 there is Distribution DSType supported.
      <Match>
          Regex "^\\S+ \"([0-9.]+)\""
          Type "response_time"
          <DSType "Distribution">
            Percentile 10
            Percentile 30
            Percentile 50
            Percentile 70
            Percentile 90
            Percentile 95
            BucketType "http_rate"
            Bucket 0.000 0.100
            Bucket 0.100 0.200
            Bucket 0.200 0.500
            Bucket 0.500 1.000
            Bucket 1.000 2.000
            Bucket 2.000 5.000
            Bucket 5.000 0
          </DSType>
      </Match>
    </File>
  </Plugin>

Nginx config:

  http {
  ... other directives ... 
        log_format  main  '[$host] "$upstream_response_time" '
           '$remote_addr - $remote_user [$time_local] $status '
           '"$request"  $body_bytes_sent "$http_referer" '
           '"$http_user_agent" "$upstream_addr"';
  ... other directives ... 
  server {
    ...
    location / {
        ...
        access_log /var/log/nginx/apache-backend1.log main;
        access_log /var/log/nginx/site.example.com-access.log;
        proxy_pass/fastcgi_pass ...
        ...
    }
    location ~* \.(gif|jpg|jpeg|ico|js|css| ... )$ {
        #access_log off;
        access_log /var/log/nginx/site.example.com-access.log;
        try_files $uri $uri/  @backend;
    }
    location @backend {
        access_log /var/log/nginx/apache-backend1.log main;
        access_log /var/log/nginx/site.example.com-access.log;
        proxy_pass/fastcgi_pass ...
        ...
    }
    ... other locations and directives ...
  }
  ... other servers ...
  }

The <code>log_format</code> setting changes formatting to add <code>$upstream_response_time</code> as a second field to the logged line. This value is then picked up by the regex at in the {{Plugin|Tail}} configuration. In the ''nginx'' config above, requests to backend separated from requests to static files, so the regex ".*" at instance "requests" is acceptable. Two <code>access_log</code> directives are used to get a complete access log (with static and dynamic requests both). 

Unfortunately, this example does not support requests which processed by several backends (step by step, by using nginx internal redirects, error handlers, etc). In that case, <code>$upstream_response_time</code> has several values, separated by commas and colons, which is not supported by the regular expression in this example.

You can tune this example for your needs, to get charts for <code>$upstream_response_length</code>, <code>$request_length</code>, <code>$request_time</code>, etc.  Refer to ''[http://nginx.org/en/docs/http/ngx_http_upstream_module.html#variables ngx_http_upstream_module#variables]''  and ''[http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format ngx_http_log_module.html#log_format]'' for a list of available variables.

== See also ==

* {{Plugin|Tail}}