V4 to v5 migration guide

From collectd Wiki
Revision as of 15:03, 11 November 2010 by Octo (talk | contribs) (Migrating existing RRD files: Information about the ZFS ARC plugin has been added.)

Jump to: navigation, search

Since version 5.* is a different major version than version 4.*, some things have been changed in a backwards incompatible manner. This migration guide is here to help you with the transition.

Upgrade strategy

If you use collectd in a client / server setup, we recommend to upgrade the server instance(s) first. This allows you to use the v5 upgrade target which transforms data sent by v4 clients to the new v5 layout. Once the server is running v5 and the v5 upgrade target you can easily upgrade the clients one by one.

This step-by-step migration suggestion assumes you have one or more dedicated collectd servers writing RRD files. The directive is to upgrade the servers and minimizing the resulting gap in the collected data.

  • If you have only one server:
    • Set up a second server identical to your production machine. You can use the Forwarding option of the Network plugin or a packet filter such as ip_tables to send a copy of the network traffic to your clone.
  • If you have two or more redundant servers:
    • Enter a downtime for one server in the monitoring and remove it from the load balancing if the server is used to create graphs. Let it still receive and handle collectd traffic, though.
  • On this separate server:
    • Install the v5 daemon.
    • Adapt the configuration for plugins collecting data locally. Usually this is fairly easy on central collectd servers.
    • Configure the v5 upgrade target. Usually this is as simple as adding the following block to your collectd configuration:
<Chain "PreCache">
  Target "v5upgrade"
    • (Re)start the daemon.
    • Look for files that are no longer upgraded and rename them or split them up. Look at #Migrating existing RRD files for details on this. Possibly rsync / scp the RRD files and run the migration script multiple times until you are satisfied with the migrated data.
  • On your original server / For each other server:
    • Install the v5 daemon.
    • Copy the v5 config and adapt hostname / IP-addresses etc.
    • Restart the daemon.
    • rsync / scp the migrated files to get a clean data set without gaps.

Migrating the configuration

This section details which changes you need to make to the configuration file.


The default value for FQDNLookup has been changed. It used to be disabled by default and it's now enabled by default. You can use "FQDNLookup false" to configure the old default behavior. Enabling this option is recommended though.

Apache plugin

The old configuration was suited for one webserver only. The new configuration (added in version 4.7) puts the same configuration options into <Instance /> blocks.

Old New
<Plugin "apache">
  URL "http://localhost/mod_status?auto"
<Plugin "apache">
  <Instance "">
    URL "http://localhost/mod_status?auto"

The string argument of the Instance blocks is used as the plugin instance. The example above used an empty string ("") as the instance name to get the same behavior the previous version (without Instance blocks) used.

Versions 4.7 through 4.10 included backwards compatibility code for configurations without Instance blocks. This code has been removed in version 5.0.

HDDTemp plugin

The TranslateDevicename configuration option has been removed.

For backwards compatibility reasons, the default behavior was to translate device names such as "/dev/sda" to major and minor device numbers, e.g. "8-0". This translation could be disabled by setting TranslateDevicename to false.

The new behavior is to not translate device names, i.e. behave as if the option was set to false.

Old New
<Plugin "hddtemp">
  … other options …
  TranslateDevicename false
<Plugin "hddtemp">
  … other options …
<Plugin "hddtemp">
  … other options …
  TranslateDevicename true
<Plugin "hddtemp">
  … other options …

+Rename RRD files!

If you used TranslateDevicename true or used the default behavior, you need to move the RRD files or rename the data by other means. If you used the recommended setting false, you only need to remove this line from the config.

MySQL plugin

In order to support collecting information from multiple MySQL instances, the configuration options have been moved into <Database /> blocks.

Old New
<Plugin "mysql">
  Host "localhost"
  User "username"
  Password "password"
<Plugin "mysql">
  <Database "">
    Host "localhost"
    User "username"
    Password "password"

The string argument of the Database block is used as plugin instance. The example above used an empty string ("") to mimic the old behavior.

Please note that the old behavior was to use the globally configured hostname for all values submitted by the MySQL plugin. This changes with the Database blocks: When the hostname is not set, an empty string ("") or "localhost", the globally defined hostname will be used as before. If the option is set to any other string, it will be used in the dispatched data, too. You may need to rename existing data accordingly.

Network plugin

The default buffer size of the Network plugin has been increased from 1024 byte to 1452 byte. If you send data from a v5 client to a v4 server, not all data sent by the client will be received, leading to gaps in the graphs.

For this and other reasons we strongly recommend to upgrade the server(s) first. If this is not an option for you, you should use the MaxPacketSize option (since version 4.8) on the server to increase the receive buffer size.

PostgreSQL plugin

The PostgreSQL plugin has been changed to its configuration is as close as possible to the configuration of the DBI and Oracle plugins. The "new" configuration options are available since version 4.6, so chances are you are already using them. Only legacy code has been removed.

The changes are:

  • The Query option has been replaced by the Statement option.
  • The Column option has been replaced with Result blocks.
  • The MinPGVersion and MaxPGVersion options have been replaced with the MinVersion and MaxVersion options.
Old New
<Plugin "postgresql">
  <Query "magic_qry">
    Query "SELECT magic_col FROM wizard;"
    Column "gauge" "magic_inst"
<Plugin "postgresql">
  <Query "magic_qry">
    Statement "SELECT magic_col FROM wizard;"
      Type "gauge"
      InstancePrefix "magic_inst"
      ValuesFrom "magic_col"

Migrating existing RRD files

The definition and use of a couple of types has been changed, which means that the appropriate RRD files need to be converted. The changes fall into three categories:

Different data source name
The convention was and is that data sets with only one data source should use the name "value" for this data source. This has been fixed for the following types:
  • absolute
  • apache_bytes
  • apache_connections
  • apache_idle_workers
  • apache_requests
  • apache_scoreboard
  • conntrack
  • contextswitch
  • delay
  • entropy
  • file_size
  • frequency
  • frequency_offset
  • http_request_methods
  • http_requests
  • http_response_codes
  • percent
  • ping
  • records
  • time_dispersion
  • timeleft
  • time_offset
  • users
  • virt_cpu_total
  • virt_vcpu
You can change the data source name using rrdtune(1):
$ rrdtool tune $FILE --data-source-rename old_name:value
Multiple data sources have been split up
A few types have multiple data sources which really don't belong in the same data set. Sometimes, this seemed reasonable at first but proved to inflexible. The DF plugin for example used one type with "used" and "free" partition space, which seems okay at first sight. If you want to collect "reserved snapshot space" or "duplicate blocks", however, this seemingly simple metric becomes a lot more complicated.
Version 5.0 corrects a couple of such mistakes. Affected plugins / types are:
DF plugin
df df_complex-used
MySQL plugin
mysql_qcache cache_result-qcache-hits
mysql_threads threads-running
ZFS ARC plugin
arc_counts-(hits/misses) cache_result-demand_data-(hit/miss)
arc_l2_bytes io_octets-L2
arc_l2_size cache_size-L2
arc_ratio-(L1/L2) cache_ratio-(arc/L2)
(only data source “current” is used.)