Difference between revisions of "V4 to v5 migration guide"

Latest revision as of 12:21, 31 March 2013

Since version 5.* is a different major version than version 4.*, some things have been changed in a backwards incompatible manner. This migration guide is here to help you with the transition.

Upgrade strategy

If you use collectd in a client / server setup, we recommend to upgrade the server instance(s) first. This allows you to use the v5 upgrade target which transforms data sent by v4 clients to the new v5 layout. Once the server is running v5 and the v5 upgrade target you can easily upgrade the clients one by one.

This step-by-step migration suggestion assumes you have one or more dedicated collectd servers writing RRD files. The objective is to upgrade the servers and minimizing the resulting gap in the collected data.

• If you have only one server:
  • Set up a second server identical to your production machine. You can use the Forwarding option of the Network plugin or a packet filter such as ip_tables to send a copy of the network traffic to your clone.
• If you have two or more redundant servers:
  • Enter a downtime for one server in the monitoring and remove it from the load balancing if the server is used to create graphs. Let it still receive and handle collectd traffic, though.
• On this separate server:
  • Install the v5 daemon.
  • Adapt the configuration for plugins collecting data locally. Usually this is fairly easy on central collectd servers.
  • Configure the v5 upgrade target. Usually this is as simple as adding the following block to your collectd configuration:

LoadPlugin target_v5upgrade
PreCacheChain "PreCache"
<Chain "PreCache">
  Target "v5upgrade"
</Chain>

• • (Re)start the daemon.
  • Look for files that are no longer upgraded and rename them or split them up. Look at #Migrating existing RRD files for details on this. Possibly rsync / scp the RRD files and run the migration script multiple times until you are satisfied with the migrated data.
• On your original server / For each other server:
  • Install the v5 daemon.
  • Copy the v5 config and adapt hostname / IP-addresses etc.
  • Restart the daemon.
  • rsync / scp the migrated files to get a clean data set without gaps.

Migrating the configuration

This section details which changes you need to make to the configuration file.

FQDNLookup

The default value for FQDNLookup has been changed. It used to be disabled by default and it's now enabled by default. You can use "FQDNLookup false" to configure the old default behavior. Enabling this option is recommended though.

Apache plugin

The old configuration was suited for one webserver only. The new configuration (added in version 4.7) puts the same configuration options into <Instance /> blocks.

<Plugin "apache">
  
  URL "http://localhost/mod_status?auto"
  
</Plugin>

<Plugin "apache">
  <Instance "">
    URL "http://localhost/mod_status?auto"
  </Instance>
</Plugin>

The string argument of the Instance blocks is used as the plugin instance. The example above used an empty string ("") as the instance name to get the same behavior the previous version (without Instance blocks) used.

Versions 4.7 through 4.10 included backwards compatibility code for configurations without Instance blocks. This code has been removed in version 5.0.

HDDTemp plugin

The TranslateDevicename configuration option has been removed.

For backwards compatibility reasons, the default behavior was to translate device names such as "/dev/sda" to major and minor device numbers, e.g. "8-0". This translation could be disabled by setting TranslateDevicename to false.

The new behavior is to not translate device names, i.e. behave as if the option was set to false.

<Plugin "hddtemp">
  … other options …
  TranslateDevicename false
</Plugin>

<Plugin "hddtemp">
  … other options …
  
</Plugin>

<Plugin "hddtemp">
  … other options …
  TranslateDevicename true
</Plugin>

<Plugin "hddtemp">
  … other options …
  
</Plugin>

If you used TranslateDevicename true or used the default behavior, you need to move the RRD files or rename the data by other means. If you used the recommended setting false, you only need to remove this line from the config.

MySQL plugin

In order to support collecting information from multiple MySQL instances, the configuration options have been moved into <Database /> blocks.

<Plugin "mysql">
  
  Host "localhost"
  User "username"
  Password "password"
  
</Plugin>

<Plugin "mysql">
  <Database "">
    Host "localhost"
    User "username"
    Password "password"
  </Database>
</Plugin>

The string argument of the Database block is used as plugin instance. The example above used an empty string ("") to mimic the old behavior.

Please note that the old behavior was to use the globally configured hostname for all values submitted by the MySQL plugin. This changes with the Database blocks: When the hostname is not set, an empty string ("") or "localhost", the globally defined hostname will be used as before. If the option is set to any other string, it will be used in the dispatched data, too. You may need to rename existing data accordingly.

Network plugin

The default buffer size of the Network plugin has been increased from 1024 byte to 1452 byte. If you send data from a v5 client to a v4 server, not all data sent by the client will be received, leading to gaps in the graphs.

For this and other reasons we strongly recommend to upgrade the server(s) first. If this is not an option for you, you should use the MaxPacketSize option (since version 4.8) on the server to increase the receive buffer size.

PostgreSQL plugin

The PostgreSQL plugin has been changed so its configuration is as close as possible to the configuration of the DBI and Oracle plugins. The "new" configuration options are available since version 4.6, so chances are you are already using them. Only legacy code has been removed.

The changes are:

• The Query option has been replaced by the Statement option.
• The Column option has been replaced with Result blocks.
• The MinPGVersion and MaxPGVersion options have been replaced with the MinVersion and MaxVersion options.

<Plugin "postgresql">
  <Query "magic_qry">
    Query "SELECT magic_col FROM wizard;"
    Column "gauge" "magic_inst"
    
    
    
    
  </Query>
</Plugin>

<Plugin "postgresql">
  <Query "magic_qry">
    Statement "SELECT magic_col FROM wizard;"
    <Result>
      Type "gauge"
      InstancePrefix "magic_inst"
      ValuesFrom "magic_col"
    </Result>
  </Query>
</Plugin>

Migrating existing RRD files

The definition and use of a couple of types has been changed, which means that the appropriate RRD files need to be converted.

This section lists all the changes that have been made to RRD files for reference. It is, however, not recommended that you migrate the RRD files “by hand”. Use the contrib/migrate-4-5.px script instead. For details see #Migration script below

The changes fall into three categories:

Different data source name

The convention was and is that data sets with only one data source should use the name "value" for this data source. This has been fixed for the following types:

• absolute
• apache_bytes
• apache_connections
• apache_idle_workers
• apache_requests
• apache_scoreboard
• conntrack
• contextswitch
• delay
• entropy
• file_size
• frequency
• frequency_offset
• http_request_methods
• http_requests
• http_response_codes
• percent
• ping
• records
• time_dispersion
• timeleft
• time_offset
• users
• virt_cpu_total
• virt_vcpu

You can change the data source name using rrdtune(1):

$ rrdtool tune $FILE --data-source-rename old_name:value

Data source type COUNTER converted to DERIVE

Traditionally, collectd supported COUNTER and GAUGE data sources. However, the DERIVE data source type is more appropriate in 99 % of all cases. Version 5 therefore uses DERIVE in all types. If you need the special behavior of COUNTER data sources, you will need to tune the RRD file by hand. The following is a list of affected types:

• apache_bytes
• apache_requests
• arc_counts
• arc_l2_bytes
• ath_stat
• compression
• connections
• cpu
• current
• df
• disk_merged
• disk_octets
• disk_ops_complex
• disk_ops
• disk_time
• dns_answer
• dns_notify
• dns_octets
• dns_opcode
• dns_qtype
• dns_query
• dns_question
• dns_rcode
• dns_reject
• dns_request
• dns_resolver
• dns_response
• dns_transfer
• dns_update
• dns_zops
• fork_rate
• fscache_stat
• http_request_methods
• http_requests
• http_response_codes
• if_collisions
• if_dropped
• if_errors
• if_multicast
• if_octets
• if_packets
• if_rx_errors
• if_tx_errors
• io_octets
• io_packets
• ipt_bytes
• ipt_packets
• irq
• memcached_command
• memcached_octets
• memcached_ops
• mysql_commands
• mysql_handler
• mysql_locks
• mysql_log_position
• mysql_octets
• nfs_procedure
• nginx_requests
• node_octets
• node_stat
• operations
• pg_blks
• pg_n_tup_c
• pg_scan
• pg_xact
• protocol_counter
• ps_code
• ps_cputime
• ps_data
• ps_pagefaults
• serial_octets
• swap_io
• virt_cpu_total
• virt_vcpu
• vmpage_action
• vmpage_faults
• vmpage_io

You can change the data source type using rrdtune(1):

$ rrdtool tune $FILE --data-source-type value:DERIVE --minimum value:0 --maximum value:U

Multiple data sources have been split up

A few types have multiple data sources which really don't belong in the same data set. Sometimes, this seemed reasonable at first but proved to inflexible. The DF plugin for example used one type with "used" and "free" partition space, which seems okay at first sight. If you want to collect "reserved snapshot space" or "duplicate blocks", however, this seemingly simple metric becomes a lot more complicated.
Version 5.0 corrects a couple of such mistakes. Affected plugins / types are:

Plugin- / type-instance changes

The usage of the plugin- and type-instance fields has not been 100 % consequent. To clean up this misuse and avoid confusion about the concept in the future, the following plugins have been changes to stick to the recommended practice:

• The DF plugin used to store the mount point in the type instance. It now uses the plugin instance instead.
• The Interface plugin used to store the interface name in the type instance. It now uses the plugin instance instead.

Migration script

You should use the contrib/migrate-4-5.px script to convert RRD files to the new layout. It takes a two-stepped approach in order to let you control every aspect of the conversion:

 # The migration script will scan your RRD directory and create
 # a shell script to do the actual conversion of files.
 contrib/migrate-4-5.px >/tmp/rrd_migrate.sh
 
 # Check that the generated shell script doesn't do
 # something stupid or unwanted.
 $EDITOR /tmp/rrd_migrate.sh
 
 # Do the actual migration by executing the shell script.
 sh /tmp/rrd_migrate.sh

!!! ATTENTION !!!

At the time of use by me, the migrate-4-5.px from collectd-5.2.1 
(installed from Debian package version 5.2.1-1) does not split 
mysql_qcache and mysql_threads into new types, and also does not know 
about arc_* types too (which was not used by me,however).
Please note to this while upgrading and update script for your needs.

Other

Exec plugin

Since the Exec plugin was written before notifications were introduced in collectd, it used to be possible to omit the PUTVAL command in the output generated by scripts. This backwards compatibility has been removed, i.e. you have to add "PUTVAL " to your script's output.

Perl plugin

• The plugin_flush_all and plugin_flush_one functions have been removed. Use plugin_flush (available since version 4.5) instead.
• Early versions of the Perl plugin expected the type as a separate argument to plugin_dispatch_values. Since version 4.4 the preferred way to pass the type was inside the value list hashref. Code for the backwards compatibility has been removed.