For obvious reasons i usually like to keep monitoring systems as independent as possible from other services (like e.g. DNS). That's why i mostly use IP addresses instead of hostnames or FQDNs when configuring hosts and services in Nagios. So a typical Nagios command configuration stanza would look like this:

define command {
    command_name    <name>
    command_line    <path>/<executable> -H $HOSTADDRESS$ <args>
}

Now, for example to check for date and time differences between the server OS and the actual time provided by our time servers, i've defined the following NRPE command on each server:

command[check_time]=<nagios plugin path>/check_ntp_time -w $ARG1$ -c $ARG2$ -H $ARG3$

On the Nagios server i've defined the following Nagios command in /etc/nagios-plugins/config/check_nrpe.cfg:

# 'check_nrpe_time' command definition
define command {
    command_name    check_nrpe_time
    command_line    $USER1$/check_nrpe -H '$HOSTADDRESS$' -c check_time -a $ARG1$ $ARG2$ $ARG3$
}

And the following Nagios service definitions are assigned to the various server groups:

# check_nrpe_time: ntp time check to NTP-1
define service {
    ...
    service_description     Check_time_NTP-1
    check_command           check_nrpe_time!<warn level>!<crit level>!10.1.1.1
}

# check_nrpe_time: ntp time check to NTP-2
define service {
    ...
    service_description     Check_time_NTP-2
    check_command           check_nrpe_time!<warn level>!<crit level>!10.1.1.2
}

The help output of check_ntp_time reads:

 ...
 -H, --hostname=ADDRESS
    Host name, IP Address, or unix socket (must be an absolute path)
 ...

which i believed to implicitly mean that the Nagios plugin knows how to differenciate between hostnames and IP addresses and behave accordingly. Unfortunately this assumption turned out to be wrong.

Recently our support group responsible for the MS ADS had trouble with the DNS service provided by the domain controllers. Probably due to a data corruption, the DNS service couldn't resolve hostnames for some, but not all, of our internally used domains anymore. Unfortunately, the time servers mentioned above are located in a domain that was affected by this issue. As a result the above checks started to fail, although they were thought to be configured to use IP addresses.

The usual head scratching, debugging, testing and putting the pieces together ensued. After looking through the Nagios Plugins source code it became clear that hostnames and IP addresses aren't treated different at all, getaddrinfo() is called indiscriminately. Since in case of an IP address this is not only unnecessary, but might also not be what was actually intended, i wrote a patch nagios-plugins_do_not_resolve_IP_address.patch for the current nagios-plugins-1.4.16 sources to fix this issue.

Leave a comment…