bityard Blog

2013-04-15 // Nagios Monitoring - IBM TS3500 Tape Library

We use several IBM TS3500 (3584) tape libraries in our datacenters to store backup copies from several IBM TSM (Tivoli Storage Manager) instances. Recently i wrote a Nagios plugin to monitor the IBM TS3500 tape libraries. In order to run the Nagios plugin, you need to have SNMP activated on the tape library. Also, a network connection from the Nagios system to the TS3500 device on port UDP/161 must be allowed.

The whole setup for monitoring IBM TS3500 tape libraries setup looks like this:

  1. Enable SNMP queries on the service processor of the TS3500. Login to the WebGUI and navigate to: 

    -> Access
   -> SNMP Settings
      -> SNMP Version: v2
      -> Check selection: "Enable SNMP Trap"
      -> Check selection: "Enable SNMP Requests"
         -> Apply
   -> SNMP System Data
      -> Community Names
         -> Read Name: <Enter the SNMP community string to access the TS3500>
      -> System Group Information
         -> Fill the form with your information
         -> Apply

    Verify the port UDP/161 on the TS3500 device can be reached from the Nagios system.

  2. Optional: Enable SNMP traps to be sent to the Nagios system from the service processor of the TS3500. This requires SNMPD and SNMPTT to be already setup on the Nagios system. Login to the WebGUI and navigate to: 

    -> Access
   -> SNMP Destinations
         -> From the drop-down menu select: "Create"
            -> Go
               -> <Enter IP and UDP port of the Nagios systems SNMPD>
                  -> OK
   -> SNMP System Data
      -> Community Names
         -> Trap Name: <Enter the SNMP community string to access the SNMPD on the Nagios system>
         -> Apply

    Verify the port UDP/162 on the Nagios system can be reached from the TS3500 devices.

  3. Download the Nagios plugin check_ts3500.sh and place it in the plugins directory of your Nagios system, in this example /usr/lib/nagios/plugins/: 

    $ mv -i check_ts3500.sh /usr/lib/nagios/plugins/
$ chmod 755 /usr/lib/nagios/plugins/check_ts3500.sh

  4. Define the following Nagios commands. In this example this is done in the file /etc/nagios-plugins/config/check_ts3500.cfg: 

    # check IBM TS3500 tape library - chassis status
define command {
        command_name    check_ts3500_chassis
        command_line    $USER1$/check_ts3500.sh -H $HOSTADDRESS$ -C chassis
}
# check IBM TS3500 tape library - changer status
define command {
        command_name    check_ts3500_changer
        command_line    $USER1$/check_ts3500.sh -H $HOSTADDRESS$ -C changer
}
# check IBM TS3500 tape library - device status
define command {
        command_name    check_ts3500_devices
        command_line    $USER1$/check_ts3500.sh -H $HOSTADDRESS$ -C devices
}
# check IBM TS3500 tape library - FC status
define command {
        command_name    check_ts3500_fc
        command_line    $USER1$/check_ts3500.sh -H $HOSTADDRESS$ -C fc
}
# check IBM TS3500 tape library - media status
define command {
        command_name    check_ts3500_media
        command_line    $USER1$/check_ts3500.sh -H $HOSTADDRESS$ -C media
}
# check IBM TS3500 tape library - SCSI status
define command {
        command_name    check_ts3500_scsi
        command_line    $USER1$/check_ts3500.sh -H $HOSTADDRESS$ -C scsi
}
# check IBM TS3500 tape library - security status
define command {
        command_name    check_ts3500_security
        command_line    $USER1$/check_ts3500.sh -H $HOSTADDRESS$ -C security
}

    If the output of the Nagios plugin – especially for devices, fc and scsi – is to verbose for your needs, just add a -q to the above command_line definitions.

  5. Verify that a generic check command for a running SNMPD is already present in your Nagios configuration. If not add a new check command like this: 

    define command {
        command_name    check_snmpd
        command_line    $USER1$/check_snmp -H $HOSTADDRESS$ -o .1.3.6.1.2.1.1.3.0 -P 1 -C public -t 30
}

  6. Define a group of services in your Nagios configuration to be checked for each TS3500 device: 

    # check snmpd
define service {
    use                     generic-service
    hostgroup_name          tapelib-ts3500
    service_description     Check_SNMPD
    check_command           check_snmpd
}
# check IBM TS3500 tape library - chassis status
define service {
    use                     generic-service
    hostgroup_name          tapelib-ts3500
    service_description     Check_TS3500_TL_Chassis
    check_command           check_ts3500_chassis
}
# check IBM TS3500 tape library - changer status
define service {
    use                     generic-service
    hostgroup_name          tapelib-ts3500
    service_description     Check_TS3500_TL_Changer
    check_command           check_ts3500_changer
}
# check IBM TS3500 tape library - devices status
define service {
    use                     generic-service-pnp
    hostgroup_name          tapelib-ts3500
    service_description     Check_TS3500_TL_Devices
    check_command           check_ts3500_devices
}
# check IBM TS3500 tape library - fc status
define service {
    use                     generic-service
    hostgroup_name          tapelib-ts3500
    service_description     Check_TS3500_TL_FC
    check_command           check_ts3500_fc
}
# check IBM TS3500 tape library - media status
define service {
    use                     generic-service-pnp
    hostgroup_name          tapelib-ts3500
    service_description     Check_TS3500_TL_Media
    check_command           check_ts3500_media
}
# check IBM TS3500 tape library - SCSI status
define service {
    use                     generic-service
    hostgroup_name          tapelib-ts3500
    service_description     Check_TS3500_TL_SCSI
    check_command           check_ts3500_scsi
}
# check IBM TS3500 tape library - security status
define service {
    use                     generic-service
    hostgroup_name          tapelib_ts3500
    service_description     Check_TS3500_TL_Security
    check_command           check_ts3500_security
}

    Replace generic-service with your Nagios service template. Replace generic-service-pnp with your Nagios service template that has performance data processing enabled.

  7. Define a service dependency to run the above checks Check_TS3500_TL_* only if the Check_SNMPD was run successfully: 

    # IBM TS3500 SNMPD dependencies
define servicedependency{
    hostgroup_name                  tapelib-ts3500
    service_description             Check_SNMPD
    dependent_service_description   Check_TS3500_TL_.*
    execution_failure_criteria      c,p,u,w
    notification_failure_criteria   c,p,u,w
}

  8. Define a host in your Nagios configuration for the service processor of each TS3500 device. In this example its named ts3500: 

    define host {
    use         tapelib
    host_name   ts3500
    alias       TS3500 Tape Library
    address     10.0.0.1
    parents     parent_lan
}

    Replace tapelib with your Nagios host template for tape library storage devices. Adjust the address and parents parameters according to your environment.

  9. Define a hostgroup in your Nagios configuration for all TS3500 devices. In this example it is named tapelib-ts3500. The above checks are run against each member of the hostgroup: 

    define hostgroup {
    hostgroup_name  tapelib-ts3500
    alias           Tape Libraries (TS3500)
    members         ts3500
}

  10. Run a configuration check and if successful reload the Nagios process: 

    $ /usr/sbin/nagios3 -v /etc/nagios3/nagios.cfg
$ /etc/init.d/nagios3 reload

The new hosts and services should soon show up in the Nagios web interface.

If the optional step number 2 in the above list was done, SNMPTT also needs to be configured to be able to understand the incoming SNMP traps from TS3500 devices. This can be achieved by the following steps:

  1. Download the SNMP MIB archive 3584-MIB.zip or 3584-MIB.tar, unpack it and transfer the files 3584v1.mib, 3584v2c.mib and SNIA-SML-rev1-21.mib to the Nagios server.

  2. Convert the SNMP MIB definitions in all three files into a format that SNMPTT can understand. 

    $ /opt/snmptt/snmpttconvertmib --in=MIB/3584v1.mib --out=/opt/snmptt/conf/snmptt.conf.ibm-ts3500_v1
...
Done

Total translations:        124
Successful translations:   124
Failed translations:       0

$ /opt/snmptt/snmpttconvertmib --in=MIB/3584v2c.mib --out=/opt/snmptt/conf/snmptt.conf.ibm-ts3500_v2c
...
Done

Total translations:        124
Successful translations:   124
Failed translations:       0

$ /opt/snmptt/snmpttconvertmib --in=MIB/SNIA-SML-rev1-21.mib --out=/opt/snmptt/conf/snmptt.conf.ibm-ts3500_SNIA-SML-rev1-21
...
Done

Total translations:        13
Successful translations:   13
Failed translations:       0

  3. The trap severity settings should be pretty reasonable by default, but you can edit them according to your requirements with: 

    $ vim /etc/snmptt/conf.d/snmptt.conf.ibm-ts3500_v1
$ vim /etc/snmptt/conf.d/snmptt.conf.ibm-ts3500_v2c
$ vim /etc/snmptt/conf.d/snmptt.conf.ibm-ts3500_SNIA-SML-rev1-21

  4. Add the new configuration file to be included in the global SNMPTT configuration and restart the SNMPTT daemon: 

    $ vim /opt/snmptt/snmptt.ini

...
[TrapFiles]
snmptt_conf_files = <<END
...
/etc/snmptt/conf.d/snmptt.conf.ibm-ts3500_v1
/etc/snmptt/conf.d/snmptt.conf.ibm-ts3500_v2c
/etc/snmptt/conf.d/snmptt.conf.ibm-ts3500_SNIA-SML-rev1-21
...
END

$ /etc/init.d/snmptt reload

  5. Download the Nagios plugin check_snmp_traps.sh and place it in the plugins directory of your Nagios system, in this example /usr/lib/nagios/plugins/: 

    $ mv -i check_snmp_traps.sh /usr/lib/nagios/plugins/
$ chmod 755 /usr/lib/nagios/plugins/check_snmp_traps.sh

  6. Define the following Nagios command to check for SNMP traps in the SNMPTT database. In this example this is done in the file /etc/nagios-plugins/config/check_snmp_traps.cfg: 

    # check for snmp traps
define command {
    command_name    check_snmp_traps
    command_line    $USER1$/check_snmp_traps.sh -H $HOSTNAME$:$HOSTADDRESS$ -u <user> -p <pass> -d <snmptt_db>
}

    Replace user, pass and snmptt_db with values suitable for your SNMPTT database environment.

  7. Add another service in your Nagios configuration to be checked for each TS3500 device: 

    # check snmptraps
define service{
    use                     generic-service
    hostgroup_name          tapelib-ts3500
    service_description     Check_SNMP_traps
    check_command           check_snmp_traps
}

  8. Optional: Define a serviceextinfo to display a folder icon next to the Check_SNMP_traps service check for each TS3500 device. This icon provides a direct link to the SNMPTT web interface with a filter for the selected host: 

    define  serviceextinfo {
    hostgroup_name          tapelib-ts3500
    service_description     Check_SNMP_traps
    notes                   SNMP Alerts
    #notes_url               http://<hostname>/nagios3/nagtrap/index.php?hostname=$HOSTNAME$
    #notes_url               http://<hostname>/nagios3/nsti/index.php?perpage=100&hostname=$HOSTNAME$
}

    Uncomment the notes_url depending on which web interface (nagtrap or nsti) is used. Replace hostname with the FQDN or IP address of the server running the web interface.

  9. Run a configuration check and if successful reload the Nagios process: 

    $ /usr/sbin/nagios3 -v /etc/nagios3/nagios.cfg
$ /etc/init.d/nagios3 reload

  10. Optional: If you're running PNP4Nagios v0.6 or later to graph Nagios performance data, you can use the check_ts3500_devices.php and check_ts3500_media.php PNP4Nagios template to beautify the graphs. Download the PNP4Nagios templates check_ts3500_devices.php and check_ts3500_media.php and place it in the PNP4Nagios template directory, in this example /usr/share/pnp4nagios/html/templates/: 

    $ mv -i check_ts3500_devices.php check_ts3500_media.php /usr/share/pnp4nagios/html/templates/
$ chmod 644 /usr/share/pnp4nagios/html/templates/check_ts3500_devices.php
$ chmod 644 /usr/share/pnp4nagios/html/templates/check_ts3500_media.php

    The following image shows an example of what the PNP4Nagios graphs look like for a IBM TS3500 unit:

    PNP4Nagios graphs for the devices in a IBM TS3500 unit

    PNP4Nagios graphs for the media in a IBM TS3500 unit

    Unfortunately there are very few values presented by the SNMP MIB to be graphed.

All done, you should now have a complete Nagios-based monitoring solution for your IBM TS3500 devices.

4 Comments | 2013-04-15 written by Frank Fegert (XING) | Permanentlink
Tags:
This website uses cookies. By using the website, you agree with storing cookies on your computer. Also you acknowledge that you have read and understand our Privacy Policy. If you do not agree leave the website. More information about cookies