bityard Blog

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision
Previous revision
2012:10:02:nagios_monitoring_emc_clariion [2012/10/02 11:58] – created Frank Fegert2012:10:02:nagios_monitoring_emc_clariion [2013/11/21 17:13] (current) Frank Fegert
Line 3: Line 3:
 Some time ago i wrote a - rather crude - Nagios plugin to monitor EMC Clariion storage arrays, specifically the CX4-120 model. The plugin isn't very pretty, but it'll do in a pinch ;-) In order to run it, you need to have the command line tools ''navicli'' and ''naviseccli'' - which are provided by EMC - installed on the Nagios system and SNMP activated on the SPs of the array. A network connection from the Nagios system to the Clariion device on ports TCP/6389 and UDP/161 must be allowed. Some time ago i wrote a - rather crude - Nagios plugin to monitor EMC Clariion storage arrays, specifically the CX4-120 model. The plugin isn't very pretty, but it'll do in a pinch ;-) In order to run it, you need to have the command line tools ''navicli'' and ''naviseccli'' - which are provided by EMC - installed on the Nagios system and SNMP activated on the SPs of the array. A network connection from the Nagios system to the Clariion device on ports TCP/6389 and UDP/161 must be allowed.
  
-Since the Nagios server in my setup runs on Debian/PPC on a IBM Power LPAR and there is no native Linux/PPC version of ''navicli'' or ''naviseccli'', i had to run those tools through the [[http://en.wikipedia.org/wiki/PowerVM_Lx86|IBM PowerVM LX86]] emulator. If it is run in a x86 environment, the variable ''RUNX86'' has to be set to an empty value.+Since the Nagios server in my setup runs on Debian/PPC on a IBM Power LPAR and there is no native Linux/PPC version of ''navicli'' or ''naviseccli'', i had to run those tools through the [[wp>PowerVM_Lx86|IBM PowerVM LX86]] emulator. If the plugin is run in a x86 environment, the variable ''RUNX86'' has to be set to an empty value.
  
 The whole setup looks like this: The whole setup looks like this:
Line 24: Line 24:
 </code> </code>
 Verify the port **UDP/161** on the Clariion device can be reached from the Nagios system. < Verify the port **UDP/161** on the Clariion device can be reached from the Nagios system. <
-  - **Optional**: Enable sending of SNMP traps to the Nagios system on each of the Clariion devices SP. This requires a already setup SNMPD and SNMPTT on the Nagios system. Login to NaviSphere and for each Clariion device navigate to:+  - **Optional**: Enable SNMP traps to be sent to the Nagios system on each of the Clariion devices SP. This requires SNMPD and SNMPTT to be already setup on the Nagios system. Login to NaviSphere and for each Clariion device navigate to:
 <code> <code>
 -> Monitors tab -> Monitors tab
Line 34: Line 34:
                   -> <Select the events you're interested in>                   -> <Select the events you're interested in>
                -> SNMP tab                -> SNMP tab
-                  -> <Enter IP of the Nagios system and the SNMPDs community string+                  -> <Enter IP of the Nagios system and the SNMPDs community string>
                -> Apply or OK                -> Apply or OK
       -> SP A       -> SP A
Line 48: Line 48:
 </code> </code>
 Verify the port **UDP/162** on the Nagios system can be reached from the Clariion devices. < Verify the port **UDP/162** on the Nagios system can be reached from the Clariion devices. <
-  - Install ''navicli'' or ''naviseccli'' on the Nagios system, in this example ''/opt/Navisphere/bin/{navicli,naviseccli}''. Verify the port **TCP/6389** on the Clariion device can be reached from the Nagios system. <+  - Install ''navicli'' or ''naviseccli'' on the Nagios system, in this example ''/opt/Navisphere/bin/navicli'' and ''/opt/Navisphere/bin/naviseccli''. Verify the port **TCP/6389** on the Clariion device can be reached from the Nagios system. <
   - Download the {{:2012:10:02:check_cx.sh|Nagios plugin check_cx.sh}} and place it in the plugins directory of your Nagios system, in this example ''/usr/lib/nagios/plugins/'':   - Download the {{:2012:10:02:check_cx.sh|Nagios plugin check_cx.sh}} and place it in the plugins directory of your Nagios system, in this example ''/usr/lib/nagios/plugins/'':
-<code+<cli
-mv -i check_cx.sh /usr/lib/nagios/plugins/ +mv -i check_cx.sh /usr/lib/nagios/plugins/ 
-chmod 755 /usr/lib/nagios/plugins/check_cx.sh +chmod 755 /usr/lib/nagios/plugins/check_cx.sh 
-</code> <+</cli> <
   - Adjust the plugin settings according to your environment. Edit the following variable assignments:   - Adjust the plugin settings according to your environment. Edit the following variable assignments:
 <code> <code>
Line 96: Line 96:
   - Define a group of services in your Nagios configuration to be checked for each Clariion device:   - Define a group of services in your Nagios configuration to be checked for each Clariion device:
 <code> <code>
-# check snmptraps 
-define service { 
-    use                     generic-service 
-    hostgroup_name          cx4-ctrl 
-    service_description     Check_SNMP_traps 
-    check_command           check_snmp_traps 
-} 
 # check snmpd # check snmpd
 define service { define service {
Line 145: Line 138:
     check_command           check_cx_sp     check_command           check_cx_sp
 } }
-</code> <+</code> 
 +Replace ''generic-service'' with your Nagios service template. <
   - Define a service dependency to run the check ''Check_CX_status'' only if the ''Check_SNMPD'' was run successfully:   - Define a service dependency to run the check ''Check_CX_status'' only if the ''Check_SNMPD'' was run successfully:
 <code> <code>
Line 173: Line 167:
     parents     parent_lan     parents     parent_lan
 } }
-</code> <+</code> 
 +Replace ''disk'' with your Nagios host template for storage devices. Adjust the ''address'' and ''parents'' parameters according to your environment. <
   - Define a hostgroup in your Nagios configuration for all Clariion devices. In this example it is named ''cx4-ctrl''. The above checks are run against each member of the hostgroup:   - Define a hostgroup in your Nagios configuration for all Clariion devices. In this example it is named ''cx4-ctrl''. The above checks are run against each member of the hostgroup:
 <code> <code>
Line 183: Line 178:
 </code> < </code> <
   - Run a configuration check and if successful reload the Nagios process:   - Run a configuration check and if successful reload the Nagios process:
 +<cli>
 +$ /usr/sbin/nagios3 -v /etc/nagios3/nagios.cfg
 +$ /etc/init.d/nagios3 reload
 +</cli> <
 +
 +The new hosts and services should soon show up in the Nagios web interface.
 +
 +If the optional step number 2 in the above list was done, SNMPTT also needs to be configured to be able to understand the incoming SNMP traps from Clariion devices. This can be achieved by the following steps:
 +  - Convert the EMC Clariion SNMP MIB definitions in ''emc-clariion.mib'' into a format that SNMPTT can understand.
 +<cli>
 +$ /opt/snmptt/snmpttconvertmib --in=MIB/emc-clariion.mib --out=/opt/snmptt/conf/snmptt.conf.emc-clariion
 +
 +...
 +Done
 +
 +Total translations:        5
 +Successful translations:   5
 +Failed translations:       0
 +</cli> <
 +  - Edit the trap severity according to your requirements, e.g.:
 +<cli>
 +$ vim /opt/snmptt/conf/snmptt.conf.emc-clariion
 +
 +...
 +EVENT EventMonitorTrapWarn .1.3.6.1.4.1.1981.0.4 "Status Events" Warning
 +...
 +EVENT EventMonitorTrapFault .1.3.6.1.4.1.1981.0.6 "Status Events" Critical
 +...
 +</cli> <
 +  - **Optional**: Apply the following patch to the configuration to reduce the number of false positives:
 +<code diff snmptt.conf.emc-clariion>
 +diff -u snmptt.conf.emc-clariion_1 snmptt.conf.emc-clariion
 +--- snmptt.conf.emc-clariion.orig   2012-10-02 19:04:15.000000000 +0200
 ++++ snmptt.conf.emc-clariion        2009-07-21 10:28:44.000000000 +0200
 +@@ -54,8 +54,31 @@
 + #
 + #
 + #
 ++EVENT EventMonitorTrapError .1.3.6.1.4.1.1981.0.5 "Status Events" Major
 ++FORMAT An Error EventMonitorTrap is generated in $*
 ++MATCH MODE=and
 ++MATCH $*: !(( Power [AB] : Faulted|Disk Array Enclosure .Bus [0-9] Enclosure [0-9]. is faulted))
 ++MATCH $X: !(0(2:5|3:0|3:1)[0-9]:[0-9][0-9])
 ++SDESC
 ++An Error EventMonitorTrap is generated in
 ++response to a user-specified event.
 ++Details can be found in Variables data.
 ++Variables:
 ++  1: hostName
 ++  2: deviceID
 ++  3: eventID
 ++  4: eventText
 ++  5: storageSystem
 ++EDESC
 ++#
 ++# Filter and ignore the following events
 ++#   02:50 - 03:15 Navisphere Power Supply Checks
 ++#
 + EVENT EventMonitorTrapError .1.3.6.1.4.1.1981.0.5 "Status Events" Normal
 + FORMAT An Error EventMonitorTrap is generated in $*
 ++MATCH MODE=and
 ++MATCH $*: (( Power [AB] : Faulted|Disk Array Enclosure .Bus [0-9] Enclosure [0-9]. is faulted))
 ++MATCH $X: (0(2:5|3:0|3:1)[0-9]:[0-9][0-9])
 + SDESC
 + An Error EventMonitorTrap is generated in
 + response to a user-specified event.
 +</code>
 +
 +The reason for this is, the Clariion performs a power supply check every friday around 3:00 am. This triggers a SNMP trap to be sent, even if the power supplies check out fine. In my opinion this behaviour is defective, but a case opened on this issue showed that EMC tends to think otherwise. Since there was very little hope for EMC to come to at least some sense, i just did the above patch to the SNMPTT configuration file. What it does is, it basically lowers the severity for all "Major" traps that are power supply related and sent around 3:00 am to "Normal". All other "Major" traps keep their original severity. <
 +  - Add the new configuration file to be included in the global SNMPTT configuration and restart the SNMPTT daemon:
 +<cli>
 +$ vim /opt/snmptt/snmptt.ini
 +
 +...
 +[TrapFiles]
 +snmptt_conf_files = <<END
 +...
 +/opt/snmptt/conf/snmptt.conf.emc-clariion
 +...
 +END
 +
 +$ /etc/init.d/snmptt reload
 +</cli> <
 +  - Download the {{:2012:10:02:check_snmp_traps.sh|Nagios plugin check_snmp_traps.sh}} and place it in the plugins directory of your Nagios system, in this example ''/usr/lib/nagios/plugins/'':
 +<cli>
 +$ mv -i check_snmp_traps.sh /usr/lib/nagios/plugins/
 +$ chmod 755 /usr/lib/nagios/plugins/check_snmp_traps.sh
 +</cli> <
 +  - Define the following Nagios command to check for SNMP traps in the SNMPTT database. In this example this is done in the file ''/etc/nagios-plugins/config/check_snmp_traps.cfg'':
 <code> <code>
-/usr/sbin/nagios3 -v /etc/nagios3/nagios.cfg +# check for snmp traps 
-/etc/init.d/nagios3 reload+define command{ 
 +    command_name    check_snmp_traps 
 +    command_line    $USER1$/check_snmp_traps.sh -H $HOSTNAME$:$HOSTADDRESS$ -u <user> -p <pass> -<snmptt_db> 
 +
 +</code> 
 +Replace ''user'', ''pass'' and ''snmptt_db'' with values suitable for your SNMPTT database environment. < 
 +  - Add another service in your Nagios configuration to be checked for each Clariion device: 
 +<code> 
 +# check snmptraps 
 +define service { 
 +    use                     generic-service 
 +    hostgroup_name          cx4-ctrl 
 +    service_description     Check_SNMP_traps 
 +    check_command           check_snmp_traps 
 +}
 </code> < </code> <
 +  - **Optional**: Define a serviceextinfo to display a folder icon next to the ''Check_SNMP_traps'' service check for each Clariion device. This icon provides a direct link to the SNMPTT web interface with a filter for the selected host:
 +<code>
 +define serviceextinfo {
 +    hostgroup_name          cx4-ctrl
 +    service_description     Check_SNMP_traps
 +    notes                   SNMP Alerts
 +    #notes_url               http://<hostname>/nagios3/nagtrap/index.php?hostname=$HOSTNAME$
 +    #notes_url               http://<hostname>/nagios3/nsti/index.php?perpage=100&hostname=$HOSTNAME$
 +}
 +</code>
 +Uncomment the ''notes_url'' depending on which web interface (nagtrap or nsti) is used. Replace ''hostname'' with the FQDN or IP address of the server running the web interface. <
 +  - Run a configuration check and if successful reload the Nagios process:
 +<cli>
 +$ /usr/sbin/nagios3 -v /etc/nagios3/nagios.cfg
 +$ /etc/init.d/nagios3 reload
 +</cli> <
  
-The new hosts and services should soon show up in the Nagios web interface.+All done, you should now have a complete Nagios-based monitoring solution for your EMC Clariion devices.