Check_MK Monitoring - RMON Interface Statistics
Check_MK provides the check rmon_stats to collect and monitor Remote Network MONitoring (RMON) statistics for interfaces of network equipment. While this stock check probably works fine with Cisco network devices, it does – out of the box – not work with network equipment from other vendors like e.g. Dell PowerConnect switches. This article shows the modifications necessary to make the rmon_stats check work with non-Cisco network equipment. Along the way, other shortcomings of the stock rmon_stats are discussed and the necessary modifications to address those issues are shown.
For the impatient and TL;DR here are the enhanced versions of the rmon_stats check here along with a slightly beautified version of the accompanying PNP4Nagios template:
Enhanced version of the rmon_stats check
Slightly beautified version of the rmon_stats PNP4Nagios template
The limitation of the rmon_stats check to Cisco network devices is due to the fact that the Check_MK inventory is being limited to vendor specific OIDs in the checks snmp_scan_function. The following code snippet shows the respective lines:
- rmon_stats
check_info["rmon_stats"] = { 'check_function' : check_rmon_stats, 'inventory_function' : inventory_rmon_stats, 'service_description' : 'RMON Stats IF %s', 'has_perfdata' : True, 'snmp_info' : ('.1.3.6.1.2.1.16.1.1.1', [ # '1', # etherStatsIndex = Item '6', # etherStatsBroadcastPkts '7', # etherStatsMulticastPkts '14', # etherStatsPkts64Octets '15', # etherStatsPkts65to127Octets '16', # etherStatsPkts128to255Octets '17', # etherStatsPkts256to511Octets '18', # etherStatsPkts512to1023Octets '19', # etherStatsPkts1024to1518Octets ]), # for the scan we need to check for any single object in the RMON tree, # we choose netDefaultGateway in the hope that it will always be present 'snmp_scan_function' : lambda oid: ( oid(".1.3.6.1.2.1.1.1.0").lower().startswith("cisco") \ or oid(".1.3.6.1.2.1.1.2.0") == ".1.3.6.1.4.1.11863.1.1.3" \ ) and oid(".1.3.6.1.2.1.16.19.12.0") != None, }
By replacing the snmp_scan_function lines at the bottom of the code above with the following lines:
- rmon_stats
# if at least one interface within the RMON tree is present (the previous # "netDefaultGateway" is not present in every device implementing the RMON # MIB 'snmp_scan_function' : lambda oid: oid(".1.3.6.1.2.1.16.1.1.1.1.1") > 0,
the check will now be able to successfully inventorize the RMON representation of interfaces even on non-Cisco network equipment.
To actually execute such an inventory, there first needs to be a Check_MK configuration rule in place, which enables the appropriate code in the inventory_rmon_stats function of the check. The following code snippet shows the respective lines:
- rmon_stats
def inventory_rmon_stats(info): settings = host_extra_conf_merged(g_hostname, inventory_if_rules) if settings.get("rmon"): [...]
I guess this is supposed to be sort of a safeguard, since the number of interfaces can grow quite large in RMON and querying them can put a lot of strain on the service processor of the network device.
The configuration option to enable RMON based checks is neatly tucked away in the WATO WebGUI at:
-> Host & Service Parameters
-> Parameters for discovered services
-> Inventory - automatic service detection
-> Network Interface and Switch Port Discovery
-> Create rule in folder ...
-> Value
[x] Collect RMON statistics data
[x] Create extra service with RMON statistics data (if available for the device)
After creating a global or folder specific configuration rule, the next run of the Check_MK inventory should discover a – possibly large – number of new interfaces and create the individual service checks, one for each new interface. If not, the network devices probably has disabled the RMON statistics feature by default or has no such feature at all. Since the procedure to enable the collection and presentation of RMON statistics via SNMP on a network device is different for each vendor, one has to check with the corresponding vendor documentation.
Taking a look at the list of newly discovered interfaces and the resulting service checks reveals two other issues of the rmon_stats check.
One is, the check is rather simple in the way that it indiscriminately collects RMON interface statistics on all interfaces of a network device, regardless of the actual link state of each interface. While the RMON-MIB lacks direct information about the link state of an interface, it fortunately carries a reference to the IF-MIB for each interface. The IF-MIB in turn provides the information about the link state (ifOperStatus), which can be used to determine if RMON statistics should be collected for a particular interface.
The other issue is, the checks use of the RMON interface index (etherStatsIndex) as a base for the name of the associated service. E.g. RMON Stats IF <etherStatsIndex> in the following example output:
OK RMON Stats IF 1 OK - bcast=0 mcast=0 0-63b=5 64-127b=124 128-255b=28 256-511b=8 512-1023b=2 1024-1518b=94 octets/sec OK RMON Stats IF 10 OK - bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Stats IF 100 OK - bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Stats IF 101 OK - bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Stats IF 102 OK - bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Stats IF 103 OK - bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Stats IF 104 OK - bcast=0 mcast=0 0-63b=0 64-127b=11 128-255b=3 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Stats IF 105 OK - bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Stats IF 106 OK - bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Stats IF 107 OK - bcast=0 mcast=1 0-63b=1 64-127b=8 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Stats IF 108 OK - bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Stats IF 109 OK - bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Stats IF 11 OK - bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Stats IF 110 OK - bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Stats IF 111 OK - bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Stats IF 112 OK - bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec [...]
Unfortunately in RMON the values of the interface index etherStatsIndex are not guaranteed to be consistent across reboots of the network device, much less the addition of new or the removal of existing interfaces. The interface numbering definition according to the original MIB-II on the other hand is much stricter in this regard. Although subsequent RFCs like the IF-MIB (see section “3.1.5. Interface Numbering”) have weakened the original definition to some degree.
Another drawback of the pure RMON interface index number as an interface identifier in the service check name is that it is less descriptive than e.g. the interface description ifDescr from the IF-MIB. This makes the visual mapping of the service check to the respective interface of the network device rather tedious and error-prone.
An obvious solution to both issues would be to use the already mentioned reference to the IF-MIB for each interface and – in the process of the Check_MK inventory – check for the interface link state (ifOperStatus). The Check_MK inventory process would return only those interfaces with a value of up(1) in the ifOperStatus variable. For those particular interfaces it would also return the interface description ifDescr to be used as a service name, instead of the previsously used RMON interface index.
As a side effect, this approach unfortunately breaks the necessary mapping from the service name – now based on the IF-MIB interface description (ifDescr) instead of the previously used RMON interface index (etherStatsIndex) – back to the RMON MIB where the interface statistic values are stored. This is mainly due to the fact that the IF-MIB itself does not provide a native reference to the RMON MIB. Without such a back reference, the statistics values collected from the RMON MIB can – during normal service check runs – not distinctly be assigned to the appropriate interface. It is therefore necessary to manually implement such a back reference within the service check. In this case the solution was to store the value of the RMON interface index etherStatsIndex in the parameters section (params) of each interface service checks inventory entry. The following excerpt shows an example of such a new inventory data structure for several RMON interface service checks:
[ [...] ('rmon_stats', 'Link Aggregate 1', {'rmon_if_idx': 89}), ('rmon_stats', 'Link Aggregate 16', {'rmon_if_idx': 104}), ('rmon_stats', 'Link Aggregate 19', {'rmon_if_idx': 107}), ('rmon_stats', 'Link Aggregate 2', {'rmon_if_idx': 90}), ('rmon_stats', 'Link Aggregate 31', {'rmon_if_idx': 119}), ('rmon_stats', 'Link Aggregate 32', {'rmon_if_idx': 120}), ('rmon_stats', 'Link Aggregate 7', {'rmon_if_idx': 95}), ('rmon_stats', 'Link Aggregate 8', {'rmon_if_idx': 96}), ('rmon_stats', 'Unit: 1 Slot: 0 Port: 1 10G - Level', {'rmon_if_idx': 1}), ('rmon_stats', 'Unit: 1 Slot: 0 Port: 10 10G - Level', {'rmon_if_idx': 10}), ('rmon_stats', 'Unit: 1 Slot: 0 Port: 16 10G - Level', {'rmon_if_idx': 16}), ('rmon_stats', 'Unit: 1 Slot: 0 Port: 19 10G - Level', {'rmon_if_idx': 19}), ('rmon_stats', 'Unit: 1 Slot: 0 Port: 2 10G - Level', {'rmon_if_idx': 2}), ('rmon_stats', 'Unit: 1 Slot: 0 Port: 20 10G - Level', {'rmon_if_idx': 20}), ('rmon_stats', 'Unit: 1 Slot: 0 Port: 7 10G - Level', {'rmon_if_idx': 7}), ('rmon_stats', 'Unit: 1 Slot: 0 Port: 8 10G - Level', {'rmon_if_idx': 8}), ('rmon_stats', 'Unit: 1 Slot: 0 Port: 9 10G - Level', {'rmon_if_idx': 9}), ('rmon_stats', 'Unit: 1 Slot: 1 Port: 3 10G - Level', {'rmon_if_idx': 23}), ('rmon_stats', 'Unit: 1 Slot: 1 Port: 4 10G - Level', {'rmon_if_idx': 24}), ('rmon_stats', 'Unit: 2 Slot: 0 Port: 1 10G - Level', {'rmon_if_idx': 25}), ('rmon_stats', 'Unit: 2 Slot: 0 Port: 10 10G - Level', {'rmon_if_idx': 34}), ('rmon_stats', 'Unit: 2 Slot: 0 Port: 16 10G - Level', {'rmon_if_idx': 40}), ('rmon_stats', 'Unit: 2 Slot: 0 Port: 19 10G - Level', {'rmon_if_idx': 43}), ('rmon_stats', 'Unit: 2 Slot: 0 Port: 2 10G - Level', {'rmon_if_idx': 26}), ('rmon_stats', 'Unit: 2 Slot: 0 Port: 20 10G - Level', {'rmon_if_idx': 44}), ('rmon_stats', 'Unit: 2 Slot: 0 Port: 7 10G - Level', {'rmon_if_idx': 31}), ('rmon_stats', 'Unit: 2 Slot: 0 Port: 8 10G - Level', {'rmon_if_idx': 32}), ('rmon_stats', 'Unit: 2 Slot: 0 Port: 9 10G - Level', {'rmon_if_idx': 33}), ('rmon_stats', 'Unit: 2 Slot: 1 Port: 3 10G - Level', {'rmon_if_idx': 47}), ('rmon_stats', 'Unit: 2 Slot: 1 Port: 4 10G - Level', {'rmon_if_idx': 48}), ('rmon_stats', 'Unit: 3 Slot: 0 Port: 1 10G - Level', {'rmon_if_idx': 49}), ('rmon_stats', 'Unit: 3 Slot: 0 Port: 2 10G - Level', {'rmon_if_idx': 50}), ('rmon_stats', 'Unit: 3 Slot: 0 Port: 9 10G - Level', {'rmon_if_idx': 57}), ('rmon_stats', 'Unit: 4 Slot: 0 Port: 1 10G - Level', {'rmon_if_idx': 69}), ('rmon_stats', 'Unit: 4 Slot: 0 Port: 2 10G - Level', {'rmon_if_idx': 70}), ('rmon_stats', 'Unit: 4 Slot: 0 Port: 9 10G - Level', {'rmon_if_idx': 77}), [...] ]
The changes to the rmon_stats check, necessary to implement the features described above are shown in the following patch:
- rmon_stats.patch
--- rmon_stats.orig 2015-12-21 13:41:46.000000000 +0100 +++ rmon_stats 2016-02-02 11:18:56.789188249 +0100 @@ -34,20 +34,43 @@ settings = host_extra_conf_merged(g_hostname, inventory_if_rules) if settings.get("rmon"): inventory = [] - for line in info: - inventory.append((line[0], None)) + for line in info[1]: + rmon_if_idx = int(re.sub('.1.3.6.1.2.1.2.2.1.1.','',line[1])) + for iface in info[0]: + if int(iface[0]) == rmon_if_idx and int(iface[2]) == 1: + params = {} + params["rmon_if_idx"] = int(line[0]) + inventory.append((iface[1], "%r" % params)) return inventory -def check_rmon_stats(item, _no_params, info): - bytes = { 1: 'bcast', 2: 'mcast', 3: '0-63b', 4: '64-127b', 5: '128-255b', 6: '256-511b', 7: '512-1023b', 8: '1024-1518b' } - perfdata = [] +def check_rmon_stats(item, params, info): + bytes = { 2: 'bcast', 3: 'mcast', 4: '0-63b', 5: '64-127b', 6: '128-255b', 7: '256-511b', 8: '512-1023b', 9: '1024-1518b' } + rmon_if_idx = str(params.get("rmon_if_idx")) + if_alias = '' infotext = '' + perfdata = [] now = time.time() - for line in info: - if line[0] == item: + + for line in info[0]: + ifIndex, ifDescr, ifOperStatus, ifAlias = line + if item == ifDescr: + if_alias = ifAlias + if_index = int(ifIndex) + + for line in info[1]: + if line[0] == rmon_if_idx: + if_mib_idx = int(re.sub('.1.3.6.1.2.1.2.2.1.1.','',line[1])) + if if_mib_idx != if_index: + return (3, "RMON interface index mapping to IF interface index mapping changed. Re-run Check_MK inventory.") + + if item != if_alias and if_alias != '': + infotext = "[%s, RMON: %s] " % (if_alias, rmon_if_idx) + else: + infotext = "[RMON: %s] " % rmon_if_idx + for i, val in bytes.items(): octets = int(re.sub(' Packets','',line[i])) - rate = get_rate("%s-%s" % (item, val), now, octets) + rate = get_rate("%s-%s" % (rmon_if_idx, val), now, octets) perfdata.append((val, rate, 0, 0, 0)) infotext += "%s=%.0f " % (val, rate) infotext += 'octets/sec' @@ -58,10 +81,18 @@ check_info["rmon_stats"] = { 'check_function' : check_rmon_stats, 'inventory_function' : inventory_rmon_stats, - 'service_description' : 'RMON Stats IF %s', + 'service_description' : 'RMON Interface %s', 'has_perfdata' : True, - 'snmp_info' : ('.1.3.6.1.2.1.16.1.1.1', [ # + 'snmp_info' : [ + ( '.1.3.6.1.2.1', [ # + '2.2.1.1', # ifNumber + '2.2.1.2', # ifDescr + '2.2.1.8', # ifOperStatus + '31.1.1.1.18', # ifAlias + ]), + ('.1.3.6.1.2.1.16.1.1.1', [ # '1', # etherStatsIndex = Item + '2', # etherStatsDataSource = Interface in the RFC1213-MIB '6', # etherStatsBroadcastPkts '7', # etherStatsMulticastPkts '14', # etherStatsPkts64Octets @@ -70,10 +101,11 @@ '17', # etherStatsPkts256to511Octets '18', # etherStatsPkts512to1023Octets '19', # etherStatsPkts1024to1518Octets - ]), - # for the scan we need to check for any single object in the RMON tree, - # we choose netDefaultGateway in the hope that it will always be present - 'snmp_scan_function' : lambda oid: ( oid(".1.3.6.1.2.1.1.1.0").lower().startswith("cisco") \ - or oid(".1.3.6.1.2.1.1.2.0") == ".1.3.6.1.4.1.11863.1.1.3" \ - ) and oid(".1.3.6.1.2.1.16.19.12.0") != None, + ]), + ], + # if at least one interface within the RMON tree is present (the previous + # "netDefaultGateway" is not present in every device implementing the RMON + # MIB + 'snmp_scan_function' : lambda oid: oid(".1.3.6.1.2.1.16.1.1.1.1.1") > 0, + }
The following output shows examples of the new service check names. The ifAlias and the etherStatsIndex have been added as auxiliary information to the service check output. The ifAlias entries in the examples have – in order to protect the innocent – been redacted with xxxxxxxx though.
OK RMON Interface Link Aggregate 1 OK - [xxxxxxxx, RMON: 89] bcast=0 mcast=0 0-63b=85 64-127b=390 128-255b=136 256-511b=25 512-1023b=20 1024-1518b=407 octets/sec OK RMON Interface Link Aggregate 16 OK - [xxxxxxxx, RMON: 104] bcast=0 mcast=0 0-63b=1 64-127b=11 128-255b=3 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec UNKN RMON Interface Link Aggregate 19 UNKNOWN - RMON interface index mapping to IF interface index mapping changed. Re-run Check_MK inventory. OK RMON Interface Link Aggregate 2 OK - [xxxxxxxx, RMON: 90] bcast=0 mcast=0 0-63b=111 64-127b=480 128-255b=207 256-511b=44 512-1023b=41 1024-1518b=484 octets/sec [...] OK RMON Interface Unit: 1 Slot: 0 Port: 1 10G - Level OK - [xxxxxxxx, RMON: 1] bcast=0 mcast=0 0-63b=49 64-127b=232 128-255b=57 256-511b=14 512-1023b=4 1024-1518b=185 octets/sec OK RMON Interface Unit: 1 Slot: 0 Port: 10 10G - Level OK - [xxxxxxxx, RMON: 10] bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Interface Unit: 1 Slot: 0 Port: 16 10G - Level OK - [xxxxxxxx, RMON: 16] bcast=0 mcast=0 0-63b=0 64-127b=2 128-255b=1 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Interface Unit: 1 Slot: 0 Port: 19 10G - Level OK - [xxxxxxxx, RMON: 19] bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Interface Unit: 1 Slot: 0 Port: 2 10G - Level OK - [xxxxxxxx, RMON: 2] bcast=0 mcast=0 0-63b=62 64-127b=293 128-255b=89 256-511b=28 512-1023b=9 1024-1518b=252 octets/sec OK RMON Interface Unit: 1 Slot: 0 Port: 20 10G - Level OK - [xxxxxxxx, RMON: 20] bcast=0 mcast=0 0-63b=75 64-127b=1640 128-255b=217 256-511b=41 512-1023b=79 1024-1518b=2096 octets/sec OK RMON Interface Unit: 1 Slot: 0 Port: 7 10G - Level OK - [xxxxxxxx, RMON: 7] bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Interface Unit: 1 Slot: 0 Port: 8 10G - Level OK - [xxxxxxxx, RMON: 8] bcast=0 mcast=0 0-63b=0 64-127b=5 128-255b=1 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Interface Unit: 1 Slot: 0 Port: 9 10G - Level OK - [xxxxxxxx, RMON: 9] bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Interface Unit: 1 Slot: 1 Port: 3 10G - Level OK - [xxxxxxxx, RMON: 23] bcast=0 mcast=0 0-63b=0 64-127b=3 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Interface Unit: 1 Slot: 1 Port: 4 10G - Level OK - [xxxxxxxx, RMON: 24] bcast=0 mcast=0 0-63b=0 64-127b=3 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Interface Unit: 2 Slot: 0 Port: 1 10G - Level OK - [xxxxxxxx, RMON: 25] bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Interface Unit: 2 Slot: 0 Port: 10 10G - Level OK - [xxxxxxxx, RMON: 34] bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Interface Unit: 2 Slot: 0 Port: 16 10G - Level OK - [xxxxxxxx, RMON: 40] bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec OK RMON Interface Unit: 2 Slot: 0 Port: 19 10G - Level OK - [xxxxxxxx, RMON: 43] bcast=0 mcast=1 0-63b=1 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec [...] OK RMON Interface Unit: 4 Slot: 0 Port: 9 10G - Level OK - [xxxxxxxx, RMON: 77] bcast=0 mcast=0 0-63b=0 64-127b=0 128-255b=0 256-511b=0 512-1023b=0 1024-1518b=0 octets/sec
The enhanced version of the rmon_stats check can be downloaded here along with a slightly beautified version of the accompanying PNP4Nagios template:
Enhanced version of the rmon_stats check
Patch to enhance the original verion of the rmon_stats check
Slightly beautified version of the rmon_stats PNP4Nagios template
Patch to beautify the rmon_stats PNP4Nagios template
Finally, the following two screenshots show examples of PNP4Nagios graphs using the modified version of the PNP4Nagios template:
The sources to both, the enhanced version of the rmon_stats check and the beautified version of the rmon_stats PNP4Nagios template, can be found on GitHub in my Check_MK Plugins repository.