2013-07-30 // Ganglia Fibre Channel Power/Attenuation Monitoring on AIX and VIO Servers
Although usually only available upon request via IBM support, efc_power is quite the handy tool when it comes to debugging or narrowing down fibre channel link issues. It provides information about the transmit and receive, power and attenuation values for a given FC port on a AIX or VIO server. Fortunately the output of efc_power:
$ /opt/freeware/bin/efc_power /dev/fscsi2 TX: 1232 -> 0.4658 mW, -3.32 dBm RX: 10a9 -> 0.4265 mW, -3.70 dBm
is very parser-friendly, so it can very easily be read by a script for further processing. In this case further processing means a continuous Ganglia monitoring of the fibre channel transmit and receive, power and attenuation values for each FC port on a AIX or VIO server. This is accomplished by the two RPM packages ganglia-addons-aix and ganglia-addons-aix-scripts:
The package ganglia-addons-aix-scripts is to be installed on the AIX or VIO server which has the FC adapter installed. It depends on the aaa_base package for the efc_power binary and on the ganglia-addons-base package, specifically on the cronjob (/opt/freeware/etc/run_parts/conf.d/ganglia-addons.sh) defined by this package. In the context of this cronjob all avaliable scripts in the directory /opt/freeware/libexec/ganglia-addons/ are executed. For this specific Ganglia addon an iteration over all fscsi devices in the system is done and efc_power is called for each fscsi device. Devices can be excluded by assigning a regex pattern to the BLACKLIST variable in the configuration file /opt/freeware/etc/ganglia-addons/ganglia-addons-efc_power.cfg. The output of each efc_power call is parsed and via the gmetric command fed into a Ganglia monitoring system that has to be already set up.
The package ganglia-addons-aix is to be installed on the host running the Ganglia webinterface. It contains templates for the customization of the FC power and attenuation metrics within the Ganglia Web 2 interface. See the README.templates file for further installation instructions. Here are samples of the two graphs created with those Ganglia monitoring templates:
In the section “1” of the graphs, the receive attenuation on FC port fscsi2 was about -7.7 dBm, which means that of the 476.6 uW sent from the Brocade switchport:
$ sfpshow 1/10 Identifier: 3 SFP Connector: 7 LC Transceiver: 540c404000000000 200,400,800_MB/s M5,M6 sw Short_dist Encoding: 1 8B10B Baud Rate: 85 (units 100 megabaud) Length 9u: 0 (units km) Length 9u: 0 (units 100 meters) Length 50u: 5 (units 10 meters) Length 62.5u:2 (units 10 meters) Length Cu: 0 (units 1 meter) Vendor Name: BROCADE Vendor OUI: 00:05:1e Vendor PN: 57-1000012-01 Vendor Rev: A Wavelength: 850 (units nm) Options: 003a Loss_of_Sig,Tx_Fault,Tx_Disable BR Max: 0 BR Min: 0 Serial No: UAF1112600001JW Date Code: 110619 DD Type: 0x68 Enh Options: 0xfa Status/Ctrl: 0x82 Alarm flags[0,1] = 0x5, 0x40 Warn Flags[0,1] = 0x5, 0x40 Alarm Warn low high low high Temperature: 41 Centigrade -10 90 -5 85 Current: 7.392 mAmps 1.000 17.000 2.000 14.000 Voltage: 3264.9 mVolts 2900.0 3700.0 3000.0 3600.0 RX Power: -4.0 dBm (400.1 uW) 10.0 uW 1258.9 uW 15.8 uW 1000.0 uW TX Power: -3.2 dBm (476.6 uW) 125.9 uW 631.0 uW 158.5 uW 562.3 uW
only about 200 uW actually made it to the FC port fscsi2 on the VIO server. Section “2” shows even worse values during the time the FC connections and cables were checked, which basically means that the FC link was down during that time period. Section “3” shows the values after the bad cable was found and replaced. Receive attenuation on FC port fscsi2 went down to about -3.7 dBm, which means that of the now 473.6 uW sent from the Brocade switchport, 427.3 uW actually make it to the FC port fscsi2 on the VIO server.
The goal with the continuous monitoring of the fibre channel transmit and receive, power and attenuation values is to catch slowly deterioration situations early on, before they become a real issue or even a service interruption. As shown above, this can be accomplished with Ganglia and the two RPM packages ganglia-addons-aix and ganglia-addons-aix-scripts. For ad hoc checks, e.g. during the debugging of the components in a suspicious FC link, efc_power is still best to be called directly from the AIX or VIO server command line.
