2013-04-21 // TCP Fast Open (TFO) support for AIX
Recently i read a quite interesting article on TCP Fast Open (TFO) support in current Linux kernel versions. The approach itself isn't really that new, but with a heavyweight like Google backing the renewed effort things seem to get more traction this time around. Although the effort to get TFO recognized as an official IETF draft is still in the experimental state i thought this would be a very nice feature to have in upcoming versions of the IBM AIX operating system. To determine if there are any plans or even a PoC implementation already in progress i opened a PMR with IBM. After some back and forth with first and second level support and AIX development – of which i'll spare you all the gory details – the digest was:
As this is in experimental state, […] this won't yet be implemented. First an official RFC would be necessary.
While i very well understand the necessity of standardization on the protocol level in an internetworked IT world, i was rather disappointed by the brush off response and really would've appreciated a bit more of a “can do” attitude. Is there really no CS student interning or an entry level developer being trained at IBM to whom this task could have been given for a PoC implementation? Not to mention the fact that i was offering my spare time to implement and test a new feature that their product would benefit from.
I think this case is a very good example of several issues:
open vs. closed source: In a closed source environment the degree of influence you have on the implementation of your individual and possibly rather obscure wish for a certain feature is very limited. Factors like available resources on vendors side, wide-spread use case, possibly large user base, chance for generation of additional revenue, etc. can contribute to the implementation of your feature, but they are to a large degree out of your direct control.
In an open source environment on the other hand you can just enhance the code with the feature you need – or hire someone to do so, if you're personally lacking the required skill set. The enhancements can flow back to the upstream version if accepted and deemed generally useful by the original developer. But even if they aren't you can keep a local fork or patch set with the enhancements you see fit.
Generally speaking, with open source the focus shifts from the “if” you can get certain features to the question of “how” and “when”. IMHO the latter results in a much more constructive, fruitful and generally positive discussion with your peers.
slow giant vs. agil midget: Large companies like IBM, Oracle, Microsoft, SAP have a rather hard time dealing with change, agility and flexibility issues. Granted, compared to smaller, more agile companies they are certainly much harder – but not impossible (see the current HP-debakel) – to sink. But due to the ever increasing internal overhead, the bigger they get, the more time they spend dealing with anything else but their primary line of work, making them harder to steer and more susceptible to a rapidly changing environment. In the current fast paced IT world this usually means, that a considerable amount of real innovation happens at smaller companies. With their hands on mentality, flat hierarchies, quick decision-making, low process overhead and high degree of motivation, they are better prepared for a rapidly changing environment and can utilize this with a shorter time to market for their products. More often than not those agil incubators for innovation get gobbled up by a slow giant (e.g. Sun vs. MySQL, IBM vs. XIV, EMC vs. VMware, etc.) which hopes to rejuvenate or reinvent itself in the process. Unfortunately this process is also usually accompanied by a considerable clash of cultures, followed by a brain-drain and any other kind of general digestion problems. Instead of catching some of that innovative virus, the slow giant diverts energy to complete the assimilation of the new aquisition by imposing every facet of its cumbersome nature onto it, thus becoming even slower and more bloated in the process.
As an example, I recently talked to an ex-IBMer who had worked in TSM development for about 10 years before going into business for himself. He described his typical work week as 90% meetings and 10% “actual developer work”. The 10% included actual coding and related work as well as work on second and third level support cases. This means that 90% of a developers time and manpower are directed towards keeping a bloated internal environment afloat that probably adds far less to the actual quality or marketability of the product than the remaining 10% of time and manpower. I figure similar numbers apply to any other IT-giant.
process-oriented vs. customer/community-oriented: Recently i read a rather nice article by The Registers Trevor Pott on the subject, that made some very interesting points on the subject of traditional vs. community based marketing. Although i personally despise EMC with every fiber of my heart, i still admire them for the bold decision to keep VMware a seperately run entity. As described in the article, VMware has a very good culture of community management, which is very sensitive to the atmosphere in the user community and to the needs of their customers. People inside VMware running this, seem to have the authority to initiate development of new and requested features or to bump development in the right direction. IBM on the other hand only tries to mimic such a community by its developerWorks, but a community management authorized to actually initiate changes in products seems to be missing. The IBM culture is in that respect still very much oriented towards its hierarchy of sales, revenue and traditional marketing. If you are a prospective customer and IBM sales has to compete with other vendors, you can get any product feature developed you can possibly dream of, because IBM sales has a strong position against development. If you are on the other hand an existing customer, generating no new revenue, you are only listened to by sales – and thus development – if you immediately take out the big guns (i.e. “… if not, we will be a new customer to IBM”). IMHO this is a increasingly failing concept, because nowadays customers have more and other options when it comes to vendor selection. Those might be more or less painful ones, but they are there nonetheless. A continuous and more immediate connection between customers and development might be a better idea.
Although this post kind of drifted off the original TFO for AIX subject, brushing several other subjects in the process, i hope it was still worth the read. I'm very curious of how IBM and several of its product dinosaurs will hold up against their new, upcoming and faster paced counterparts from smaller vendors and from open source projects.
2013-04-15 // Nagios Monitoring - IBM TS3500 Tape Library
We use several IBM TS3500 (3584) tape libraries in our datacenters to store backup copies from several IBM TSM (Tivoli Storage Manager) instances. Recently i wrote a Nagios plugin to monitor the IBM TS3500 tape libraries. In order to run the Nagios plugin, you need to have SNMP activated on the tape library. Also, a network connection from the Nagios system to the TS3500 device on port UDP/161 must be allowed.
The whole setup for monitoring IBM TS3500 tape libraries setup looks like this:
Enable SNMP queries on the service processor of the TS3500. Login to the WebGUI and navigate to:
-> Access -> SNMP Settings -> SNMP Version: v2 -> Check selection: "Enable SNMP Trap" -> Check selection: "Enable SNMP Requests" -> Apply -> SNMP System Data -> Community Names -> Read Name: <Enter the SNMP community string to access the TS3500> -> System Group Information -> Fill the form with your information -> ApplyVerify the port UDP/161 on the TS3500 device can be reached from the Nagios system.
Optional: Enable SNMP traps to be sent to the Nagios system from the service processor of the TS3500. This requires SNMPD and SNMPTT to be already setup on the Nagios system. Login to the WebGUI and navigate to:
-> Access -> SNMP Destinations -> From the drop-down menu select: "Create" -> Go -> <Enter IP and UDP port of the Nagios systems SNMPD> -> OK -> SNMP System Data -> Community Names -> Trap Name: <Enter the SNMP community string to access the SNMPD on the Nagios system> -> ApplyVerify the port UDP/162 on the Nagios system can be reached from the TS3500 devices.
Download the Nagios plugin check_ts3500.sh and place it in the plugins directory of your Nagios system, in this example
/usr/lib/nagios/plugins/:$ mv -i check_ts3500.sh /usr/lib/nagios/plugins/ $ chmod 755 /usr/lib/nagios/plugins/check_ts3500.sh
Define the following Nagios commands. In this example this is done in the file
/etc/nagios-plugins/config/check_ts3500.cfg:# check IBM TS3500 tape library - chassis status define command { command_name check_ts3500_chassis command_line $USER1$/check_ts3500.sh -H $HOSTADDRESS$ -C chassis } # check IBM TS3500 tape library - changer status define command { command_name check_ts3500_changer command_line $USER1$/check_ts3500.sh -H $HOSTADDRESS$ -C changer } # check IBM TS3500 tape library - device status define command { command_name check_ts3500_devices command_line $USER1$/check_ts3500.sh -H $HOSTADDRESS$ -C devices } # check IBM TS3500 tape library - FC status define command { command_name check_ts3500_fc command_line $USER1$/check_ts3500.sh -H $HOSTADDRESS$ -C fc } # check IBM TS3500 tape library - media status define command { command_name check_ts3500_media command_line $USER1$/check_ts3500.sh -H $HOSTADDRESS$ -C media } # check IBM TS3500 tape library - SCSI status define command { command_name check_ts3500_scsi command_line $USER1$/check_ts3500.sh -H $HOSTADDRESS$ -C scsi } # check IBM TS3500 tape library - security status define command { command_name check_ts3500_security command_line $USER1$/check_ts3500.sh -H $HOSTADDRESS$ -C security }If the output of the Nagios plugin – especially for
devices,fcandscsi– is to verbose for your needs, just add a-qto the abovecommand_linedefinitions.Verify that a generic check command for a running SNMPD is already present in your Nagios configuration. If not add a new check command like this:
define command { command_name check_snmpd command_line $USER1$/check_snmp -H $HOSTADDRESS$ -o .1.3.6.1.2.1.1.3.0 -P 1 -C public -t 30 }Define a group of services in your Nagios configuration to be checked for each TS3500 device:
# check snmpd define service { use generic-service hostgroup_name tapelib-ts3500 service_description Check_SNMPD check_command check_snmpd } # check IBM TS3500 tape library - chassis status define service { use generic-service hostgroup_name tapelib-ts3500 service_description Check_TS3500_TL_Chassis check_command check_ts3500_chassis } # check IBM TS3500 tape library - changer status define service { use generic-service hostgroup_name tapelib-ts3500 service_description Check_TS3500_TL_Changer check_command check_ts3500_changer } # check IBM TS3500 tape library - devices status define service { use generic-service-pnp hostgroup_name tapelib-ts3500 service_description Check_TS3500_TL_Devices check_command check_ts3500_devices } # check IBM TS3500 tape library - fc status define service { use generic-service hostgroup_name tapelib-ts3500 service_description Check_TS3500_TL_FC check_command check_ts3500_fc } # check IBM TS3500 tape library - media status define service { use generic-service-pnp hostgroup_name tapelib-ts3500 service_description Check_TS3500_TL_Media check_command check_ts3500_media } # check IBM TS3500 tape library - SCSI status define service { use generic-service hostgroup_name tapelib-ts3500 service_description Check_TS3500_TL_SCSI check_command check_ts3500_scsi } # check IBM TS3500 tape library - security status define service { use generic-service hostgroup_name tapelib_ts3500 service_description Check_TS3500_TL_Security check_command check_ts3500_security }Replace
generic-servicewith your Nagios service template. Replacegeneric-service-pnpwith your Nagios service template that has performance data processing enabled.Define a service dependency to run the above checks
Check_TS3500_TL_*only if theCheck_SNMPDwas run successfully:# IBM TS3500 SNMPD dependencies define servicedependency{ hostgroup_name tapelib-ts3500 service_description Check_SNMPD dependent_service_description Check_TS3500_TL_.* execution_failure_criteria c,p,u,w notification_failure_criteria c,p,u,w }Define a host in your Nagios configuration for the service processor of each TS3500 device. In this example its named
ts3500:define host { use tapelib host_name ts3500 alias TS3500 Tape Library address 10.0.0.1 parents parent_lan }Replace
tapelibwith your Nagios host template for tape library storage devices. Adjust theaddressandparentsparameters according to your environment.Define a hostgroup in your Nagios configuration for all TS3500 devices. In this example it is named
tapelib-ts3500. The above checks are run against each member of the hostgroup:define hostgroup { hostgroup_name tapelib-ts3500 alias Tape Libraries (TS3500) members ts3500 }Run a configuration check and if successful reload the Nagios process:
$ /usr/sbin/nagios3 -v /etc/nagios3/nagios.cfg $ /etc/init.d/nagios3 reload
The new hosts and services should soon show up in the Nagios web interface.
If the optional step number 2 in the above list was done, SNMPTT also needs to be configured to be able to understand the incoming SNMP traps from TS3500 devices. This can be achieved by the following steps:
Download the SNMP MIB archive 3584-MIB.zip or 3584-MIB.tar, unpack it and transfer the files
3584v1.mib,3584v2c.mibandSNIA-SML-rev1-21.mibto the Nagios server.Convert the SNMP MIB definitions in all three files into a format that SNMPTT can understand.
$ /opt/snmptt/snmpttconvertmib --in=MIB/3584v1.mib --out=/opt/snmptt/conf/snmptt.conf.ibm-ts3500_v1 ... Done Total translations: 124 Successful translations: 124 Failed translations: 0 $ /opt/snmptt/snmpttconvertmib --in=MIB/3584v2c.mib --out=/opt/snmptt/conf/snmptt.conf.ibm-ts3500_v2c ... Done Total translations: 124 Successful translations: 124 Failed translations: 0 $ /opt/snmptt/snmpttconvertmib --in=MIB/SNIA-SML-rev1-21.mib --out=/opt/snmptt/conf/snmptt.conf.ibm-ts3500_SNIA-SML-rev1-21 ... Done Total translations: 13 Successful translations: 13 Failed translations: 0
The trap severity settings should be pretty reasonable by default, but you can edit them according to your requirements with:
$ vim /etc/snmptt/conf.d/snmptt.conf.ibm-ts3500_v1 $ vim /etc/snmptt/conf.d/snmptt.conf.ibm-ts3500_v2c $ vim /etc/snmptt/conf.d/snmptt.conf.ibm-ts3500_SNIA-SML-rev1-21
Add the new configuration file to be included in the global SNMPTT configuration and restart the SNMPTT daemon:
$ vim /opt/snmptt/snmptt.ini ... [TrapFiles] snmptt_conf_files = <<END ... /etc/snmptt/conf.d/snmptt.conf.ibm-ts3500_v1 /etc/snmptt/conf.d/snmptt.conf.ibm-ts3500_v2c /etc/snmptt/conf.d/snmptt.conf.ibm-ts3500_SNIA-SML-rev1-21 ... END $ /etc/init.d/snmptt reload
Download the Nagios plugin check_snmp_traps.sh and place it in the plugins directory of your Nagios system, in this example
/usr/lib/nagios/plugins/:$ mv -i check_snmp_traps.sh /usr/lib/nagios/plugins/ $ chmod 755 /usr/lib/nagios/plugins/check_snmp_traps.sh
Define the following Nagios command to check for SNMP traps in the SNMPTT database. In this example this is done in the file
/etc/nagios-plugins/config/check_snmp_traps.cfg:# check for snmp traps define command { command_name check_snmp_traps command_line $USER1$/check_snmp_traps.sh -H $HOSTNAME$:$HOSTADDRESS$ -u <user> -p <pass> -d <snmptt_db> }Replace
user,passandsnmptt_dbwith values suitable for your SNMPTT database environment.Add another service in your Nagios configuration to be checked for each TS3500 device:
# check snmptraps define service{ use generic-service hostgroup_name tapelib-ts3500 service_description Check_SNMP_traps check_command check_snmp_traps }Optional: Define a serviceextinfo to display a folder icon next to the
Check_SNMP_trapsservice check for each TS3500 device. This icon provides a direct link to the SNMPTT web interface with a filter for the selected host:define serviceextinfo { hostgroup_name tapelib-ts3500 service_description Check_SNMP_traps notes SNMP Alerts #notes_url http://<hostname>/nagios3/nagtrap/index.php?hostname=$HOSTNAME$ #notes_url http://<hostname>/nagios3/nsti/index.php?perpage=100&hostname=$HOSTNAME$ }Uncomment the
notes_urldepending on which web interface (nagtrap or nsti) is used. Replacehostnamewith the FQDN or IP address of the server running the web interface.Run a configuration check and if successful reload the Nagios process:
$ /usr/sbin/nagios3 -v /etc/nagios3/nagios.cfg $ /etc/init.d/nagios3 reload
Optional: If you're running PNP4Nagios v0.6 or later to graph Nagios performance data, you can use the
check_ts3500_devices.phpandcheck_ts3500_media.phpPNP4Nagios template to beautify the graphs. Download the PNP4Nagios templates check_ts3500_devices.php and check_ts3500_media.php and place it in the PNP4Nagios template directory, in this example/usr/share/pnp4nagios/html/templates/:$ mv -i check_ts3500_devices.php check_ts3500_media.php /usr/share/pnp4nagios/html/templates/ $ chmod 644 /usr/share/pnp4nagios/html/templates/check_ts3500_devices.php $ chmod 644 /usr/share/pnp4nagios/html/templates/check_ts3500_media.php
The following image shows an example of what the PNP4Nagios graphs look like for a IBM TS3500 unit:
Unfortunately there are very few values presented by the SNMP MIB to be graphed.
All done, you should now have a complete Nagios-based monitoring solution for your IBM TS3500 devices.
2013-04-14 // Nagios Monitoring - IBM TS3310 or Adic / Quantum Scalar i500 Tape Library
We use several IBM TS3310 tape libraries in our datacenters to store backup copies from several IBM TSM (Tivoli Storage Manager) instances. Some time ago i wrote a – rather crude – Nagios plugin to monitor the IBM TS3310 tape libraryies. Since the TS3310 is actually a IBM OEMed version of the ADIC (now Quantum) Scalar i500 tape library, i guess the Nagios plugin should also work for the ADIC/Quantum systems. In order to run the Nagios plugin, you need to have SNMP activated on the tape library. Also, a network connection from the Nagios system to the TS3310 device on port UDP/161 must be allowed.
The whole setup for monitoring IBM TS3310 – and probably ADIC/Quantum Scalar i500 – tape libraries setup looks like this:
Enable SNMP queries on the service processor of the TS3310. Login to the WebGUI and navigate to:
-> Manage Library -> Settings -> SNMP -> SNMP Community: <Enter the SNMP community string to access the TS3310> -> Check selection: "Enable SNMP v1 & v2" -> Submit ChangesVerify the port UDP/161 on the TS3310 device can be reached from the Nagios system.
Optional: Enable SNMP traps to be sent to the Nagios system from the service processor of the TS3310. This requires SNMPD and SNMPTT to be already setup on the Nagios system. Login to the WebGUI and navigate to:
-> Manage Library -> Settings -> SNMP Traps -> From the drop-down menu select: "Add ..." -> Go -> <Enter IP and UDP port of the Nagios systems SNMPD> -> OKVerify the port UDP/162 on the Nagios system can be reached from the TS3310 devices.
Download the two Nagios plugins check_adic.sh and check_adic_drive.sh and place them in the plugins directory of your Nagios system, in this example
/usr/lib/nagios/plugins/:$ mv -i check_adic.sh check_adic_drive.sh /usr/lib/nagios/plugins/ $ chmod 755 /usr/lib/nagios/plugins/check_adic.sh $ chmod 755 /usr/lib/nagios/plugins/check_adic_drive.sh
Adjust the plugins settings according to your environment. Edit the following variable assignments:
SNMPGETNEXT_ARGS="-On -v 1 -c public -t 15"
Define the following Nagios commands. In this example this is done in the file
/etc/nagios-plugins/config/check_adic.cfg:# check ADIC tape library health define command { command_name check_adic command_line $USER1$/check_adic.sh -H $HOSTADDRESS$ } # check ADIC tape library drive health define command { command_name check_adic_drive command_line $USER1$/check_adic_drive.sh -H $HOSTADDRESS$ }Verify that a generic check command for a running SNMPD is already present in your Nagios configuration. If not add a new check command like this:
define command { command_name check_snmpd command_line $USER1$/check_snmp -H $HOSTADDRESS$ -o .1.3.6.1.2.1.1.3.0 -P 1 -C public -t 30 }Define a group of services in your Nagios configuration to be checked for each TS3310 device:
# check snmpd define service{ use generic-service hostgroup_name tapelib-ts3310 service_description Check_SNMPD check_command check_snmpd } # check ADIC tape library health define service{ use generic-service hostgroup_name tapelib-ts3310 service_description Check_ADIC_TL_Health check_command check_adic } # check ADIC tape library drive health define service{ use generic-service hostgroup_name tapelib-ts3310 service_description Check_ADIC_TL_Drives check_command check_adic_drive }Replace
generic-servicewith your Nagios service template.Define a service dependency to run the checks
Check_ADIC_TL_HealthandCheck_ADIC_TL_Drivesonly if theCheck_SNMPDwas run successfully:# ADIC/TS3310 SNMPD dependencies define servicedependency{ hostgroup_name tapelib service_description Check_SNMPD dependent_service_description Check_ADIC_TL_.* execution_failure_criteria c,p,u,w notification_failure_criteria c,p,u,w }Define a host in your Nagios configuration for the service processor of each TS3310 device. In this example its named
ts3310:define host { use tapelib host_name ts3310 alias TS3310 Tape Library address 10.0.0.1 parents parent_lan }Replace
tapelibwith your Nagios host template for tape library storage devices. Adjust theaddressandparentsparameters according to your environment.Define a hostgroup in your Nagios configuration for all TS3310 devices. In this example it is named
tapelib-ts3310. The above checks are run against each member of the hostgroup:define hostgroup { hostgroup_name tapelib-ts3310 alias Tape Libraries members ts3310 }Run a configuration check and if successful reload the Nagios process:
$ /usr/sbin/nagios3 -v /etc/nagios3/nagios.cfg $ /etc/init.d/nagios3 reload
The new hosts and services should soon show up in the Nagios web interface.
If the optional step number 2 in the above list was done, SNMPTT also needs to be configured to be able to understand the incoming SNMP traps from TS3310 devices. This can be achieved by the following steps:
Download the SNMP MIB file from the WebGUI of the TS3310 (→
Manage Library→Settings→SNMP→Download SNMP MIB File), save the file asTS3310_MIB.miband transfer it to the Nagios server.Convert the SNMP MIB definitions in
TS3310_MIB.mibinto a format that SNMPTT can understand.$ /opt/snmptt/snmpttconvertmib --in=MIB/TS3310_MIB.mib --out=/opt/snmptt/conf/snmptt.conf.ibm-ts3310 ... Done Total translations: 17 Successful translations: 17 Failed translations: 0
The trap severity settings should be pretty reasonable by default, but you can edit them according to your requirements with:
$ vim /opt/snmptt/conf/snmptt.conf.ibm-ts3310
Add the new configuration file to be included in the global SNMPTT configuration and restart the SNMPTT daemon:
$ vim /opt/snmptt/snmptt.ini ... [TrapFiles] snmptt_conf_files = <<END ... /opt/snmptt/conf/snmptt.conf.ibm-ts3310 ... END $ /etc/init.d/snmptt reload
Download the Nagios plugin check_snmp_traps.sh and place it in the plugins directory of your Nagios system, in this example
/usr/lib/nagios/plugins/:$ mv -i check_snmp_traps.sh /usr/lib/nagios/plugins/ $ chmod 755 /usr/lib/nagios/plugins/check_snmp_traps.sh
Define the following Nagios command to check for SNMP traps in the SNMPTT database. In this example this is done in the file
/etc/nagios-plugins/config/check_snmp_traps.cfg:# check for snmp traps define command { command_name check_snmp_traps command_line $USER1$/check_snmp_traps.sh -H $HOSTNAME$:$HOSTADDRESS$ -u <user> -p <pass> -d <snmptt_db> }Replace
user,passandsnmptt_dbwith values suitable for your SNMPTT database environment.Add another service in your Nagios configuration to be checked for each TS3310 device:
# check snmptraps define service{ use generic-service hostgroup_name tapelib-ts3310 service_description Check_SNMP_traps check_command check_snmp_traps }Optional: Define a serviceextinfo to display a folder icon next to the
Check_SNMP_trapsservice check for each TS3310 device. This icon provides a direct link to the SNMPTT web interface with a filter for the selected host:define serviceextinfo { hostgroup_name tapelib-ts3310 service_description Check_SNMP_traps notes SNMP Alerts #notes_url http://<hostname>/nagios3/nagtrap/index.php?hostname=$HOSTNAME$ #notes_url http://<hostname>/nagios3/nsti/index.php?perpage=100&hostname=$HOSTNAME$ }Uncomment the
notes_urldepending on which web interface (nagtrap or nsti) is used. Replacehostnamewith the FQDN or IP address of the server running the web interface.Run a configuration check and if successful reload the Nagios process:
$ /usr/sbin/nagios3 -v /etc/nagios3/nagios.cfg $ /etc/init.d/nagios3 reload
All done, you should now have a complete Nagios-based monitoring solution for your IBM TS3310 or ADIC/Quantum Scalar i500 devices.
2013-03-13 // LPM Performance with 802.3ad Etherchannel on VIOS
Due to new network gear (Cisco Nexus 5596) being available in the datacenters, i recently switched most of the 1Gbps links on our IBM Power systems over to 10Gbps links. This freed up a lot of interfaces, patch panel ports and switch ports. So i thought it would be a nice idea to give some of the now free 1Gbps network links to the VIOS management interfaces, configure an 802.3ad etherchannel and thus aggregate the multiple links. The plan was to have some additional bandwith available for the resource intensive data transfer during live partition mobility (LPM) operations. Another 10Gbps link for each VIOS would probably have been the better solution, but it's hard to justify the additional investment “just” for the usecase of LPM traffic.
While i was aware that a single LPM operation would not benefit from the etherchannel (one sender, one receiver, one used link), i figured in case of multiple parallel LPM operations the additional bandwidth would speed up the data transfer almost linearly with the number of links in the etherchannel.
In a test setup i configured four VIOS on two IBM Power systems to each have an etherchannel consisting of two 1Gbps links:
root@vios1:/$ ifconfig -a
en27: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
tcp_sendspace 262144 tcp_recvspace 262144 tcp_nodelay 1 rfc1323 1
root@vios1:/$ lsattr -EHl ent27
attribute value description user_settable
adapter_names ent0,ent2 EtherChannel Adapters True
alt_addr 0x000000000000 Alternate EtherChannel Address True
auto_recovery yes Enable automatic recovery after failover True
backup_adapter NONE Adapter used when whole channel fails True
hash_mode src_port Determines how outgoing adapter is chosen True
interval long Determines interval value for IEEE 802.3ad mode True
mode 8023ad EtherChannel mode of operation True
netaddr 0 Address to ping True
noloss_failover yes Enable lossless failover after ping failure True
num_retries 3 Times to retry ping before failing True
retry_time 1 Wait time (in seconds) between pings True
use_alt_addr no Enable Alternate EtherChannel Address True
use_jumbo_frame no Enable Gigabit Ethernet Jumbo Frames True
All three combinations of the ethercannels hash_mode attribute (dst_port, src_port and src_dst_port) were tested in a szenario with four parallel LPM processes. The four LPARs used for the LPM tests had 32GB memory assigned and were only moderately active at the time.
Even with a rought timing measurement taken by hand it pretty soon became clear, that the transfer rate was still hovering slightly below 1Gbps. A look at the performance counters of the interfaces involved in the etherchannel before the LPM operation:
root@vios1:/$ entstat -d ent27 | grep "Bytes: " Bytes: 16892 Bytes: 12594 # Etherchannel: ent27 Bytes: 11958 Bytes: 8402 # Physical interface: ent0 Bytes: 5100 Bytes: 4262 # Physical interface: ent2
and after the LPM operation:
root@vios1:/$ entstat -d ent27 | grep "Bytes: " Bytes: 12018850788 Bytes: 210768159909 # Etherchannel: ent27 Bytes: 137243 Bytes: 210767988779 # Physical interface: ent0 Bytes: 12018713727 Bytes: 171200 # Physical interface: ent2
confirmed the suspicion that only one physical link was actually used for the data transfer, no matter what hash_mode was chosen.
Looking further into the processes and network connections involved in the LPM operation at VIOS level, showed that four migmover processes were started on the source and destination VIOS, one process for each LPM operation. Each migmover process opened two network connections from the source VIOS (192.168.1.1) to the destination VIOS (192.168.1.2), which in the netstat output looked something like this:
root@vios1:/$ netstat -na | grep EST tcp4 0 0 192.168.1.1.32788 192.168.1.2.32791 ESTABLISHED # Socket pair 1 tcp4 0 13888 192.168.1.1.32789 192.168.1.2.32792 ESTABLISHED # Socket pair 2 tcp4 0 0 192.168.1.1.32790 192.168.1.2.32793 ESTABLISHED # Socket pair 3 tcp4 0 12768 192.168.1.1.32791 192.168.1.2.32794 ESTABLISHED # Socket pair 4 tcp4 0 0 192.168.1.1.32792 192.168.1.2.32795 ESTABLISHED # Socket pair 5 tcp4 18824 9744 192.168.1.1.32793 192.168.1.2.32796 ESTABLISHED # Socket pair 6 tcp4 0 0 192.168.1.1.32794 192.168.1.2.32797 ESTABLISHED # Socket pair 7 tcp4 0 5264 192.168.1.1.32795 192.168.1.2.32798 ESTABLISHED # Socket pair 8
The first socket pair (numbers 1, 3, 5, 7) of each migmover process did not see much data transfer and thus seemed to be some sort of control channel. The second socket pair (numbers 2, 4, 6, 8) on the other hand saw a lot of data transfer and thus seemed to be responsible for the actual LPM data transfer. The TCP source and destination ports for the very first socket pair seemed to start at a random value somewhere in the ephemeral port range1). All the subsequently created sockets pairs seemed to have TCP source and destination ports that are an increment by one from the ones used in the previously allocated socket pair.
The combination of the three factors:
sequential port allocation from a random starting point in the ephemeral range
alternating port allocation for control and data channels
same port allocation strategy on the source and the destination VIOS
leads to the observed behaviour of only half the number of physical links being fully used in a etherchannel configuration with an even number of interfaces. Due to this TCP port allocation strategy, all control channels consistently get placed on one physical link, while all data channels consistently get placed on the next physical link, regardless of the hash_mode being used.
After figuring this out, i opened a PMR with IBM support, which pretty soon was transfered to development. After some rather fruitless back and forth, the developer finally agreed to accept a DCR (MR030413328) to implement an alternative TCP port allocation strategy for the LPM data transfer:
User Marketing Field Requirement Number: MR030413328
*Title: LPM performance enhancement with etherchannel 802.3ad
*Description: When the Mover Service Partition managing LPM is using an Etherchannel Adapter built on 2 physical adapters and this Etherchannel is configured in 8023ad mode, the port selection of LPM process makes that all the data connection to the same physical adapter and the control connection to the other adapter. This cause an highly unbalanced usage of those adapters. (The data connection will manage nearly all data transfered, where control connection only handle a little)
While LPM operation, mover is first creating the control socket and then the data socket. Hence this is more than possible that src_data = src_ctrl + 1 and dst_data = dst_ctrl + 1, which will cause in an etherchannel build on 2 adapters, configured in 8023ad mode, that all control socket use the same adapter and all data socket will use the same other adapter. (The parity of “src_data + dst_data” will be the same as “(src_ctrl + 1) + (dst_ctrl + 1)” )
I would suggest to enforce the parity of the port number depending on the fact we are the source or destination side mover service partition:
- data and control socket with same port parity if we are the source mover :
→ i.e. : data port 63122 and control port 63120
or : data port 63113 and control port 63115
- data and control socket with a different port parity if we are the destination mover :
→ i.e. : data port 63122 and control port 63121
or : data port 63113 and control port 63114
This should cause a more balanced load on etherchannel adapter.
*Priority: Medium
*Requested Completion Date: 01.01.2014
(Original Requested Date) 01.01.2014
*Resolving Pipe: AIX, LoP, System p Hardware
*Resolving Release: AIX
*Resolving Entity: Virtualization
*IBM Business Justification (Include SIEBEL number for a priority): This is a problem that has been seen many time, and this will happen for any customer with VIOS configured with etherchannel.
*Requirement Type Performance
Suggested Solution:
Source: Customer
Problem Management Record (PMR):
PMR Number: xxxxx, Branch Code xxx, Country Code xxx
Other Related Information:
Environment: VIOS Level is 2.2.2.1
The by far most interesting part about this DCR being the item “IBM Business Justification”! Remind me again, why this wasn't already fixed if it has been seen many times as an issue? Well, fingers crossed the code change will make it into a VIOS release this time …
tcp_port_low and tcp_port_high of the vioslpm0 device2013-03-10 // AIX and VIOS DLPAR Operation fails on LHEA Adapters
After upgrading our VIOS from v2.2.1.4 (aka FixPack 25, ServicePack 2) to v2.2.2.1 (aka FixPack 26) i noticed that add/remove DLPAR operations on LHEA adapters would fail with the following error message being displayed at the HMC:
The dynamic logical partitioning operation failed. - <name of VIOS> The dynamic logical partitioning requested could not be completed. Logical Port 1 belonging to Port Group 1 of HEA 23000000 failed to be added. Vary on of the LHEA failed. Please run the command rsthwres to recover the HEA configuration. ..build_tree ROOT DIR=/usr/lib/dr/scripts Syslog ch=DRMGR cal_dr_scriptinfo_file_checksum : Checksum : 0x94b71f75720f0132 File read: string table s_script: file_name: 0x200bf758(/usr/lib/dr/scripts/all/IBM.CSMAgentRM_dr.sh) script_version: 0x200bf7a8(2) script_vendor_info: 0x200bf7b3(IBM) script_creation_date: 0x200bf7aa(05252010) script_info: 0x200bf785(DR script to manage IBM.CSMAgentRM) s_resource(Before - offsets): resource_name: 0x5f resource_use_description: 0x69 s_resource:(After - ptrs) resource_name: 0x200bf7b7(pmig) resource_use_description: 0x200bf7c1(Partition migration for IBM.CSMAgentRM) s_resource(Before - offsets): resource_name: 0x64 resource_use_description: 0x90 s_resource:(After - ptrs) resource_name: 0x200bf7bc(phib) resource_use_description: 0x200bf7e8(Partition hibernation for IBM.CSMAgentRM) s_script: file_name: 0x200bf811(/usr/lib/dr/scripts/all/aud_acct_dr) script_version: 0x200bf860(1) script_vendor_info: 0x200bf86b(IBM) script_creation_date: 0x200bf862(03232007) script_info: 0x200bf835(WPAR DR Script for Auditing and Accounting) s_resource(Before - offsets): resource_name: 0x117 resource_use_description: 0x134 s_resource:(After - ptrs) resource_name: 0x200bf86f(wmig-checkpoint) resource_use_description: 0x200bf88c(Checkpoint of Auditing and Accounting within a WPAR) s_resource(Before - offsets): resource_name: 0x127 resource_use_description: 0x168 s_resource:(After - ptrs) resource_name: 0x200bf87f(wmig-restart) resource_use_description: 0x200bf8c0(Restart of Auditing and Accounting within a WPAR) s_script: file_name: 0x200bf8f1(/usr/lib/dr/scripts/all/ctrmc_MDdr) script_version: 0x200bf949(2) script_vendor_info: 0x200bf954(IBM) script_creation_date: 0x200bf94b(05252010) script_info: 0x200bf914(DR script to refresh Management Domain configuration) s_resource(Before - offsets): resource_name: 0x200 resource_use_description: 0x20a s_resource:(After - ptrs) resource_name: 0x200bf958(pmig) resource_use_description: 0x200bf962(Partition migration for RSCT Management Domain) s_resource(Before - offsets): resource_name: 0x205 resource_use_description: 0x239 s_resource:(After - ptrs) resource_name: 0x200bf95d(phib) resource_use_description: 0x200bf991(Partition hibernation for RSCT Management Domain) s_script: file_name: 0x200bf9c2(/usr/lib/dr/scripts/all/viosdr) script_version: 0x200bf9f1(1) script_vendor_info: 0x200bf9fc(IBM Corp.) script_creation_date: 0x200bf9f3(11192012) script_info: 0x200bf9e1(VIOS DR Handler) s_resource(Before - offsets): resource_name: 0x2ae resource_use_description: 0x2b3 s_resource:(After - ptrs) resource_name: 0x200bfa06(slot) resource_use_description: 0x200bfa0b(Virtual I/O Slot Handler) s_script: file_name: 0x200bfa24(/usr/lib/dr/scripts/all/wpar_drs) script_version: 0x200bfa62(1) script_vendor_info: 0x200bfa6d(IBM) script_creation_date: 0x200bfa64(05312007) script_info: 0x200bfa45(WPAR DR script for DR Events) s_resource(Before - offsets): resource_name: 0x319 resource_use_description: 0x335 s_resource:(After - ptrs) resource_name: 0x200bfa71(cpu) resource_use_description: 0x200bfa8d(Propagate CPU add/remove) s_resource(Before - offsets): resource_name: 0x31d resource_use_description: 0x34e s_resource:(After - ptrs) resource_name: 0x200bfa75(mem) resource_use_description: 0x200bfaa6(Propagate Memory add/remove) s_resource(Before - offsets): resource_name: 0x321 resource_use_description: 0x36a s_resource:(After - ptrs) resource_name: 0x200bfa79(capacity) resource_use_description: 0x200bfac2(Propagate Capacity changes) s_resource(Before - offsets): resource_name: 0x32a resource_use_description: 0x385 s_resource:(After - ptrs) resource_name: 0x200bfa82(var_weight) resource_use_description: 0x200bfadd(Propagate Var Weight changes) ..add_a_slot do_scripts_if_DR_pre_setup: bp_resource:slot invoke_DR_scripts: command:7, bp_resource:slot Invoking script (/usr/lib/dr/scripts/all/viosdr) with command (checkacquire) invoke_one_DR_script: entry->command:7, timeout:20, script idx:3, resource name: slot set_env_vars: entry: cmd : checkacquire set_env_vars: exit: DR_FORCE=FALSE DR_DRC_NAME=HEA 1 Parent: Waiting on the pipe for any info:out_line:0x2000f5f8, pipeinp: 0xf11674e8 Script returned string:(Execing:(DR_FORCE=FALSE DR_DRC_NAME=HEA 1 /usr/lib/dr/scripts/all/viosdr) with cmd:checkacquire, resource_name: slot) out of fgets while loop waiting for status child returned status: 127 Script returned status: 127 Check acquire phase failed, rc:0x7f HSCL297B
After wading through a lot of not really interesting output, the actual culprit for the failure can be found almost at the end of the output. The 7th line from the bottom, starting with Script returned string: … shows that the value which is supposed to be assigned to the environment variable DR_DRC_NAME contains a space and is probably not correctly quoted. The shell tries - in this example - to execute the command “1”, which of course cannot be found. So the correct command line should probably look something like this:
DR_FORCE="FALSE" DR_DRC_NAME="HEA 1" /usr/lib/dr/scripts/all/viosdr ...
I opened a PMR with IBM support and was given a modified drslot_chrp_slot binary, which has a temorary fix for this issue. The APAR IV36448 was opened to deal with this issue in future AIX and VIOS versions.
Although the DLPAR operation now works with the modified drslot_chrp_slot binary, the cfglog may still contain errors which look like this:
... C4 4325452 15:04:29 drscript_main.c 544 Invoking script (/usr/lib/dr/scripts/all/viosdr) with command (postacquire) C4 4325452 15:04:29 drscript_main.c 546 invoke_one_DR_script: entry->command:9, timeout:20, script idx:3, resource name: slot C4 4325452 15:04:29 drscript_main.c 1673 set_env_vars: entry: cmd : postacquire C4 4325452 15:04:29 drscript_main.c 1719 set_env_vars: exit: DR_FORCE=FALSE DR_DRC_NAME="HEA 1" C4 4325452 15:04:29 drscript_main.c 577 Environment var list: DR_FORCE=FALSE DR_DRC_NAME="HEA 1" C4 4325452 15:04:29 drscript_main.c 615 Parent: Waiting on the pipe for any info:out_line:0x20010398, pipeinp: 0xf06324e8 C4 4325452 15:04:29 drscript_main.c 782 Execing:(DR_FORCE=FALSE DR_DRC_NAME="HEA 1" /usr/lib/dr/scripts/all/viosdr) with cmd:postacquire, resource_name: slot C4 4325452 15:04:29 drscript_main.c 630 Script returned string: (Execing:(DR_FORCE=FALSE DR_DRC_NAME="HEA 1" /usr/lib/dr/scripts/all/viosdr) with cmd:postacquire, resource_name: slot ) C4 4325452 15:04:29 drscript_main.c 690 out of fgets while loop C4 4325452 15:04:29 drscript_main.c 697 waiting for status C4 4325452 15:04:29 drscript_main.c 710 child returned status: 1 C4 4325452 15:04:29 drscript_main.c 711 Script returned status: 1 ...
The problem here is with the /usr/lib/dr/scripts/all/viosdr script. Another PMR with IBM support was opened and according to the developer:
Its (N.B.:viosdr) purpose is to identify virtual adapters that are added to the system with DLPAR and automatically run the correct cfgmgr method to configure them.
LHEA adapters are not currently supported by this method so it should have ignored them and just returnedVIOSDR_SUCCESS(zero)“.
Unfortunately there is currently no fix available for this issue.
