Following up on the previous article (IBM SVC and experiences with Real-time Compression) about our experiences with Real-time compression (RTC) in a IBM SVC environment, we did some additional and further testing with regard to RTC being used in a VMware environment hosting soley database instances of Microsoft SQL Server.

With the implementation of the seperation of all of our Microsoft SQL Server instances into an own VMware ESX cluster – which had to be done due to licensing constraints – an opportunity presented itself, to take a bit more detailed look into how well the SVCs RTC feature would perform on this particular type of workload.

Usually, the contents of a VMware datastore are by their very nature opaque to the lower levels of the storage hierarchy like the SVC which is providing the block storage for the datastores. On the one hand an advantage, this lack of transparency on the other hand also prohibits the SVC from providing a fine grained, per VM set of its usual statistics (e.g. the actual amount of storage used with thin-provisioning or RTC enabled, the amount of storage in the individual storage tier classes, etc.) and performance metrics. By not only moving the compute part of the VMs over to the new VMware ESX hosts in a seperate VMware ESX cluster, but also individually moving the storage part of one VM after the other onto new and empty datastores, which in turn were at the SVC based on newly created, empty VDisks with RTC enabled, we were able to measure the impact of RTC on a per VM level. It was a bit of tedious work and the precision of the numbers has to be taken with a grain of salt because I/O was concurrently happening on the database and operating system level, probably causing additional extends to be allocated and thus some inaccuracy to a small extent.

The environment consists of the following four datastores, four VDisks and a total number of 43 VMs running various versions of Microsoft SQL Server:

Usually the VMs are configured with one or two disks or drives for the OS and one or more disks or drives dedicated to the database contents. In some legacy cases, specifically VM #7, #10 and #12 on datastore “san-ssb-sql-oben-35C”, VM #6 on datastore “san-ssb-sql-unten-360” and VM #8 on datastore “san-ssb-sql-unten-361”, there is also a local application instance – in our case different versions of SAP – installed on the system. The operating systems used are Windows Server 2003, 2008R2 and 2012R2.

For each of the four datastores or VDisks, the results of the tests are shown in the following four sections. Each section consists of a table representation of the datastore as it was being populated with VMs in chronological order. The columns of the tables are pretty self-explainatory, with the most interesting columns being “Rel. Compressed Size”, “Rel. Compression Saving” and “Compression Ratio”. If JavaScript is enabled in your browser, click on a tables column headers to sort by that particular column. Each section also contains a graph of the absolute allocation values (table columns “VM Provisioned”, “VM Used” and “SVC Used”) per VM and a graph of the relative allocation values (values from the table column “SVC Used” divided by values from the table column “VM Used” in %) per VM. The latter ones also feature the “Compression Ratio” metric which is graphed against a second y-axis on the right-hand side of the graphs.

• Datastore: san-ssb-sql-oben-35C
  VDisk: sas-4096G-35C

  VM	VM Provisioned (GB)	VM Used (GB)	SVC Used (GB)	SVC Delta Used (GB)	Rel. Compressed Size (%)	Rel. Compression Saving (%)	Compression Ratio	SQL Server Version	Comment
  Empty Datastore	–	–	6.81 MB	6.81 MB	–	–	–	–
  Number 1	134.11	44.00	20.78	20.10	45.68	54.32	2.19	10.50.4000.0
  Number 2	144.11	55.04	48.45	28.35	51.51	48.49	1.94	11.0.5058.0
  Number 3	94.18	49.79	75.88	27.43	55.09	44.91	1.82	11.0.5058.0
  –	–	–	76.30	0.42	–	–	–	–
  Number 4	454.11	52.22	92.47	16.17	30.97	69.03	3.23	10.50.4000.0
  Number 5	456.11	391.14	337.74	245.27	62.71	37.29	1.59	10.50.4000.0	SharePoint with file uploads
  Number 6	238.14	156.29	450.89	113.15	72.40	27.60	1.38	10.50.6000.34	HR Applicant Management
  Number 7	488.11	304.32	660.05	209.16	68.73	31.27	1.45	10.50.4000.0	SQL Page Compression
  Number 8	348.09	259.32	703.47	43.42	16.74	83.26	5.97	9.00.4035.00
  Number 9	185.24	149.33	744.68	41.21	27.60	72.40	3.62	10.50.4000.0
  Number 10	292.12	104.31	804.14	59.46	57.00	43.00	1.75	11.0.3349.0	SQL Page Compression
  Number 11	214.11	148.73	882.44	78.30	52.65	47.35	1.90	10.50.2500.0
  Number 12	496.09	463.15	993.53	111.09	23.99	76.01	4.17	9.00.3228.00
  Number 13	202.12	176.96	1058.76	65.23	36.86	63.14	2.71	9.00.4035.00
  Number 14	442.41	363.83	1200.11	141.35	38.85	61.15	2.57	11.0.3412.0	SQL Page Compression
  Number 15	395.38	374.78	1300.00	97.96	26.14	73.86	3.83	11.0.3412.0	SQL Page Compression
  Number 16	739.31	464.30	1520.00	220.00	47.38	52.62	2.11	11.0.3412.0	SQL Page Compression
  Sum or Average	5189.63	3513.51		1497.55	42.10	57.90	2.38

  
  
  

• Datastore: san-ssb-sql-unten-360
  VDisk: sas-4096G-360

  VM	VM Provisioned (GB)	VM Used (GB)	SVC Used (GB)	SVC Delta Used (GB)	Rel. Compressed Size (%)	Rel. Compression Saving (%)	Compression Ratio	SQL Server Version	Comment
  Empty Datastore	–	–	5.47 MB	5.47 MB	–	–	–	–
  Number 1	377.68	241.77	98.13	97.58	40.36	59.64	2.48	11.0.3412.0	SQL Page Compression
  Number 2	439.29	256.54	198.76	100.63	39.23	60.77	2.55	11.0.3412.0	SQL Page Compression
  Number 3	776.76	507.24	448.04	249.28	49.14	50.86	2.03	11.0.3412.0	SQL Page Compression
  Number 4	195.21	154.10	533.52	85.48	55.47	44.53	1.80	11.0.5058.0
  Number 5	34.11	23.91	546.66	13.14	54.96	45.04	1.82	9.00.4035.00
  Number 6	497.11	394.46	718.35	171.69	43.53	56.47	2.30	10.50.1702.0	SQL Page Compression
  Number 7	219.91	53.87	740.20	21.85	40.56	59.44	2.47	11.0.3000.0
  Number 8	276.11	238.67	850.77	110.57	46.33	53.67	2.16	10.50.2500.0
  Number 9	228.11	188.26	986.25	135.48	71.96	28.04	1.39	10.50.2500.0	HR Applicant Management
  Sum or Average	3044.29	2058.82		986.25	47.90	52.10	2.09

  
  
  

• Datastore: san-ssb-sql-unten-361
  VDisk: sas-2048G-361

  VM	VM Provisioned (GB)	VM Used (GB)	SVC Used (GB)	SVC Delta Used (GB)	Rel. Compressed Size (%)	Rel. Compression Saving (%)	Compression Ratio	SQL Server Version	Comment
  Empty Datastore	–	–	6.66 MB	6.66 MB	–	–	–	–
  Number 1	188.11	162.98	51.46	51.45	31.57	68.43	3.17	10.50.4000.0
  Number 2	180.31	130.82	90.79	39.33	30.06	69.94	3.33	10.50.4000.0
  Number 3	178.11	142.63	131.17	40.38	28.31	71.69	3.53	10.50.4000.0
  Number 4	156.09	127.26	177.30	46.13	36.25	63.75	2.76	9.00.4035.00
  Number 5	100.86	90.40	202.76	25.46	28.16	71.84	3.55	9.00.4035.00
  –	–	–	205.91	3.15	–	–	–	–
  Number 6	139.09	79.17	233.85	27.94	35.29	64.71	2.83	11.0.5058.0
  Number 7	34.11	24.16	246.37	12.52	51.82	48.18	1.93	9.00.4035.00
  Number 8	696.32	289.09	402.07	155.07	53.86	46.14	1.86	11.0.5058.0	SQL Page Compression
  Sum or Average	1673.00	1046.51		402.07	38.42	61.58	2.60

  
  
  

• Datastore: san-ssb-sql-oben-362
  VDisk: sas-2048G-362

  VM	VM Provisioned (GB)	VM Used (GB)	SVC Used (GB)	SVC Delta Used (GB)	Rel. Compressed Size (%)	Rel. Compression Saving (%)	Compression Ratio	SQL Server Version	Comment
  Empty Datastore	–	–	–	–	–	–	–	–
  Number 1	218.09	37.19	19.93	19.93	53.59	46.41	1.87	9.00.4035.00
  Number 2	122.12	112.53	62.32	42.39	37.67	62.33	2.65	10.50.1600.1
  Number 3	273.62	77.70	97.04	34.72	44.68	55.32	2.24	10.50.4000.0
  Number 4	143.09	37.98	116.47	19.43	51.16	48.84	1.95	11.0.5058.0
  Number 5	137.43	115.29	157.45	40.98	35.55	64.45	2.81	10.50.4000.0
  Number 6	80.09	76.04	178.76	21.31	28.02	71.98	3.57	9.00.4035.00
  –	–	–	182.30	3.54	–	–	–	–
  Number 7	136.09	117.59	229.95	47.65	40.52	59.48	2.47	9.00.4035.00
  Number 8	194.11	53.55	253.40	23.45	43.79	56.21	2.28	10.50.2500.0
  Number 9	78.11	52.60	273.07	19.67	37.40	62.60	2.67	–
  Number 10	53.11	31.42	287.33	14.26	45.39	54.61	2.20	10.50.4000.0
  Sum or Average	1435.86	711.89		283.82	40.36	59.64	2.48

  
  
  

From the detailed tabular representation of each datastore on a per VM basis that is shown above, an aggregated and summarized view, shown in the table below was created. The columns of the table are again pretty self-explainatory. Some of the less interesting columns from the detailed tables above have been dropped in the aggregated result table. The most interesting columns are again “Rel. Compressed Size”, “Rel. Compression Saving” and “Compression Ratio”. Again, if JavaScript is enabled in your browser, click on a tables column headers to sort by that particular column.

On the whole, the compression results achived with RTC at SVC level are quite good. Considering the fact that VMware already achieved a sizeable amount of storage reduction through its own thin-provisioning algorithms, the average RTC compression ratio of 2.34 – with a minimum of 1.38 and a maximum of 5.97 – at the SVC is even more impressive. This means that on average only 42.76% of the storage space allocated by VMware is actually used on the SVC level to store the data. Taking VMware thin-provisioning into account, the relative amount of actual storage space needed sinks even further down to 28.28%, or in other words a reduction by a factor of 3.54.

Looking at the individual graphs offers several other interesting insights, which can be indicators for further investigations on the database layer. In the graphs showing the relative allocation values and titled “SVC Real-Time Compression Relative Storage Allocation and Compression Ratio on VMware with MS SQL Server”, those data samples with a high “Rel. Compressed Size” value (purple bars) or a low “Compression Ratio” value (red line) are of particular interest in this case. Selecting those systems from the above results, with an – arbitrarily choosen – value of well below 2.00 for the “Compression Ratio” metric, gives us the following list of systems to take a closer look at:

The reasons for why those particular systems are exhibiting a subaverage compression ratio could be manifold. Together with our DBAs we went over the list and came up with the following, not exhaustive nor exclusive list of explainations which are already hinted in the “Comment” columns of the above table:

• Samples 1 and 6: Those systems were recently used to test the influence of 4k vs. 64k NTFS cluster size on the database I/O performance. Over the course of these tests, a number of dummy files were written with the SQLIO benchmark tool. Altough the dummy files were deleted after the tests were concluded, the storage blocks remained allocated with apparently less compressible contents.

• Samples 2, 3 and 8: Those systems host the databases for SharePoint (sample #2) and a HR/HCM applicant management and employee training management system (samples #3 and #8). Both applications seem to be designed or configured to store files uploaded to the application as binary blobs into the database. Both usecases suggest that the uploaded file data is to some extent already in some kind of compressed format (e.g. xlsx, pptx, jpeg, png, mpeg, etc.), thus limiting the efficiency of any further compression attempt like RTC. Storing large amounts of unstructured data together with structured data in a database is subject of frequent, ongoing and probably never ending, controversial discussions. There are several points of view to this question and good arguments are being made from either side. Personally and from a purely operational point of view, i'd favour the unstructured binary data not to be stored in a structured database.

• Sample 4: This system hosts the legacy design described above, where a local application instance of SAP is installed along Microsoft SQL Server on the same system. The database tables of this SAP release already use the SQL Server page compression feature. Although this could have very well been the cause for the subaverage compression ratio, a look into the OS offered a different plausible explanation. Inside the OS the sum of space allocated to the various filesystems is only approximately 202 GB, while at same time the amount of space allocated by VMware is a little more than 304 GB. It seems that at one point in time there was a lot more data of unknown compressibility present on the system. The data has since been deleted, but the previously allocated storage blocks have not been properly reclaimed.

  In order to put this theory to test, the procedure described in VMware KB 2004155 to reclaim the unused, but not de-allocated storage blocks, was applied. The following table shows the storage allocation for the system in sample #4 before and after the reclaim procedure:

  Sample	Datastore / VDisk	VM #	VM Provisioned (GB)	VM Used (GB)	SVC Delta Used (GB)	Compression Ratio	Comment
  4	san-rtc-ssb-sql-unten-366 sas-1024G-366	7	488.11	304.32	209.16	1.45	SAP and MSSQL (with SQL Page Compression) on the same VM - before reclaiming unused storage space
  4		7	488.11	172.53	81.29	2.12	SAP and MSSQL (with SQL Page Compression) on the same VM - after reclaiming unused storage space

  The numbers show that the above assumption was correct. After zeroing unused disk space within the VM and reclaiming unused storage blocks by re-thin-provisioning the virtual disks in VMware, the RTC compression ratio rose from a meager 1.45 to a near average 2.12. Now, only 47.11% – instead of the previous 68.73% – of the storage space allocated by VMware is actually used on the SVC level. Again, taking VMware thin-provisioning into account, the relative amount of actual storage space needed sinks even further down to 16.65%, or in other words a reduction by a factor of 6.00.

• Samples 5 and 10: These systems basically share the same circumstances as the previously examined system in sample #4. The only difference is, that there is still some old, probably unused data sitting in the filesystems. In case of sample #5 it's about 20 GB of compressed SAP installation media and in case of sample #10 it's about 84.3 GB of partially compressed SAP installation media and SAP SUM update files. Of course we'd like to also reclaim this storage space and – hopefully – in the course of this some deleted, but up to now not de-allocated storage blocks, too. By this we hope to see equally good end results as in the case of sample #4. We're currently waiting on clearance from our SAP team to dispose of some of the old and probably unused data.

• Samples 7, 9, 11 and 12: Systems in this sample category are probably best described as just being “small”. To better illustrate what is meant by this, the following table shows a compiled view of these four samples. The data was taken from the detailed tabular representation of each datastore on a per VM basis that is shown above:

  Sample	Datastore / VDisk	VM #	VM Provisioned (GB)	VM Used (GB)	SVC Delta Used (GB)	Rel. Compressed Size (%)	Rel. Compression Saving (%)	Compression Ratio	Windows Version
  7	san-ssb-sql-unten-360 sas-4096G-360	5	34.11	23.91	13.14	54.96	45.04	1.82	2003
  9	san-ssb-sql-unten-361 sas-2048G-361	7	34.11	24.16	12.52	51.82	48.18	1.93	2003
  11	san-ssb-sql-oben-362 sas-2048G-362	1	218.09	37.19	19.93	53.59	46.41	1.87	2012 R2
  12		4	143.09	37.98	19.43	51.16	48.84	1.95	2012 R2

  Inside the systems there is very little user or installation data besides the base Windows operating system, the SQL Server binaries and some standard management tools which are used in our environment. VMware thin-provisioning already does a significant amount of space reduction. It reduces the storage space that would be used on the SVC level to approximately 24 GB for Windows 2003 and approximately 37.5 GB for Windows 2012 R2 systems. Although the number of samples is rather low, the almost consistent values within each Windows release category suggest that these values represent – in our environment – the minimal amount of storage space needed for each Windows release. The compressed size of approximately 12.8 GB for Windows 2003 and approximately 19.7 GB for Windows 2012 R2 systems as well as the average compression ratio of approximately 1.89 seem to support this theory in the way that they show very similar values within each Windows release category. It appears that – in our environment – these values are the bottom line with regard to minimal allocated storage capacity. There simply seems not to be enough other data with a good compressability in those systems in order to achieve better results with regard to the overall compression ratio.

By and large, we're quite pleased with the results we're seeing from the SVCs real-time compression feature in our VMware and MS SQL server environment. The amount of storage space saved is – especially with VMware thin-provisioning taken into account – significant. Even in those cases where the SQL servers page compression feature is heavily used, we're still seeing quite good compression results with the use of the additional real-time compression at the SVC level. In part, this is very likely due to the fact that real-time compression at the SVC level also covers the VMs data that is outside of the scope which is covered by the SQL servers page compression. On the other hand, this does not entirely suffice to explain the amount of saved storage space – between 50.86% and 73.86%, if we disregard some special cases for which the reasons of a low compression ratio were discussed above – we're seeing in cases where SQL server page compression is used. From the data collected and shown above, it would appear that the algorithms used in SQL server page compression still leave enough redundant data of low entropy for the algorithms used in the RACE at the SVC level to perform rather well.

In general and with regard to performance in particular we have – up to now – not noticed any negative side effects, like e.g. noticeable increases in I/O latency, by using RTC for the various SQL server and SAP systems shown above.

The use of RTC not only promises, but quite matter of factly offers a significant reduction of the storage space needed. Even with seemingly already compressed workloads, like e.g. SQL server page compressed databases, it deals very well. It thus enables a delayed procurement of additional storage resources, lower operational costs (administration, rack space, power, cooling, etc.), a reduced amount of I/O to the backend storage systems and a more efficient use of tier-1 storage like e.g. flash based storage systems.

It is on the other hand also no out-of-the-box, fire-and-forget solution when used sensibly. The selected examples of “interesting” systems, which were shown and discussed above, illustrates very well that there are always usecases which require special attention. They also point out the increased necessity of a good overall, interdisciplinary technical knowledge or a very good and open communication between the organisational units responsible for application, database, operating system, virtualization and storage administration.

Due to the rather old release level of our VMware environment we unfortunately weren't able to cover the interaction of TRIM, UNMAP, VAAI and RTC. It'd be very interesting to see if and how well those storage block reclaimation technologies work together with the SVCs real-time compression feature.

Comments and own experiences are – as always – very welcome!

Under certain conditions it is possible to inadvertently run into an out of memory situation on IBM Storwize V3700 systems, by simply running a Download Support Package procedure or the respective CLI command. This will – of course – bring all I/O on the affected system to a grinding halt.

A few days ago, the Nagios monitoring plugin introduced in “Nagios Monitoring - IBM SVC and Storwize” reported a failed PSU on one of our IBM Storwize V3700 systems. After raising a PMR with IBM in order to get the seemingly defective PSU replaced, i was told to simply reseat the PSU. According to IBM support this would usually fix this – apparently known – issue.

This was the first WTF moment and it turned out to not be the last one. So either IBM produces and sells subpar components – in this case the PSU – which need to be given a boot – yes, PSUs nowadays have their own firmware too – in order to be persuaded to cooperate again. Or it means IBM produces and sells subpar software which is not at all able to properly detect a component failure and distinguish between a faulty and a good PSU. Or perhaps its an unfortunate combination of both.

In any case, the procdure to reseat the PSU was carried out, which fixed the PSU issue. During the course of the fix procedure the system would become unreachable via TCP/IP for a rather long time, though. Definately over two minutes, but i haven't had a chance for an exact measurement. After the system was reachable again i followed this strange behaviour up and had a look at the systems event log. There were quite a lot of “Error Code: 1370, Error Code Text: SCSI ERP occurred” messages, so i decided to bother the IBM support again and send them a support collection in order to get an analysis with regard to the reachablilty issue as well as the 1370 errors.

From previous occasions i knew that the IBM support would most likely request a support collection which was run with the “svc_livedump” CLI command or with the “Standard logs plus new statesaves” option from the WebUI. The latter one is marked red in the following screenshot example:

So i decided to pull a support collection with this option. After some time into the support collection process, the SVC sitting in front of the V3700 and other storage systems, started to show very high latencies (~60 sec.) on the primary VDisks backed by the V3700. On other VDisks which “only” had their secondary VDisk-Mirror located on MDiskGroups of the V3700, the latency peak was less dramatic, but still very noticable. Eventually degraded paths to the MDisks located on the V3700 started showing up on the SVC. After the support collection process finished the situation went back to normal. The latency on both primary and secondary VDisks instantly dropped down to the usual values and after running the fix procedures on the SVC, the degraded paths came back online.

The performance issues were in magnitude and duration severe enough to affect several applications pretty badly. Although the immediate issue was resolved, i still needed an analysis and written statement from IBM support for an action plan on how to prevent this kind of situation in the future and for compliance reasons as well. Here is the digest of what – according to IBM – happened:

1. During the procedure to reseat the reportedly defective PSU, one or more power surges occured.

2. These power surges apparently caused issues on the internal disk buses, which lead to the 1370 errors to be logged.

3. The power surges or the resulting 1370 errors are probably the cause for a failover of the config node too. Hence the connectivity issues with the CLI and the WebUI via TCP/IP.

4. More 1370 errors were logged during the runtime of the subsequent “Standard logs plus new statesaves” support collection process.

5. The issue of very high latency and MDisk paths becoming degraded was caused by the “Standard logs plus new statesaves” support collection process using up all of the memory – yes, including the data cache – on the V3700 system. This behaviour is specific and limited to the V3700 systems with “only” 4GB of memory.

As one can imagine, the last item was my second WTF moment. Apparently there are no programmatical safeguards to prevent the support collection process at a “Standard logs plus new statesaves” level from using up all of the systems memory. This would normally not be that bad at all, if the “Standard logs plus new statesaves” wasn't the particular level which IBM support would usually request on support cases concerning SVC and Storwize systems. On the phone the IBM support technician admitted that this common practice is in general probably a bad idea. But he also mentioned that he up to now hadn't heard of the known side effects actually occuring, like in this case.

The suggestion on how to prevent this kind of situation in the future was to either upgrade the V3700 systems from 4GB to 8GB memory – a solution i would gladly take provided it came free of charge – or to only run the support collection process only with the “svc_snap” CLI command or the “Standard logs” option from the WebUI. Since the memory upgrade for free isn't likely to happen, i'll stick with the second suggestion for now.

Incidently a third option came up over the last weekend. Looking at the IBM System Storage SAN Volume Controller V7.3.0.9 Release Note, it could be construed that someone at IBM SVC and Storwize development came to the realization that the issue could also be addressed in software, by altering the resource utilization of the support collection process:

HU00636          Livedump prepare fails on V3500 & V3700 systems with 4GB
                 memory when cache partition fullness is less than 35%

Fingers crossed, this fix really addresses and resolves the issue described above.

After upgrading our test IBM SAN Volume Controller (SVC) systems from version 7.3.0.7 to 7.3.0.8 or later, the previously described Nagios monitoring plugin (Nagios Monitoring - IBM SVC and Storwize) ceased to work. A quick check revealed that the “wbemcli” command line tool from the Standards Based Linux Instrumentation project, which is used in the Nagios plugin to query the CIMOM server on the SVC or Storwize systems, would fail with the following error message:

$ /opt/sblim-wbemcli/bin/wbemcli -dx -noverify ecn https://user:pass@svc-test:5989/root/ibm
*
* /opt/sblim-wbemcli/bin/wbemcli: Http Exception: SSL connect error
*

Re-checking the release notes, this sudden change in behaviour seemed to be explained by the fix:

SSL vulnerability CVE-2014-3566

Not really being that verbose a description, a quick look at the CVE-2014-3566 showed that this is a fix for the “POODLE” issue. So IBM probably switched off the support for the SSLv3 protocol in the SVC and Storwize code. But why would this cause the “wbemcli” command line tool to fail? Here are the steps taken in an analysis of the issue:

1. First, i was trying to get the “wbemcli” command line tool to be a tad more verbose about what it is actually doing:

   $ /opt/sblim-wbemcli/bin/wbemcli -dx -noverify ecn https://user:pass@svc-test:5989/root/ibm
   To server: <?xml version="1.0" encoding="utf-8" ?>
   <CIM CIMVERSION="2.0" DTDVERSION="2.0">
   <MESSAGE ID="4711" PROTOCOLVERSION="1.0"><SIMPLEREQ><IMETHODCALL NAME="EnumerateClassNames"><LOCALNAMESPACEPATH><NAMESPACE NAME="root"></NAMESPACE><NAMESPACE NAME="ibm"></NAMESPACE></LOCALNAMESPACEPATH>
   <IPARAMVALUE NAME="DeepInheritance"><VALUE>TRUE</VALUE></IPARAMVALUE>
   </IMETHODCALL></SIMPLEREQ>
   </MESSAGE></CIM>
   *
   * /opt/sblim-wbemcli/bin/wbemcli: Http Exception: SSL connect error
   *

   Not really an abundance of information in here too.

2. Getting the source code for and, while we're at it, updating the “wbemcli” command line tool from version 1.6.0 to 1.6.3. Spent some time looking through the source code and with the “gdb” debugger to get a feeling for the general program flow and functions/methods being called. While looking through the source code i noticed the cURL debugging options are being set if a environment variable named “CURLDEBUG” is set to “true”. Later also found this mentioned in the ChangeLog file:

   $ CURLDEBUG=true /opt/sblim-wbemcli/bin/wbemcli -dx -noverify ecn https://user:pass@svc-test:5989/root/ibm
   To server: <?xml version="1.0" encoding="utf-8" ?>
   <CIM CIMVERSION="2.0" DTDVERSION="2.0">
   <MESSAGE ID="4711" PROTOCOLVERSION="1.0"><SIMPLEREQ><IMETHODCALL NAME="EnumerateClassNames"><LOCALNAMESPACEPATH><NAMESPACE NAME="root"></NAMESPACE><NAMESPACE NAME="ibm"></NAMESPACE></LOCALNAMESPACEPATH>
   <IPARAMVALUE NAME="DeepInheritance"><VALUE>TRUE</VALUE></IPARAMVALUE>
   </IMETHODCALL></SIMPLEREQ>
   </MESSAGE></CIM>
   * About to connect() to svc-test port 5989 (#0)
   *   Trying 192.168.x.x...
   * connected
   * Connected to svc-test (192.168.x.x) port 5989 (#0)
   * successfully set certificate verify locations:
   *   CAfile: none
     CApath: /etc/ssl/certs
   * Unknown SSL protocol error in connection to svc-test:5989
   * Closing connection #0
   * SSL connect error
   *
   * /opt/sblim-wbemcli/bin/wbemcli: Http Exception: SSL connect error
   *

   Now we know that we're encountering a “Unknown SSL protocol error” – not that helpful either.

3. Searched the web for further information on how to debug the cURL library and found the debug.c example source code, which was very helpful. Incorperated it into the file “CimCurl.cpp”:

   sblim-wbemcli-1.6.3_debug.patch

   --- sblim-wbemcli-1.6.3_orig/CimCurl.cpp    2013-09-21 01:26:32.000000000 +0200
   +++ sblim-wbemcli-1.6.3_new/CimCurl.cpp     2014-11-26 16:30:19.000000000 +0100
   @@ -37,6 +37,100 @@
    extern int waitTime;
    extern int expect100;
    
   +// Trace Begin
   +struct data {
   +  char trace_ascii; /* 1 or 0 */ 
   +};
   +
   +static
   +void dump(const char *text,
   +          FILE *stream, unsigned char *ptr, size_t size,
   +          char nohex)
   +{ 
   +  size_t i;
   +  size_t c;
   +  
   +  unsigned int width=0x10;
   +  
   +  if(nohex)
   +    /* without the hex output, we can fit more on screen */
   +    width = 0x40;
   +
   +  fprintf(stream, "%s, %010.10ld bytes (0x%08.8lx)\n",
   +          text, (long)size, (long)size);
   +
   +  for(i=0; i<size; i+= width) {
   +
   +    fprintf(stream, "%04.4lx: ", (long)i);
   +
   +    if(!nohex) {
   +      /* hex not disabled, show it */
   +      for(c = 0; c < width; c++)
   +        if(i+c < size)
   +          fprintf(stream, "%02x ", ptr[i+c]);
   +        else
   +          fputs("   ", stream);
   +    }
   +
   +    for(c = 0; (c < width) && (i+c < size); c++) {
   +      /* check for 0D0A; if found, skip past and start a new line of output */
   +      if (nohex && (i+c+1 < size) && ptr[i+c]==0x0D && ptr[i+c+1]==0x0A) {
   +        i+=(c+2-width);
   +        break;
   +      }
   +      fprintf(stream, "%c",
   +              (ptr[i+c]>=0x20) && (ptr[i+c]<0x80)?ptr[i+c]:'.');
   +      /* check again for 0D0A, to avoid an extra \n if it's at width */
   +      if (nohex && (i+c+2 < size) && ptr[i+c+1]==0x0D && ptr[i+c+2]==0x0A) {
   +        i+=(c+3-width);
   +        break;
   +      }
   +    }
   +    fputc('\n', stream); /* newline */
   +  }
   +  fflush(stream);
   +}
   +
   +static
   +int my_trace(CURL *handle, curl_infotype type,
   +             char *data, size_t size,
   +             void *userp)
   +{
   +  struct data *config = (struct data *)userp;
   +  const char *text;
   +  (void)handle; /* prevent compiler warning */
   +
   +  switch (type) {
   +  case CURLINFO_TEXT:
   +    fprintf(stderr, "== Info: %s", data);
   +  default: /* in case a new one is introduced to shock us */
   +    return 0;
   +
   +  case CURLINFO_HEADER_OUT:
   +    text = "=> Send header";
   +    break;
   +  case CURLINFO_DATA_OUT:
   +    text = "=> Send data";
   +    break;
   +  case CURLINFO_SSL_DATA_OUT:
   +    text = "=> Send SSL data";
   +    break;
   +  case CURLINFO_HEADER_IN:
   +    text = "<= Recv header";
   +    break;
   +  case CURLINFO_DATA_IN:
   +    text = "<= Recv data";
   +    break;
   +  case CURLINFO_SSL_DATA_IN:
   +    text = "<= Recv SSL data";
   +    break;
   +  }
   +
   +  dump(text, stderr, (unsigned char *)data, size, config->trace_ascii);
   +  return 0;
   +}
   +// Trace End
   +
    // These are the constant headers added to all requests
    static const char *headers[] = {
        "Content-Type: application/xml; charset=\"utf-8\"",
   @@ -152,6 +246,11 @@
        CURLcode rv;
        string sb;
    
   +// Trace Begin
   +struct data config;
   +config.trace_ascii = 1;
   +// Trace End
   +
        mUri = url.scheme + "://" + url.host + ":" + url.port + "/cimom";
        url.ns.toStringBuffer(sb,"%2F");
    
   @@ -248,6 +347,11 @@
    
        rv = curl_easy_setopt(mHandle, CURLOPT_WRITEHEADER, &mErrorData);
        rv = curl_easy_setopt(mHandle, CURLOPT_HEADERFUNCTION, headerCb);
   +
   +// Trace Begin
   +    rv = curl_easy_setopt(mHandle, CURLOPT_DEBUGFUNCTION, my_trace);
   +    rv = curl_easy_setopt(mHandle, CURLOPT_DEBUGDATA, &config);
   +// Trace End
    }
    
    static string getErrorMessage(CURLcode err)

   rebuild the “wbemcli” command line tool and tried again:

   $ CURLDEBUG=true /opt/sblim-wbemcli/bin/wbemcli -dx -noverify ecn https://user:pass@svc-test:5989/root/ibm
   To server: <?xml version="1.0" encoding="utf-8" ?>
   <CIM CIMVERSION="2.0" DTDVERSION="2.0">
   <MESSAGE ID="4711" PROTOCOLVERSION="1.0"><SIMPLEREQ><IMETHODCALL NAME="EnumerateClassNames"><LOCALNAMESPACEPATH><NAMESPACE NAME="root"></NAMESPACE><NAMESPACE NAME="ibm"></NAMESPACE></LOCALNAMESPACEPATH>
   <IPARAMVALUE NAME="DeepInheritance"><VALUE>TRUE</VALUE></IPARAMVALUE>
   </IMETHODCALL></SIMPLEREQ>
   </MESSAGE></CIM>
   == Info: About to connect() to svc-test port 5989 (#0)
   == Info:   Trying 192.168.x.x...
   == Info: connected
   == Info: Connected to svc-test (192.168.x.x) port 5989 (#0)
   == Info: found 172 certificates in /etc/ssl/certs/ca-certificates.crt
   == Info: gnutls_handshake() failed: A TLS packet with unexpected length was received.
   == Info: Closing connection #0
   == Info: SSL connect error
   *
   * /opt/sblim-wbemcli/bin/wbemcli: Http Exception: SSL connect error
   *

   Now we know there is an issue in the way the GnuTLS library used by libcURL interacts with the CIMOM server on the SVC or Storwize systems.

4. Again, searching the web for similar issues with the error message “gnutls_handshake() failed: A TLS packet with unexpected length was received.”, we find a rebuild against the OpenSSL library instead of the GnuTLS library could solve this issue:

   $ dpkg -l | grep curl
   ii  curl                        7.26.0-1+wheezy11   powerpc     command line tool for transferring data with URL syntax
   ii  libcurl3:powerpc            7.26.0-1+wheezy11   powerpc     easy-to-use client-side URL transfer library (OpenSSL flavour)
   ii  libcurl3-gnutls:powerpc     7.26.0-1+wheezy11   powerpc     easy-to-use client-side URL transfer library (GnuTLS flavour)
   ii  libcurl4-gnutls-dev         7.26.0-1+wheezy11   powerpc     development files and documentation for libcurl (GnuTLS flavour)
   
   $ apt-get install libcurl4-openssl-dev
   Reading package lists... Done
   Building dependency tree
   Reading state information... Done
   Suggested packages:
     libcurl3-dbg
   The following packages will be REMOVED:
     libcurl4-gnutls-dev
   The following NEW packages will be installed:
     libcurl4-openssl-dev
   0 upgraded, 1 newly installed, 1 to remove and 0 not upgraded.
   Need to get 0 B/1,259 kB of archives.
   After this operation, 28.7 kB of additional disk space will be used.
   Do you want to continue [Y/n]? y
   (Reading database ... 65717 files and directories currently installed.)
   Removing libcurl4-gnutls-dev ...
   Processing triggers for man-db ...
   Selecting previously unselected package libcurl4-openssl-dev.
   (Reading database ... 65463 files and directories currently installed.)
   Unpacking libcurl4-openssl-dev (from .../libcurl4-openssl-dev_7.26.0-1+wheezy11_powerpc.deb) ...
   Processing triggers for man-db ...
   Setting up libcurl4-openssl-dev (7.26.0-1+wheezy11) ...
   
   $ dpkg -l | grep curl
   ii  curl                        7.26.0-1+wheezy11   powerpc     command line tool for transferring data with URL syntax
   ii  libcurl3:powerpc            7.26.0-1+wheezy11   powerpc     easy-to-use client-side URL transfer library (OpenSSL flavour)
   ii  libcurl3-gnutls:powerpc     7.26.0-1+wheezy11   powerpc     easy-to-use client-side URL transfer library (GnuTLS flavour)
   ii  libcurl4-openssl-dev        7.26.0-1+wheezy11   powerpc     development files and documentation for libcurl (OpenSSL flavour)

   Rebuild the “wbemcli” command line tool and tried again:

   $ CURLDEBUG=true /opt/sblim-wbemcli/bin/wbemcli -dx -noverify ecn https://user:pass@svc-test:5989/root/ibm
   To server: <?xml version="1.0" encoding="utf-8" ?>
   <CIM CIMVERSION="2.0" DTDVERSION="2.0">
   <MESSAGE ID="4711" PROTOCOLVERSION="1.0"><SIMPLEREQ><IMETHODCALL NAME="EnumerateClassNames"><LOCALNAMESPACEPATH><NAMESPACE NAME="root"></NAMESPACE><NAMESPACE NAME="ibm"></NAMESPACE></LOCALNAMESPACEPATH>
   <IPARAMVALUE NAME="DeepInheritance"><VALUE>TRUE</VALUE></IPARAMVALUE>
   </IMETHODCALL></SIMPLEREQ>
   </MESSAGE></CIM>
   == Info: About to connect() to svc-test port 5989 (#0)
   == Info:   Trying 192.168.x.x...
   == Info: connected
   == Info: Connected to svc-test (192.168.x.x) port 5989 (#0)
   == Info: successfully set certificate verify locations:
   == Info:   CAfile: none
     CApath: /etc/ssl/certs
   == Info: SSLv3, TLS handshake, Client hello (1):
   => Send SSL data, 0000000134 bytes (0x00000086)
   0000: ......T.."\.~....`...K4.......p..7"&4...Z.....9.8.........5.....
   0040: ................3.2.....E.D...../...A...........................
   0080: ......
   == Info: Unknown SSL protocol error in connection to svc-test:5989
   == Info: Closing connection #0
   == Info: SSL connect error
   *
   * /opt/sblim-wbemcli/bin/wbemcli: Http Exception: SSL connect error
   *

   Now we know that the “wbemcli” command line tool is actually – as already suspected – trying to initiate a SSLv3 connection.

5. In order to confirm we're on the right track, try to first verify manually that we're unable to connct with a SSLv3 secured connection:

   $ openssl s_client -host svc-test -port 5989 -ssl3
   CONNECTED(00000003)
   write:errno=104
   ---
   no peer certificate available
   ---
   No client certificate CA names sent
   ---
   SSL handshake has read 0 bytes and written 0 bytes
   ---
   New, (NONE), Cipher is (NONE)
   Secure Renegotiation IS NOT supported
   Compression: NONE
   Expansion: NONE
   SSL-Session:
       Protocol  : SSLv3
       Cipher    : 0000
       Session-ID:
       Session-ID-ctx:
       Master-Key:
       Key-Arg   : None
       PSK identity: None
       PSK identity hint: None
       SRP username: None
       Start Time: 1418397594
       Timeout   : 7200 (sec)
       Verify return code: 0 (ok)
   ---
   quit

   And that we're instead able to connect with a TLS secured connection:

   $ openssl s_client -host svc-test -port 5989
   CONNECTED(00000003)
   depth=0 C = GB, L = Hursley, O = IBM, OU = SSG, CN = 2145, emailAddress = support@ibm.com
   verify error:num=18:self signed certificate
   verify return:1
   depth=0 C = GB, L = Hursley, O = IBM, OU = SSG, CN = 2145, emailAddress = support@ibm.com
   verify return:1
   ---
   Certificate chain
    0 s:/C=GB/L=Hursley/O=IBM/OU=SSG/CN=2145/emailAddress=support@ibm.com
      i:/C=GB/L=Hursley/O=IBM/OU=SSG/CN=2145/emailAddress=support@ibm.com
   ---
   Server certificate
   -----BEGIN CERTIFICATE-----
   MIICyDCCAjGgAwIBAgIEUAPzmTANBgkqhkiG9w0BAQUFADBqMQswCQYDVQQGEwJH
   QjEQMA4GA1UEBxMHSHVyc2xleTEMMAoGA1UEChMDSUJNMQwwCgYDVQQLEwNTU0cx
   DTALBgNVBAMTBDIxNDUxHjAcBgkqhkiG9w0BCQEWD3N1cHBvcnRAaWJtLmNvbTAe
   Fw0xMjA3MTYxMDU3MjlaFw0yNzA3MTMxMDU3MjlaMGoxCzAJBgNVBAYTAkdCMRAw
   DgYDVQQHEwdIdXJzbGV5MQwwCgYDVQQKEwNJQk0xDDAKBgNVBAsTA1NTRzENMAsG
   A1UEAxMEMjE0NTEeMBwGCSqGSIb3DQEJARYPc3VwcG9ydEBpYm0uY29tMIGfMA0G
   CSqGSIb3DQEBAQUAA4GNADCBiQKBgQC3E7+7mE2GAID/35o5/s7cnzoqu9PQdOGB
   ryGMa8adD4Wd9hpmTkrsgyNvkUB6sPIifbFstGooOkQtK9ZNgP5OHOorZmqINSxM
   9goCkSCQG9xRKAvNt2tA8gujaV+p42oVEhIH6naJUul96qZI31y3GffUu2CRrJL7
   4wG/8cv0BQIDAQABo3sweTAJBgNVHRMEAjAAMCwGCWCGSAGG+EIBDQQfFh1PcGVu
   U1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUkPMkXUjn0YHlfQW8
   TJiRC5jWQO4wHwYDVR0jBBgwFoAUkPMkXUjn0YHlfQW8TJiRC5jWQO4wDQYJKoZI
   hvcNAQEFBQADgYEAKqu7KpVxnOXonQE3unC1O7qUHKoyQUEWqcKsM/4tPI+lsBMZ
   jvoPwn8yQRWiLehFmVc8VSZfdFPLzshNabXp5qbZo/EFberXrgI2CbtPiULYyyyH
   DUhWF+vhwb6uqwfBbGncvTvI2ewU8+0oTXsuTkSjumJ7+chpaHFWWyj2cJA=
   -----END CERTIFICATE-----
   subject=/C=GB/L=Hursley/O=IBM/OU=SSG/CN=2145/emailAddress=support@ibm.com
   issuer=/C=GB/L=Hursley/O=IBM/OU=SSG/CN=2145/emailAddress=support@ibm.com
   ---
   No client certificate CA names sent
   ---
   SSL handshake has read 1029 bytes and written 498 bytes
   ---
   New, TLSv1/SSLv3, Cipher is AES256-GCM-SHA384
   Server public key is 1024 bit
   Secure Renegotiation IS supported
   Compression: NONE
   Expansion: NONE
   SSL-Session:
       Protocol  : TLSv1.2
       Cipher    : AES256-GCM-SHA384
       Session-ID: 48291D368E0A8584A8DFA00A9881B8979BDE370FC6C9439294C670695D031239
       Session-ID-ctx:
       Master-Key: 71BD12A161FC595CD056DA8E6D6E27420F37468E47498B7591A403A86844C55F61FF02B2FEC7739FAAEDCE3DFEA0F217
       Key-Arg   : None
       PSK identity: None
       PSK identity hint: None
       SRP username: None
       TLS session ticket lifetime hint: 300 (seconds)
       TLS session ticket:
       0000 - a0 e0 b1 9b c2 37 9a ca-49 1c 54 f5 26 4b d6 24   .....7..I.T.&K.$
       0010 - af 6a 7d cc 5e 4a 97 a8-b3 6d b7 66 0b b7 0a 65   .j}.^J...m.f...e
       0020 - 47 af ef 47 76 fc c7 e9-38 ff 84 28 ca 8e 73 25   G..Gv...8..(..s%
       0030 - 47 25 f6 0d 36 01 04 f1-f9 f7 0c b6 42 ef cf 09   G%..6.......B...
       0040 - 64 8f df ff 89 38 ed 7c-ae 1d 0e 25 d1 c1 77 86   d....8.|...%..w.
       0050 - b6 61 88 15 cf fe 9f 20-86 0d 17 74 18 da ea c0   .a..... ...t....
       0060 - 33 3a 47 f5 f9 51 24 ae-48 37 8a 3f 19 dd c6 04   3:G..Q$.H7.?....
       0070 - 7e d1 20 78 35 99 0b 9f-3b 1f ce 7c bc 11 93 e4   ~. x5...;..|....
       0080 - 0f 94 de 94 f1 0d 0c da-64 ca 0d f6 10 2a c8 fa   ........d....*..
       0090 - dc 3e e4 1a 97 d1 34 7a-9c f5 c3 00 e8 1b 10 d7   .>....4z........
   
       Start Time: 1418397614
       Timeout   : 300 (sec)
       Verify return code: 18 (self signed certificate)
   ---
   quit

   The secured connection negotiated to TLS succeeded, so we're on the right track!

6. Now we need to find the spot in the source code, where the “wbemcli” command line tool is forced to initiate a SSLv3 connection. We know this is probably done in a cURL related function call, since libcURL is used for the network connection. So lets first look into the cURL code for all the lines showing any sign of SSL related operations:

   $ grep -n curl CimCurl.cpp | grep -i ssl
   185:    // Assume we support SSL if we don't have the curl_version_info API
   276:    rv = curl_easy_setopt(mHandle, CURLOPT_SSL_VERIFYHOST, 0);
   277:    //    rv = curl_easy_setopt(mHandle, CURLOPT_SSL_VERIFYPEER, 0);
   280:    rv = curl_easy_setopt(mHandle, CURLOPT_SSLVERSION, 3);
   441:     if ((rv=curl_easy_setopt(mHandle,CURLOPT_SSL_VERIFYPEER,0))) {
   448:       if ((rv=curl_easy_setopt(mHandle,CURLOPT_SSL_VERIFYPEER,1))) {
   466:     if ((rv=curl_easy_setopt(mHandle,CURLOPT_SSLCERT,certificate))) {
   470:     if ((rv=curl_easy_setopt(mHandle,CURLOPT_SSLKEY,key))) {

   Looks like a perfect match in line 280 of the file “CimCurl.cpp”. The line numbers are a bit off from the original source code, since the file “CimCurl.cpp” was patched with our above debugging code. The code in the original, unpatched source file “CimCurl.cpp”, within the function “CimomCurl::genRequest” looks like this:

   CimCurl.cpp

   175     [...]
   176     /* Disable SSL host verification */
   177     rv = curl_easy_setopt(mHandle, CURLOPT_SSL_VERIFYHOST, 0);
   178     //    rv = curl_easy_setopt(mHandle, CURLOPT_SSL_VERIFYPEER, 0);
   179
   180     /* Force using SSL V3 */
   181     rv = curl_easy_setopt(mHandle, CURLOPT_SSLVERSION, 3);
   182
   183     /* Set username and password */
   184     if (url.user.length() > 0 && url.password.length() > 0) {
   185         mUserPass = url.user + ":" + url.password;
   186         rv = curl_easy_setopt(mHandle, CURLOPT_USERPWD, mUserPass.c_str());
   187     }
   188     [...]

   In line 181 the cURL option “CURLOPT_SSLVERSION” is indiscriminately set to use SSLv3 and nothing else, which we know from the above deduction is bound to fail on systems adressing the “POODLE” issues.

7. With the knowledge where the issue is actually caused, an easy quick'n'dirty fix can be implemented:

   sblim-wbemcli-1.6.3_debug.patch

   --- sblim-wbemcli-1.6.3_orig/CimCurl.cpp    2013-09-21 01:26:32.000000000 +0200
   +++ sblim-wbemcli-1.6.3/CimCurl.cpp         2014-11-26 16:46:09.000000000 +0100
   @@ -178,7 +178,7 @@
        //    rv = curl_easy_setopt(mHandle, CURLOPT_SSL_VERIFYPEER, 0);
    
        /* Force using SSL V3 */
   -    rv = curl_easy_setopt(mHandle, CURLOPT_SSLVERSION, 3);
   +    //rv = curl_easy_setopt(mHandle, CURLOPT_SSLVERSION, 3);
    
        /* Set username and password */
        if (url.user.length() > 0 && url.password.length() > 0) {

   Inserting the comment at the line where the cURL option “CURLOPT_SSLVERSION” is forced to SSLv3 causes libcURL to fall back to its default value, which is now TLS.

   Rebuild the “wbemcli” command line tool and tried again:

   $ CURLDEBUG=true /opt/sblim-wbemcli/bin/wbemcli -dx -noverify ecn https://user:pass@svc-test:5989/root/ibm
   To server: <?xml version="1.0" encoding="utf-8" ?>
   <CIM CIMVERSION="2.0" DTDVERSION="2.0">
   <MESSAGE ID="4711" PROTOCOLVERSION="1.0"><SIMPLEREQ><IMETHODCALL NAME="EnumerateClassNames"><LOCALNAMESPACEPATH><NAMESPACE NAME="root"></NAMESPACE><NAMESPACE NAME="ibm"></NAMESPACE></LOCALNAMESPACEPATH>
   <IPARAMVALUE NAME="DeepInheritance"><VALUE>TRUE</VALUE></IPARAMVALUE>
   </IMETHODCALL></SIMPLEREQ>
   </MESSAGE></CIM>
   * About to connect() to svc-test port 5989 (#0)
   *   Trying 192.168.x.x...
   * connected
   * Connected to svc-test (192.168.x.x) port 5989 (#0)
   * found 172 certificates in /etc/ssl/certs/ca-certificates.crt
   *    server certificate verification SKIPPED
   *    common name: 2145 (does not match 'svc-test')
   *    server certificate expiration date OK
   *    server certificate activation date OK
   *    certificate public key: RSA
   *    certificate version: #3
   *    subject: C=GB,L=Hursley,O=IBM,OU=SSG,CN=2145,EMAIL=support@ibm.com
   *    start date: Mon, 16 Jul 2012 10:57:29 GMT
   
   *    expire date: Tue, 13 Jul 2027 10:57:29 GMT
   
   *    issuer: C=GB,L=Hursley,O=IBM,OU=SSG,CN=2145,EMAIL=support@ibm.com
   *    compression: NULL
   *    cipher: AES-128-CBC
   *    MAC: SHA1
   * Server auth using Basic with user 'user'
   > POST /cimom HTTP/1.1
   Authorization: Basic enp6bmFnaW9zOm5hZ2lvcw==
   Host: svc-test:5989
   Content-Type: application/xml; charset="utf-8"
   Connection: Keep-Alive, TE
   CIMProtocolVersion: 1.0
   CIMOperation: MethodCall
   CIMMethod: EnumerateClassNames
   CIMObject: root%2Fibm
   Content-Length: 396
   
   * upload completely sent off: 396 out of 396 bytes
   * additional stuff not fine transfer.c:1037: 0 0
   * HTTP 1.1 or later with persistent connection, pipelining supported
   < HTTP/1.1 200 OK
   < Content-Type: application/xml; charset="utf-8"
   From server: Content-Type: application/xml; charset="utf-8"
   < content-length: 0000072284
   From server: content-length: 0000072284
   < CIMOperation: MethodResponse
   From server: CIMOperation: MethodResponse
   <
   * Connection #0 to host svc-test left intact
   From server: <?xml version="1.0" encoding="utf-8" ?>
   <CIM CIMVERSION="2.0" DTDVERSION="2.0">
   <MESSAGE ID="4711" PROTOCOLVERSION="1.0">
   <SIMPLERSP>
   <IMETHODRESPONSE NAME="EnumerateClassNames">
   <IRETURNVALUE>
   <CLASSNAME NAME="CIM_ConcreteIdentity"/>
   <CLASSNAME NAME="CIM_NetworkPacketAction"/>
   <CLASSNAME NAME="CIM_CollectionInSystem"/>
   [...]
   
   $ /opt/sblim-wbemcli/bin/wbemcli -noverify ecn https://user:pass@svc-test:5989/root/ibm
   svc-test:5989/root/ibm:CIM_ConcreteIdentity
   svc-test:5989/root/ibm:CIM_NetworkPacketAction
   svc-test:5989/root/ibm:CIM_CollectionInSystem
   svc-test:5989/root/ibm:CIM_DeviceSAPImplementation
   svc-test:5989/root/ibm:CIM_ProtocolControllerAccessesUnit
   svc-test:5989/root/ibm:CIM_ControlledBy
   [...]

   Great, now we're finally able to query and monitor the IBM SVC or Storwize systems again with the Nagios monitoring plugin (Nagios Monitoring - IBM SVC and Storwize)!

Between first noticing and researching the issue and creating the quick'n'dirty fix shown above, an official bug report has been filed on the issue and a patch has already been submitted to the source code repository in order to adress the issue more thoroughly. Hopefully an updated official source code package will be released soon.

Nonetheless, the process of researching and debugging this issue was an excellent hands on exercise for me, which i enjoyed very much. Hopefully the steps taken and described here, will turn out to be of use for others as well.

We've been using the IBM SAN Volume Controller (SVC) for several years now (since about 2006). While the technological progress was a tad slow in the early years, development – probably along with popularity and more widespread use – of the SVC fortunately picked up considerable pace in recent years. For us the most interesting new features that emerged over time are the:

• stretched cluster configuration, which seems to be far more popular in europe with its traditionally shorter distances between datacenters. It allows us to provide active-active access to host LUNs and thus considerably simplifying a resilient VMware environment.

• synchronous VDisk mirroring, which allows us to provide mirrored LUNs that are independent of the storage vendor in the backend.

• EasyTier, which allows us to utilize our TMS RamSan (now IBM Flash System) flash storage much more efficiently.

• Real-time compression (RTC), which as well allows us to utilize our overall backend storage much more efficiently.

We also went through several cycles of node hardware, starting with our initial 2145-4F2 via the 2145-8F2 and the 2145-8G4 to our current 2145-CG8. Currently our overall environment looks like this:

• Six 2145-CG8 nodes in three I/O groups.

• A stretched cluster configuration covering two datacenters which are within a distance of about 300 meters of single-mode fibre length. A quorum storage on a third location on the same campus.

• Dual fabric 8GB FC SAN consisting of four Brocade DCX-4S directors.

• A total of six disk based storage systems from different vendors (HDS AMS2300, Fujitsu DX90, IBM V3700), one of each vendor in each datacenter, with a total capacity of about 260TB. The systems are typically sized and configured for multiple 4+1 or 8+1 RAID-5 arrays. The arrays are subsequently mapped entirely as LUNs to the SVC. All the LUNs from one disk storage system are pooled together into one MDiskGroup, forming a single, striped failure group.

• A total of six flash based storage systems (TMS RamSan-630, TMS RamSan-810, IBM Flashsystem 820), one of each model in each datacenter, with a total capacity of about 34TB. The systems LUNs are also mapped to the SVC and all the LUNs from one flash storage are added to one disk based MDiskGroup for use as a SSD tier with EasyTier.

• 65 host systems (IBM xSeries, IBM BladeCenter and IBM Power), running either non-virtualized or virtualized (VMware, Xen, PowerVM) workloads.

• 291 VDisks, ranging from sizes of 1GB up to 4TB. Almost all VDisks are mirrored, exceptions are made where the application on the host prefers to do the mirroring or the replication itself (e.g. MS Exchange).

For about seven months – since the end of april 2014 – we've now been using Real-time compression (RTC). We started by implementing RPQ 8S1296 on all of our nodes. This added an additional 6-core CPU and 24GB memory to each node. As a result the “normal” SVC operation still uses four of the CPU-cores on the initially installed CPU and the RTC algorithm is dedicated eight CPU-cores – two on the initially and six on the additionally installed CPU. The SVC code version was v7.1.0.3 when we initially started testing and implementing RTC (see the screenshots below). After gradually converting more and more VDisks to compressed volumes, the CPU load of the RTC algorithm became quite noticable (30-60%). Along with that, a substancially increased latency (>10ms) could be observed. After updating the SVC code to v7.2.0.7, the CPU utilization dropped noticably to 10% and below. The latency also went back to the values (<2ms) usually observed before implementing RTC. There must have been quite an improvement in the RTC code, unfortunately IBM does not publish any such interesting details.

Over the course of implementing RTC, i took four series of screenshots from the SVC WebUI showing the compression allocation view at that particular point in time. They have been combined into the four following images showing all information at once, with the last one being a representation of the current situation:

One thing about the visual representation in the compression allocation view and the volume of RTC licensed storage – which confused me too at first – has to be noted. It seems that with regard to RTC licensing only the storage space allocated to primary VDisk volumes i.e. the unmirrored storage capacity is considered. From the information available on the matter at the time, both the corresponding sections from the “IBM EMEA Software Announcement ZP12-0235”:

The license entitles users for the quantity of terabytes of SVC volumes created with real-time compression enabled. The volume size (and not necessarily the amount of data you are able to store on that volume compressed) is the measure used to determine how many terabytes of 5641-CP1 to license.

and from the “Real-time Compression in SAN Volume Controller and Storwize V7000” redpaper:

In SAN Volume Controller, real-time compression is licensed by the total number of terabytes of virtual capacity, which is the amount of storage that the host sees.

are a bit ambiguous. At first the emphasis seems to be on the matter of uncompressed vs. compressed storage size. But on second thought, the key words “[…] the amount of storage that the host sees.” from the second quote come into play. Those basically say, that enabling RTC on secondary VDisk copies is free, since a host only sees the unmirrored amount of storage.

Example: In our setup each VDisk compressed with RTC has also a secondary copy, also compressed with RTC. There is currently only one exception in the form of a flash copy volume of 2750GB which is compressed, but not mirrored. The currently compressed virtual capacity is 62.67TB. The currently used RTC licenses is being reported as 32.68TB. The calculation is: ( 62.67TB - 2750GB ) / 2 + 2750GB = 32.68TB. Makes sense, doesn't it?

For the four cases shown in the images above, the average RTC+TP and RTC-only compression ratio, relative storage size after compression and relative storage saving have been calculated and are shown in the following table:

Although thin provisioning (TP) alone would amount to 16-20% storage saving on the overall average, the larger and quiet substancial effect of 64-67% of storage saving on the overall average can only be achieved with RTC. Needless to say, we're currently only using RTC on VDisks that promise to yield rather good compression results. This is done in order to save valueable licensed compression volume and to minimize the number of CPU cycles wasted on uncompressible data. For guidelines please see the recommendations in the redbook referenced below in the “Links & Resources” section. The VDisks have either been chosen by estimating the compressability with the comprestimator utility or by knowing the application data very well. VDisks which contain data with an inherent high entropy, like e.g. encrypted and compressed MS Exchange DAGs, repositories of already compressed installation media, compressed Windows system images or compressed AIX mksysb images, page compressed MS SQL data files or fileserver volumes with user data (docx, xlsx, pptx, zip, jpeg, mpeg, etc.) have been excluded from RTC for now. We're still investigating which way to ultimately go on the topic of MS SQL page compression vs. IBM SVC RTC. We're also still sorting out the “mess” of compressible and uncompressible data being mixed together within the same VMware datastores before converting some of them into RTC enabled VDisks. We'll probably end up with a mix of RTC enabled VDisks backing VMware datastores containing compression-friendly VMs and regular VDisks backing datastores containing high-entropy, less compressible VMs. It'll be interesting to see what the ratio between the two will be and which VM will fall into which category.

A detailed compilation of storage saving with RTC on a per VDisk basis is shown in the section Detailed per VDisk RTC data. The columns are pretty self-explainatory, with the most interesting columns being “Application Type”, “Size (%)” and “Saving (%)”. To get a better overview, a condensed view aggregated by “Application Type” was compliled. The following table shows the values aggregated by “Application Type”. The columns “Avg. Size (GB)” and “Avg. Compressed Size (GB)” show the arithmetic average for the VDisk size and compressed size. To be able to compare the different application types against each other, relative arithmetic averages were calculated (columns “Avg. Size (%)” and “vg. Saving (%)”). The columns “Min./Max. Size/Saving (%)” and “Samples” are provided to get an impression about the variability and the significance of the arithmetic averages.

A graphical representation of the above aggregated data is shown in the following graph:

The values “Avg. Size (%)” and “Avg. Saving (%)” are shown as colored bars, the error bars are their respective relative minimum and maximum values, the numbers at the bottom of the colored bars are the number of samples per application type.

The “AIX rootvg” and “Database” application types show rather good compression results, but also a high range of variability. For the case of “AIX rootvg” the compression results depend mainly on the amount of paging space being used, since on all systems the paging space is part of the rootvg. Systems with very low compressed VDisk size also show no or almost no paging space usage. For the case of “Database” the compression results obviously depend on the data stored within the database tablespaces, unfortunately i have no further information on what kind of data is stored in the less compressible cases. In the most compressible cases (5.38% and 7.32% of the original VDisk size) though, the databases contain large amounts of text data and fulltext search indexes, which explains the extraordinary compression results. The “ERP” application type shows equally good compression results, but with a much lower variability, which seems to be largely caused by the current amount of free unused database space. The sample from the “MS SQL” application type is unfortunately the only VDisk directly allocated to a MS SQL Server instance, so it's not really a representative sample. All other MS SQL Server instance in our environment run on VMware and the VDisks there are currently not RTC enabled due to the application and use case mixture described above. The “Code Server” and “Fileserver” application types show good compression results, considering the data usually associated with those. The low sample number indicated that they have specifically been chosen as candidates for RTC enabled VDisks. Especially the other available fileserver VDisks showed much worse results in the preliminary comprestimator runs. Only the the fileserver VDisk containing the Windows server saved user profiles showed good enough results to justify enabling RTC. The “Linux Package” and “Windows Package” application types show the expected worst compression results, since the data on those VDisks is already stored in compressed formats. Still, compression savings of approximately 40% on this kind of data is still pretty good. The “Apache & MySQL” application type is by its compression behaviour kind of a mixture between the “AIX rootvg” and “Fileserver” application types, since it has the same paging space configuration and has large amounts of already compressed data (bz2, gz, docx, xlsx, pptx, zip, jpeg, mpeg, etc.) stored on the VDisks. The “VMware ESX” and “Xen Server” also show average compression results, considering that those are run by 3rd parties where there is currently no distinction between and seperation of compression-friendly from compression-unfriendly VM data being made.

Overall, we're up to now quite happy with the SVCs RTC feature, since it gives us some breathing room and on the overall takes off the pressure to evaluate, purchase, implement and maintain additional storage units for the foreseeable future. On the downside, there is a pretty hefty price tag to the RTC volume licenses. Especially compared to other vendors (e.g. Pure Storage) where the compression functionality is part of the systems base package. One is advised to do a thorough TCO calculation to determine whether the investment in SVC RTC licenses outways the benefits of operating storage systems with actually allocateable space. In the end this boils down to a bet about the current and future compressability of the data actually stored. Besides licensing terms another issue for us was, that we unfortunately pretty quickly reached the limit for the allowed number of compressed VDisks per I/O group on our current 2145-CG8 hardware (max. 200 RTC enabled VDisks). This limit is put in place by SVC development as a safeguard, supposedly in order to keep the nodes from being overloaded with RTC workload. While understandable from a technical point of view, this limit can be reached pretty quickly if you're for example using a lot RTC enabled rootvgs or if you're following the SAP or Oracle disk layout guidelines pretty closely. For the rootvgs and possibly test and development SAP or Oracle systems a consolidation of the system and data disks into fewer VDisks using AIX/VIOS shared storage pools could be an option. But then again, the same downsides for e.g. VMware datastores apply to shared storage pools as well.

Real-time Compression in SAN Volume Controller and Storwize V7000
Implementing IBM Real-time Compression in SAN Volume Controller and IBM Storwize V7000
IBM System Storage SAN Volume Controller software V6.4.0 improves storage efficiency - IBM Europe, Middle East, and Africa Software Announcement ZP12-0235 - June 4, 2012

Detailed compilation of storage saving with RTC on a per VDisk basis. If JavaScript is enabled in your browser, click on a column headers to sort by that particular column.

Some time ago i wrote a – rather crude – Nagios plugin to monitor IBM SAN Volume Controller (SVC) systems. The plugin was initially targeted at version 4.3.x of the SVC software on 2145-8F2 nodes, we used back then. Since the initial implementation of the plugin we upgraded the hard- and software of our SVC systems several times and are now at version 7.1.x of the SVC software on 2145-CG8 nodes. Recently we also got some IBM Storwize V3700 storage arrays, which share the same code as the SVC, but are missing some of the features and provide additional other features. A code and functional review of the original plugin for the SVC as well as an adaption for the Storwize arrays seemed to be in order. The result were the two plugins check_ibm_svc.pl and check_ibm_storwize.pl. They share a lot of common code with the original plugin, but are still maintained seperately for the simple reason that IBM might develop the SVC and the Storwize code in slightly different, incompatible directions.

In order to run the plugins, you need to have the command line tool wbemcli from the Standards Based Linux Instrumentation project installed on the Nagios system. In my case the wbemcli command line tool is placed in /opt/sblim-wbemcli/bin/wbemcli. If you use a different path, adapt the configuration hash entry “%conf{'wbemcli'}” according to your environment. The plugins use wbemcli to query the CIMOM service on the SVC or Storwize system for the necessary information. Therefor a network connection from the Nagios system to the SVC or Storwize systems on port TCP/5989 must be allowed and a user with the “Monitor” authorization must be created on the SVC or Storwize systems:

IBM_2145:svc:admin$ mkuser -name nagios -usergrp Monitor -password <password>

Or in the WebUI:

1. Optional: Enable SNMP traps to be sent to the Nagios system on each of the SVC or Storwize device. This requires SNMPD and SNMPTT to be already setup on the Nagios system. Login to the SVC or Storwize CLI and issue the command:

   IBM_2145:svc:admin$ mksnmpserver -ip <IP adress> -community public -error on -warning on -info on -port 162

   Where <IP> is the IP address of your Nagios system. Or in the SVC or Storwize WebUI navigate to:

   -> Settings
      -> Event Notifications
         -> SNMP
            -> <Enter IP of the Nagios system and the SNMPDs community string>

   Verify the port UDP/162 on the Nagios system can be reached from the SVC or Storwize devices.

For SAN Volume Controller (SVC) devices the whole setup looks like this:

1. Download the Nagios plugin check_ibm_svc.pl and place it in the plugins directory of your Nagios system, in this example /usr/lib/nagios/plugins/:

   $ mv -i check_ibm_svc.pl /usr/lib/nagios/plugins/
   $ chmod 755 /usr/lib/nagios/plugins/check_ibm_svc.pl

2. Adjust the plugin settings according to your environment. Edit the following variable assignments:

   my %conf = (
       wbemcli => '/opt/sblim-wbemcli/bin/wbemcli',

3. Define the following Nagios commands. In this example this is done in the file /etc/nagios-plugins/config/check_svc.cfg:

   # check SVC Backend Controller status
   define command {
       command_name    check_svc_bc
       command_line    $USER1$/check_ibm_svc.pl -H $HOSTNAME$ -u <user> -p <password> -C BackendController
   }
   # check SVC Backend SCSI Status
   define command {
       command_name    check_svc_btspe
       command_line    $USER1$/check_ibm_svc.pl -H $HOSTNAME$ -u <user> -p <password> -C BackendTargetSCSIPE
   }
   # check SVC MDisk status
   define command {
       command_name    check_svc_bv
       command_line    $USER1$/check_ibm_svc.pl -H $HOSTNAME$ -u <user> -p <password> -C BackendVolume
   }
   # check SVC Cluster status
   define command {
       command_name    check_svc_cl
       command_line    $USER1$/check_ibm_svc.pl -H $HOSTNAME$ -u <user> -p <password> -C Cluster
   }
   # check SVC MDiskGroup status
   define command {
       command_name    check_svc_csp
       command_line    $USER1$/check_ibm_svc.pl -H $HOSTNAME$ -u <user> -p <password> -C ConcreteStoragePool
   }
   # check SVC Ethernet Port status
   define command {
       command_name    check_svc_eth
       command_line    $USER1$/check_ibm_svc.pl -H $HOSTNAME$ -u <user> -p <password> -C EthernetPort
   }
   # check SVC FC Port status
   define command {
       command_name    check_svc_fcp
       command_line    $USER1$/check_ibm_svc.pl -H $HOSTNAME$ -u <user> -p <password> -C FCPort
   }
   # check SVC FC Port statistics
   define command {
       command_name    check_svc_fcp_stats
       command_line    $USER1$/check_ibm_svc.pl -H $HOSTNAME$ -u <user> -p <password> -C FCPortStatistics
   }
   # check SVC I/O Group status and memory allocation
   define command {
       command_name    check_svc_iogrp
       command_line    $USER1$/check_ibm_svc.pl -H $HOSTNAME$ -u <user> -p <password> -C IOGroup -w $ARG1$ -c $ARG2$
   }
   # check SVC WebUI status
   define command {
       command_name    check_svc_mc
       command_line    $USER1$/check_ibm_svc.pl -H $HOSTNAME$ -u <user> -p <password> -C MasterConsole
   }
   # check SVC VDisk Mirror status
   define command {
       command_name    check_svc_mirror
       command_line    $USER1$/check_ibm_svc.pl -H $HOSTNAME$ -u <user> -p <password> -C MirrorExtent
   }
   # check SVC Node status
   define command {
       command_name    check_svc_node
       command_line    $USER1$/check_ibm_svc.pl -H $HOSTNAME$ -u <user> -p <password> -C Node
   }
   # check SVC Quorum Disk status
   define command {
       command_name    check_svc_quorum
       command_line    $USER1$/check_ibm_svc.pl -H $HOSTNAME$ -u <user> -p <password> -C QuorumDisk
   }
   # check SVC Storage Volume status
   define command {
       command_name    check_svc_sv
       command_line    $USER1$/check_ibm_svc.pl -H $HOSTNAME$ -u <user> -p <password> -C StorageVolume
   }

   Replace <user> and <password> with name and password of the CIMOM user created above.

4. Define a group of services in your Nagios configuration to be checked for each SVC system:

   # check sshd
   define service {
       use                     generic-service
       hostgroup_name          svc
       service_description     Check_SSH
       check_command           check_ssh
   }
   # check_tcp CIMOM
   define service {
       use                     generic-service-pnp
       hostgroup_name          svc
       service_description     Check_CIMOM
       check_command           check_tcp!5989
   }
   # check_svc_bc
   define service {
       use                     generic-service-pnp
       hostgroup_name          svc
       service_description     Check_Backend_Controller
       check_command           check_svc_bc
   }
   # check_svc_btspe
   define service {
       use                     generic-service-pnp
       hostgroup_name          svc
       service_description     Check_Backend_Target
       check_command           check_svc_btspe
   }
   # check_svc_bv
   define service {
       use                     generic-service-pnp
       hostgroup_name          svc
       service_description     Check_Backend_Volume
       check_command           check_svc_bv
   }
   # check_svc_cl
   define service {
       use                     generic-service-pnp
       hostgroup_name          svc
       service_description     Check_Cluster
       check_command           check_svc_cl
   }
   # check_svc_csp
   define service {
       use                     generic-service-pnp
       hostgroup_name          svc
       service_description     Check_Storage_Pool
       check_command           check_svc_csp
   }
   # check_svc_eth
   define service {
       use                     generic-service-pnp
       hostgroup_name          svc
       service_description     Check_Ethernet_Port
       check_command           check_svc_eth
   }
   # check_svc_fcp
   define service {
       use                     generic-service-pnp
       hostgroup_name          svc
       service_description     Check_FC_Port
       check_command           check_svc_fcp
   }
   # check_svc_fcp_stats
   define service {
       use                     generic-service-pnp
       hostgroup_name          svc
       service_description     Check_FC_Port_Statistics
       check_command           check_svc_fcp_stats
   }
   # check_svc_iogrp
   define service {
       use                     generic-service-pnp
       hostgroup_name          svc
       service_description     Check_IO_Group
       check_command           check_svc_iogrp!102400!204800
   }
   # check_svc_mc
   define service {
       use                     generic-service-pnp
       hostgroup_name          svc
       service_description     Check_Master_Console
       check_command           check_svc_mc
   }
   # check_svc_mirror
   define service {
       use                     generic-service-pnp
       hostgroup_name          svc
       service_description     Check_Mirror_Extents
       check_command           check_svc_mirror
   }
   # check_svc_node
   define service {
       use                     generic-service-pnp
       hostgroup_name          svc
       service_description     Check_Node
       check_command           check_svc_node
   }
   # check_svc_quorum
   define service {
       use                     generic-service-pnp
       hostgroup_name          svc
       service_description     Check_Quorum
       check_command           check_svc_quorum
   }
   # check_svc_sv
   define service {
       use                     generic-service-pnp
       hostgroup_name          svc
       service_description     Check_Storage_Volume
       check_command           check_svc_sv
   }

   Replace generic-service with your Nagios service template. Replace generic-service-pnp with your Nagios service template that has performance data processing enabled.

5. Define hosts in your Nagios configuration for each SVC device. In this example its named svc1:

   define host {
       use         svc
       host_name   svc1
       alias       SAN Volume Controller 1
       address     10.0.0.1
       parents     parent_lan
   }

   Replace svc with your Nagios host template for SVC devices. Adjust the address and parents parameters according to your environment.

6. Define a hostgroup in your Nagios configuration for all SVC systems. In this example it is named svc. The above checks are run against each member of the hostgroup:

   define hostgroup {
       hostgroup_name  svc
       alias           IBM SVC Clusters
       members         svc1
   }

7. Run a configuration check and if successful reload the Nagios process:

   $ /usr/sbin/nagios3 -v /etc/nagios3/nagios.cfg
   $ /etc/init.d/nagios3 reload

The new hosts and services should soon show up in the Nagios web interface.

For Storwize devices the whole setup looks like this:

1. Download the Nagios plugin check_ibm_storwize.pl and place it in the plugins directory of your Nagios system, in this example /usr/lib/nagios/plugins/:

   $ mv -i check_ibm_storwize.pl /usr/lib/nagios/plugins/
   $ chmod 755 /usr/lib/nagios/plugins/check_ibm_storwize.pl

2. Adjust the plugin settings according to your environment. Edit the following variable assignments:

   my %conf = (
       wbemcli => '/opt/sblim-wbemcli/bin/wbemcli',

3. Define the following Nagios commands. In this example this is done in the file /etc/nagios-plugins/config/check_storwize.cfg:

   # check Storwize RAID Array status
   define command {
       command_name    check_storwize_array
       command_line    $USER1$/check_ibm_storwize.pl -H $HOSTNAME$ -u <user> -p <password> -C Array
   }
   # check Storwize Hot Spare coverage
   define command {
       command_name    check_storwize_asc
       command_line    $USER1$/check_ibm_storwize.pl -H $HOSTNAME$ -u <user> -p <password> -C ArrayBasedOnDiskDrive
   }
   # check Storwize MDisk status
   define command {
       command_name    check_storwize_bv
       command_line    $USER1$/check_ibm_storwize.pl -H $HOSTNAME$ -u <user> -p <password> -C BackendVolume
   }
   # check Storwize Cluster status
   define command {
       command_name    check_storwize_cl
       command_line    $USER1$/check_ibm_storwize.pl -H $HOSTNAME$ -u <user> -p <password> -C Cluster
   }
   # check Storwize MDiskGroup status
   define command {
       command_name    check_storwize_csp
       command_line    $USER1$/check_ibm_storwize.pl -H $HOSTNAME$ -u <user> -p <password> -C ConcreteStoragePool
   }
   # check Storwize Disk status
   define command {
       command_name    check_storwize_disk
       command_line    $USER1$/check_ibm_storwize.pl -H $HOSTNAME$ -u <user> -p <password> -C DiskDrive
   }
   # check Storwize Enclosure status
   define command {
       command_name    check_storwize_enc
       command_line    $USER1$/check_ibm_storwize.pl -H $HOSTNAME$ -u <user> -p <password> -C Enclosure
   }
   # check Storwize Ethernet Port status
   define command {
       command_name    check_storwize_eth
       command_line    $USER1$/check_ibm_storwize.pl -H $HOSTNAME$ -u <user> -p <password> -C EthernetPort
   }
   # check Storwize FC Port status
   define command {
       command_name    check_storwize_fcp
       command_line    $USER1$/check_ibm_storwize.pl -H $HOSTNAME$ -u <user> -p <password> -C FCPort
   }
   # check Storwize I/O Group status and memory allocation
   define command {
       command_name    check_storwize_iogrp
       command_line    $USER1$/check_ibm_storwize.pl -H $HOSTNAME$ -u <user> -p <password> -C IOGroup -w $ARG1$ -c $ARG2$
   }
   # check Storwize Hot Spare status
   define command {
       command_name    check_storwize_is
       command_line    $USER1$/check_ibm_storwize.pl -H $HOSTNAME$ -u <user> -p <password> -C IsSpare
   }
   # check Storwize WebUI status
   define command {
       command_name    check_storwize_mc
       command_line    $USER1$/check_ibm_storwize.pl -H $HOSTNAME$ -u <user> -p <password> -C MasterConsole
   }
   # check Storwize VDisk Mirror status
   define command {
       command_name    check_storwize_mirror
       command_line    $USER1$/check_ibm_storwize.pl -H $HOSTNAME$ -u <user> -p <password> -C MirrorExtent
   }
   # check Storwize Node status
   define command {
       command_name    check_storwize_node
       command_line    $USER1$/check_ibm_storwize.pl -H $HOSTNAME$ -u <user> -p <password> -C Node
   }
   # check Storwize Quorum Disk status
   define command {
       command_name    check_storwize_quorum
       command_line    $USER1$/check_ibm_storwize.pl -H $HOSTNAME$ -u <user> -p <password> -C QuorumDisk
   }
   # check Storwize Storage Volume status
   define command {
       command_name    check_storwize_sv
       command_line    $USER1$/check_ibm_storwize.pl -H $HOSTNAME$ -u <user> -p <password> -C StorageVolume
   }

   Replace <user> and <password> with name and password of the CIMOM user created above.

4. Define a group of services in your Nagios configuration to be checked for each Storwize system:

   # check sshd
   define service {
       use                     generic-service
       hostgroup_name          storwize
       service_description     Check_SSH
       check_command           check_ssh
   }
   # check_tcp CIMOM
   define service {
       use                     generic-service-pnp
       hostgroup_name          storwize
       service_description     Check_CIMOM
       check_command           check_tcp!5989
   }
   # check_storwize_array
   define service {
       use                     generic-service-pnp
       hostgroup_name          storwize
       service_description     Check_Array
       check_command           check_storwize_array
   }
   # check_storwize_asc
   define service {
       use                     generic-service-pnp
       hostgroup_name          storwize
       service_description     Check_Array_Spare_Coverage
       check_command           check_storwize_asc
   }
   # check_storwize_bv
   define service {
       use                     generic-service-pnp
       hostgroup_name          storwize
       service_description     Check_Backend_Volume
       check_command           check_storwize_bv
   }
   # check_storwize_cl
   define service {
       use                     generic-service-pnp
       hostgroup_name          storwize
       service_description     Check_Cluster
       check_command           check_storwize_cl
   }
   # check_storwize_csp
   define service {
       use                     generic-service-pnp
       hostgroup_name          storwize
       service_description     Check_Storage_Pool
       check_command           check_storwize_csp
   }
   # check_storwize_disk
   define service {
       use                     generic-service-pnp
       hostgroup_name          storwize
       service_description     Check_Disk_Drive
       check_command           check_storwize_disk
   }
   # check_storwize_enc
   define service {
       use                     generic-service-pnp
       hostgroup_name          storwize
       service_description     Check_Enclosure
       check_command           check_storwize_enc
   }
   # check_storwize_eth
   define service {
       use                     generic-service-pnp
       hostgroup_name          storwize
       service_description     Check_Ethernet_Port
       check_command           check_storwize_eth
   }
   # check_storwize_fcp
   define service {
       use                     generic-service-pnp
       hostgroup_name          storwize
       service_description     Check_FC_Port
       check_command           check_storwize_fcp
   }
   # check_storwize_iogrp
   define service {
       use                     generic-service-pnp
       hostgroup_name          storwize
       service_description     Check_IO_Group
       check_command           check_storwize_iogrp!102400!204800
   }
   # check_storwize_is
   define service {
       use                     generic-service-pnp
       hostgroup_name          storwize
       service_description     Check_Hot_Spare
       check_command           check_storwize_is
   }
   # check_storwize_mc
   define service {
       use                     generic-service-pnp
       hostgroup_name          storwize
       service_description     Check_Master_Console
       check_command           check_storwize_mc
   }
   # check_storwize_mirror
   define service {
       use                     generic-service-pnp
       hostgroup_name          storwize
       service_description     Check_Mirror_Extents
       check_command           check_storwize_mirror
   }
   # check_storwize_node
   define service {
       use                     generic-service-pnp
       hostgroup_name          storwize
       service_description     Check_Node
       check_command           check_storwize_node
   }
   # check_storwize_quorum
   define service {
       use                     generic-service-pnp
       hostgroup_name          storwize
       service_description     Check_Quorum
       check_command           check_storwize_quorum
   }
   # check_storwize_sv
   define service {
       use                     generic-service-pnp
       hostgroup_name          storwize
       service_description     Check_Storage_Volume
       check_command           check_storwize_sv
   }

   Replace generic-service with your Nagios service template. Replace generic-service-pnp with your Nagios service template that has performance data processing enabled.

5. Define hosts in your Nagios configuration for each Storwize device. In this example its named storwize1:

   define host {
       use         disk
       host_name   storwize1
       alias       Storwize Disk Storage 1
       address     10.0.0.1
       parents     parent_lan
   }

   Replace disk with your Nagios host template for storage devices. Adjust the address and parents parameters according to your environment.

6. Define a hostgroup in your Nagios configuration for all SVC systems. In this example it is named storwize. The above checks are run against each member of the hostgroup:

   define hostgroup {
       hostgroup_name  storwize
       alias           IBM Storwize Devices
       members         storwize1
   }

7. Run a configuration check and if successful reload the Nagios process:

   $ /usr/sbin/nagios3 -v /etc/nagios3/nagios.cfg
   $ /etc/init.d/nagios3 reload

The new hosts and services should soon show up in the Nagios web interface.

If the optional step in the “Generic” section above was done, SNMPTT also needs to be configured to be able to understand the incoming SNMP traps from Storwize systems. This can be achieved by the following steps:

1. Download the IBM SVC/Storwize SNMP MIB matching your software version from ftp://ftp.software.ibm.com/storage/san/sanvc/.

2. Convert the IBM SVC/Storwize SNMP MIB definitions in SVC_MIB_<version>.MIB into a format that SNMPTT can understand.

   $ /opt/snmptt/snmpttconvertmib --in=MIB/SVC_MIB_7.1.0.MIB --out=/opt/snmptt/conf/snmptt.conf.ibm-svc-710
   
   ...
   Done
   
   Total translations:        3
   Successful translations:   3
   Failed translations:       0

3. Edit the trap severity according to your requirements, e.g.:

   $ vim /opt/snmptt/conf/snmptt.conf.ibm-svc-710
   
   ...
   EVENT tsveETrap .1.3.6.1.4.1.2.6.190.1 "Status Events" Critical
   
   ...
   EVENT tsveWTrap .1.3.6.1.4.1.2.6.190.2 "Status Events" Warning
   ...

4. Optional: Apply the following patch to the configuration to reduce the number of false positives:

   snmptt.conf.ibm-svc-710

   -- /opt/snmptt/conf/snmptt.conf.ibm-svc-710.orig       2013-12-28 21:16:25.000000000 +0100
   +++ /opt/snmptt/conf/snmptt.conf.ibm-svc-710    2013-12-28 21:17:55.000000000 +0100
   @@ -29,11 +29,21 @@
      16: tsveMPNO
      17: tsveOBJN
    EDESC
   +# Filter and ignore the following events that are not really warnings
   +#   "Error ID = 980440": Failed to transfer file from remote node
   +#   "Error ID = 981001": Cluster Fabric View updated by fabric discovery
   +#   "Error ID = 981014": LUN Discovery failed
   +#   "Error ID = 982009": Migration complete
    #
   +EVENT tsveWTrap .1.3.6.1.4.1.2.6.190.2 "Status Events" Normal
   +FORMAT tsve information trap $*
   +MATCH $3: (Error ID = 980440|981001|981014|982009)
    #
   +# All remaining events with this OID are actually warnings
    #
    EVENT tsveWTrap .1.3.6.1.4.1.2.6.190.2 "Status Events" Warning
    FORMAT tsve warning trap $*
   +MATCH $3: !(Error ID = 980440|981001|981014|982009)
    SDESC
    tsve warning trap
    Variables:

5. Add the new configuration file to be included in the global SNMPTT configuration and restart the SNMPTT daemon:

   $ vim /opt/snmptt/snmptt.ini
   
   ...
   [TrapFiles]
   snmptt_conf_files = <<END
   ...
   /opt/snmptt/conf/snmptt.conf.ibm-svc-710
   ...
   END
   
   $ /etc/init.d/snmptt reload

6. Download the Nagios plugin check_snmp_traps.sh and place it in the plugins directory of your Nagios system, in this example /usr/lib/nagios/plugins/:

   $ mv -i check_snmp_traps.sh /usr/lib/nagios/plugins/
   $ chmod 755 /usr/lib/nagios/plugins/check_snmp_traps.sh

7. Define the following Nagios command to check for SNMP traps in the SNMPTT database. In this example this is done in the file /etc/nagios-plugins/config/check_snmp_traps.cfg:

   # check for snmp traps
   define command {
       command_name    check_snmp_traps
       command_line    $USER1$/check_snmp_traps.sh -H $HOSTNAME$:$HOSTADDRESS$ -u <user> -p <pass> -d <snmptt_db>
   }

   Replace user, pass and snmptt_db with values suitable for your SNMPTT database environment.

8. Add another service in your Nagios configuration to be checked for each SVC:

   # check snmptraps
   define service {
       use                     generic-service
       hostgroup_name          svc
       service_description     Check_SNMP_traps
       check_command           check_snmp_traps
   }

   or Storwize system:

   # check snmptraps
   define service {
       use                     generic-service
       hostgroup_name          storwize
       service_description     Check_SNMP_traps
       check_command           check_snmp_traps
   }

9. Optional: Define a serviceextinfo to display a folder icon next to the Check_SNMP_traps service check for each SVC:

   define  serviceextinfo {
       hostgroup_name          svc
       service_description     Check_SNMP_traps
       notes                   SNMP Alerts
       #notes_url               http://<hostname>/nagios3/nagtrap/index.php?hostname=$HOSTNAME$
       #notes_url               http://<hostname>/nagios3/nsti/index.php?perpage=100&hostname=$HOSTNAME$
   }

   or Storwize system:

   define  serviceextinfo {
       hostgroup_name          storwize
       service_description     Check_SNMP_traps
       notes                   SNMP Alerts
       #notes_url               http://<hostname>/nagios3/nagtrap/index.php?hostname=$HOSTNAME$
       #notes_url               http://<hostname>/nagios3/nsti/index.php?perpage=100&hostname=$HOSTNAME$
   }

   device. This icon provides a direct link to the SNMPTT web interface with a filter for the selected host. Uncomment the notes_url depending on which web interface (nagtrap or nsti) is used. Replace hostname with the FQDN or IP address of the server running the web interface.

10. Run a configuration check and if successful reload the Nagios process:

    $ /usr/sbin/nagios3 -v /etc/nagios3/nagios.cfg
    $ /etc/init.d/nagios3 reload

11. Optional: If you're running PNP4Nagios v0.6 or later to graph Nagios performance data, you can use the PNP4Nagios template in pnp4nagios_storwize.tar.bz2 and pnp4nagios_svc.tar.bz2 to beautify the graphs. Download the PNP4Nagios templates pnp4nagios_svc.tar.bz2 and pnp4nagios_storwize.tar.bz2 and place them in the PNP4Nagios template directory, in this example /usr/share/pnp4nagios/html/templates/:

    $ tar jxf pnp4nagios_storwize.tar.bz2
    $ mv -i check_storwize_*.php /usr/share/pnp4nagios/html/templates/
    $ chmod 644 /usr/share/pnp4nagios/html/templates/check_storwize_*.php
    
    $ tar jxf pnp4nagios_svc.tar.bz2
    $ mv -i check_svc_*.php /usr/share/pnp4nagios/html/templates/
    $ chmod 644 /usr/share/pnp4nagios/html/templates/check_svc_*.php

All done, you should now have a complete Nagios-based monitoring solution for your IBM SVC and Storwize systems.