bityard Blog

// Nagios Monitoring - IBM SVC and Storwize (Update)

After upgrading our test IBM SAN Volume Controller (SVC) systems from version 7.3.0.7 to 7.3.0.8 or later, the previously described Nagios monitoring plugin (Nagios Monitoring - IBM SVC and Storwize) ceased to work. A quick check revealed that the “wbemcli” command line tool from the Standards Based Linux Instrumentation project, which is used in the Nagios plugin to query the CIMOM server on the SVC or Storwize systems, would fail with the following error message:

$ /opt/sblim-wbemcli/bin/wbemcli -dx -noverify ecn https://user:pass@svc-test:5989/root/ibm
*
* /opt/sblim-wbemcli/bin/wbemcli: Http Exception: SSL connect error
*

Re-checking the release notes, this sudden change in behaviour seemed to be explained by the fix:

SSL vulnerability CVE-2014-3566

Not really being that verbose a description, a quick look at the CVE-2014-3566 showed that this is a fix for the “POODLE” issue. So IBM probably switched off the support for the SSLv3 protocol in the SVC and Storwize code. But why would this cause the “wbemcli” command line tool to fail? Here are the steps taken in an analysis of the issue:

  1. First, i was trying to get the “wbemcli” command line tool to be a tad more verbose about what it is actually doing:

    $ /opt/sblim-wbemcli/bin/wbemcli -dx -noverify ecn https://user:pass@svc-test:5989/root/ibm
    To server: <?xml version="1.0" encoding="utf-8" ?>
    <CIM CIMVERSION="2.0" DTDVERSION="2.0">
    <MESSAGE ID="4711" PROTOCOLVERSION="1.0"><SIMPLEREQ><IMETHODCALL NAME="EnumerateClassNames"><LOCALNAMESPACEPATH><NAMESPACE NAME="root"></NAMESPACE><NAMESPACE NAME="ibm"></NAMESPACE></LOCALNAMESPACEPATH>
    <IPARAMVALUE NAME="DeepInheritance"><VALUE>TRUE</VALUE></IPARAMVALUE>
    </IMETHODCALL></SIMPLEREQ>
    </MESSAGE></CIM>
    *
    * /opt/sblim-wbemcli/bin/wbemcli: Http Exception: SSL connect error
    *
    

    Not really an abundance of information in here too.

  2. Getting the source code for and, while we're at it, updating the “wbemcli” command line tool from version 1.6.0 to 1.6.3. Spent some time looking through the source code and with the “gdb” debugger to get a feeling for the general program flow and functions/methods being called. While looking through the source code i noticed the cURL debugging options are being set if a environment variable named “CURLDEBUG” is set to “true”. Later also found this mentioned in the ChangeLog file:

    $ CURLDEBUG=true /opt/sblim-wbemcli/bin/wbemcli -dx -noverify ecn https://user:pass@svc-test:5989/root/ibm
    To server: <?xml version="1.0" encoding="utf-8" ?>
    <CIM CIMVERSION="2.0" DTDVERSION="2.0">
    <MESSAGE ID="4711" PROTOCOLVERSION="1.0"><SIMPLEREQ><IMETHODCALL NAME="EnumerateClassNames"><LOCALNAMESPACEPATH><NAMESPACE NAME="root"></NAMESPACE><NAMESPACE NAME="ibm"></NAMESPACE></LOCALNAMESPACEPATH>
    <IPARAMVALUE NAME="DeepInheritance"><VALUE>TRUE</VALUE></IPARAMVALUE>
    </IMETHODCALL></SIMPLEREQ>
    </MESSAGE></CIM>
    * About to connect() to svc-test port 5989 (#0)
    *   Trying 192.168.x.x...
    * connected
    * Connected to svc-test (192.168.x.x) port 5989 (#0)
    * successfully set certificate verify locations:
    *   CAfile: none
      CApath: /etc/ssl/certs
    * Unknown SSL protocol error in connection to svc-test:5989
    * Closing connection #0
    * SSL connect error
    *
    * /opt/sblim-wbemcli/bin/wbemcli: Http Exception: SSL connect error
    *
    

    Now we know that we're encountering a “Unknown SSL protocol error” – not that helpful either.

  3. Searched the web for further information on how to debug the cURL library and found the debug.c example source code, which was very helpful. Incorperated it into the file “CimCurl.cpp”:

    sblim-wbemcli-1.6.3_debug.patch
    --- sblim-wbemcli-1.6.3_orig/CimCurl.cpp    2013-09-21 01:26:32.000000000 +0200
    +++ sblim-wbemcli-1.6.3_new/CimCurl.cpp     2014-11-26 16:30:19.000000000 +0100
    @@ -37,6 +37,100 @@
     extern int waitTime;
     extern int expect100;
     
    +// Trace Begin
    +struct data {
    +  char trace_ascii; /* 1 or 0 */ 
    +};
    +
    +static
    +void dump(const char *text,
    +          FILE *stream, unsigned char *ptr, size_t size,
    +          char nohex)
    +{ 
    +  size_t i;
    +  size_t c;
    +  
    +  unsigned int width=0x10;
    +  
    +  if(nohex)
    +    /* without the hex output, we can fit more on screen */
    +    width = 0x40;
    +
    +  fprintf(stream, "%s, %010.10ld bytes (0x%08.8lx)\n",
    +          text, (long)size, (long)size);
    +
    +  for(i=0; i<size; i+= width) {
    +
    +    fprintf(stream, "%04.4lx: ", (long)i);
    +
    +    if(!nohex) {
    +      /* hex not disabled, show it */
    +      for(c = 0; c < width; c++)
    +        if(i+c < size)
    +          fprintf(stream, "%02x ", ptr[i+c]);
    +        else
    +          fputs("   ", stream);
    +    }
    +
    +    for(c = 0; (c < width) && (i+c < size); c++) {
    +      /* check for 0D0A; if found, skip past and start a new line of output */
    +      if (nohex && (i+c+1 < size) && ptr[i+c]==0x0D && ptr[i+c+1]==0x0A) {
    +        i+=(c+2-width);
    +        break;
    +      }
    +      fprintf(stream, "%c",
    +              (ptr[i+c]>=0x20) && (ptr[i+c]<0x80)?ptr[i+c]:'.');
    +      /* check again for 0D0A, to avoid an extra \n if it's at width */
    +      if (nohex && (i+c+2 < size) && ptr[i+c+1]==0x0D && ptr[i+c+2]==0x0A) {
    +        i+=(c+3-width);
    +        break;
    +      }
    +    }
    +    fputc('\n', stream); /* newline */
    +  }
    +  fflush(stream);
    +}
    +
    +static
    +int my_trace(CURL *handle, curl_infotype type,
    +             char *data, size_t size,
    +             void *userp)
    +{
    +  struct data *config = (struct data *)userp;
    +  const char *text;
    +  (void)handle; /* prevent compiler warning */
    +
    +  switch (type) {
    +  case CURLINFO_TEXT:
    +    fprintf(stderr, "== Info: %s", data);
    +  default: /* in case a new one is introduced to shock us */
    +    return 0;
    +
    +  case CURLINFO_HEADER_OUT:
    +    text = "=> Send header";
    +    break;
    +  case CURLINFO_DATA_OUT:
    +    text = "=> Send data";
    +    break;
    +  case CURLINFO_SSL_DATA_OUT:
    +    text = "=> Send SSL data";
    +    break;
    +  case CURLINFO_HEADER_IN:
    +    text = "<= Recv header";
    +    break;
    +  case CURLINFO_DATA_IN:
    +    text = "<= Recv data";
    +    break;
    +  case CURLINFO_SSL_DATA_IN:
    +    text = "<= Recv SSL data";
    +    break;
    +  }
    +
    +  dump(text, stderr, (unsigned char *)data, size, config->trace_ascii);
    +  return 0;
    +}
    +// Trace End
    +
     // These are the constant headers added to all requests
     static const char *headers[] = {
         "Content-Type: application/xml; charset=\"utf-8\"",
    @@ -152,6 +246,11 @@
         CURLcode rv;
         string sb;
     
    +// Trace Begin
    +struct data config;
    +config.trace_ascii = 1;
    +// Trace End
    +
         mUri = url.scheme + "://" + url.host + ":" + url.port + "/cimom";
         url.ns.toStringBuffer(sb,"%2F");
     
    @@ -248,6 +347,11 @@
     
         rv = curl_easy_setopt(mHandle, CURLOPT_WRITEHEADER, &mErrorData);
         rv = curl_easy_setopt(mHandle, CURLOPT_HEADERFUNCTION, headerCb);
    +
    +// Trace Begin
    +    rv = curl_easy_setopt(mHandle, CURLOPT_DEBUGFUNCTION, my_trace);
    +    rv = curl_easy_setopt(mHandle, CURLOPT_DEBUGDATA, &config);
    +// Trace End
     }
     
     static string getErrorMessage(CURLcode err)

    rebuild the “wbemcli” command line tool and tried again:

    $ CURLDEBUG=true /opt/sblim-wbemcli/bin/wbemcli -dx -noverify ecn https://user:pass@svc-test:5989/root/ibm
    To server: <?xml version="1.0" encoding="utf-8" ?>
    <CIM CIMVERSION="2.0" DTDVERSION="2.0">
    <MESSAGE ID="4711" PROTOCOLVERSION="1.0"><SIMPLEREQ><IMETHODCALL NAME="EnumerateClassNames"><LOCALNAMESPACEPATH><NAMESPACE NAME="root"></NAMESPACE><NAMESPACE NAME="ibm"></NAMESPACE></LOCALNAMESPACEPATH>
    <IPARAMVALUE NAME="DeepInheritance"><VALUE>TRUE</VALUE></IPARAMVALUE>
    </IMETHODCALL></SIMPLEREQ>
    </MESSAGE></CIM>
    == Info: About to connect() to svc-test port 5989 (#0)
    == Info:   Trying 192.168.x.x...
    == Info: connected
    == Info: Connected to svc-test (192.168.x.x) port 5989 (#0)
    == Info: found 172 certificates in /etc/ssl/certs/ca-certificates.crt
    == Info: gnutls_handshake() failed: A TLS packet with unexpected length was received.
    == Info: Closing connection #0
    == Info: SSL connect error
    *
    * /opt/sblim-wbemcli/bin/wbemcli: Http Exception: SSL connect error
    *
    

    Now we know there is an issue in the way the GnuTLS library used by libcURL interacts with the CIMOM server on the SVC or Storwize systems.

  4. Again, searching the web for similar issues with the error message “gnutls_handshake() failed: A TLS packet with unexpected length was received.”, we find a rebuild against the OpenSSL library instead of the GnuTLS library could solve this issue:

    $ dpkg -l | grep curl
    ii  curl                        7.26.0-1+wheezy11   powerpc     command line tool for transferring data with URL syntax
    ii  libcurl3:powerpc            7.26.0-1+wheezy11   powerpc     easy-to-use client-side URL transfer library (OpenSSL flavour)
    ii  libcurl3-gnutls:powerpc     7.26.0-1+wheezy11   powerpc     easy-to-use client-side URL transfer library (GnuTLS flavour)
    ii  libcurl4-gnutls-dev         7.26.0-1+wheezy11   powerpc     development files and documentation for libcurl (GnuTLS flavour)
    
    $ apt-get install libcurl4-openssl-dev
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    Suggested packages:
      libcurl3-dbg
    The following packages will be REMOVED:
      libcurl4-gnutls-dev
    The following NEW packages will be installed:
      libcurl4-openssl-dev
    0 upgraded, 1 newly installed, 1 to remove and 0 not upgraded.
    Need to get 0 B/1,259 kB of archives.
    After this operation, 28.7 kB of additional disk space will be used.
    Do you want to continue [Y/n]? y
    (Reading database ... 65717 files and directories currently installed.)
    Removing libcurl4-gnutls-dev ...
    Processing triggers for man-db ...
    Selecting previously unselected package libcurl4-openssl-dev.
    (Reading database ... 65463 files and directories currently installed.)
    Unpacking libcurl4-openssl-dev (from .../libcurl4-openssl-dev_7.26.0-1+wheezy11_powerpc.deb) ...
    Processing triggers for man-db ...
    Setting up libcurl4-openssl-dev (7.26.0-1+wheezy11) ...
    
    $ dpkg -l | grep curl
    ii  curl                        7.26.0-1+wheezy11   powerpc     command line tool for transferring data with URL syntax
    ii  libcurl3:powerpc            7.26.0-1+wheezy11   powerpc     easy-to-use client-side URL transfer library (OpenSSL flavour)
    ii  libcurl3-gnutls:powerpc     7.26.0-1+wheezy11   powerpc     easy-to-use client-side URL transfer library (GnuTLS flavour)
    ii  libcurl4-openssl-dev        7.26.0-1+wheezy11   powerpc     development files and documentation for libcurl (OpenSSL flavour)
    

    Rebuild the “wbemcli” command line tool and tried again:

    $ CURLDEBUG=true /opt/sblim-wbemcli/bin/wbemcli -dx -noverify ecn https://user:pass@svc-test:5989/root/ibm
    To server: <?xml version="1.0" encoding="utf-8" ?>
    <CIM CIMVERSION="2.0" DTDVERSION="2.0">
    <MESSAGE ID="4711" PROTOCOLVERSION="1.0"><SIMPLEREQ><IMETHODCALL NAME="EnumerateClassNames"><LOCALNAMESPACEPATH><NAMESPACE NAME="root"></NAMESPACE><NAMESPACE NAME="ibm"></NAMESPACE></LOCALNAMESPACEPATH>
    <IPARAMVALUE NAME="DeepInheritance"><VALUE>TRUE</VALUE></IPARAMVALUE>
    </IMETHODCALL></SIMPLEREQ>
    </MESSAGE></CIM>
    == Info: About to connect() to svc-test port 5989 (#0)
    == Info:   Trying 192.168.x.x...
    == Info: connected
    == Info: Connected to svc-test (192.168.x.x) port 5989 (#0)
    == Info: successfully set certificate verify locations:
    == Info:   CAfile: none
      CApath: /etc/ssl/certs
    == Info: SSLv3, TLS handshake, Client hello (1):
    => Send SSL data, 0000000134 bytes (0x00000086)
    0000: ......T.."\.~....`...K4.......p..7"&4...Z.....9.8.........5.....
    0040: ................3.2.....E.D...../...A...........................
    0080: ......
    == Info: Unknown SSL protocol error in connection to svc-test:5989
    == Info: Closing connection #0
    == Info: SSL connect error
    *
    * /opt/sblim-wbemcli/bin/wbemcli: Http Exception: SSL connect error
    *
    

    Now we know that the “wbemcli” command line tool is actually – as already suspected – trying to initiate a SSLv3 connection.

  5. In order to confirm we're on the right track, try to first verify manually that we're unable to connct with a SSLv3 secured connection:

    $ openssl s_client -host svc-test -port 5989 -ssl3
    CONNECTED(00000003)
    write:errno=104
    ---
    no peer certificate available
    ---
    No client certificate CA names sent
    ---
    SSL handshake has read 0 bytes and written 0 bytes
    ---
    New, (NONE), Cipher is (NONE)
    Secure Renegotiation IS NOT supported
    Compression: NONE
    Expansion: NONE
    SSL-Session:
        Protocol  : SSLv3
        Cipher    : 0000
        Session-ID:
        Session-ID-ctx:
        Master-Key:
        Key-Arg   : None
        PSK identity: None
        PSK identity hint: None
        SRP username: None
        Start Time: 1418397594
        Timeout   : 7200 (sec)
        Verify return code: 0 (ok)
    ---
    quit
    

    And that we're instead able to connect with a TLS secured connection:

    $ openssl s_client -host svc-test -port 5989
    CONNECTED(00000003)
    depth=0 C = GB, L = Hursley, O = IBM, OU = SSG, CN = 2145, emailAddress = support@ibm.com
    verify error:num=18:self signed certificate
    verify return:1
    depth=0 C = GB, L = Hursley, O = IBM, OU = SSG, CN = 2145, emailAddress = support@ibm.com
    verify return:1
    ---
    Certificate chain
     0 s:/C=GB/L=Hursley/O=IBM/OU=SSG/CN=2145/emailAddress=support@ibm.com
       i:/C=GB/L=Hursley/O=IBM/OU=SSG/CN=2145/emailAddress=support@ibm.com
    ---
    Server certificate
    -----BEGIN CERTIFICATE-----
    MIICyDCCAjGgAwIBAgIEUAPzmTANBgkqhkiG9w0BAQUFADBqMQswCQYDVQQGEwJH
    QjEQMA4GA1UEBxMHSHVyc2xleTEMMAoGA1UEChMDSUJNMQwwCgYDVQQLEwNTU0cx
    DTALBgNVBAMTBDIxNDUxHjAcBgkqhkiG9w0BCQEWD3N1cHBvcnRAaWJtLmNvbTAe
    Fw0xMjA3MTYxMDU3MjlaFw0yNzA3MTMxMDU3MjlaMGoxCzAJBgNVBAYTAkdCMRAw
    DgYDVQQHEwdIdXJzbGV5MQwwCgYDVQQKEwNJQk0xDDAKBgNVBAsTA1NTRzENMAsG
    A1UEAxMEMjE0NTEeMBwGCSqGSIb3DQEJARYPc3VwcG9ydEBpYm0uY29tMIGfMA0G
    CSqGSIb3DQEBAQUAA4GNADCBiQKBgQC3E7+7mE2GAID/35o5/s7cnzoqu9PQdOGB
    ryGMa8adD4Wd9hpmTkrsgyNvkUB6sPIifbFstGooOkQtK9ZNgP5OHOorZmqINSxM
    9goCkSCQG9xRKAvNt2tA8gujaV+p42oVEhIH6naJUul96qZI31y3GffUu2CRrJL7
    4wG/8cv0BQIDAQABo3sweTAJBgNVHRMEAjAAMCwGCWCGSAGG+EIBDQQfFh1PcGVu
    U1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUkPMkXUjn0YHlfQW8
    TJiRC5jWQO4wHwYDVR0jBBgwFoAUkPMkXUjn0YHlfQW8TJiRC5jWQO4wDQYJKoZI
    hvcNAQEFBQADgYEAKqu7KpVxnOXonQE3unC1O7qUHKoyQUEWqcKsM/4tPI+lsBMZ
    jvoPwn8yQRWiLehFmVc8VSZfdFPLzshNabXp5qbZo/EFberXrgI2CbtPiULYyyyH
    DUhWF+vhwb6uqwfBbGncvTvI2ewU8+0oTXsuTkSjumJ7+chpaHFWWyj2cJA=
    -----END CERTIFICATE-----
    subject=/C=GB/L=Hursley/O=IBM/OU=SSG/CN=2145/emailAddress=support@ibm.com
    issuer=/C=GB/L=Hursley/O=IBM/OU=SSG/CN=2145/emailAddress=support@ibm.com
    ---
    No client certificate CA names sent
    ---
    SSL handshake has read 1029 bytes and written 498 bytes
    ---
    New, TLSv1/SSLv3, Cipher is AES256-GCM-SHA384
    Server public key is 1024 bit
    Secure Renegotiation IS supported
    Compression: NONE
    Expansion: NONE
    SSL-Session:
        Protocol  : TLSv1.2
        Cipher    : AES256-GCM-SHA384
        Session-ID: 48291D368E0A8584A8DFA00A9881B8979BDE370FC6C9439294C670695D031239
        Session-ID-ctx:
        Master-Key: 71BD12A161FC595CD056DA8E6D6E27420F37468E47498B7591A403A86844C55F61FF02B2FEC7739FAAEDCE3DFEA0F217
        Key-Arg   : None
        PSK identity: None
        PSK identity hint: None
        SRP username: None
        TLS session ticket lifetime hint: 300 (seconds)
        TLS session ticket:
        0000 - a0 e0 b1 9b c2 37 9a ca-49 1c 54 f5 26 4b d6 24   .....7..I.T.&K.$
        0010 - af 6a 7d cc 5e 4a 97 a8-b3 6d b7 66 0b b7 0a 65   .j}.^J...m.f...e
        0020 - 47 af ef 47 76 fc c7 e9-38 ff 84 28 ca 8e 73 25   G..Gv...8..(..s%
        0030 - 47 25 f6 0d 36 01 04 f1-f9 f7 0c b6 42 ef cf 09   G%..6.......B...
        0040 - 64 8f df ff 89 38 ed 7c-ae 1d 0e 25 d1 c1 77 86   d....8.|...%..w.
        0050 - b6 61 88 15 cf fe 9f 20-86 0d 17 74 18 da ea c0   .a..... ...t....
        0060 - 33 3a 47 f5 f9 51 24 ae-48 37 8a 3f 19 dd c6 04   3:G..Q$.H7.?....
        0070 - 7e d1 20 78 35 99 0b 9f-3b 1f ce 7c bc 11 93 e4   ~. x5...;..|....
        0080 - 0f 94 de 94 f1 0d 0c da-64 ca 0d f6 10 2a c8 fa   ........d....*..
        0090 - dc 3e e4 1a 97 d1 34 7a-9c f5 c3 00 e8 1b 10 d7   .>....4z........
    
        Start Time: 1418397614
        Timeout   : 300 (sec)
        Verify return code: 18 (self signed certificate)
    ---
    quit
    

    The secured connection negotiated to TLS succeeded, so we're on the right track!

  6. Now we need to find the spot in the source code, where the “wbemcli” command line tool is forced to initiate a SSLv3 connection. We know this is probably done in a cURL related function call, since libcURL is used for the network connection. So lets first look into the cURL code for all the lines showing any sign of SSL related operations:

    $ grep -n curl CimCurl.cpp | grep -i ssl
    185:    // Assume we support SSL if we don't have the curl_version_info API
    276:    rv = curl_easy_setopt(mHandle, CURLOPT_SSL_VERIFYHOST, 0);
    277:    //    rv = curl_easy_setopt(mHandle, CURLOPT_SSL_VERIFYPEER, 0);
    280:    rv = curl_easy_setopt(mHandle, CURLOPT_SSLVERSION, 3);
    441:     if ((rv=curl_easy_setopt(mHandle,CURLOPT_SSL_VERIFYPEER,0))) {
    448:       if ((rv=curl_easy_setopt(mHandle,CURLOPT_SSL_VERIFYPEER,1))) {
    466:     if ((rv=curl_easy_setopt(mHandle,CURLOPT_SSLCERT,certificate))) {
    470:     if ((rv=curl_easy_setopt(mHandle,CURLOPT_SSLKEY,key))) {
    

    Looks like a perfect match in line 280 of the file “CimCurl.cpp”. The line numbers are a bit off from the original source code, since the file “CimCurl.cpp” was patched with our above debugging code. The code in the original, unpatched source file “CimCurl.cpp”, within the function “CimomCurl::genRequest” looks like this:

    CimCurl.cpp
    175     [...]
    176     /* Disable SSL host verification */
    177     rv = curl_easy_setopt(mHandle, CURLOPT_SSL_VERIFYHOST, 0);
    178     //    rv = curl_easy_setopt(mHandle, CURLOPT_SSL_VERIFYPEER, 0);
    179
    180     /* Force using SSL V3 */
    181     rv = curl_easy_setopt(mHandle, CURLOPT_SSLVERSION, 3);
    182
    183     /* Set username and password */
    184     if (url.user.length() > 0 && url.password.length() > 0) {
    185         mUserPass = url.user + ":" + url.password;
    186         rv = curl_easy_setopt(mHandle, CURLOPT_USERPWD, mUserPass.c_str());
    187     }
    188     [...]

    In line 181 the cURL option “CURLOPT_SSLVERSION” is indiscriminately set to use SSLv3 and nothing else, which we know from the above deduction is bound to fail on systems adressing the “POODLE” issues.

  7. With the knowledge where the issue is actually caused, an easy quick'n'dirty fix can be implemented:

    sblim-wbemcli-1.6.3_debug.patch
    --- sblim-wbemcli-1.6.3_orig/CimCurl.cpp    2013-09-21 01:26:32.000000000 +0200
    +++ sblim-wbemcli-1.6.3/CimCurl.cpp         2014-11-26 16:46:09.000000000 +0100
    @@ -178,7 +178,7 @@
         //    rv = curl_easy_setopt(mHandle, CURLOPT_SSL_VERIFYPEER, 0);
     
         /* Force using SSL V3 */
    -    rv = curl_easy_setopt(mHandle, CURLOPT_SSLVERSION, 3);
    +    //rv = curl_easy_setopt(mHandle, CURLOPT_SSLVERSION, 3);
     
         /* Set username and password */
         if (url.user.length() > 0 && url.password.length() > 0) {

    Inserting the comment at the line where the cURL option “CURLOPT_SSLVERSION” is forced to SSLv3 causes libcURL to fall back to its default value, which is now TLS.

    Rebuild the “wbemcli” command line tool and tried again:

    $ CURLDEBUG=true /opt/sblim-wbemcli/bin/wbemcli -dx -noverify ecn https://user:pass@svc-test:5989/root/ibm
    To server: <?xml version="1.0" encoding="utf-8" ?>
    <CIM CIMVERSION="2.0" DTDVERSION="2.0">
    <MESSAGE ID="4711" PROTOCOLVERSION="1.0"><SIMPLEREQ><IMETHODCALL NAME="EnumerateClassNames"><LOCALNAMESPACEPATH><NAMESPACE NAME="root"></NAMESPACE><NAMESPACE NAME="ibm"></NAMESPACE></LOCALNAMESPACEPATH>
    <IPARAMVALUE NAME="DeepInheritance"><VALUE>TRUE</VALUE></IPARAMVALUE>
    </IMETHODCALL></SIMPLEREQ>
    </MESSAGE></CIM>
    * About to connect() to svc-test port 5989 (#0)
    *   Trying 192.168.x.x...
    * connected
    * Connected to svc-test (192.168.x.x) port 5989 (#0)
    * found 172 certificates in /etc/ssl/certs/ca-certificates.crt
    *    server certificate verification SKIPPED
    *    common name: 2145 (does not match 'svc-test')
    *    server certificate expiration date OK
    *    server certificate activation date OK
    *    certificate public key: RSA
    *    certificate version: #3
    *    subject: C=GB,L=Hursley,O=IBM,OU=SSG,CN=2145,EMAIL=support@ibm.com
    *    start date: Mon, 16 Jul 2012 10:57:29 GMT
    
    *    expire date: Tue, 13 Jul 2027 10:57:29 GMT
    
    *    issuer: C=GB,L=Hursley,O=IBM,OU=SSG,CN=2145,EMAIL=support@ibm.com
    *    compression: NULL
    *    cipher: AES-128-CBC
    *    MAC: SHA1
    * Server auth using Basic with user 'user'
    > POST /cimom HTTP/1.1
    Authorization: Basic enp6bmFnaW9zOm5hZ2lvcw==
    Host: svc-test:5989
    Content-Type: application/xml; charset="utf-8"
    Connection: Keep-Alive, TE
    CIMProtocolVersion: 1.0
    CIMOperation: MethodCall
    CIMMethod: EnumerateClassNames
    CIMObject: root%2Fibm
    Content-Length: 396
    
    * upload completely sent off: 396 out of 396 bytes
    * additional stuff not fine transfer.c:1037: 0 0
    * HTTP 1.1 or later with persistent connection, pipelining supported
    < HTTP/1.1 200 OK
    < Content-Type: application/xml; charset="utf-8"
    From server: Content-Type: application/xml; charset="utf-8"
    < content-length: 0000072284
    From server: content-length: 0000072284
    < CIMOperation: MethodResponse
    From server: CIMOperation: MethodResponse
    <
    * Connection #0 to host svc-test left intact
    From server: <?xml version="1.0" encoding="utf-8" ?>
    <CIM CIMVERSION="2.0" DTDVERSION="2.0">
    <MESSAGE ID="4711" PROTOCOLVERSION="1.0">
    <SIMPLERSP>
    <IMETHODRESPONSE NAME="EnumerateClassNames">
    <IRETURNVALUE>
    <CLASSNAME NAME="CIM_ConcreteIdentity"/>
    <CLASSNAME NAME="CIM_NetworkPacketAction"/>
    <CLASSNAME NAME="CIM_CollectionInSystem"/>
    [...]
    
    $ /opt/sblim-wbemcli/bin/wbemcli -noverify ecn https://user:pass@svc-test:5989/root/ibm
    svc-test:5989/root/ibm:CIM_ConcreteIdentity
    svc-test:5989/root/ibm:CIM_NetworkPacketAction
    svc-test:5989/root/ibm:CIM_CollectionInSystem
    svc-test:5989/root/ibm:CIM_DeviceSAPImplementation
    svc-test:5989/root/ibm:CIM_ProtocolControllerAccessesUnit
    svc-test:5989/root/ibm:CIM_ControlledBy
    [...]
    

    Great, now we're finally able to query and monitor the IBM SVC or Storwize systems again with the Nagios monitoring plugin (Nagios Monitoring - IBM SVC and Storwize)!

Between first noticing and researching the issue and creating the quick'n'dirty fix shown above, an official bug report has been filed on the issue and a patch has already been submitted to the source code repository in order to adress the issue more thoroughly. Hopefully an updated official source code package will be released soon.

Nonetheless, the process of researching and debugging this issue was an excellent hands on exercise for me, which i enjoyed very much. Hopefully the steps taken and described here, will turn out to be of use for others as well.

Comments

Sven
No. 1 @ 2015/03/05 13:16

Thank you! You saved my day :)

Oliver
No. 2 @ 2015/03/17 21:48

Great post, thanks for the detailed steps on how you found the cause of the problem!

Glenn
No. 3 @ 2016/10/17 20:43

This article details precisely the problem I ran into when attempting to use the wbemcli tool to communicate with an IBM Power system's FSP to get thermal information. Unfortunately the fix didn't work for me.

Glenn
No. 4 @ 2016/10/17 22:02

@Glenn: I was too quick to post. Realized that I had forgotten to update my RHEL 6.5 server with latest fixes. For current levels of TLS (and SSLv3) to work properly, this code needs to be compiled on a system that has updated libcurl, etc. Once I updated, the program seems to be working fine. Thank you very much.

Thomas
No. 5 @ 2016/11/02 09:00

Hey! I just wanted to let you know that I really appreciate you sharing this well-researched article. You show a very thorough approach to troubleshooting.

Leave a comment…




L​ M D P O
  • E-Mail address will not be published.
  • Formatting:
    //italic//  __underlined__
    **bold**  ''preformatted''
  • Links:
    [[http://example.com]]
    [[http://example.com|Link Text]]
  • Quotation:
    > This is a quote. Don't forget the space in front of the text: "> "
  • Code:
    <code>This is unspecific source code</code>
    <code [lang]>This is specifc [lang] code</code>
    <code php><?php echo 'example'; ?></code>
    Available: html, css, javascript, bash, cpp, …
  • Lists:
    Indent your text by two spaces and use a * for
    each unordered list item or a - for ordered ones.
This website uses cookies for visitor traffic analysis. By using the website, you agree with storing the cookies on your computer.More information