bityard Blog

// DIY - Lifter for a Coffee Pad Tin

As a christmas gift i got a new tin to store a different brand of single portion coffee pads for Senseo machines in. Those are the kind of attentive and useful gifts i really love and appreciate the most! They are used and enjoyed regularly, instead of just collecting dust in some closet. Every time you use them, they remind you in a positive way of the person who gave them to you and put a big, bright smile on your face :-)

The old coffee pad tin consists of three parts:

  • A tin can body with about the same diameter as the coffee pads. This is in order to keep a vertical stack of pads from falling over.
  • A tin can lid.
  • A lifter – see the exclamation mark in the picture below – to ease the retrieval of individual coffee pads from the tin can body. The pads are put in a vertical stack onto the spiral made from wire at the bottom of the lifter. The horizontal top part of the lifter acts as a handle by which the lifter and the vertical stack of pads is pulled from the tin can body.

All three parts are shown in the following picture:

Old coffee pad tin with the original lifter made from a thick wire

The new coffee pad tin is from a different manufacturer and only consists of two parts – a tin can body and a tin can lid. It was unfortunately missing the very convenient lifter described above – see the question mark in the picture below. So there was nothing to put the coffee pads on and pull them out of the tin can body for an easier individual retrieval. Instead the tin can had to be shaken until one coffee pad fell out or topped over to get the whole stack out. Usually this resulted in the remaining pads not being vertically stacked anymore and instead being wedged together in a pile, which in turn hindered the subsequent retrieval of pads from the tin.

The new coffee pad tin is shown in the following picure:

New coffee pad tin without a lifter

In order to remedy this deficiency i decided to do a little low-tech DIY. So i picked up a spool of thick metal wire (Gardol Gartendraht Classic, length: 25m, diameter:1,8mm) at the local hardware store for €3,99. Together with the tools shown in the following picture:

Tools and materials used to build an DIY lifter from a thick wire

i made a replica from the lifter that came with the old coffee pad tin. The original lifter shown on the far right was used as a template. The yarn second to the right was used to measure the lenght of wire needed off the spool of wire, shown on the left side of the picture. This was done by following the contour of the original lifter with the thread and then measuring the length of the straightened thread against a straightened portion of the wire from the spool. A length of about 50cm of wire was used, so the cost of material was about €0,08. The pliers shown second to the left in the picture were used to cut off the measured piece of wire and to bend around the sharp cutoff edges of the wire. They were also used to hold and form the straightened piece of wire while bending it into roughtly the same shape as the template.

The result was a lifter replica with roughtly the same shape as the original. The following picture shows them side by side. The replica on the left side and the original on the right side:

Left new DIY lifter, right original lifter

Now, the new coffee pad tin has also three parts, same as the old coffee pad tin:

New coffee pad tin with the DIY lifter made from a thick wire

Two more pictures. The fist one with the lifter replica in action in the empty new coffee pad tin:

New coffee pad tin with the DIY lifter in action (empty tin)

And the second one with the lifter replica in action in the new coffee pad tin filled with a few pads:

New coffee pad tin with the DIY lifter in action (tin filled with a few pads)

Enhancing the new coffee pad tin by the simple means of a self-made lifter added even more value to it than what has already been described at the beginning of this post. It provided for a very enjoyable do-it-yourself time, the pleasurable feeling of accomplishment and success and the outcome is something that is very useful in everyday life. Best gift, ever :-)

// IBM Storwize V3700 out of Memory

Under certain conditions it is possible to inadvertently run into an out of memory situation on IBM Storwize V3700 systems, by simply running a Download Support Package procedure or the respective CLI command. This will – of course – bring all I/O on the affected system to a grinding halt.

A few days ago, the Nagios monitoring plugin introduced in “Nagios Monitoring - IBM SVC and Storwize” reported a failed PSU on one of our IBM Storwize V3700 systems. After raising a PMR with IBM in order to get the seemingly defective PSU replaced, i was told to simply reseat the PSU. According to IBM support this would usually fix this – apparently known – issue.

This was the first WTF moment and it turned out to not be the last one. So either IBM produces and sells subpar components – in this case the PSU – which need to be given a boot – yes, PSUs nowadays have their own firmware too – in order to be persuaded to cooperate again. Or it means IBM produces and sells subpar software which is not at all able to properly detect a component failure and distinguish between a faulty and a good PSU. Or perhaps its an unfortunate combination of both.

In any case, the procdure to reseat the PSU was carried out, which fixed the PSU issue. During the course of the fix procedure the system would become unreachable via TCP/IP for a rather long time, though. Definately over two minutes, but i haven't had a chance for an exact measurement. After the system was reachable again i followed this strange behaviour up and had a look at the systems event log. There were quite a lot of “Error Code: 1370, Error Code Text: SCSI ERP occurred” messages, so i decided to bother the IBM support again and send them a support collection in order to get an analysis with regard to the reachablilty issue as well as the 1370 errors.

From previous occasions i knew that the IBM support would most likely request a support collection which was run with the “svc_livedumpCLI command or with the “Standard logs plus new statesaves” option from the WebUI. The latter one is marked red in the following screenshot example:

SVC and Storwize "Download Support Package" Dialog from the WebUI

So i decided to pull a support collection with this option. After some time into the support collection process, the SVC sitting in front of the V3700 and other storage systems, started to show very high latencies (~60 sec.) on the primary VDisks backed by the V3700. On other VDisks which “only” had their secondary VDisk-Mirror located on MDiskGroups of the V3700, the latency peak was less dramatic, but still very noticable. Eventually degraded paths to the MDisks located on the V3700 started showing up on the SVC. After the support collection process finished the situation went back to normal. The latency on both primary and secondary VDisks instantly dropped down to the usual values and after running the fix procedures on the SVC, the degraded paths came back online.

The performance issues were in magnitude and duration severe enough to affect several applications pretty badly. Although the immediate issue was resolved, i still needed an analysis and written statement from IBM support for an action plan on how to prevent this kind of situation in the future and for compliance reasons as well. Here is the digest of what – according to IBM – happened:

  1. During the procedure to reseat the reportedly defective PSU, one or more power surges occured.

  2. These power surges apparently caused issues on the internal disk buses, which lead to the 1370 errors to be logged.

  3. The power surges or the resulting 1370 errors are probably the cause for a failover of the config node too. Hence the connectivity issues with the CLI and the WebUI via TCP/IP.

  4. More 1370 errors were logged during the runtime of the subsequent “Standard logs plus new statesaves” support collection process.

  5. The issue of very high latency and MDisk paths becoming degraded was caused by the “Standard logs plus new statesaves” support collection process using up all of the memory – yes, including the data cache – on the V3700 system. This behaviour is specific and limited to the V3700 systems with “only” 4GB of memory.

As one can imagine, the last item was my second WTF moment. Apparently there are no programmatical safeguards to prevent the support collection process at a “Standard logs plus new statesaves” level from using up all of the systems memory. This would normally not be that bad at all, if the “Standard logs plus new statesaves” wasn't the particular level which IBM support would usually request on support cases concerning SVC and Storwize systems. On the phone the IBM support technician admitted that this common practice is in general probably a bad idea. But he also mentioned that he up to now hadn't heard of the known side effects actually occuring, like in this case.

The suggestion on how to prevent this kind of situation in the future was to either upgrade the V3700 systems from 4GB to 8GB memory – a solution i would gladly take provided it came free of charge – or to only run the support collection process only with the “svc_snapCLI command or the “Standard logs” option from the WebUI. Since the memory upgrade for free isn't likely to happen, i'll stick with the second suggestion for now.

Incidently a third option came up over the last weekend. Looking at the IBM System Storage SAN Volume Controller V7.3.0.9 Release Note, it could be construed that someone at IBM SVC and Storwize development came to the realization that the issue could also be addressed in software, by altering the resource utilization of the support collection process:

HU00636          Livedump prepare fails on V3500 & V3700 systems with 4GB
                 memory when cache partition fullness is less than 35%

Fingers crossed, this fix really addresses and resolves the issue described above.

// Diff File Icon for Patch Files in DokuWiki

Essentially “.patch” files are – with regard to their content – not different from “.diff” files. Still, throughout this blog and all of my other documentation i like to and also have grown accustomed to using the “.diff” filename suffix in cases where i want to focus on the semantics of a pure comparison. On the other hand, in cases where i want to focus on the semantics of actually applying a patch to a certain source file, the “.patch” filename suffix is used. I guess it's more of a personal quirk, rather than a well funded scientific theory.

Unfortunately in the stock DokuWiki distribution there is a little deficiency in displaying the appropriate diff file icon not only for “.diff” files, but for “.patch” files also. Here is an example screenshot of what this looks like:

DokuWiki default file icon for .patch files

Note the plain white paper sheet icon in front of the filename “sblim-wbemcli-1.6.3_debug.patch”, which is the default file icon in case of an unknown filename suffix.

This behaviour basically exists, because the “.patch” filename suffix has no appropriate icon in the “<dokuwiki-path>/lib/images/fileicons/” directory:

$ find <dokuwiki-path>/lib/images/ -type f -name "patch.*" -ls | wc -l
0

See also the DokuWiki documentation on MIME configuration.

This can be quickly fixed though. By just creating appropriate directory enties for the “.patch” filename suffix via symbolic links to the file icons for the “.diff” filename suffix:

$ ln -s <dokuwiki-path>/lib/images/fileicons/diff.png <dokuwiki-path>/lib/images/fileicons/patch.png
$ ln -s <dokuwiki-path>/lib/images/fileicons/32x32/diff.png <dokuwiki-path>/lib/images/fileicons/32x32/patch.png
$ find <dokuwiki-path>/lib/images/ -name "patch.*" -ls
 ... lrwxrwxrwx   ...   <dokuwiki-path>/lib/images/fileicons/patch.png -> <dokuwiki-path>/lib/images/fileicons/diff.png
 ... lrwxrwxrwx   ...   <dokuwiki-path>/lib/images/fileicons/32x32/patch.png -> <dokuwiki-path>/lib/images/fileicons/32x32/diff.png
$ touch <dokuwiki-path>/conf/local.php

The final “touch” command is necessary, since the icons are part of the DokuWiki cache and the application logic has to be notified to rebuild the now invalid cache.

After reloading the respective DokuWiki page in the browser, note the now “Diff”-labeled paper sheet icon in front of the filename “sblim-wbemcli-1.6.3_debug.patch”:

DokuWiki diff file icon for .patch files

This may be just a small change in the display of the DokuWiki page, but in my humble opinion it makes things much more pleasant to look at.

// Nagios Monitoring - IBM SVC and Storwize (Update)

After upgrading our test IBM SAN Volume Controller (SVC) systems from version 7.3.0.7 to 7.3.0.8 or later, the previously described Nagios monitoring plugin (Nagios Monitoring - IBM SVC and Storwize) ceased to work. A quick check revealed that the “wbemcli” command line tool from the Standards Based Linux Instrumentation project, which is used in the Nagios plugin to query the CIMOM server on the SVC or Storwize systems, would fail with the following error message:

$ /opt/sblim-wbemcli/bin/wbemcli -dx -noverify ecn https://user:pass@svc-test:5989/root/ibm
*
* /opt/sblim-wbemcli/bin/wbemcli: Http Exception: SSL connect error
*

Re-checking the release notes, this sudden change in behaviour seemed to be explained by the fix:

SSL vulnerability CVE-2014-3566

Not really being that verbose a description, a quick look at the CVE-2014-3566 showed that this is a fix for the “POODLE” issue. So IBM probably switched off the support for the SSLv3 protocol in the SVC and Storwize code. But why would this cause the “wbemcli” command line tool to fail? Here are the steps taken in an analysis of the issue:

  1. First, i was trying to get the “wbemcli” command line tool to be a tad more verbose about what it is actually doing:

    $ /opt/sblim-wbemcli/bin/wbemcli -dx -noverify ecn https://user:pass@svc-test:5989/root/ibm
    To server: <?xml version="1.0" encoding="utf-8" ?>
    <CIM CIMVERSION="2.0" DTDVERSION="2.0">
    <MESSAGE ID="4711" PROTOCOLVERSION="1.0"><SIMPLEREQ><IMETHODCALL NAME="EnumerateClassNames"><LOCALNAMESPACEPATH><NAMESPACE NAME="root"></NAMESPACE><NAMESPACE NAME="ibm"></NAMESPACE></LOCALNAMESPACEPATH>
    <IPARAMVALUE NAME="DeepInheritance"><VALUE>TRUE</VALUE></IPARAMVALUE>
    </IMETHODCALL></SIMPLEREQ>
    </MESSAGE></CIM>
    *
    * /opt/sblim-wbemcli/bin/wbemcli: Http Exception: SSL connect error
    *
    

    Not really an abundance of information in here too.

  2. Getting the source code for and, while we're at it, updating the “wbemcli” command line tool from version 1.6.0 to 1.6.3. Spent some time looking through the source code and with the “gdb” debugger to get a feeling for the general program flow and functions/methods being called. While looking through the source code i noticed the cURL debugging options are being set if a environment variable named “CURLDEBUG” is set to “true”. Later also found this mentioned in the ChangeLog file:

    $ CURLDEBUG=true /opt/sblim-wbemcli/bin/wbemcli -dx -noverify ecn https://user:pass@svc-test:5989/root/ibm
    To server: <?xml version="1.0" encoding="utf-8" ?>
    <CIM CIMVERSION="2.0" DTDVERSION="2.0">
    <MESSAGE ID="4711" PROTOCOLVERSION="1.0"><SIMPLEREQ><IMETHODCALL NAME="EnumerateClassNames"><LOCALNAMESPACEPATH><NAMESPACE NAME="root"></NAMESPACE><NAMESPACE NAME="ibm"></NAMESPACE></LOCALNAMESPACEPATH>
    <IPARAMVALUE NAME="DeepInheritance"><VALUE>TRUE</VALUE></IPARAMVALUE>
    </IMETHODCALL></SIMPLEREQ>
    </MESSAGE></CIM>
    * About to connect() to svc-test port 5989 (#0)
    *   Trying 192.168.x.x...
    * connected
    * Connected to svc-test (192.168.x.x) port 5989 (#0)
    * successfully set certificate verify locations:
    *   CAfile: none
      CApath: /etc/ssl/certs
    * Unknown SSL protocol error in connection to svc-test:5989
    * Closing connection #0
    * SSL connect error
    *
    * /opt/sblim-wbemcli/bin/wbemcli: Http Exception: SSL connect error
    *
    

    Now we know that we're encountering a “Unknown SSL protocol error” – not that helpful either.

  3. Searched the web for further information on how to debug the cURL library and found the debug.c example source code, which was very helpful. Incorperated it into the file “CimCurl.cpp”:

    sblim-wbemcli-1.6.3_debug.patch
    --- sblim-wbemcli-1.6.3_orig/CimCurl.cpp    2013-09-21 01:26:32.000000000 +0200
    +++ sblim-wbemcli-1.6.3_new/CimCurl.cpp     2014-11-26 16:30:19.000000000 +0100
    @@ -37,6 +37,100 @@
     extern int waitTime;
     extern int expect100;
     
    +// Trace Begin
    +struct data {
    +  char trace_ascii; /* 1 or 0 */ 
    +};
    +
    +static
    +void dump(const char *text,
    +          FILE *stream, unsigned char *ptr, size_t size,
    +          char nohex)
    +{ 
    +  size_t i;
    +  size_t c;
    +  
    +  unsigned int width=0x10;
    +  
    +  if(nohex)
    +    /* without the hex output, we can fit more on screen */
    +    width = 0x40;
    +
    +  fprintf(stream, "%s, %010.10ld bytes (0x%08.8lx)\n",
    +          text, (long)size, (long)size);
    +
    +  for(i=0; i<size; i+= width) {
    +
    +    fprintf(stream, "%04.4lx: ", (long)i);
    +
    +    if(!nohex) {
    +      /* hex not disabled, show it */
    +      for(c = 0; c < width; c++)
    +        if(i+c < size)
    +          fprintf(stream, "%02x ", ptr[i+c]);
    +        else
    +          fputs("   ", stream);
    +    }
    +
    +    for(c = 0; (c < width) && (i+c < size); c++) {
    +      /* check for 0D0A; if found, skip past and start a new line of output */
    +      if (nohex && (i+c+1 < size) && ptr[i+c]==0x0D && ptr[i+c+1]==0x0A) {
    +        i+=(c+2-width);
    +        break;
    +      }
    +      fprintf(stream, "%c",
    +              (ptr[i+c]>=0x20) && (ptr[i+c]<0x80)?ptr[i+c]:'.');
    +      /* check again for 0D0A, to avoid an extra \n if it's at width */
    +      if (nohex && (i+c+2 < size) && ptr[i+c+1]==0x0D && ptr[i+c+2]==0x0A) {
    +        i+=(c+3-width);
    +        break;
    +      }
    +    }
    +    fputc('\n', stream); /* newline */
    +  }
    +  fflush(stream);
    +}
    +
    +static
    +int my_trace(CURL *handle, curl_infotype type,
    +             char *data, size_t size,
    +             void *userp)
    +{
    +  struct data *config = (struct data *)userp;
    +  const char *text;
    +  (void)handle; /* prevent compiler warning */
    +
    +  switch (type) {
    +  case CURLINFO_TEXT:
    +    fprintf(stderr, "== Info: %s", data);
    +  default: /* in case a new one is introduced to shock us */
    +    return 0;
    +
    +  case CURLINFO_HEADER_OUT:
    +    text = "=> Send header";
    +    break;
    +  case CURLINFO_DATA_OUT:
    +    text = "=> Send data";
    +    break;
    +  case CURLINFO_SSL_DATA_OUT:
    +    text = "=> Send SSL data";
    +    break;
    +  case CURLINFO_HEADER_IN:
    +    text = "<= Recv header";
    +    break;
    +  case CURLINFO_DATA_IN:
    +    text = "<= Recv data";
    +    break;
    +  case CURLINFO_SSL_DATA_IN:
    +    text = "<= Recv SSL data";
    +    break;
    +  }
    +
    +  dump(text, stderr, (unsigned char *)data, size, config->trace_ascii);
    +  return 0;
    +}
    +// Trace End
    +
     // These are the constant headers added to all requests
     static const char *headers[] = {
         "Content-Type: application/xml; charset=\"utf-8\"",
    @@ -152,6 +246,11 @@
         CURLcode rv;
         string sb;
     
    +// Trace Begin
    +struct data config;
    +config.trace_ascii = 1;
    +// Trace End
    +
         mUri = url.scheme + "://" + url.host + ":" + url.port + "/cimom";
         url.ns.toStringBuffer(sb,"%2F");
     
    @@ -248,6 +347,11 @@
     
         rv = curl_easy_setopt(mHandle, CURLOPT_WRITEHEADER, &mErrorData);
         rv = curl_easy_setopt(mHandle, CURLOPT_HEADERFUNCTION, headerCb);
    +
    +// Trace Begin
    +    rv = curl_easy_setopt(mHandle, CURLOPT_DEBUGFUNCTION, my_trace);
    +    rv = curl_easy_setopt(mHandle, CURLOPT_DEBUGDATA, &config);
    +// Trace End
     }
     
     static string getErrorMessage(CURLcode err)

    rebuild the “wbemcli” command line tool and tried again:

    $ CURLDEBUG=true /opt/sblim-wbemcli/bin/wbemcli -dx -noverify ecn https://user:pass@svc-test:5989/root/ibm
    To server: <?xml version="1.0" encoding="utf-8" ?>
    <CIM CIMVERSION="2.0" DTDVERSION="2.0">
    <MESSAGE ID="4711" PROTOCOLVERSION="1.0"><SIMPLEREQ><IMETHODCALL NAME="EnumerateClassNames"><LOCALNAMESPACEPATH><NAMESPACE NAME="root"></NAMESPACE><NAMESPACE NAME="ibm"></NAMESPACE></LOCALNAMESPACEPATH>
    <IPARAMVALUE NAME="DeepInheritance"><VALUE>TRUE</VALUE></IPARAMVALUE>
    </IMETHODCALL></SIMPLEREQ>
    </MESSAGE></CIM>
    == Info: About to connect() to svc-test port 5989 (#0)
    == Info:   Trying 192.168.x.x...
    == Info: connected
    == Info: Connected to svc-test (192.168.x.x) port 5989 (#0)
    == Info: found 172 certificates in /etc/ssl/certs/ca-certificates.crt
    == Info: gnutls_handshake() failed: A TLS packet with unexpected length was received.
    == Info: Closing connection #0
    == Info: SSL connect error
    *
    * /opt/sblim-wbemcli/bin/wbemcli: Http Exception: SSL connect error
    *
    

    Now we know there is an issue in the way the GnuTLS library used by libcURL interacts with the CIMOM server on the SVC or Storwize systems.

  4. Again, searching the web for similar issues with the error message “gnutls_handshake() failed: A TLS packet with unexpected length was received.”, we find a rebuild against the OpenSSL library instead of the GnuTLS library could solve this issue:

    $ dpkg -l | grep curl
    ii  curl                        7.26.0-1+wheezy11   powerpc     command line tool for transferring data with URL syntax
    ii  libcurl3:powerpc            7.26.0-1+wheezy11   powerpc     easy-to-use client-side URL transfer library (OpenSSL flavour)
    ii  libcurl3-gnutls:powerpc     7.26.0-1+wheezy11   powerpc     easy-to-use client-side URL transfer library (GnuTLS flavour)
    ii  libcurl4-gnutls-dev         7.26.0-1+wheezy11   powerpc     development files and documentation for libcurl (GnuTLS flavour)
    
    $ apt-get install libcurl4-openssl-dev
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    Suggested packages:
      libcurl3-dbg
    The following packages will be REMOVED:
      libcurl4-gnutls-dev
    The following NEW packages will be installed:
      libcurl4-openssl-dev
    0 upgraded, 1 newly installed, 1 to remove and 0 not upgraded.
    Need to get 0 B/1,259 kB of archives.
    After this operation, 28.7 kB of additional disk space will be used.
    Do you want to continue [Y/n]? y
    (Reading database ... 65717 files and directories currently installed.)
    Removing libcurl4-gnutls-dev ...
    Processing triggers for man-db ...
    Selecting previously unselected package libcurl4-openssl-dev.
    (Reading database ... 65463 files and directories currently installed.)
    Unpacking libcurl4-openssl-dev (from .../libcurl4-openssl-dev_7.26.0-1+wheezy11_powerpc.deb) ...
    Processing triggers for man-db ...
    Setting up libcurl4-openssl-dev (7.26.0-1+wheezy11) ...
    
    $ dpkg -l | grep curl
    ii  curl                        7.26.0-1+wheezy11   powerpc     command line tool for transferring data with URL syntax
    ii  libcurl3:powerpc            7.26.0-1+wheezy11   powerpc     easy-to-use client-side URL transfer library (OpenSSL flavour)
    ii  libcurl3-gnutls:powerpc     7.26.0-1+wheezy11   powerpc     easy-to-use client-side URL transfer library (GnuTLS flavour)
    ii  libcurl4-openssl-dev        7.26.0-1+wheezy11   powerpc     development files and documentation for libcurl (OpenSSL flavour)
    

    Rebuild the “wbemcli” command line tool and tried again:

    $ CURLDEBUG=true /opt/sblim-wbemcli/bin/wbemcli -dx -noverify ecn https://user:pass@svc-test:5989/root/ibm
    To server: <?xml version="1.0" encoding="utf-8" ?>
    <CIM CIMVERSION="2.0" DTDVERSION="2.0">
    <MESSAGE ID="4711" PROTOCOLVERSION="1.0"><SIMPLEREQ><IMETHODCALL NAME="EnumerateClassNames"><LOCALNAMESPACEPATH><NAMESPACE NAME="root"></NAMESPACE><NAMESPACE NAME="ibm"></NAMESPACE></LOCALNAMESPACEPATH>
    <IPARAMVALUE NAME="DeepInheritance"><VALUE>TRUE</VALUE></IPARAMVALUE>
    </IMETHODCALL></SIMPLEREQ>
    </MESSAGE></CIM>
    == Info: About to connect() to svc-test port 5989 (#0)
    == Info:   Trying 192.168.x.x...
    == Info: connected
    == Info: Connected to svc-test (192.168.x.x) port 5989 (#0)
    == Info: successfully set certificate verify locations:
    == Info:   CAfile: none
      CApath: /etc/ssl/certs
    == Info: SSLv3, TLS handshake, Client hello (1):
    => Send SSL data, 0000000134 bytes (0x00000086)
    0000: ......T.."\.~....`...K4.......p..7"&4...Z.....9.8.........5.....
    0040: ................3.2.....E.D...../...A...........................
    0080: ......
    == Info: Unknown SSL protocol error in connection to svc-test:5989
    == Info: Closing connection #0
    == Info: SSL connect error
    *
    * /opt/sblim-wbemcli/bin/wbemcli: Http Exception: SSL connect error
    *
    

    Now we know that the “wbemcli” command line tool is actually – as already suspected – trying to initiate a SSLv3 connection.

  5. In order to confirm we're on the right track, try to first verify manually that we're unable to connct with a SSLv3 secured connection:

    $ openssl s_client -host svc-test -port 5989 -ssl3
    CONNECTED(00000003)
    write:errno=104
    ---
    no peer certificate available
    ---
    No client certificate CA names sent
    ---
    SSL handshake has read 0 bytes and written 0 bytes
    ---
    New, (NONE), Cipher is (NONE)
    Secure Renegotiation IS NOT supported
    Compression: NONE
    Expansion: NONE
    SSL-Session:
        Protocol  : SSLv3
        Cipher    : 0000
        Session-ID:
        Session-ID-ctx:
        Master-Key:
        Key-Arg   : None
        PSK identity: None
        PSK identity hint: None
        SRP username: None
        Start Time: 1418397594
        Timeout   : 7200 (sec)
        Verify return code: 0 (ok)
    ---
    quit
    

    And that we're instead able to connect with a TLS secured connection:

    $ openssl s_client -host svc-test -port 5989
    CONNECTED(00000003)
    depth=0 C = GB, L = Hursley, O = IBM, OU = SSG, CN = 2145, emailAddress = support@ibm.com
    verify error:num=18:self signed certificate
    verify return:1
    depth=0 C = GB, L = Hursley, O = IBM, OU = SSG, CN = 2145, emailAddress = support@ibm.com
    verify return:1
    ---
    Certificate chain
     0 s:/C=GB/L=Hursley/O=IBM/OU=SSG/CN=2145/emailAddress=support@ibm.com
       i:/C=GB/L=Hursley/O=IBM/OU=SSG/CN=2145/emailAddress=support@ibm.com
    ---
    Server certificate
    -----BEGIN CERTIFICATE-----
    MIICyDCCAjGgAwIBAgIEUAPzmTANBgkqhkiG9w0BAQUFADBqMQswCQYDVQQGEwJH
    QjEQMA4GA1UEBxMHSHVyc2xleTEMMAoGA1UEChMDSUJNMQwwCgYDVQQLEwNTU0cx
    DTALBgNVBAMTBDIxNDUxHjAcBgkqhkiG9w0BCQEWD3N1cHBvcnRAaWJtLmNvbTAe
    Fw0xMjA3MTYxMDU3MjlaFw0yNzA3MTMxMDU3MjlaMGoxCzAJBgNVBAYTAkdCMRAw
    DgYDVQQHEwdIdXJzbGV5MQwwCgYDVQQKEwNJQk0xDDAKBgNVBAsTA1NTRzENMAsG
    A1UEAxMEMjE0NTEeMBwGCSqGSIb3DQEJARYPc3VwcG9ydEBpYm0uY29tMIGfMA0G
    CSqGSIb3DQEBAQUAA4GNADCBiQKBgQC3E7+7mE2GAID/35o5/s7cnzoqu9PQdOGB
    ryGMa8adD4Wd9hpmTkrsgyNvkUB6sPIifbFstGooOkQtK9ZNgP5OHOorZmqINSxM
    9goCkSCQG9xRKAvNt2tA8gujaV+p42oVEhIH6naJUul96qZI31y3GffUu2CRrJL7
    4wG/8cv0BQIDAQABo3sweTAJBgNVHRMEAjAAMCwGCWCGSAGG+EIBDQQfFh1PcGVu
    U1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUkPMkXUjn0YHlfQW8
    TJiRC5jWQO4wHwYDVR0jBBgwFoAUkPMkXUjn0YHlfQW8TJiRC5jWQO4wDQYJKoZI
    hvcNAQEFBQADgYEAKqu7KpVxnOXonQE3unC1O7qUHKoyQUEWqcKsM/4tPI+lsBMZ
    jvoPwn8yQRWiLehFmVc8VSZfdFPLzshNabXp5qbZo/EFberXrgI2CbtPiULYyyyH
    DUhWF+vhwb6uqwfBbGncvTvI2ewU8+0oTXsuTkSjumJ7+chpaHFWWyj2cJA=
    -----END CERTIFICATE-----
    subject=/C=GB/L=Hursley/O=IBM/OU=SSG/CN=2145/emailAddress=support@ibm.com
    issuer=/C=GB/L=Hursley/O=IBM/OU=SSG/CN=2145/emailAddress=support@ibm.com
    ---
    No client certificate CA names sent
    ---
    SSL handshake has read 1029 bytes and written 498 bytes
    ---
    New, TLSv1/SSLv3, Cipher is AES256-GCM-SHA384
    Server public key is 1024 bit
    Secure Renegotiation IS supported
    Compression: NONE
    Expansion: NONE
    SSL-Session:
        Protocol  : TLSv1.2
        Cipher    : AES256-GCM-SHA384
        Session-ID: 48291D368E0A8584A8DFA00A9881B8979BDE370FC6C9439294C670695D031239
        Session-ID-ctx:
        Master-Key: 71BD12A161FC595CD056DA8E6D6E27420F37468E47498B7591A403A86844C55F61FF02B2FEC7739FAAEDCE3DFEA0F217
        Key-Arg   : None
        PSK identity: None
        PSK identity hint: None
        SRP username: None
        TLS session ticket lifetime hint: 300 (seconds)
        TLS session ticket:
        0000 - a0 e0 b1 9b c2 37 9a ca-49 1c 54 f5 26 4b d6 24   .....7..I.T.&K.$
        0010 - af 6a 7d cc 5e 4a 97 a8-b3 6d b7 66 0b b7 0a 65   .j}.^J...m.f...e
        0020 - 47 af ef 47 76 fc c7 e9-38 ff 84 28 ca 8e 73 25   G..Gv...8..(..s%
        0030 - 47 25 f6 0d 36 01 04 f1-f9 f7 0c b6 42 ef cf 09   G%..6.......B...
        0040 - 64 8f df ff 89 38 ed 7c-ae 1d 0e 25 d1 c1 77 86   d....8.|...%..w.
        0050 - b6 61 88 15 cf fe 9f 20-86 0d 17 74 18 da ea c0   .a..... ...t....
        0060 - 33 3a 47 f5 f9 51 24 ae-48 37 8a 3f 19 dd c6 04   3:G..Q$.H7.?....
        0070 - 7e d1 20 78 35 99 0b 9f-3b 1f ce 7c bc 11 93 e4   ~. x5...;..|....
        0080 - 0f 94 de 94 f1 0d 0c da-64 ca 0d f6 10 2a c8 fa   ........d....*..
        0090 - dc 3e e4 1a 97 d1 34 7a-9c f5 c3 00 e8 1b 10 d7   .>....4z........
    
        Start Time: 1418397614
        Timeout   : 300 (sec)
        Verify return code: 18 (self signed certificate)
    ---
    quit
    

    The secured connection negotiated to TLS succeeded, so we're on the right track!

  6. Now we need to find the spot in the source code, where the “wbemcli” command line tool is forced to initiate a SSLv3 connection. We know this is probably done in a cURL related function call, since libcURL is used for the network connection. So lets first look into the cURL code for all the lines showing any sign of SSL related operations:

    $ grep -n curl CimCurl.cpp | grep -i ssl
    185:    // Assume we support SSL if we don't have the curl_version_info API
    276:    rv = curl_easy_setopt(mHandle, CURLOPT_SSL_VERIFYHOST, 0);
    277:    //    rv = curl_easy_setopt(mHandle, CURLOPT_SSL_VERIFYPEER, 0);
    280:    rv = curl_easy_setopt(mHandle, CURLOPT_SSLVERSION, 3);
    441:     if ((rv=curl_easy_setopt(mHandle,CURLOPT_SSL_VERIFYPEER,0))) {
    448:       if ((rv=curl_easy_setopt(mHandle,CURLOPT_SSL_VERIFYPEER,1))) {
    466:     if ((rv=curl_easy_setopt(mHandle,CURLOPT_SSLCERT,certificate))) {
    470:     if ((rv=curl_easy_setopt(mHandle,CURLOPT_SSLKEY,key))) {
    

    Looks like a perfect match in line 280 of the file “CimCurl.cpp”. The line numbers are a bit off from the original source code, since the file “CimCurl.cpp” was patched with our above debugging code. The code in the original, unpatched source file “CimCurl.cpp”, within the function “CimomCurl::genRequest” looks like this:

    CimCurl.cpp
    175     [...]
    176     /* Disable SSL host verification */
    177     rv = curl_easy_setopt(mHandle, CURLOPT_SSL_VERIFYHOST, 0);
    178     //    rv = curl_easy_setopt(mHandle, CURLOPT_SSL_VERIFYPEER, 0);
    179
    180     /* Force using SSL V3 */
    181     rv = curl_easy_setopt(mHandle, CURLOPT_SSLVERSION, 3);
    182
    183     /* Set username and password */
    184     if (url.user.length() > 0 && url.password.length() > 0) {
    185         mUserPass = url.user + ":" + url.password;
    186         rv = curl_easy_setopt(mHandle, CURLOPT_USERPWD, mUserPass.c_str());
    187     }
    188     [...]

    In line 181 the cURL option “CURLOPT_SSLVERSION” is indiscriminately set to use SSLv3 and nothing else, which we know from the above deduction is bound to fail on systems adressing the “POODLE” issues.

  7. With the knowledge where the issue is actually caused, an easy quick'n'dirty fix can be implemented:

    sblim-wbemcli-1.6.3_debug.patch
    --- sblim-wbemcli-1.6.3_orig/CimCurl.cpp    2013-09-21 01:26:32.000000000 +0200
    +++ sblim-wbemcli-1.6.3/CimCurl.cpp         2014-11-26 16:46:09.000000000 +0100
    @@ -178,7 +178,7 @@
         //    rv = curl_easy_setopt(mHandle, CURLOPT_SSL_VERIFYPEER, 0);
     
         /* Force using SSL V3 */
    -    rv = curl_easy_setopt(mHandle, CURLOPT_SSLVERSION, 3);
    +    //rv = curl_easy_setopt(mHandle, CURLOPT_SSLVERSION, 3);
     
         /* Set username and password */
         if (url.user.length() > 0 && url.password.length() > 0) {

    Inserting the comment at the line where the cURL option “CURLOPT_SSLVERSION” is forced to SSLv3 causes libcURL to fall back to its default value, which is now TLS.

    Rebuild the “wbemcli” command line tool and tried again:

    $ CURLDEBUG=true /opt/sblim-wbemcli/bin/wbemcli -dx -noverify ecn https://user:pass@svc-test:5989/root/ibm
    To server: <?xml version="1.0" encoding="utf-8" ?>
    <CIM CIMVERSION="2.0" DTDVERSION="2.0">
    <MESSAGE ID="4711" PROTOCOLVERSION="1.0"><SIMPLEREQ><IMETHODCALL NAME="EnumerateClassNames"><LOCALNAMESPACEPATH><NAMESPACE NAME="root"></NAMESPACE><NAMESPACE NAME="ibm"></NAMESPACE></LOCALNAMESPACEPATH>
    <IPARAMVALUE NAME="DeepInheritance"><VALUE>TRUE</VALUE></IPARAMVALUE>
    </IMETHODCALL></SIMPLEREQ>
    </MESSAGE></CIM>
    * About to connect() to svc-test port 5989 (#0)
    *   Trying 192.168.x.x...
    * connected
    * Connected to svc-test (192.168.x.x) port 5989 (#0)
    * found 172 certificates in /etc/ssl/certs/ca-certificates.crt
    *    server certificate verification SKIPPED
    *    common name: 2145 (does not match 'svc-test')
    *    server certificate expiration date OK
    *    server certificate activation date OK
    *    certificate public key: RSA
    *    certificate version: #3
    *    subject: C=GB,L=Hursley,O=IBM,OU=SSG,CN=2145,EMAIL=support@ibm.com
    *    start date: Mon, 16 Jul 2012 10:57:29 GMT
    
    *    expire date: Tue, 13 Jul 2027 10:57:29 GMT
    
    *    issuer: C=GB,L=Hursley,O=IBM,OU=SSG,CN=2145,EMAIL=support@ibm.com
    *    compression: NULL
    *    cipher: AES-128-CBC
    *    MAC: SHA1
    * Server auth using Basic with user 'user'
    > POST /cimom HTTP/1.1
    Authorization: Basic enp6bmFnaW9zOm5hZ2lvcw==
    Host: svc-test:5989
    Content-Type: application/xml; charset="utf-8"
    Connection: Keep-Alive, TE
    CIMProtocolVersion: 1.0
    CIMOperation: MethodCall
    CIMMethod: EnumerateClassNames
    CIMObject: root%2Fibm
    Content-Length: 396
    
    * upload completely sent off: 396 out of 396 bytes
    * additional stuff not fine transfer.c:1037: 0 0
    * HTTP 1.1 or later with persistent connection, pipelining supported
    < HTTP/1.1 200 OK
    < Content-Type: application/xml; charset="utf-8"
    From server: Content-Type: application/xml; charset="utf-8"
    < content-length: 0000072284
    From server: content-length: 0000072284
    < CIMOperation: MethodResponse
    From server: CIMOperation: MethodResponse
    <
    * Connection #0 to host svc-test left intact
    From server: <?xml version="1.0" encoding="utf-8" ?>
    <CIM CIMVERSION="2.0" DTDVERSION="2.0">
    <MESSAGE ID="4711" PROTOCOLVERSION="1.0">
    <SIMPLERSP>
    <IMETHODRESPONSE NAME="EnumerateClassNames">
    <IRETURNVALUE>
    <CLASSNAME NAME="CIM_ConcreteIdentity"/>
    <CLASSNAME NAME="CIM_NetworkPacketAction"/>
    <CLASSNAME NAME="CIM_CollectionInSystem"/>
    [...]
    
    $ /opt/sblim-wbemcli/bin/wbemcli -noverify ecn https://user:pass@svc-test:5989/root/ibm
    svc-test:5989/root/ibm:CIM_ConcreteIdentity
    svc-test:5989/root/ibm:CIM_NetworkPacketAction
    svc-test:5989/root/ibm:CIM_CollectionInSystem
    svc-test:5989/root/ibm:CIM_DeviceSAPImplementation
    svc-test:5989/root/ibm:CIM_ProtocolControllerAccessesUnit
    svc-test:5989/root/ibm:CIM_ControlledBy
    [...]
    

    Great, now we're finally able to query and monitor the IBM SVC or Storwize systems again with the Nagios monitoring plugin (Nagios Monitoring - IBM SVC and Storwize)!

Between first noticing and researching the issue and creating the quick'n'dirty fix shown above, an official bug report has been filed on the issue and a patch has already been submitted to the source code repository in order to adress the issue more thoroughly. Hopefully an updated official source code package will be released soon.

Nonetheless, the process of researching and debugging this issue was an excellent hands on exercise for me, which i enjoyed very much. Hopefully the steps taken and described here, will turn out to be of use for others as well.

// HMC Update to 7.7.9.0 SP1

Like before with HMC Update to 7.7.5.0, HMC Update to 7.7.7.0 SP1 and HMC Update to 7.7.8.0, the recent HMC update to v7.7.9.0 was again not installable directly from the ISO images via the HMC GUI. Along with the HMC network installation images which are now mentioned in the release notes, there is now also an official documentation of the update procedure using the HMC network installation images. It's called “HMC network installation” and provides a more remote admin friendly way of performing the update. Since it's only a slightly shortened version of the procedure i already tested and used in HMC Update to 7.7.7.0 SP1, i decided to stick with my own procedure.

Also a turn for the better is, that now the release notes as well as FixCentral clearly point out the dependencies between the fixpacks that are supposed to go on top of the update release and the order they are supposed to be applied in. In case of MH01405 (aka V7R7.9.0.0) this is currently only MH01428 (aka V7R7.9.0.1 or v7.7.9.0 SP1.

As always, be sure to study the release notes thoroughly before an update attempt. Depending on your environment and HMC hardware there might be a road block in there. Special attention deserves item 2 in the “upgrade notes” section of the release notes of MH01405. Due to a bug in certain Power Systems firmware levels, there is a dependency between the usable HMC version and the firmware version on the managed system. Be sure to check if your managed systems firmware levels fall in the affected version ranges. For me this meant first upgrading certain managed systems to firmware levels not affected by the above bug. Then upgrading the HMCs to v7.7.9.0 SP1. And finally upgrading certain other managed systems to firmware levels depending on HMC version v7.7.9.0 or later. The latter dependency was - besides the fixed security issues - the main reason to upgrade the HMC in the first place.

For reference purposes, here are some example screen shots from a KVM session to the HMC during a update to MH01405:

HMC network based upgrade to v7.7.9.0 - 1

HMC network based upgrade to v7.7.9.0 - 2

HMC network based upgrade to v7.7.9.0 - 3

HMC network based upgrade to v7.7.9.0 - 4

HMC network based upgrade to v7.7.9.0 - 5

HMC network based upgrade to v7.7.9.0 - 6

HMC network based upgrade to v7.7.9.0 - 7

HMC network based upgrade to v7.7.9.0 - 8

HMC network based upgrade to v7.7.9.0 - 9

HMC network based upgrade to v7.7.9.0 - 10

HMC network based upgrade to v7.7.9.0 - 11

After the upgrade to v7.7.9.0 (MH01405) was complete and the HMC had rebooted, the following errors showed up:

Error after HMC upgrade to v7.7.9.0 - 1

Error after HMC upgrade to v7.7.9.0 - 2

Error after HMC upgrade to v7.7.9.0 - 3

There are not a lot of resources on this particular error, but it seems to occationally have occurred on earlier HMC upgrades as well. Up to now there seems to be no negative effect resulting from this error.

After the upgrade to V7R7.9.0.0 (MH01405) is complete, you can apply the service pack V7R7.9.0.1 (MH01428) in the usual way via the HMC GUI. For me the service pack showed the following output during the update process:

  • MH01428:

    Management console corrective service installation in progress. Please wait...
    Corrective service file offload from remote server in progress...
    The corrective service file offload was successful. Continuing with HMC service installation...
    Verifying Certificate Information
    Authenticating Install Packages
    Installing Packages
    --- Installing ptf-req ....
    --- Installing RSCT ....
    src-3.1.4.10-14056
    rsct.core.utils-3.1.4.10-14056
    rsct.core-3.1.4.10-14056
    rsct.service-3.5.0.0-1
    rsct.basic-3.1.4.10-14056
    --- Installing CSM ....
    csm.core-1.7.1.20-1
    csm.deploy-1.7.1.20-1
    csm_hmc.server-1.7.1.20-1
    csm_hmc.hdwr_svr-7.0-3.4.0
    csm_hmc.client-1.7.1.20-1
    csm.server.hsc-1.7.1.20-1
    --- Installing LPARCMD ....
    hsc.lparcmd-3.3.0.1-3
    ln: creating symbolic link `/usr/hmcrbin/lsnodeid': File exists
    ln: creating symbolic link `/usr/hmcrbin/lsrsrc-api': File exists
    ln: creating symbolic link `/usr/hmcrbin/mkrsrc-api': File exists
    ln: creating symbolic link `/usr/hmcrbin/rmrsrc-api': File exists
    --- Installing Pegasus ....
    --- Updating baseOS ....
    cp: cannot stat `.dev': No such file or directory
    PreInstalling HMC REST Web Services ...
    Installing HMC REST Web Services ...
    pmc.core-7.7.9.0-20140312T2001
    pmc.soliddb-7.7.9.0-20140312T2001
    pmc.wlp-7.7.9.0-20140312T2001
    pmc.wlp.soliddriver-7.7.9.0-20140312T2001
    pmc.wlp.log4j-7.7.9.0-20140312T2010
    pmc.wlp.guava-7.7.9.0-20140312T2010
    pmc.wlp.jaxb2.runtime-7.7.9.0-20140312T2010
    pmc.wlp.slf4j.api-7.7.9.0-20140312T2002
    pmc.wlp.quartz-7.7.9.0-20140312T2015
    pmc.wlp.commons-7.7.9.0-20140312T2013
    pmc.war.rest-7.7.9.0-20140312T2002
    pmc.soliddb.rest.sql-7.7.9.0-20140312T2010
    pmc.soliddb.pcm.sql-7.7.9.0-20140312T2015
    pmc.pcm.rest-7.7.9.0-20140312T2010
    pmc.ui.developer-7.7.9.0-20140312T2015
    D: opening db environment /var/lib/rpm/Packages create:cdb:mpool:private
    D: opening db index /var/lib/rpm/Packages rdonly mode=0x0
    D: locked db index /var/lib/rpm/Packages
    D: opening db index /var/lib/rpm/Name rdonly:nofsync mode=0x0
    D: read h# 849 Header SHA1 digest: OK (f280a38b206c3192103a5f4050991e1690ed43ea)
    D: ========== recording tsort relations
    D: ========== tsorting packages (order, #predecessors, #succesors, tree, depth, breadth)
    D: 0 0 0 0 1 0 -pmc.sem.war-7.7.9.0-20140312T2009.i386
    D: closed db index /var/lib/rpm/Name
    D: closed db index /var/lib/rpm/Packages
    D: closed db environment /var/lib/rpm/Packages
    D: opening db environment /var/lib/rpm/Packages create:cdb:mpool:private
    D: opening db index /var/lib/rpm/Packages create mode=0x42
    D: locked db index /var/lib/rpm/Packages
    D: sanity checking 1 elements
    D: running pre-transaction scripts
    D: computing 95 file fingerprints
    D: computing file dispositions
    D: opening db index /var/lib/rpm/Basenames create:nofsync mode=0x42
    D: read h# 931 Header SHA1 digest: OK (91c4128792b6b1d92318612e35f0d99c2b09bc41)
    D: read h# 932 Header SHA1 digest: OK (28745eb2d1d7a376c9cd9b457d024887849da559)
    D: read h# 933 Header SHA1 digest: OK (37f012c206c4a6cf6335be71d34ff09c0538d5b5)
    D: read h# 934 Header SHA1 digest: OK (9561ea0f53cff788ccf4a3f58655aff9a4d63bf3)
    D: read h# 935 Header SHA1 digest: OK (5718f74492d5e0af6858146bb5b72f588b0da4ab)
    D: read h# 936 Header SHA1 digest: OK (3a41ff5853613da4930e18097f657efb3a3b296f)
    D: read h# 937 Header SHA1 digest: OK (473cf0fe9c2b6f3458c180f1a8b206ed12f97536)
    D: read h# 938 Header SHA1 digest: OK (8b83dc36fa9ba3ee8a3454f00c0c1d3456794463)
    D: read h# 939 Header SHA1 digest: OK (76e2814d26d19030ca280e3a87c64a63f6db4939)
    D: read h# 940 Header SHA1 digest: OK (0bbd68866852e41a9813d4e95ae6f4af33316ad8)
    D: read h# 941 Header SHA1 digest: OK (88ca29e0d1812cccfd2a8729a9ae474366840cd0)
    D: read h# 942 Header SHA1 digest: OK (11fb71feb49795a55b8d4775dd3a87efa23ae085)
    D: read h# 943 Header SHA1 digest: OK (a65291baf06e105125843c7d9345284a4d036225)
    D: read h# 944 Header SHA1 digest: OK (fcf30711bf354a554a7a639bf634865a4a07dc35)
    D: read h# 945 Header SHA1 digest: OK (29769b20afd49242a01ba84d069521dac74bcb99)
    D: ========== --- pmc.sem.war-7.7.9.0-20140312T2009 i386-linux 0x0
    D: erase: pmc.sem.war-7.7.9.0-20140312T2009 has 95 files, test = 0
    D: opening db index /var/lib/rpm/Name create:nofsync mode=0x42
    D: read h# 849 Header SHA1 digest: OK (f280a38b206c3192103a5f4050991e1690ed43ea)
    D: opening db index /var/lib/rpm/Triggername create:nofsync mode=0x42
    D: erase: %preun(pmc.sem.war-7.7.9.0-20140312T2009.i386) asynchronous scriptlet start
    D: erase: %preun(pmc.sem.war-7.7.9.0-20140312T2009.i386) execv(/bin/sh) pid 16142
    D: erase: waitpid(16142) rc 16142 status 0 secs 0.001
    D: fini 100755 1 ( 504,1004) 662 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/theme/feed.css
    D: fini 100755 1 ( 504,1004) 625 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/theme/Master.css
    D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/theme
    D: fini 100755 1 ( 504,1004) 2121 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/feed.xsl
    D: fini 100755 1 ( 504,1004) 1704 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/web.xml
    D: fini 100755 1 ( 504,1004) 57211 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/wink-client-1.2.0-incubating.jar
    D: fini 100755 1 ( 504,1004) 679719 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.workhorse.templates-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 404664 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.workhorse.templates-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 35865 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.servlet.templates-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 29188 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.servlet.templates-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 26675 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.servlet.jdbc-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 128797 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.servlet.common-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 92119 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.servlet.common-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 41062 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.schema.web.src-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 93348 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.schema.web-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 349323 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.schema.vios-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 182187 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.schema.uom.src-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 509804 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.schema.uom-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 120510 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.schema.templates.src-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 415957 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.schema.templates-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 90846 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.schema.serviceableeventmanager.src-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 62845 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.schema.serviceableeventmanager-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 613724 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.westbound-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 643149 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.viosapi.rmc.handler-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 343001 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.viosapi.rmc.handler-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 55846 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.utils-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 31166 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.utils-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 1812 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.uom.vios-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 1787 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.uom.vios-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 399799 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.uom.phyp-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 247481 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.uom.phyp-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 15743 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.spi-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 10126 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.spi-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 22369 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.sem-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 13576 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.sem-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 62648 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.provider-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 14607 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.resources-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 12268 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.resources-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 32880 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.logging.log4j-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 13132 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.logging-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 10638 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.logging-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 877816 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.web-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 294774 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.web-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 762319 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.vios-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 582466 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.vios-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 11266109 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.uom-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 2401072 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.uom-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 5042339 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.templates-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 1587400 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.templates-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 581709 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.serviceableeventmanager-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 197153 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.serviceableeventmanager-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 472045 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.reflection-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 145769 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.reflection-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 493817 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.api.server-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 280894 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.api.server-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 76060 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.api.common-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 41833 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.api.common-7.7.9.0-src.jar
    D: fini 100755 1 ( 504,1004) 368989 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.cli.adapter-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 2061 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.annotations-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 1078246 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/dpsmapi-7.7.9.0.jar
    D: fini 100755 1 ( 504,1004) 243016 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/commons-lang-2.2.jar
    D: fini 100755 1 ( 504,1004) 65621 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/commons-io-1.2.jar
    D: fini 100755 1 ( 504,1004) 232771 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/commons-codec-1.6.jar
    D: fini 100755 1 ( 504,1004) 713 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/.jazzignore
    D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib
    D: fini 100755 1 ( 504,1004) 4892 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/geronimo-web.xml
    D: fini 100755 1 ( 504,1004) 5963 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/log4j2.xml
    D: fini 100755 1 ( 504,1004) 1378 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/com/ibm/pmc/war/servlet/applications/sem/SemServletApplication.class
    D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/com/ibm/pmc/war/servlet/applications/sem
    D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/com/ibm/pmc/war/servlet/applications
    D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/com/ibm/pmc/war/servlet
    D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/com/ibm/pmc/war
    D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/com/ibm/pmc
    D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/com/ibm
    D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/com
    D: fini 100755 1 ( 504,1004) 764 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/ServletDaemonClassnames.properties
    D: fini 100755 1 ( 504,1004) 771 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/REST_Servlet.properties.SAMPLE
    D: fini 100755 1 ( 504,1004) 967 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/META-INF/wink-alternate-shortcuts.properties
    D: fini 100755 1 ( 504,1004) 0 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/META-INF/server/wink-providers
    D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/META-INF/server
    D: fini 100755 1 ( 504,1004) 363 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/META-INF/persistence.xml
    D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/META-INF
    D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes
    D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF
    D: fini 100755 1 ( 504,1004) 275 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/META-INF/MANIFEST.MF
    D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/META-INF
    D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war
    D: fini 040755 7 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps skip
    D: fini 040755 8 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc skip
    D: fini 040755 6 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers skip
    D: fini 040755 4 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr skip
    D: fini 040755 9 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5 skip
    D: fini 040755 4 ( 504,1004) 4096 /opt/pmc/lib/share skip
    D: fini 040755 3 ( 504,1004) 4096 /opt/pmc/lib skip
    D: fini 040755 8 ( 504,1004) 4096 /opt/pmc skip
    D: erase: %postun(pmc.sem.war-7.7.9.0-20140312T2009.i386) asynchronous scriptlet start
    D: erase: %postun(pmc.sem.war-7.7.9.0-20140312T2009.i386) execv(/bin/sh) pid 16145
    + . /opt/pmc/bin/rpmScripts.include
    ++ EZLOG=/var/hsc/log/EZBuild.rpm.log
    ++ BASENAME=/usr/bin/basename
    ++ CAT=/bin/cat
    ++ CHMOD=/bin/chmod
    ++ CHOWN=/bin/chown
    ++ CP=/bin/cp
    ++ DATE='/bin/date --rfc-3339=seconds'
    ++ DIRNAME=/usr/bin/dirname
    ++ ECHO=/bin/echo
    ++ FIND=/usr/bin/find
    ++ GREP=/usr/bin/grep
    ++ LN=/bin/ln
    ++ MKDIR=/bin/mkdir
    ++ RM='/bin/rm -v'
    ++ RUNASROOT=/opt/ccfw/bin/framework/runAsRoot
    ++ TAR=/bin/tar
    ++ WC=/usr/bin/wc
    ++ UNZIP=/usr/bin/unzip
    ++ MV=/bin/mv
    + setupPMC
    + PMC_ROOT=/opt/pmc
    + PMC_BIN=/opt/pmc/bin
    + PMC_CONFIG=/opt/pmc/config
    + PMC_LIB=/opt/pmc/lib
    + PMC_LIB_SHARE=/opt/pmc/lib/share
    + PMC_LIB_WLP=/opt/pmc/lib/wlp
    + PMC_LIB_SOLIDDB=/opt/pmc/lib/soliddb
    + PMC_LOG=/opt/pmc/log
    + PMC_TMP=/opt/pmc/tmp
    + setupWLP
    + WLP_ROOT=/opt/pmc/lib/wlp
    + WLP_CONFIG_DIR=/opt/pmc/config/wlp
    + WLP_LOG_DIR=/opt/pmc/log/wlp
    + WLP_CONFIG_DIR=/opt/pmc/config/wlp
    + WLP_VERSION=8.5.5
    + WLP_SHARE_ROOT=/opt/pmc/lib/share/wlp-8.5.5
    + WLP_SERVER_ROOT=/opt/pmc/lib/wlp/usr/servers/pmc
    + WLP_PMC_CONTRIB=/opt/pmc/lib/wlp/usr/servers/pmc/libs/pmc-contrib/
    + WLP_SERVER_LOGS=/var/hsc/log/wlp
    + WLP_SERVER_APPS=/opt/pmc/lib/wlp/usr/servers/pmcapps/
    + HMC_PLUGINJARS_DIR=/usr/websm/codebase/pluginjars
    + RPM_NAME=pmc.sem.war
    + write_to_log START: RPM postUnInstall for pmc.sem.war
    ++ /bin/date --rfc-3339=seconds
    + echo '2014-12-04 22:24:13+01:00 START: RPM postUnInstall for pmc.sem.war'
    + /opt/pmc/bin/removeWARfromWLPserverxml.sh PMC_SEM_WAR
    + write_to_log removing of REST war is done
    ++ /bin/date --rfc-3339=seconds
    + echo '2014-12-04 22:24:13+01:00 removing of REST war is done'
    + write_to_log STOP: RPM postUnInstall for pmc.sem.war
    ++ /bin/date --rfc-3339=seconds
    + echo '2014-12-04 22:24:13+01:00 STOP: RPM postUnInstall for pmc.sem.war'
    D: erase: waitpid(16145) rc 16145 status 0 secs 0.016
    D: --- h# 849 pmc.sem.war-7.7.9.0-20140312T2009
    D: removing "pmc.sem.war" from Name index.
    D: removing 95 entries from Basenames index.
    D: opening db index /var/lib/rpm/Group create:nofsync mode=0x42
    D: removing "pmc" from Group index.
    D: opening db index /var/lib/rpm/Requirename create:nofsync mode=0x42
    D: removing 5 entries from Requirename index.
    D: opening db index /var/lib/rpm/Providename create:nofsync mode=0x42
    D: removing "pmc.sem.war" from Providename index.
    D: opening db index /var/lib/rpm/Dirnames create:nofsync mode=0x42
    D: removing 24 entries from Dirnames index.
    D: opening db index /var/lib/rpm/Requireversion create:nofsync mode=0x42
    D: removing 5 entries from Requireversion index.
    D: opening db index /var/lib/rpm/Provideversion create:nofsync mode=0x42
    D: removing "0:7.7.9.0-20140312T2009" from Provideversion index.
    D: opening db index /var/lib/rpm/Installtid create:nofsync mode=0x42
    D: removing 1 entries from Installtid index.
    D: opening db index /var/lib/rpm/Sigmd5 create:nofsync mode=0x42
    D: removing 1 entries from Sigmd5 index.
    D: opening db index /var/lib/rpm/Sha1header create:nofsync mode=0x42
    D: removing "f280a38b206c3192103a5f4050991e1690ed43ea" from Sha1header index.
    D: opening db index /var/lib/rpm/Filemd5s create:nofsync mode=0x42
    D: removing 95 entries from Filemd5s index.
    D: running post-transaction scripts
    D: closed db index /var/lib/rpm/Filemd5s
    D: closed db index /var/lib/rpm/Sha1header
    D: closed db index /var/lib/rpm/Sigmd5
    D: closed db index /var/lib/rpm/Installtid
    D: closed db index /var/lib/rpm/Provideversion
    D: closed db index /var/lib/rpm/Requireversion
    D: closed db index /var/lib/rpm/Dirnames
    D: closed db index /var/lib/rpm/Triggername
    D: closed db index /var/lib/rpm/Providename
    D: closed db index /var/lib/rpm/Requirename
    D: closed db index /var/lib/rpm/Group
    D: closed db index /var/lib/rpm/Basenames
    D: closed db index /var/lib/rpm/Name
    D: closed db index /var/lib/rpm/Packages
    D: closed db environment /var/lib/rpm/Packages
    D: May free Score board((nil))
    package pmc.sem.war is not installed
    Corrective service installation was successful.

    For reference purposes, here are some example screen shots from a KVM session to the HMC during a update to MH01428:

    HMC network based upgrade to v7.7.9.0 SP1 - 1

    HMC network based upgrade to v7.7.9.0 SP1 - 2

The MH01428 update still shows error messages with regard to symlink creation appearing during the update process, which can be savely ignored. The error message “cp: cannot stat `.dev': No such file or directory” is also becoming an old friend and originates from the shell script /images/installImages inside the MH01428 installation ISO image, where non-existing files are attempted to be copied.

The strange output lines starting with “D:”, “+” and “++” characters seemed awfully familiar, just like the output of a “rpm -vv” command. From tracing down the call order of /images/installImages inside the MH01428 installation ISO image:

/images/installImages
448
449 if [ -f $image/installK2Payloads.sh ]
450 then
451     echo "Installing HMC REST Web Services ..."
452     $image/./installK2Payloads.sh $image
453 fi
454

to:

/images/installK2Payloads.sh
  4
  5 LogFile=/tmp/K2Install.log
  6 IndexFile=K2Payloads.txt
  7
[...]
 73     while read line
 74     do
 75     rpmToInstall=`echo $line | cut -d '/' -f 4`
 76     rpmToUpdate=`echo $line | cut -d '/' -f 4 | cut -d '-' -f 1`
 77 
 78     #   pmc.sem.war needs to be uninstalled and then installed
 79     if [ "$rpmToUpdate" == "pmc.sem.war" ]; then
 80         /bin/rpm -evv $rpmToUpdate --nodeps --allmatches
 81     fi
 82 
 83     /bin/rpm -q $rpmToUpdate
 84     if [ $? -eq 0 ]; then
 85         /bin/rpm -vvh -U $rpmToInstall >> $LogFile 2>&1
 86     else
 87         /bin/rpm -vv -i $rpmToInstall >> $LogFile 2>&1
 88     fi
 89     done < $IndexFile

it seems the dubious output is due to a missing output redirection at the “rpm -evv pmc.sem.war” command in line 80 in the above code snippet of “/images/installK2Payloads.sh”. Since the uninstall of pmc.sem.war is mentioned in the comment in line 78 and the uninstall process apparently finished successfully it can be savely ignored. Although from the above output and without privileged access to the HMCs OS its hard to verify, whether the RPM “pmc.sem.war-7.7.9.1-20140905T1656.i386.rpm” was subsequently successfully re-installed. It can indirectly be verified by looking at the log entries written to the file “/tmp/K2Install.log” in line 87 of the above code snippet. The contents of the log file “/var/hsc/log/EZBuild.rpm.log” written from within the RPM install-/erase-time scriptlets could also be interesting in order to verify everything was re-installed properly.

Aside from the error and dubious messages mentioned above, we're currently not experiencing any issues with the new HMC version v7.7.9.0 SP1 (V7R7.9.0.1).

This website uses cookies for visitor traffic analysis. By using the website, you agree with storing the cookies on your computer.More information