2014-12-09 // HMC Update to 7.7.9.0 SP1
Like before with HMC Update to 7.7.5.0, HMC Update to 7.7.7.0 SP1 and HMC Update to 7.7.8.0, the recent HMC update to v7.7.9.0 was again not installable directly from the ISO images via the HMC GUI. Along with the HMC network installation images which are now mentioned in the release notes, there is now also an official documentation of the update procedure using the HMC network installation images. It's called “HMC network installation” and provides a more remote admin friendly way of performing the update. Since it's only a slightly shortened version of the procedure i already tested and used in HMC Update to 7.7.7.0 SP1, i decided to stick with my own procedure.
Also a turn for the better is, that now the release notes as well as FixCentral clearly point out the dependencies between the fixpacks that are supposed to go on top of the update release and the order they are supposed to be applied in. In case of MH01405 (aka V7R7.9.0.0) this is currently only MH01428 (aka V7R7.9.0.1 or v7.7.9.0 SP1.
As always, be sure to study the release notes thoroughly before an update attempt. Depending on your environment and HMC hardware there might be a road block in there. Special attention deserves item 2 in the “upgrade notes” section of the release notes of MH01405. Due to a bug in certain Power Systems firmware levels, there is a dependency between the usable HMC version and the firmware version on the managed system. Be sure to check if your managed systems firmware levels fall in the affected version ranges. For me this meant first upgrading certain managed systems to firmware levels not affected by the above bug. Then upgrading the HMCs to v7.7.9.0 SP1. And finally upgrading certain other managed systems to firmware levels depending on HMC version v7.7.9.0 or later. The latter dependency was - besides the fixed security issues - the main reason to upgrade the HMC in the first place.
For reference purposes, here are some example screen shots from a KVM session to the HMC during a update to MH01405:
After the upgrade to v7.7.9.0 (MH01405) was complete and the HMC had rebooted, the following errors showed up:
There are not a lot of resources on this particular error, but it seems to occationally have occurred on earlier HMC upgrades as well. Up to now there seems to be no negative effect resulting from this error.
After the upgrade to V7R7.9.0.0 (MH01405) is complete, you can apply the service pack V7R7.9.0.1 (MH01428) in the usual way via the HMC GUI. For me the service pack showed the following output during the update process:
MH01428:
Management console corrective service installation in progress. Please wait... Corrective service file offload from remote server in progress... The corrective service file offload was successful. Continuing with HMC service installation... Verifying Certificate Information Authenticating Install Packages Installing Packages --- Installing ptf-req .... --- Installing RSCT .... src-3.1.4.10-14056 rsct.core.utils-3.1.4.10-14056 rsct.core-3.1.4.10-14056 rsct.service-3.5.0.0-1 rsct.basic-3.1.4.10-14056 --- Installing CSM .... csm.core-1.7.1.20-1 csm.deploy-1.7.1.20-1 csm_hmc.server-1.7.1.20-1 csm_hmc.hdwr_svr-7.0-3.4.0 csm_hmc.client-1.7.1.20-1 csm.server.hsc-1.7.1.20-1 --- Installing LPARCMD .... hsc.lparcmd-3.3.0.1-3 ln: creating symbolic link `/usr/hmcrbin/lsnodeid': File exists ln: creating symbolic link `/usr/hmcrbin/lsrsrc-api': File exists ln: creating symbolic link `/usr/hmcrbin/mkrsrc-api': File exists ln: creating symbolic link `/usr/hmcrbin/rmrsrc-api': File exists --- Installing Pegasus .... --- Updating baseOS .... cp: cannot stat `.dev': No such file or directory PreInstalling HMC REST Web Services ... Installing HMC REST Web Services ... pmc.core-7.7.9.0-20140312T2001 pmc.soliddb-7.7.9.0-20140312T2001 pmc.wlp-7.7.9.0-20140312T2001 pmc.wlp.soliddriver-7.7.9.0-20140312T2001 pmc.wlp.log4j-7.7.9.0-20140312T2010 pmc.wlp.guava-7.7.9.0-20140312T2010 pmc.wlp.jaxb2.runtime-7.7.9.0-20140312T2010 pmc.wlp.slf4j.api-7.7.9.0-20140312T2002 pmc.wlp.quartz-7.7.9.0-20140312T2015 pmc.wlp.commons-7.7.9.0-20140312T2013 pmc.war.rest-7.7.9.0-20140312T2002 pmc.soliddb.rest.sql-7.7.9.0-20140312T2010 pmc.soliddb.pcm.sql-7.7.9.0-20140312T2015 pmc.pcm.rest-7.7.9.0-20140312T2010 pmc.ui.developer-7.7.9.0-20140312T2015 D: opening db environment /var/lib/rpm/Packages create:cdb:mpool:private D: opening db index /var/lib/rpm/Packages rdonly mode=0x0 D: locked db index /var/lib/rpm/Packages D: opening db index /var/lib/rpm/Name rdonly:nofsync mode=0x0 D: read h# 849 Header SHA1 digest: OK (f280a38b206c3192103a5f4050991e1690ed43ea) D: ========== recording tsort relations D: ========== tsorting packages (order, #predecessors, #succesors, tree, depth, breadth) D: 0 0 0 0 1 0 -pmc.sem.war-7.7.9.0-20140312T2009.i386 D: closed db index /var/lib/rpm/Name D: closed db index /var/lib/rpm/Packages D: closed db environment /var/lib/rpm/Packages D: opening db environment /var/lib/rpm/Packages create:cdb:mpool:private D: opening db index /var/lib/rpm/Packages create mode=0x42 D: locked db index /var/lib/rpm/Packages D: sanity checking 1 elements D: running pre-transaction scripts D: computing 95 file fingerprints D: computing file dispositions D: opening db index /var/lib/rpm/Basenames create:nofsync mode=0x42 D: read h# 931 Header SHA1 digest: OK (91c4128792b6b1d92318612e35f0d99c2b09bc41) D: read h# 932 Header SHA1 digest: OK (28745eb2d1d7a376c9cd9b457d024887849da559) D: read h# 933 Header SHA1 digest: OK (37f012c206c4a6cf6335be71d34ff09c0538d5b5) D: read h# 934 Header SHA1 digest: OK (9561ea0f53cff788ccf4a3f58655aff9a4d63bf3) D: read h# 935 Header SHA1 digest: OK (5718f74492d5e0af6858146bb5b72f588b0da4ab) D: read h# 936 Header SHA1 digest: OK (3a41ff5853613da4930e18097f657efb3a3b296f) D: read h# 937 Header SHA1 digest: OK (473cf0fe9c2b6f3458c180f1a8b206ed12f97536) D: read h# 938 Header SHA1 digest: OK (8b83dc36fa9ba3ee8a3454f00c0c1d3456794463) D: read h# 939 Header SHA1 digest: OK (76e2814d26d19030ca280e3a87c64a63f6db4939) D: read h# 940 Header SHA1 digest: OK (0bbd68866852e41a9813d4e95ae6f4af33316ad8) D: read h# 941 Header SHA1 digest: OK (88ca29e0d1812cccfd2a8729a9ae474366840cd0) D: read h# 942 Header SHA1 digest: OK (11fb71feb49795a55b8d4775dd3a87efa23ae085) D: read h# 943 Header SHA1 digest: OK (a65291baf06e105125843c7d9345284a4d036225) D: read h# 944 Header SHA1 digest: OK (fcf30711bf354a554a7a639bf634865a4a07dc35) D: read h# 945 Header SHA1 digest: OK (29769b20afd49242a01ba84d069521dac74bcb99) D: ========== --- pmc.sem.war-7.7.9.0-20140312T2009 i386-linux 0x0 D: erase: pmc.sem.war-7.7.9.0-20140312T2009 has 95 files, test = 0 D: opening db index /var/lib/rpm/Name create:nofsync mode=0x42 D: read h# 849 Header SHA1 digest: OK (f280a38b206c3192103a5f4050991e1690ed43ea) D: opening db index /var/lib/rpm/Triggername create:nofsync mode=0x42 D: erase: %preun(pmc.sem.war-7.7.9.0-20140312T2009.i386) asynchronous scriptlet start D: erase: %preun(pmc.sem.war-7.7.9.0-20140312T2009.i386) execv(/bin/sh) pid 16142 D: erase: waitpid(16142) rc 16142 status 0 secs 0.001 D: fini 100755 1 ( 504,1004) 662 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/theme/feed.css D: fini 100755 1 ( 504,1004) 625 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/theme/Master.css D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/theme D: fini 100755 1 ( 504,1004) 2121 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/feed.xsl D: fini 100755 1 ( 504,1004) 1704 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/web.xml D: fini 100755 1 ( 504,1004) 57211 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/wink-client-1.2.0-incubating.jar D: fini 100755 1 ( 504,1004) 679719 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.workhorse.templates-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 404664 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.workhorse.templates-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 35865 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.servlet.templates-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 29188 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.servlet.templates-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 26675 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.servlet.jdbc-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 128797 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.servlet.common-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 92119 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.servlet.common-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 41062 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.schema.web.src-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 93348 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.schema.web-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 349323 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.schema.vios-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 182187 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.schema.uom.src-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 509804 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.schema.uom-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 120510 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.schema.templates.src-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 415957 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.schema.templates-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 90846 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.schema.serviceableeventmanager.src-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 62845 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.schema.serviceableeventmanager-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 613724 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.westbound-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 643149 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.viosapi.rmc.handler-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 343001 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.viosapi.rmc.handler-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 55846 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.utils-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 31166 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.utils-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 1812 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.uom.vios-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 1787 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.uom.vios-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 399799 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.uom.phyp-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 247481 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.uom.phyp-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 15743 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.spi-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 10126 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.spi-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 22369 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.sem-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 13576 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.servlet.sem-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 62648 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.rest.provider-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 14607 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.resources-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 12268 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.resources-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 32880 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.logging.log4j-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 13132 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.logging-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 10638 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.logging-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 877816 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.web-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 294774 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.web-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 762319 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.vios-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 582466 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.vios-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 11266109 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.uom-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 2401072 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.uom-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 5042339 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.templates-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 1587400 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.templates-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 581709 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.serviceableeventmanager-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 197153 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.serviceableeventmanager-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 472045 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.reflection-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 145769 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.reflection-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 493817 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.api.server-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 280894 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.api.server-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 76060 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.api.common-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 41833 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.jaxb.api.common-7.7.9.0-src.jar D: fini 100755 1 ( 504,1004) 368989 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.cli.adapter-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 2061 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/pmc.annotations-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 1078246 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/dpsmapi-7.7.9.0.jar D: fini 100755 1 ( 504,1004) 243016 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/commons-lang-2.2.jar D: fini 100755 1 ( 504,1004) 65621 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/commons-io-1.2.jar D: fini 100755 1 ( 504,1004) 232771 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/commons-codec-1.6.jar D: fini 100755 1 ( 504,1004) 713 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib/.jazzignore D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/lib D: fini 100755 1 ( 504,1004) 4892 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/geronimo-web.xml D: fini 100755 1 ( 504,1004) 5963 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/log4j2.xml D: fini 100755 1 ( 504,1004) 1378 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/com/ibm/pmc/war/servlet/applications/sem/SemServletApplication.class D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/com/ibm/pmc/war/servlet/applications/sem D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/com/ibm/pmc/war/servlet/applications D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/com/ibm/pmc/war/servlet D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/com/ibm/pmc/war D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/com/ibm/pmc D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/com/ibm D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/com D: fini 100755 1 ( 504,1004) 764 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/ServletDaemonClassnames.properties D: fini 100755 1 ( 504,1004) 771 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/REST_Servlet.properties.SAMPLE D: fini 100755 1 ( 504,1004) 967 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/META-INF/wink-alternate-shortcuts.properties D: fini 100755 1 ( 504,1004) 0 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/META-INF/server/wink-providers D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/META-INF/server D: fini 100755 1 ( 504,1004) 363 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/META-INF/persistence.xml D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes/META-INF D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF/classes D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/WEB-INF D: fini 100755 1 ( 504,1004) 275 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/META-INF/MANIFEST.MF D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war/META-INF D: fini 040755 2 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps/PMC_SEM.war D: fini 040755 7 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc/apps skip D: fini 040755 8 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers/pmc skip D: fini 040755 6 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr/servers skip D: fini 040755 4 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5/usr skip D: fini 040755 9 ( 504,1004) 4096 /opt/pmc/lib/share/wlp-8.5.5 skip D: fini 040755 4 ( 504,1004) 4096 /opt/pmc/lib/share skip D: fini 040755 3 ( 504,1004) 4096 /opt/pmc/lib skip D: fini 040755 8 ( 504,1004) 4096 /opt/pmc skip D: erase: %postun(pmc.sem.war-7.7.9.0-20140312T2009.i386) asynchronous scriptlet start D: erase: %postun(pmc.sem.war-7.7.9.0-20140312T2009.i386) execv(/bin/sh) pid 16145 + . /opt/pmc/bin/rpmScripts.include ++ EZLOG=/var/hsc/log/EZBuild.rpm.log ++ BASENAME=/usr/bin/basename ++ CAT=/bin/cat ++ CHMOD=/bin/chmod ++ CHOWN=/bin/chown ++ CP=/bin/cp ++ DATE='/bin/date --rfc-3339=seconds' ++ DIRNAME=/usr/bin/dirname ++ ECHO=/bin/echo ++ FIND=/usr/bin/find ++ GREP=/usr/bin/grep ++ LN=/bin/ln ++ MKDIR=/bin/mkdir ++ RM='/bin/rm -v' ++ RUNASROOT=/opt/ccfw/bin/framework/runAsRoot ++ TAR=/bin/tar ++ WC=/usr/bin/wc ++ UNZIP=/usr/bin/unzip ++ MV=/bin/mv + setupPMC + PMC_ROOT=/opt/pmc + PMC_BIN=/opt/pmc/bin + PMC_CONFIG=/opt/pmc/config + PMC_LIB=/opt/pmc/lib + PMC_LIB_SHARE=/opt/pmc/lib/share + PMC_LIB_WLP=/opt/pmc/lib/wlp + PMC_LIB_SOLIDDB=/opt/pmc/lib/soliddb + PMC_LOG=/opt/pmc/log + PMC_TMP=/opt/pmc/tmp + setupWLP + WLP_ROOT=/opt/pmc/lib/wlp + WLP_CONFIG_DIR=/opt/pmc/config/wlp + WLP_LOG_DIR=/opt/pmc/log/wlp + WLP_CONFIG_DIR=/opt/pmc/config/wlp + WLP_VERSION=8.5.5 + WLP_SHARE_ROOT=/opt/pmc/lib/share/wlp-8.5.5 + WLP_SERVER_ROOT=/opt/pmc/lib/wlp/usr/servers/pmc + WLP_PMC_CONTRIB=/opt/pmc/lib/wlp/usr/servers/pmc/libs/pmc-contrib/ + WLP_SERVER_LOGS=/var/hsc/log/wlp + WLP_SERVER_APPS=/opt/pmc/lib/wlp/usr/servers/pmcapps/ + HMC_PLUGINJARS_DIR=/usr/websm/codebase/pluginjars + RPM_NAME=pmc.sem.war + write_to_log START: RPM postUnInstall for pmc.sem.war ++ /bin/date --rfc-3339=seconds + echo '2014-12-04 22:24:13+01:00 START: RPM postUnInstall for pmc.sem.war' + /opt/pmc/bin/removeWARfromWLPserverxml.sh PMC_SEM_WAR + write_to_log removing of REST war is done ++ /bin/date --rfc-3339=seconds + echo '2014-12-04 22:24:13+01:00 removing of REST war is done' + write_to_log STOP: RPM postUnInstall for pmc.sem.war ++ /bin/date --rfc-3339=seconds + echo '2014-12-04 22:24:13+01:00 STOP: RPM postUnInstall for pmc.sem.war' D: erase: waitpid(16145) rc 16145 status 0 secs 0.016 D: --- h# 849 pmc.sem.war-7.7.9.0-20140312T2009 D: removing "pmc.sem.war" from Name index. D: removing 95 entries from Basenames index. D: opening db index /var/lib/rpm/Group create:nofsync mode=0x42 D: removing "pmc" from Group index. D: opening db index /var/lib/rpm/Requirename create:nofsync mode=0x42 D: removing 5 entries from Requirename index. D: opening db index /var/lib/rpm/Providename create:nofsync mode=0x42 D: removing "pmc.sem.war" from Providename index. D: opening db index /var/lib/rpm/Dirnames create:nofsync mode=0x42 D: removing 24 entries from Dirnames index. D: opening db index /var/lib/rpm/Requireversion create:nofsync mode=0x42 D: removing 5 entries from Requireversion index. D: opening db index /var/lib/rpm/Provideversion create:nofsync mode=0x42 D: removing "0:7.7.9.0-20140312T2009" from Provideversion index. D: opening db index /var/lib/rpm/Installtid create:nofsync mode=0x42 D: removing 1 entries from Installtid index. D: opening db index /var/lib/rpm/Sigmd5 create:nofsync mode=0x42 D: removing 1 entries from Sigmd5 index. D: opening db index /var/lib/rpm/Sha1header create:nofsync mode=0x42 D: removing "f280a38b206c3192103a5f4050991e1690ed43ea" from Sha1header index. D: opening db index /var/lib/rpm/Filemd5s create:nofsync mode=0x42 D: removing 95 entries from Filemd5s index. D: running post-transaction scripts D: closed db index /var/lib/rpm/Filemd5s D: closed db index /var/lib/rpm/Sha1header D: closed db index /var/lib/rpm/Sigmd5 D: closed db index /var/lib/rpm/Installtid D: closed db index /var/lib/rpm/Provideversion D: closed db index /var/lib/rpm/Requireversion D: closed db index /var/lib/rpm/Dirnames D: closed db index /var/lib/rpm/Triggername D: closed db index /var/lib/rpm/Providename D: closed db index /var/lib/rpm/Requirename D: closed db index /var/lib/rpm/Group D: closed db index /var/lib/rpm/Basenames D: closed db index /var/lib/rpm/Name D: closed db index /var/lib/rpm/Packages D: closed db environment /var/lib/rpm/Packages D: May free Score board((nil)) package pmc.sem.war is not installed Corrective service installation was successful.
For reference purposes, here are some example screen shots from a KVM session to the HMC during a update to MH01428:
The MH01428 update still shows error messages with regard to symlink creation appearing during the update process, which can be savely ignored. The error message “cp: cannot stat `.dev': No such file or directory” is also becoming an old friend and originates from the shell script /images/installImages inside the MH01428 installation ISO image, where non-existing files are attempted to be copied.
The strange output lines starting with “D:”, “+” and “++” characters seemed awfully familiar, just like the output of a “rpm -vv” command. From tracing down the call order of /images/installImages inside the MH01428 installation ISO image:
- /images/installImages
448 449 if [ -f $image/installK2Payloads.sh ] 450 then 451 echo "Installing HMC REST Web Services ..." 452 $image/./installK2Payloads.sh $image 453 fi 454
to:
- /images/installK2Payloads.sh
4 5 LogFile=/tmp/K2Install.log 6 IndexFile=K2Payloads.txt 7 [...] 73 while read line 74 do 75 rpmToInstall=`echo $line | cut -d '/' -f 4` 76 rpmToUpdate=`echo $line | cut -d '/' -f 4 | cut -d '-' -f 1` 77 78 # pmc.sem.war needs to be uninstalled and then installed 79 if [ "$rpmToUpdate" == "pmc.sem.war" ]; then 80 /bin/rpm -evv $rpmToUpdate --nodeps --allmatches 81 fi 82 83 /bin/rpm -q $rpmToUpdate 84 if [ $? -eq 0 ]; then 85 /bin/rpm -vvh -U $rpmToInstall >> $LogFile 2>&1 86 else 87 /bin/rpm -vv -i $rpmToInstall >> $LogFile 2>&1 88 fi 89 done < $IndexFile
it seems the dubious output is due to a missing output redirection at the “rpm -evv pmc.sem.war” command in line 80 in the above code snippet of “/images/installK2Payloads.sh”. Since the uninstall of pmc.sem.war is mentioned in the comment in line 78 and the uninstall process apparently finished successfully it can be savely ignored. Although from the above output and without privileged access to the HMCs OS its hard to verify, whether the RPM “pmc.sem.war-7.7.9.1-20140905T1656.i386.rpm” was subsequently successfully re-installed. It can indirectly be verified by looking at the log entries written to the file “/tmp/K2Install.log” in line 87 of the above code snippet. The contents of the log file “/var/hsc/log/EZBuild.rpm.log” written from within the RPM install-/erase-time scriptlets could also be interesting in order to verify everything was re-installed properly.
Aside from the error and dubious messages mentioned above, we're currently not experiencing any issues with the new HMC version v7.7.9.0 SP1 (V7R7.9.0.1).
2014-04-26 // HMC Update to 7.7.8.0 SP1
Updating the HMC from v7.7.8.0 to v7.7.8.0 SP1 was once again and like the previous HMC Update to 7.7.6.0 SP1 and HMC Update to 7.7.7.0 SP2 very painless. The service pack MH01397 was easily installable from the ISO images via the HMC GUI.
The service pack and the additional efixes showed the following output during the update process:
MH01397:
Management console corrective service installation in progress. Please wait... Corrective service file offload from remote server in progress... The corrective service file offload was successful. Continuing with HMC service installation... Verifying Certificate Information Authenticating Install Packages Installing Packages --- Installing ptf-req .... --- Installing RSCT .... src-3.1.4.9-13275 rsct.core.utils-3.1.4.9-13275 rsct.core-3.1.4.9-13275 rsct.service-3.5.0.0-1 rsct.basic-3.1.4.9-13275 --- Installing CSM .... csm.core-1.7.1.20-1 csm.deploy-1.7.1.20-1 csm_hmc.server-1.7.1.20-1 csm_hmc.hdwr_svr-7.0-3.4.0 csm_hmc.client-1.7.1.20-1 csm.server.hsc-1.7.1.20-1 --- Installing LPARCMD .... hsc.lparcmd-3.3.0.1-1 ln: creating symbolic link `/usr/hmcrbin/lsnodeid': File exists ln: creating symbolic link `/usr/hmcrbin/lsrsrc-api' : File exists ln: creating symbolic link `/usr/hmcrbin/mkrsrc-api' : File exists ln: creating symbolic link `/usr/hmcrbin/rmrsrc-api' : File exists --- Installing InventoryScout .... --- Installing Pegasus .... --- Updating baseOS .... cp: cannot stat `.dev': No such file or directory cp: cannot stat `.image.updates': No such file or directory PreInstalling HMC REST Web Services ... Installing HMC REST Web Services ... /dump/hsc_install.images/images/./installK2Payloads.sh: line 84: K2Payloads.txt: No such file or directory Corrective service installation was successful.
MH01416:
Management console corrective service installation in progress. Please wait... Corrective service file offload from remote server in progress... The corrective service file offload was successful. Continuing with HMC service installation... Verifying Certificate Information Authenticating Install Packages Installing Packages --- Installing ptf-req .... Corrective service installation was successful.
MH01423:
Management console corrective service installation in progress. Please wait... Corrective service file offload from remote server in progress... The corrective service file offload was successful. Continuing with HMC service installation... Verifying Certificate Information Authenticating Install Packages Installing Packages --- Installing ptf-req .... Verifying archive integrity... All good. Uncompressing Updating from 2014-02-17 to 2014-04-08. ................................................................................................... ......................................................................................................................... ... .. ........ .. .. .. .. .. ........ ... .. .. .. .. ........ .. .. ... . ... . .. .. . ......... . . . . . Updating to hmc61_1-mcp-2014-04-08-203505... upgrading distro_id... Preparing... ################################################## distro_id ######### ################################### # # ## ## Preparing... ########## ########## ########## ########## ########## openssl # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # ###### # binutils_static # # # # # # # # # # # ### # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # distro_id ######### ################################## # # ### ## openssl-fips ## # ##### ###### ###### ##### ###### ###### ###### ##### ## Updating distribution id... Updating manifest... Corrective service installation was successful.
The MH01397 update still shows error messages with regard to symlink creation appearing during the update process, which can be savely ignored. The error message “cp: cannot stat `.dev': No such file or directory” is also becoming an old friend and originates from the shell script /images/installImages inside the MH01397 installation ISO image, where non-existing files are attempted to be copied. The newly introduced error message “cp: cannot stat `.image.updates': No such file or directory” falls into the same category.
The error message “/dump/hsc_install.images/images/./installK2Payloads.sh: line 84: K2Payloads.txt: No such file or directory” seems to be more serious. From tracing down the call order of /images/installImages inside the MH01397 installation ISO image:
- /images/installImages
448 449 if [ -f $image/installK2Payloads.sh ] 450 then 451 echo "Installing HMC REST Web Services ..." 452 $image/./installK2Payloads.sh $image 453 fi 454
to:
- /images/installK2Payloads.sh
4 5 LogFile=/tmp/K2Install.log 6 IndexFile=K2Payloads.txt 7 [...] 68 69 # Install the K2 rpms if K2 directory exists 70 if [ -d $image/K2 ]; then 71 cd $image/K2 72 73 while read line 74 do 75 rpmToInstall=`echo $line | cut -d '/' -f 4` 76 rpmToUpdate=`echo $line | cut -d '/' -f 4 | cut -d '-' -f 1` 77 /bin/rpm -q $rpmToUpdate 78 if [ $? -eq 0 ]; then 79 /bin/rpm -vvh -U $rpmToInstall >> $LogFile 2>&1 80 else 81 /bin/rpm -vv -i $rpmToInstall >> $LogFile 2>&1 82 fi 83 done < $IndexFile 84 fi 85
it seems the file K2Payloads.txt is missing from the “K2” directory containing the RPM files for the HMC REST web services:
$ mount -o loop HMC_Update_V7R780_SP1.iso /mnt $ ls -al /mnt/images/K2/ total 251394 dr-xr-xr-x 2 root root 4096 Mar 5 19:57 . dr-xr-xr-x 8 root root 2048 Mar 5 19:57 .. -r--r--r-- 1 root root 30 Feb 18 03:52 k2.version -r--r--r-- 1 root root 13398 Feb 18 04:12 pmc.core-7.7.8.1-20140217T2012.i386.rpm -r--r--r-- 1 root root 14242627 Feb 18 04:00 pmc.pcm.rest-7.7.8.1-20140217T2019.i386.rpm -r--r--r-- 1 root root 33865936 Feb 18 03:58 pmc.soliddb-7.7.8.1-20140217T2012.i386.rpm -r--r--r-- 1 root root 15375 Feb 18 04:01 pmc.soliddb.pcm.sql-7.7.8.1-20140217T2020.i386.rpm -r--r--r-- 1 root root 46559 Feb 18 04:10 pmc.soliddb.rest.sql-7.7.8.1-20140217T2019.i386.rpm -r--r--r-- 1 root root 54565876 Feb 18 03:59 pmc.ui.developer-7.7.8.1-20140217T2021.i386.rpm -r--r--r-- 1 root root 74508025 Feb 18 03:57 pmc.war.rest-7.7.8.1-20140217T2013.i386.rpm -r--r--r-- 1 root root 76751466 Feb 18 03:58 pmc.wlp-7.7.8.1-20140217T2012.i386.rpm -r--r--r-- 1 root root 478438 Feb 18 04:08 pmc.wlp.commons-7.7.8.1-20140217T2013.i386.rpm -r--r--r-- 1 root root 1695200 Feb 18 04:08 pmc.wlp.guava-7.7.8.1-20140217T2019.i386.rpm -r--r--r-- 1 root root 109789 Feb 18 04:09 pmc.wlp.jaxb2.runtime-7.7.8.1-20140217T2019.i386.rpm -r--r--r-- 1 root root 446974 Feb 18 04:08 pmc.wlp.log4j-7.7.8.1-20140217T2019.i386.rpm -r--r--r-- 1 root root 428041 Feb 18 04:09 pmc.wlp.quartz-7.7.8.1-20140217T2020.i386.rpm -r--r--r-- 1 root root 29861 Feb 18 04:09 pmc.wlp.slf4j.api-7.7.8.1-20140217T2013.i386.rpm -r--r--r-- 1 root root 219413 Feb 18 03:58 pmc.wlp.soliddriver-7.7.8.1-20140217T2012.i386.rpm
Without privileged access to the HMCs OS its hard to verify, but it appears that the updates to the HMC REST web services have not been installed. So i guess i'll be opening up a PMR about that now.
The weird “.” and “#” characters in the output of the MH01423 update can also savely be ignored. They're just unhandled output from the update script and the RPM commands called therein. To take a look for yourself:
$ mkdir /tmp/hmc && cd /tmp/hmc $ mount -o loop MH01423.iso /mnt $ /mnt/images/update-hmc61_1-20140217-20140408.sh --tar xvf $ ls -al total 4100 drwxr-xr-x 3 root root 4096 Apr 26 23:07 . drwxrwxrwt 35 root root 24576 Apr 26 23:07 .. -rw-rw-r-- 1 root root 2524328 Jun 19 2010 binutils_static-2.20.0-21.i686.rpm drwxr-xr-x 2 root root 20480 Apr 9 17:48 DEPENDS -rw-rw-r-- 1 root root 24564 Aug 23 2012 distro_id-1.0-9.i686.rpm -rwxr-xr-x 1 root root 56125 Apr 9 17:49 hmc61_1-mcp-2014-04-08-203505-manifest.txt -rw-rw-r-- 1 root root 1316850 Apr 8 22:56 openssl-1.0.1e-20.i686.rpm -rw-rw-r-- 1 root root 198060 Apr 8 22:56 openssl-fips-1.0.1e-20.i686.rpm -rwxr-xr-x 1 root root 3462 Apr 9 17:49 setup.sh $ less setup.sh
In the file setup.sh look for the RPM install (“rpm -Uvh”) or update (“rpm -ivh”) calls being done explicitly with the verbose and hash option. This seems to be a bad choice, because the WebUI seems not to be able to handle the output produced by the RPM commands. Certainly not pretty to have that in the output of the update process, where it could be cause for some confusion. But in the wake of the OpenSSL heartbleed bug, which MH01423 addresses, a quick and dirty solution probably seemed more appropriate.
More importantly though, going through the HMC WebUI and checking for any differences and functional breakage, i stumpled upon a long missed feature that IBM apparently had already sneaked in with the previous HMC version 7.7.8.0: Sync current configuration Capability. And without me even noticing! Finally, no more bad surprises after a LPAR shutdown and re-activation!
2014-01-20 // HMC Update to 7.7.8.0
Like before with HMC Update to 7.7.5.0 and HMC Update to 7.7.7.0 SP1, the recent HMC update to v7.7.8.0 was again not installable directly from the ISO images via the HMC GUI. Along with the HMC network installation images which are now mentioned in the release notes, there is now also an official documentation of the update procedure using the HMC network installation images. It's called “HMC network installation” and provides a more remote admin friendly way of performing the update. Since it's only a slightly shortened version of the procedure i already tested and used in HMC Update to 7.7.7.0 SP1, i decided to stick with my own procedure.
Also a turn for the better is, that now the release notes as well as FixCentral clearly point out the dependencies between the fixpacks that are supposed to go on top of the update release and the order they are supposed to be applied in. In case of MH01377 (aka V7R7.8.0.0) these are MH01388 (aka "Required fix for HMC V7R7.8.0 (11-25-2013)") and MH01396 (aka "Fix for HMC V7R7.8.0 (12-10-2013)").
Compared to earlier updates, the restriction to have to shut down or disconnect the second HMC in a dual HMC setup has been weakened to:
When two HMCs manage the same server, both HMCs must be at the same version. Once the server is connected to the higher version of the management console, the partition configuration is migrated to the latest version. Lower management consoles will not be able to understand the data properly. […]
To me this reads “ensure that you don't do any configuration from the second HMC while the first HMC has already been updated”, which still is some kind of restriction, but a far less intrusive and thus a much more manageable one compared to before.
As always, be sure to study the release notes thoroughly before an update attempt. Depending on your environment and HMC hardware there might be a road block in there. Special attention deserves the document “HMC Version 7 Release 7.8.0 Upgrade sizing”, mentioned in the release notes of MH01377.
For me, the good news gathered from the release notes was:
Power Enterprise Pools to share CoD resources among the machines in your environment without the need of a special RPQ (
chcodpool,lscodpoolandmkcodpool). I still dare to dream that some day IBM will allow a fully dynamic sharing, even of non-CoD resources for us mere mortals.LPM evacuation to support the migration of all LPARs off a system within a single command, much like the VMware maintenance mode (
migrlpar […] --all).Regeneration and XML export of the hypervisor runtime configuration (
mkprofdata).Refresh of OS level information on the HMC in case the OS within the LPAR was updated (
lssyscfg -r lpar -m <managed system> --osrefresh).
As already mentioned above, i used the update procedure described earlier in HMC Update to 7.7.7.0 SP1. In my setup this worked well for all HMCs and showed no fatal or noticeable errors. It seems others might not have been so lucky. I did the update on 2014/01/17 and checked back to FixCentral on 2014/01/19. By then the following warning messages have been put up:
So it's probably best to hold off the update for just now, if it hasn't been already done. For reference purposes, here are some example screen shots from a KVM session to the HMC during a – eventually successful – update to MH01377:
After the upgrade to V7R7.8.0.0 is complete, you can apply the additional efixes in the usual way via the HMC GUI. For me the additional efixes showed the following output during the update process:
MH01388:
Management console corrective service installation in progress. Please wait... Corrective service file offload from remote server in progress... The corrective service file offload was successful. Continuing with HMC service installation... Verifying Certificate Information Authenticating Install Packages Installing Packages --- Installing ptf-req .... --- Installing RSCT .... src-3.1.4.9-13275 rsct.core.utils-3.1.4.9-13275 rsct.core-3.1.4.9-13275 rsct.service-3.5.0.0-1 rsct.basic-3.1.4.9-13275 --- Installing CSM .... csm.core-1.7.1.20-1 csm.deploy-1.7.1.20-1 csm_hmc.server-1.7.1.20-1 csm_hmc.hdwr_svr-7.0-3.4.0 csm_hmc.client-1.7.1.20-1 csm.server.hsc-1.7.1.20-1 --- Installing LPARCMD .... hsc.lparcmd-3.3.0.1-1 ln: creating symbolic link `/usr/hmcrbin/lsnodeid': File exists ln: creating symbolic link `/usr/hmcrbin/lsrsrc-api': File exists ln: creating symbolic link `/usr/hmcrbin/mkrsrc-api': File exists ln: creating symbolic link `/usr/hmcrbin/rmrsrc-api': File exists --- Installing InventoryScout .... --- Installing Pegasus .... --- Installing service documentation .... cp: cannot stat `.dev': No such file or directory PreInstalling HMC REST Web Services ... Installing HMC REST Web Services ... pmc.core-7.7.8.0-20131027T1102 pmc.soliddb-7.7.8.0-20131027T1102 pmc.wlp-7.7.8.0-20131027T1102 pmc.wlp.soliddriver-7.7.8.0-20131027T1102 pmc.wlp.log4j-7.7.8.0-20131027T1110 pmc.wlp.guava-7.7.8.0-20131027T1110 pmc.wlp.jaxb2.runtime-7.7.8.0-20131027T1111 pmc.wlp.slf4j.api-7.7.8.0-20131027T1103 pmc.wlp.quartz-7.7.8.0-20131027T1111 pmc.wlp.commons-7.7.8.0-20131027T1103 pmc.war.rest-7.7.8.0-20131027T1103 pmc.soliddb.rest.sql-7.7.8.0-20131027T1110 pmc.soliddb.pcm.sql-7.7.8.0-20131027T1111 pmc.pcm.rest-7.7.8.0-20131027T1110 pmc.ui.developer-7.7.8.0-20131027T1112 Corrective service installation was successful.
MH01396:
Management console corrective service installation in progress. Please wait... Corrective service file offload from remote server in progress... The corrective service file offload was successful. Continuing with HMC service installation... Verifying Certificate Information Authenticating Install Packages Installing Packages --- Installing ptf-req .... PreInstalling HMC REST Web Services ... Installing HMC REST Web Services ... pmc.core-7.7.8.0-20131124T1024 pmc.soliddb-7.7.8.0-20131124T1024 pmc.wlp-7.7.8.0-20131124T1024 pmc.wlp.soliddriver-7.7.8.0-20131124T1024 pmc.wlp.log4j-7.7.8.0-20131124T1038 pmc.wlp.guava-7.7.8.0-20131124T1038 pmc.wlp.jaxb2.runtime-7.7.8.0-20131124T1038 pmc.wlp.slf4j.api-7.7.8.0-20131124T1025 pmc.wlp.quartz-7.7.8.0-20131124T1039 pmc.wlp.commons-7.7.8.0-20131124T1025 pmc.war.rest-7.7.8.0-20131124T1025 pmc.soliddb.rest.sql-7.7.8.0-20131124T1038 pmc.soliddb.pcm.sql-7.7.8.0-20131124T1039 pmc.pcm.rest-7.7.8.0-20131124T1038 pmc.ui.developer-7.7.8.0-20131124T1039 Corrective service installation was successful.
The MH01388 update still shows error messages with regard to symlink creation appearing during the update process, so i guess my DCR MR0809134336 went straight into the circular file. The newly introduced error message “cp: cannot stat `.dev': No such file or directory” probably also originates from the shell script /images/installImages inside the MH01388 installation ISO image:
- /images/installImages
380 cp -p finishUpdate /console/HSC/ 381 cp -p postinstall /console/HSC/ 382 cp -p .VERSION /console/HSC/.VERSION 383 cp -p .dev /console/HSC/.dev 384 385 cp -p -r baseHMC /console/HSC/
which is trying to copy the .dev file which doesn't exist on the MH01388 ISO image:
$ mount -o loop MH01388.iso /mnt $ ls -al /mnt/images/ total 121 dr-xr-xr-x 8 root root 4096 Nov 25 17:32 . dr-xr-xr-x 3 root root 2048 Nov 22 23:50 .. dr-xr-xr-x 2 root root 10240 Nov 25 17:32 baseHMC -r-xr-xr-x 1 root root 40644 Nov 5 22:45 finishUpdate -r--r--r-- 1 root root 2035 Nov 25 17:30 IBMhmc.MH01388_d1-7.0-7.8.0.i386.rpm -r--r--r-- 1 root root 9 Oct 29 01:45 .image.updates dr-xr-xr-x 2 root root 2048 Nov 22 23:47 info -r-xr-xr-x 1 root root 15397 Nov 6 18:56 installImages -r-xr-xr-x 1 root root 2644 Oct 29 22:05 installK2Payloads.sh -r--r--r-- 1 root root 28 Oct 29 01:45 inventory dr-xr-xr-x 2 root root 4096 Nov 25 17:30 K2 dr-xr-xr-x 2 root root 2048 Nov 25 17:32 pegasus -r-xr-xr-x 1 root root 10805 Apr 16 2013 postinstall -r-xr-xr-x 1 root root 1520 Nov 7 17:30 preInstallK2Payloads.sh dr-xr-xr-x 2 root root 4096 Nov 25 17:32 rmc dr-xr-xr-x 2 root root 2048 Nov 25 17:32 service -r--r--r-- 1 root root 17 Oct 29 01:45 .signature -r--r--r-- 2 root root 37 Nov 25 17:32 .VERSION
So either someone forgot to remove the not necessary copy command from the install script, or the file .dev supposed to be existing was forgotten during creation of the ISO image. Either way lets hope the file .dev doesn't contain any vital information. Aside from that, up to now no issues with the new HMC version.
2013-11-11 // Nagios Monitoring - IBM Power Systems and HMC
We use several IBM Power Systems servers in our datacenters to run mission critical application and database services. Some time ago i wrote a – rather crude – Nagios plugin to monitor the IBM Power Systems hardware via the Hardware Management Console (HMC) appliance, as well as the HMC itself. In order to run the Nagios plugin, you need to have “Remote Command Execution” via SSH activated on the HMC appliances. Also, a network connection from the Nagios system to the HMC appliances on port TCP/22 must be allowed.
The whole setup for monitoring IBM Power Systems servers and HMC appliances looks like this:
Enable remote command execution on the HMC. Login to the HMC WebGUI and navigate to:
-> HMC Management -> Administration -> Remote Command Execution -> Check selection: "Enable remote command execution using the ssh facility" -> OKVerify the port TCP/22 on the HMC appliance can be reached from the Nagios system.
Create a SSH private/public key pair on the Nagios system to authenticate the monitoring user against the HMC appliance:
$ mkdir -p /etc/nagios3/.ssh/ $ chown -R nagios:nagios /etc/nagios3/.ssh/ $ chmod 700 /etc/nagios3/.ssh/ $ cd /etc/nagios3/.ssh/ $ ssh-keygen -t rsa -b 2048 -f id_rsa_hmc Generating public/private rsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in id_rsa_hmc. Your public key has been saved in id_rsa_hmc.pub. The key fingerprint is: xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx user@host $ chown -R nagios:nagios /etc/nagios3/.ssh/id_rsa_hmc*
Prepare a SSH authorized keys file and transfer it to the HMC.
Optional Create a separate user on the HMC for monitoring purposes. The new user needs to have permission to run the HMC commands
lsled,lssvcevents,lssyscfgandmonhmc. If a separate user should be used, replacehscrootin the following text with the name of the new user.Optional If there is already a SSH authorized keys file set up on the HMC, be sure to retrieve it first so it doesn't get overwritten:
$ cd /tmp $ scp -r hscroot@hmc:~/.ssh /tmp/ Password: authorized_keys2 100% 600 0.6KB/s 00:00
Insert the SSH public key created above into a SSH authorized keys file and push it back to the HMC:
$ echo -n 'from="<IP of your Nagios system>",no-agent-forwarding,no-port-forwarding,no-X11-forwarding ' \ >> /tmp/.ssh/authorized_keys2 $ cat /etc/nagios3/.ssh/id_rsa_hmc.pub >> /tmp/.ssh/authorized_keys2 $ scp -r /tmp/.ssh hscroot@hmc:~/ Password: authorized_keys2 100% 1079 1.1KB/s 00:00 $ rm -r /tmp/.ssh
Add the SSH host public key from the HMC appliance:
HMC$ cat /etc/ssh/ssh_host_rsa_key.pub ssh-rsa AAAAB3N...[cut]...jxFAz+4O1X root@hmc
to the SSH
ssh_known_hostsfile of the Nagios system:$ vi /etc/ssh/ssh_known_hosts hmc,FQDN,<IP of your HMC> ssh-rsa AAAAB3N...[cut]...jxFAz+4O1X
Verify the login on the HMC appliance is possible from the Nagios system with the SSH public key created above:
$ ssh -i /etc/nagios3/.ssh/id_rsa_hmc hscroot@hmc 'lshmc -V' "version= Version: 7 Release: 7.7.0 Service Pack: 2 HMC Build level 20130730.1 MH01373: Fix for HMC V7R7.7.0 SP2 (08-06-2013) ","base_version=V7R7.7.0"
Download the Nagios plugin check_hmc.sh and place it in the plugins directory of your Nagios system, in this example
/usr/lib/nagios/plugins/:$ mv -i check_hmc.sh /usr/lib/nagios/plugins/ $ chmod 755 /usr/lib/nagios/plugins/check_hmc.sh
Define the following Nagios commands. In this example this is done in the file
/etc/nagios-plugins/config/check_hmc.cfg:# check HMC disk define command { command_name check_hmc_disk command_line $USER1$/check_hmc.sh -H $HOSTADDRESS$ -C disk -w $ARG1$ -c $ARG2$ -a $ARG3$ } # check HMC processor define command { command_name check_hmc_proc command_line $USER1$/check_hmc.sh -H $HOSTADDRESS$ -C proc -w $ARG1$ -c $ARG2$ } # check HMC memory define command { command_name check_hmc_mem command_line $USER1$/check_hmc.sh -H $HOSTADDRESS$ -C mem -w $ARG1$ -c $ARG2$ } # check HMC swap define command { command_name check_hmc_swap command_line $USER1$/check_hmc.sh -H $HOSTADDRESS$ -C swap -w $ARG1$ -c $ARG2$ } # check HMC rmc processes define command { command_name check_hmc_rmc command_line $USER1$/check_hmc.sh -H $HOSTADDRESS$ -C rmc -w $ARG1$ -c $ARG2$ -a $ARG3$ } # check HMC System Attention LED for physical system define command { command_name check_hmc_ledphys command_line $USER1$/check_hmc.sh -H $HOSTADDRESS$ -C ledphys } # check HMC System Attention LED for virtual partition define command { command_name check_hmc_ledvlpar command_line $USER1$/check_hmc.sh -H $HOSTADDRESS$ -C ledvlpar } # check HMC System Attention LED for virtual system define command { command_name check_hmc_ledvsys command_line $USER1$/check_hmc.sh -H $HOSTADDRESS$ -C ledvsys } # check HMC serviceable events define command { command_name check_hmc_events command_line $USER1$/check_hmc.sh -H $HOSTADDRESS$ -C events }Define a group of services in your Nagios configuration to be checked for each HMC appliance:
# check sshd define service { use generic-service hostgroup_name hmc service_description Check_SSH check_command check_ssh } # check_hmc_mem define service { use generic-service-pnp servicegroups sshchecks hostgroup_name hmc service_description Check_memory check_command check_hmc_mem!20!10 } # check_hmc_proc define service { use generic-service-pnp servicegroups sshchecks hostgroup_name hmc service_description Check_CPU_statistics check_command check_hmc_proc!10!5 } # check_hmc_disk define service { use generic-service-pnp servicegroups sshchecks hostgroup_name hmc service_description Check_space_on_/ check_command check_hmc_disk!10%!5%!'/$' } define service { use generic-service-pnp servicegroups sshchecks hostgroup_name hmc service_description Check_space_on_/dump check_command check_hmc_disk!10%!5%!/dump } define service { use generic-service-pnp servicegroups sshchecks hostgroup_name hmc service_description Check_space_on_/extra check_command check_hmc_disk!10%!5%!/extra } define service { use generic-service-pnp servicegroups sshchecks hostgroup_name hmc service_description Check_space_on_/var check_command check_hmc_disk!10%!5%!/var } # check_hmc_swap define service { use generic-service-pnp servicegroups sshchecks hostgroup_name hmc service_description Check_space_on_swap check_command check_hmc_swap!200!100 } # check_hmc_rmc: IBM.CSMAgentRMd define service { use generic-service servicegroups sshchecks hostgroup_name hmc service_description Check_process_IBM.CSMAgentRMd check_command check_hmc_rmc!1!1!IBM.CSMAgentRMd } # check_hmc_rmc: IBM.LparCmdRMd define service { use generic-service servicegroups sshchecks hostgroup_name hmc service_description Check_process_IBM.LparCmdRMd check_command check_hmc_rmc!1!1!IBM.LparCmdRMd } # check_hmc_rmc: IBM.ServiceRMd define service { use generic-service servicegroups sshchecks hostgroup_name hmc service_description Check_process_IBM.ServiceRMd check_command check_hmc_rmc!1!1!IBM.ServiceRMd } # check_hmc_ledphys define service { use generic-service servicegroups sshchecks hostgroup_name hmc service_description Check_SA_LED_physical check_command check_hmc_ledphys } # check_hmc_ledvlpar define service { use generic-service servicegroups sshchecks hostgroup_name hmc service_description Check_SA_LED_vpartition check_command check_hmc_ledvlpar } # check_hmc_ledvsys define service { use generic-service servicegroups sshchecks hostgroup_name hmc service_description Check_SA_LED_vsystem check_command check_hmc_ledvsys } # check_hmc_events define service { use generic-service servicegroups sshchecks hostgroup_name hmc service_description Check_serviceable_events check_command check_hmc_events }Replace
generic-servicewith your Nagios service template. Replacegeneric-service-pnpwith your Nagios service template that has performance data processing enabled.Define a service dependency to run the above checks only if the
Check_SSHwas run successfully:# HMC SSH dependencies define servicedependency { hostgroup_name hmc service_description Check_SSH dependent_service_description Check_process_.* execution_failure_criteria c,p,u,w notification_failure_criteria c,p,u,w } define servicedependency { hostgroup_name hmc service_description Check_SSH dependent_service_description Check_space_on_.* execution_failure_criteria c,p,u,w notification_failure_criteria c,p,u,w } define servicedependency { hostgroup_name hmc service_description Check_SSH dependent_service_description Check_memory execution_failure_criteria c,p,u,w notification_failure_criteria c,p,u,w } define servicedependency { hostgroup_name hmc service_description Check_SSH dependent_service_description Check_CPU_statistics execution_failure_criteria c,p,u,w notification_failure_criteria c,p,u,w } define servicedependency { hostgroup_name hmc service_description Check_SSH dependent_service_description Check_SA_LED_.* execution_failure_criteria c,p,u,w notification_failure_criteria c,p,u,w } define servicedependency { hostgroup_name hmc service_description Check_SSH dependent_service_description Check_serviceable_events execution_failure_criteria c,p,u,w notification_failure_criteria c,p,u,w }Define a host in your Nagios configuration for each HMC appliance. In this example its named
hmc1:define host { use hmc host_name hmc1 alias HMC 1 address 10.0.0.1 parents parent_lan }Replace
hmcwith your Nagios host template for HMC appliances. Adjust theaddressandparentsparameters according to your environment.Define a hostgroup in your Nagios configuration for all HMC appliances. In this example it is named
hmc. The above checks are run against each member of the hostgroup:define hostgroup{ hostgroup_name hmc alias Hardware Management Console members hmc1 }Run a configuration check and if successful reload the Nagios process:
$ /usr/sbin/nagios3 -v /etc/nagios3/nagios.cfg $ /etc/init.d/nagios3 reload
The new hosts and services should soon show up in the Nagios web interface.
Optional: If you're running PNP4Nagios v0.6 or later to graph Nagios performance data, you can use the
check_hmc_disk.php,check_hmc_mem.php,check_hmc_proc.phpandcheck_hmc_swap.phpPNP4Nagios template to beautify the graphs. Download the PNP4Nagios templates check_hmc_disk.php, check_hmc_mem.php, check_hmc_proc.php and check_hmc_swap.php and place them in the PNP4Nagios template directory, in this example/usr/share/pnp4nagios/html/templates/:$ mv -i check_hmc_disk.php check_hmc_mem.php check_hmc_proc.php check_hmc_swap.php \ /usr/share/pnp4nagios/html/templates/ $ chmod 644 /usr/share/pnp4nagios/html/templates/check_hmc_disk.php $ chmod 644 /usr/share/pnp4nagios/html/templates/check_hmc_mem.php $ chmod 644 /usr/share/pnp4nagios/html/templates/check_hmc_proc.php $ chmod 644 /usr/share/pnp4nagios/html/templates/check_hmc_swap.php
The following image shows an example of what the PNP4Nagios graphs look like for a IBM HMC appliance:
All done, you should now have a complete Nagios-based monitoring solution for your IBM HMC appliances and IBM Power Systems servers.
2013-09-03 // HMC Update to 7.7.7.0 SP2
Updating the HMC from v7.7.7.0 SP1 to v7.7.7.0 SP2 was once again and like the previous HMC Update to 7.7.6.0 SP1 very painless. The service pack MH01354 was easily installable from the ISO images via the HMC GUI. The previous restriction of having to disconnect one of the HMCs in a redundant setup seems to have been dropped, at least it wasn't mentioned in the release notes anymore. The readme of the eFixes (MH01367 and MH01373) to be installed on top of the service pack was this time very clear on MH01373 superseding MH01367.
This package includes a fix for HMC Version 7 Release 7.7.0 Service Pack 2. You can reference this package by APAR# MB03714. This fix must be installed on top of HMC Version 7 Release 7.7.0 Service Pack 2 with or without PTF MH01367 installed. PTF MH01373 supersedes PTF MH01367.
The service pack and the additional efixes showed the following output during the update process:
MH01354:
Mangement console corrective service installation in progress. Please wait... Corrective service file offload from remote server in progress... The corrective service file offload was successful. Continuing with HMC service installation... Verifying Certificate Information Authenticating Install Packages Installing Packages --- Installing ptf-req .... --- Installing RSCT .... src-3.1.4.4-13032 rsct.core.utils-3.1.4.4-13032 rsct.core-3.1.4.4-13032 rsct.service-3.5.0.0-1 rsct.basic-3.1.4.4-13032 --- Installing CSM .... csm.core-1.7.1.20-1 csm.deploy-1.7.1.20-1 csm_hmc.server-1.7.1.20-1 csm_hmc.hdwr_svr-7.0-3.4.0 csm_hmc.client-1.7.1.20-1 csm.server.hsc-1.7.1.20-1 --- Installing LPARCMD .... hsc.lparcmd-3.0.0.1-1 ln: creating symbolic link `/usr/hmcrbin/lsnodeid' : File exists ln: creating symbolic link `/usr/hmcrbin/lsrsrc-api' : File exists ln: creating symbolic link `/usr/hmcrbin/mkrsrc-api' : File exists ln: creating symbolic link `/usr/hmcrbin/rmrsrc-api' : File exists --- Installing InventoryScout .... --- Installing Pegasus .... --- Installing service documentation .... --- Updating baseOS .... Corrective service installation was successful.
MH01367: Skipped, because - as mentioned in the release notes of MH01373 - it is superseded by MH01373.
MH01373:
Mangement console corrective service installation in progress. Please wait... Corrective service file offload from remote server in progress... The corrective service file offload was successful. Continuing with HMC service installation... Verifying Certificate Information Authenticating Install Packages Installing Packages --- Installing ptf-req .... Corrective service installation was successful.
The MH01354 updates still shows error messages with regard to symlink creation appearing during the update process. This – admittedly minor – issue and the associated PMR i opened with IBM support are virtually racing towards my all time top ten of downright ridiculous experiences. After a lengthy back and forth with L2 support/development things concluded in their position:
Most of our customers want to see the verbose output during upgrades/updates. There is no defect here.
If the customer wishes to do so he can open a DCR, but there is no defect.
Well, verbose output would indeed be nice, i suppose. An option to choose whether to show or to hide said verbose output would be even nicer. What i really don't care for are constant visual reminders of your lack of knowledge in proper shell scripting, not to mention your awful coding style! So here we are again, a simple, non-critical issue which could be fixed by a trivial change in a matter of minutes and still were spending a disproportionate amount of time arguing about whether it's really an issue or not. But alright, lets waste the time of even more people and open a DCR (MR0809134336) on this issue. And while we're at it, have another one (MR0618131954) for botching the mksysplan command in case a failover SEA is used on the VIO server.
Aside from that, up to now no issues with the new HMC version.
