====== Recover from a broken VIO Server Update ======

If you're -- like me -- running Ganglia to monitor the performance metrics of your AIX and VIOS systems, it can make sense to disable the, now kind of redundant, default collection method for performance metrics in order not to waste recources and to prevent unnecessary cluttering of the local VIOS filesystem. This can be done via the "''xmdaily''" entry in ''/etc/inittab'':

<code bash /etc/inittab>
:xmdaily:2:once:/usr/bin/topasrec -L -s 300 -R 1 -r 6 -o /home/ios/perf/topas/ -ypersistent=1 2>&1 >/dev/null
</code>

<cli>
$ lsitab xmdaily; echo $?
1
</cli>

Unfortunately this breaks the VIOS upgrade process, since the good folks at IBM insist on the "''xmdaily''" inittab entry being vital for a properly functioning VIOS system. Specifically, this is caused by the ''/usr/lpp/ios.cli/ios.cli.rte/<VERSION>/inst_root/ios.cli.rte.post_u'' script in the ''ios.cli.rte'' package. E.g.:

<code bash /usr/lpp/ios.cli/ios.cli.rte/6.1.9.1/inst_root/ios.cli.rte.post_u>
507 rt=`/usr/sbin/lsitab xmdaily`
508 if [[ $rt != "" ]]
[...]
528 else
529    echo "Warning: lsitab failed..."
530    exit 1
531 fi
</code>

I beg to differ on the whole subject of "''xmdaily''" being necessary at all, but one could file this under the category of "philosophical differences". To fail the whole package update with the "''exit 1''" in line 530 of the above code sample seems to be a bit too harsh though.

So normally i would just put the "''xmdaily''" entry back into the inittab right before an VIOS update. Unfortunately on the update to 2.2.3.1-FP27 and subsequently to 2.2.3.2-FP27-SP02 i forgot to do that on a few VIOS systems. The result of this negligence were failure messages during the update process (RBAC output omitted for better readability!):

<code>
[...]
installp:  APPLYING software for:
        ios.cli.rte 6.1.9.2

. . . . . << Copyright notice for ios.cli >> . . . . . . .
 Licensed Materials - Property of IBM

 5765G3400
   Copyright International Business Machines Corp. 2004, 2014.

 All rights reserved.
 US Government Users Restricted Rights - Use, duplication or disclosure
 restricted by GSA ADP Schedule Contract with IBM Corp.
. . . . . << End of copyright notice for ios.cli >>. . . .

sysck: 3001-022 The file
        /usr/ios/utils/part
        was not found.

Start Creating VIOS Authorizations
Completed Creating VIOS Authorizations

Warning: lsitab failed...
update:  Failed while executing the ios.cli.rte.post_u script.

0503-464 installp:  The installation has FAILED for the "root" part
        ios.cli.rte 6.1.9.2

installp:  Cleaning up software for:
        ios.cli.rte 6.1.9.2

[...]
</code>

as well as at the end of the update process:

<code>
[...]
Installation Summary
--------------------
Name                        Level           Part        Event       Result
-------------------------------------------------------------------------------
ios.cli.rte                 6.1.9.2         USR         APPLY       SUCCESS
ios.cli.rte                 6.1.9.2         ROOT        APPLY       FAILED
ios.cli.rte                 6.1.9.2         ROOT        CLEANUP     SUCCESS
devices.vtdev.scsi.rte      6.1.9.2         USR         APPLY       SUCCESS
devices.vtdev.scsi.rte      6.1.9.2         ROOT        APPLY       SUCCESS
[...]
</code>

This would manifest through the fact that no IOS command would work properly anymore:

<cli>
IBM Virtual I/O Server

login: padmin
padmin's Password:
Last login: Sun Apr 27 11:40:41 DFT 2014 on /dev/vty0

Access to run command is not valid.

[vios1-p550-b1] /home/padmin $ license -accept
Access to run command is not valid.

[vios2-p550-b1] /home/padmin $ lsmap -all
Access to run command is not valid.

[vios2-p550-b1] /home/padmin $ 
</cli>

A quick check with a login to the VIOS as ''root'' via SSH confirmed that the "root" part of the package ''ios.cli.rte'' had been rolled back entirely:

<cli>
root@vios2-p550-b1:/$ lppchk -v
lppchk:  The following filesets need to be installed or corrected to bring
         the system to a consistent state:

  ios.cli.rte 6.1.9.2                     (usr: APPLIED, root: not installed)
</cli>

To fix this issue, it worked for me to just run the "''ios.cli.rte.pre_u''" script of the ''ios.cli.rte'' package manually to redefine the now rolled back RBAC authorizations:

<cli>
root@vios2-p550-b1:/$ SAVEDIR=/tmp/ /usr/lpp/ios.cli/ios.cli.rte/6.1.9.2/inst_root/ios.cli.rte.pre_u
root@vios2-p550-b1:/$ rm /tmp/org_*
</cli>


For future reference, make sure you have a valid "''xmdaily''" entry in ''/etc/inittab'' before attempting to update any VIOS system:

<cli>
$ lsitab xmdaily; echo $?
xmdaily:2:once:/usr/bin/topasrec -L -s 300 -R 1 -r 6 -o /home/ios/perf/topas/ -ypersistent=1 2>&1 >/dev/null
0
</cli>