bityard Blog

// Recover from a broken VIO Server Update

If you're – like me – running Ganglia to monitor the performance metrics of your AIX and VIOS systems, it can make sense to disable the, now kind of redundant, default collection method for performance metrics in order not to waste recources and to prevent unnecessary cluttering of the local VIOS filesystem. This can be done via the “xmdaily” entry in /etc/inittab:

:xmdaily:2:once:/usr/bin/topasrec -L -s 300 -R 1 -r 6 -o /home/ios/perf/topas/ -ypersistent=1 2>&1 >/dev/null
$ lsitab xmdaily; echo $?

Unfortunately this breaks the VIOS upgrade process, since the good folks at IBM insist on the “xmdaily” inittab entry being vital for a properly functioning VIOS system. Specifically, this is caused by the /usr/lpp/ios.cli/ios.cli.rte/<VERSION>/inst_root/ios.cli.rte.post_u script in the ios.cli.rte package. E.g.:

507 rt=`/usr/sbin/lsitab xmdaily`
508 if [[ $rt != "" ]]
528 else
529    echo "Warning: lsitab failed..."
530    exit 1
531 fi

I beg to differ on the whole subject of “xmdaily” being necessary at all, but one could file this under the category of “philosophical differences”. To fail the whole package update with the “exit 1” in line 530 of the above code sample seems to be a bit too harsh though.

So normally i would just put the “xmdaily” entry back into the inittab right before an VIOS update. Unfortunately on the update to and subsequently to i forgot to do that on a few VIOS systems. The result of this negligence were failure messages during the update process (RBAC output omitted for better readability!):

installp:  APPLYING software for:

. . . . . << Copyright notice for ios.cli >> . . . . . . .
 Licensed Materials - Property of IBM

   Copyright International Business Machines Corp. 2004, 2014.

 All rights reserved.
 US Government Users Restricted Rights - Use, duplication or disclosure
 restricted by GSA ADP Schedule Contract with IBM Corp.
. . . . . << End of copyright notice for ios.cli >>. . . .

sysck: 3001-022 The file
        was not found.

Start Creating VIOS Authorizations
Completed Creating VIOS Authorizations

Warning: lsitab failed...
update:  Failed while executing the ios.cli.rte.post_u script.

0503-464 installp:  The installation has FAILED for the "root" part

installp:  Cleaning up software for:


as well as at the end of the update process:

Installation Summary
Name                        Level           Part        Event       Result
ios.cli.rte                USR         APPLY       SUCCESS
ios.cli.rte                ROOT        APPLY       FAILED
ios.cli.rte                ROOT        CLEANUP     SUCCESS
devices.vtdev.scsi.rte         USR         APPLY       SUCCESS
devices.vtdev.scsi.rte         ROOT        APPLY       SUCCESS

This would manifest through the fact that no IOS command would work properly anymore:

IBM Virtual I/O Server

login: padmin
padmin's Password:
Last login: Sun Apr 27 11:40:41 DFT 2014 on /dev/vty0

Access to run command is not valid.

[vios1-p550-b1] /home/padmin $ license -accept
Access to run command is not valid.

[vios2-p550-b1] /home/padmin $ lsmap -all
Access to run command is not valid.

[vios2-p550-b1] /home/padmin $ 

A quick check with a login to the VIOS as root via SSH confirmed that the “root” part of the package ios.cli.rte had been rolled back entirely:

root@vios2-p550-b1:/$ lppchk -v
lppchk:  The following filesets need to be installed or corrected to bring
         the system to a consistent state:

  ios.cli.rte                     (usr: APPLIED, root: not installed)

To fix this issue, it worked for me to just run the “ios.cli.rte.pre_u” script of the ios.cli.rte package manually to redefine the now rolled back RBAC authorizations:

root@vios2-p550-b1:/$ SAVEDIR=/tmp/ /usr/lpp/ios.cli/ios.cli.rte/
root@vios2-p550-b1:/$ rm /tmp/org_*

For future reference, make sure you have a valid “xmdaily” entry in /etc/inittab before attempting to update any VIOS system:

$ lsitab xmdaily; echo $?
xmdaily:2:once:/usr/bin/topasrec -L -s 300 -R 1 -r 6 -o /home/ios/perf/topas/ -ypersistent=1 2>&1 >/dev/null

Leave a comment…

X F S​ G H
  • E-Mail address will not be published.
  • Formatting:
    //italic//  __underlined__
    **bold**  ''preformatted''
  • Links:
    [[|Link Text]]
  • Quotation:
    > This is a quote. Don't forget the space in front of the text: "> "
  • Code:
    <code>This is unspecific source code</code>
    <code [lang]>This is specifc [lang] code</code>
    <code php><?php echo 'example'; ?></code>
    Available: html, css, javascript, bash, cpp, …
  • Lists:
    Indent your text by two spaces and use a * for
    each unordered list item or a - for ordered ones.
This website uses cookies for visitor traffic analysis. By using the website, you agree with storing the cookies on your computer.More information