2018-02-10 // Brocade Fabric OS Authentication Failure with SSH Public Key
With the update to Fabric OS v7.4.1d on Brocade fibre channel SAN switches, the CLI login via SSH public key authentication will sometimes be broken for administrative users. This blog post describes a manual workaround which can be used in order to temporarily correct this issue without the immediate need for another Fabric OS update.
During the preparation phase for migrating from our aging Brocade 5100 and 5300 Gen4 fibre channel SAN switches to the shiny new Brocade G620 Gen6 fibre channel SAN Switches, we needed to update the Fabric OS on the old switches to a v7.4.x version. Due to compatibility and support constraints with the IBM SAN Volume Controller (SVC), we decided to go with the Fabric OS v7.4.1d version.
After the successful Fabric OS update, the CLI login via SSH public key authentication was broken for some, but not all users with admin level priviledges on some but not all switches. A re-upload of the SSH public key for those users with the sshUtil importpubkey command didn't solve the issue. Debugging this further with a strace attached to the SSH daemon process on an affected switch revealed why the SSH public key authentication was failing:
[...]
[pid 27941] connect(8, {sa_family=AF_FILE, path="/dev/log"}, 16) = -1 EPROTOTYPE (Protocol wrong type for socket)
[pid 27941] close(8) = 0
[pid 27941] socket(PF_FILE, SOCK_STREAM, 0) = 8
[pid 27941] fcntl64(8, F_SETFD, FD_CLOEXEC) = 0
[pid 27941] connect(8, {sa_family=AF_FILE, path="/dev/log"}, 16) = 0
[pid 27941] send(8, "<39>Feb 1 22:07:00 sshd[27941]: debug1: trying public key file /fabos/users/admin/.ssh/authorized_keys.<USERNAME>\0", 117, MSG_NOSIGNAL) = 117
[pid 27941] close(8) = 0
[pid 27941] open("/fabos/users/admin/.ssh/authorized_keys.<USERNAME_2>", O_RDONLY|O_NONBLOCK|O_LARGEFILE) = -1 EACCES (Permission denied)
[...]
This was done by using the root account on the Brocade switch. The same account was also used for the following research and the temporary workaround derived from this. Beware that using the root account on Brocade switches might have serious implications on the warranty or support for the devices. Be extra careful what you are doing as root on the Brocade switch, since it might easily affect the operational status of the device.
The last line from the above snippet of the strace output show, that the cause of the issue with SSH public key authentication was in the permissions of the users authorized_keys file. Looking at the permissions of the directory containing the file and the file itself showed:
switch:FID128:root> cd /fabos/users/admin/ switch:FID128:root> ls -al total 28 drwxr-xr-x 4 root admin 4096 Jan 23 12:36 ./ drwxr-xr-x 12 root sys 4096 Jul 15 2016 ../ -rw-r--r-- 1 root admin 507 Jul 15 2016 .bash_logout -rw-r--r-- 1 root admin 27 Jul 15 2016 .inputrc -rw-r--r-- 1 root admin 1275 Jul 15 2016 .profile drwxr-xr-x 2 root admin 4096 Feb 1 22:54 .ssh/ drwxrwxrwx 3 root sys 4096 Aug 11 2011 .terminfo/ switch:FID128:root> cd .ssh switch:FID128:root> pwd /fabos/users/admin/.ssh switch:FID128:root> ls -al total 44 drwxr-xr-x 2 root admin 4096 Feb 1 22:54 ./ drwxr-xr-x 4 root admin 4096 Jan 23 12:36 ../ -rw-r--r-- 1 root admin 10240 Feb 1 22:54 authorizedKeys.tar -rw------- 1 root root 408 Jan 23 12:43 authorized_keys -rw-r--r-- 1 root admin 755 Dec 8 11:13 authorized_keys.<USERNAME_1> -rw------- 1 root admin 1230 Feb 1 22:54 authorized_keys.<USERNAME_2> -rw------- 1 root root 408 Mar 22 2016 authorized_keys.<USERNAME_3> -rw-r--r-- 1 root root 605 Sep 19 11:06 authorized_keys.<USERNAME_4> -rw-r--r-- 1 root admin 134 Jul 15 2016 environment switch:FID128:root> tar tvf authorizedKeys.tar -rw-r--r-- root/admin 755 2017-12-08 11:13:06 authorized_keys.<USERNAME_1> -rw------- root/admin 1230 2018-02-01 22:54:01 authorized_keys.<USERNAME_2> -rw------- root/root 408 2016-03-22 10:12:37 authorized_keys.<USERNAME_3> -rw-r--r-- root/root 606 2017-09-19 11:06:10 authorized_keys.<USERNAME_4> switch:FID128:root> tar tvf /mnt/fabos/users/admin/.ssh/authorizedKeys.tar -rw-r--r-- root/admin 755 2017-12-08 11:13:06 authorized_keys.<USERNAME_1> -rw------- root/admin 1230 2018-02-01 22:54:01 authorized_keys.<USERNAME_2> -rw------- root/root 408 2016-03-22 10:12:37 authorized_keys.<USERNAME_3> -rw-r--r-- root/root 606 2017-09-19 11:06:10 authorized_keys.<USERNAME_4>
The permissions on some, but not all of the authorized_keys.<USERNAME_*> files were being too restrictive, since the SSH daemon was trying to read them as an effective user of the admin group. An immediate fix for this issue was to alter the permissions on the authorized_keys.<USERNAME_*> files in order to allow the admin group to read the content of the files:
switch:FID128:root> cd /fabos/users/admin/ switch:FID128:root> chmod 640 authorized_keys.* switch:FID128:root> chown root:admin authorized_keys.* switch:FID128:root> tar cpf authorizedKeys.tar authorized_keys.* switch:FID128:root> cd /mnt/fabos/users/admin/.ssh/ switch:FID128:root> chmod 640 authorized_keys.* switch:FID128:root> chown root:admin authorized_keys.* switch:FID128:root> tar cpf authorizedKeys.tar authorized_keys.*
Again looking at the permissions of the directory containing the authorized_keys.<USERNAME_*> files and the files itself showed now:
switch:FID128:root> cd /fabos/users/admin/ switch:FID128:root> ls -la total 44 drwxr-xr-x 2 root admin 4096 Feb 1 22:54 ./ drwxr-xr-x 4 root admin 4096 Jan 23 12:36 ../ -rw-r--r-- 1 root admin 10240 Feb 9 06:59 authorizedKeys.tar -rw------- 1 root root 408 Jan 23 12:43 authorized_keys -rw-r----- 1 root admin 755 Dec 8 11:13 authorized_keys.<USERNAME_1> -rw-r----- 1 root admin 1230 Feb 1 22:54 authorized_keys.<USERNAME_2> -rw-r----- 1 root admin 408 Mar 22 2016 authorized_keys.<USERNAME_3> -rw-r----- 1 root admin 605 Sep 19 11:06 authorized_keys.<USERNAME_4> -rw-r--r-- 1 root admin 134 Jul 15 2016 environment switch:FID128:root> tar tvf /fabos/users/admin/.ssh/authorizedKeys.tar -rw-r----- root/admin 755 2017-12-08 11:13:06 authorized_keys.<USERNAME_1> -rw-r----- root/admin 1230 2018-02-01 22:54:01 authorized_keys.<USERNAME_2> -rw-r----- root/admin 408 2016-03-22 10:12:37 authorized_keys.<USERNAME_3> -rw-r----- root/admin 606 2017-09-19 11:06:10 authorized_keys.<USERNAME_4> switch:FID128:root> cd /mnt/fabos/users/admin/.ssh/ switch:FID128:root> ls -la total 44 drwxr-xr-x 2 root admin 4096 Feb 1 22:54 ./ drwxr-xr-x 4 root admin 4096 Jan 23 12:50 ../ -rw-r--r-- 1 root admin 10240 Feb 1 22:54 authorizedKeys.tar -rw------- 1 root root 408 Jan 23 12:43 authorized_keys -rw-r----- 1 root admin 755 Dec 8 11:13 authorized_keys.<USERNAME_1> -rw-r----- 1 root admin 1230 Feb 1 22:54 authorized_keys.<USERNAME_2> -rw-r----- 1 root admin 408 Mar 22 2016 authorized_keys.<USERNAME_3> -rw-r----- 1 root admin 605 Sep 19 11:06 authorized_keys.<USERNAME_4> -rw-r--r-- 1 root admin 134 Jan 23 12:50 environment switch:FID128:root> tar tvf /mnt/fabos/users/admin/.ssh/authorizedKeys.tar -rw-r----- root/admin 755 2017-12-08 11:13:06 authorized_keys.<USERNAME_1> -rw-r----- root/admin 1230 2018-02-01 22:54:01 authorized_keys.<USERNAME_2> -rw-r----- root/admin 408 2016-03-22 10:12:37 authorized_keys.<USERNAME_3> -rw-r----- root/admin 606 2017-09-19 11:06:10 authorized_keys.<USERNAME_4>
With the corrected permissions on the authorized_keys.<USERNAME_*> files, CLI login via SSH public key authentication was now possible again.
Unfortunately this is only a temporary workaround, since the next upload of a SSH public key with the sshUtil importpubkey command will likely set the wrong permissions on the newly created or replaced authorized_keys.<USERNAME_*> file. This is due to the root cause of the issue actually being with the sshUtil importpubkey command. The snippet of a strace output show below was captured from a running sshUtil importpubkey command:
[...]
chdir("/fabos/users/admin/.ssh") = 0
[...]
[pid 10611] execve("/bin/cat", ["cat", "<USERNAME_2>_brocade_dsa.pub"], [/* 45 vars */]) = 0
[...]
[pid 10612] execve("/bin/chmod", ["/bin/chmod", "600", "authorized_keys.<USERNAME_2>"], [/* 45 vars */]) = 0
[...]
[pid 10612] lstat64("authorized_keys.<USERNAME_2>", {st_mode=S_IFREG|0640, st_size=1230, ...}) = 0
[pid 10612] chmod("authorized_keys.<USERNAME_2>", 0600) = 0
[...]
[pid 10613] execve("/bin/cp", ["cp", "-f", "authorized_keys.<USERNAME_2>", "/mnt/fabos/users/admin/.ssh/"], [/* 45 vars */]) = 0
[...]
[pid 10613] chmod("/mnt/fabos/users/admin/.ssh/authorized_keys.<USERNAME_2>", 0100600) = 0
[...]
[pid 10618] execve("/bin/tar", ["tar", "-cf", "authorizedKeys.tar", "authorized_keys.<USERNAME_1>", "authorized_keys.<USERNAME_2>", "authorized_keys.<USERNAME_3>"], [/* 45 vars */]) = 0
[...]
The /bin/chmod command on the third line of the above strace output shows that the file permission for the authorized_keys.<USERNAME_2> file is mistakenly set to 600 (-rw-------) instead to at least 640 (-rw-r-----). Exactly why this is sometimes happening can't be further analyzed, since the source code to the sshUtil command is not available.
A permanent resolution to this issue will be to update to at least Fabric OS v7.4.1e. The Release Notes for Fabric OS v7.4.1e indicate this in the following known defect:
Defect ID: DEFECT000616486
Technical Severity: Medium
Probability: Medium
Product: Brocade Fabric OS
Technology Group: Security
Reported In Release: FOS7.4.1
Technology: SSH - Secure Shell
Symptom: Unable to authenticate an SSH session after importing public key to switch.
Condition: This is encountered by admin level users on a switch running Fabric OS v7.4.1d
