bityard Blog

// Dell iDRAC and Limitation in Certificate Signing Request Field Lengths

Current versions of the Dell iDRAC have limitations with regard to the field length for the SSL CSR. These limitations violate RFC 5280 and lead to an invalid CSR and possibly an invalid SSL certificate.


The Dell iDRAC allows the protection of browser based access via HTTPS. In order to facilitate this, SSL certificates are being used. Initially and by default there is a self-signed SSL certificate installed on the iDRAC which is signed by the Dell CA. For additional security this self-signed SSL certificate can be replaced with one provided by the systems administrator. There are two ways to create an own SSL certificate for the Dell iDRAC.

The first one is to use the iDRACs integrated facility to create a SSL CSR. The appropriate form can be found in the iDRAC WebUI under:

iDRAC Settings → Network → SSL → Generate Certificate Signing Request (CSR) → Next

After filling in the fields Common Name, Organization Name, Organization Unit, Locality, State Name, Country Code and Email with the appropriate values and clicking on Generate a CSR is generated in the background and after a short time offered for download. This CSR can now be fed into a CA – own or service provider operated – in order to generate the actual certificate. The generated certificate is then uploaded into the iDRAC via the WebUI.

So far so easy, and done countless times without any issues. Until the day, one of the values of the aforementioned fields Common Name, Organization Name, Organization Unit, Locality, State Name, Country Code or Email for the CSR changes to a significantly longer string. In my case the value of Organization Name changed from 11 characters to 60 characters. It appeares that unfortunately the developers of the iDRAC didn't expect values of that length, although those are absolutely in conformance with the appropriate standard RFC 5280 and well within the upper bounds mentioned there:

[...]
--  specifications of Upper Bounds MUST be regarded as mandatory
--  from Annex B of ITU-T X.411 Reference Definition of MTS Parameter
--  Upper Bounds

-- Upper Bounds
ub-name INTEGER ::= 32768
ub-common-name INTEGER ::= 64
ub-locality-name INTEGER ::= 128
ub-state-name INTEGER ::= 128
ub-organization-name INTEGER ::= 64
ub-organizational-unit-name INTEGER ::= 64
[...]
ub-emailaddress-length INTEGER ::= 255
ub-common-name-length INTEGER ::= 64
ub-country-name-alpha-length INTEGER ::= 2
[...]

-- Note - upper bounds on string types, such as TeletexString, are
-- measured in characters.  Excepting PrintableString or IA5String, a
-- significantly greater number of octets will be required to hold
-- such a value.  As a minimum, 16 octets, or twice the specified
-- upper bound, whichever is the larger, should be allowed for
-- TeletexString.  For UTF8String or UniversalString at least four
-- times the upper bound should be allowed.
[...]

Even worse, the iDRAC doesn't issue an error or a warning if it's given an exceedingly long value. On the contrary, it silently accepts the longer value and just cuts off everything after the 48th or 51st character.

I've checked various versions of the Dell iDRAC and encountered the issue on at least the following Dell PowerEdge hardware and iDRAC software versions:

Host Model iDRAC iDRAC Version
host1 R815 6 1.95 (Build 05)
host2 R815 6 2.85 (Build 04)
host3 M915 6 3.75 (Build 5)
host4 M620 7 2.21.21.21
host5 M630 8 2.30.30.30
host6 M630 8 2.41.40.40

Here are some examples of the Subject: from the CSRs generated on different Dell PowerEdge hardware and iDRAC software versions:

user@host:~$ for FL in *.csr; do echo -n "${FL}: "; openssl req -text -in ${FL} -noout | grep "Subject: "; done
host1.csr: Subject: C=CC, ST=SSSSSS, L=LLLLLL, O=OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO, OU=OUOUOUOUOUOUOUOU, CN=FQDN/emailAddress=EMAIL
host2.csr: Subject: C=CC, ST=SSSSSS, L=LLLLLL, O=OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO, OU=OUOUOUOUOUOUOUOU, CN=FQDN/emailAddress=EMAIL
host3.csr: Subject: C=CC, ST=SSSSSS, L=LLLLLL, O=OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO, OU=OUOUOUOUOUOUOUOU, FQDN/emailAddress=EMAIL
host4.csr: Subject: C=CC, ST=SSSSSS, L=LLLLLL, O=OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO, OU=OUOUOUOUOUOUOUOU, CN=FQDN/emailAddress=EMAIL
host5.csr: Subject: C=CC, ST=SSSSSS, L=LLLLLL, O=OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO, OU=OUOUOUOUOUOUOUOU, CN=FQDN/emailAddress=EMAIL
host6.csr: Subject: C=CC, ST=SSSSSS, L=LLLLLL, O=OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO, OU=OUOUOUOUOUOUOUOU, CN=FQDN/emailAddress=EMAIL

The values in the different fields have been anonymized with the characters of the field name, but have otherwise been preserved in the length they appeared in the CSR. The observed behaviour is mostly consistent, with only one exception where the field value is strangely cut off after the 51st character.

Needless to say, cropping the field values like this results in an invalid CSR which in turn leads to issues in the certificate generation process at the CA.

After raising a support case with Dell, the problem was verified and the issue was acknowledged as a defect. A future iDRAC update is supposed to fix this issue. Unfortunately i have no information what version that will be nor when it will be released.

In the meantime, until there is a proper fix for this issue, the second way can be used. This is to manually create all three components – an own SSL private key, an own CSR and a certificate – for the Dell iDRAC. Unfortunately this involves significantly more steps, which are described below.

  • If necessary install some additional software packages. For Debian based systems:

    root@host:~# apt-get install openssl wsmancli rpl
    
  • In the following steps and commands, replace <FQDN> and <IP> with the FQDN and IP address of the Dell iDRAC where necessary.

  • Create a SSL private key:

    user@host:~$ openssl genrsa -out <FQDN>.key.pem 4096
    
  • Generate a CSR for the FQDN and IP address of the Dell iDRAC. Replace <C>, <ST>, <L>, <O>, <OU>, and <EMAIL> with the country, state, locality, organization, organizational-unit and a valid email address respectively.

    user@host:~$ openssl req -new -sha256 -nodes -key <FQDN>.key.pem -out <FQDN>.csr.pem -subj '/C=<C>/ST=<ST>/L=<L>/O=<O>/OU=<OU>/CN=<FDQN>/emailAddress=<EMAIL>' -config <(
    cat <<-EOF
    [req]
    req_extensions = req_ext
    distinguished_name = dn
    [ dn ]
    [ req_ext ]
    subjectAltName = @alt_names
    [alt_names]
    DNS.1 = <FQDN>
    IP.1 = <IP>
    EOF
    )
    

    Additional DNS.X = <FQDN> or IP.X = <IP> lines can be added if necessary. Increment the value of X accordingly.

  • With the generated CSR, request a certificate from your own CA or the one of your service provider. The range of different CAs and organizational processes involved is just too broad to cover this step in greater detail.

  • Merge the received certificate and the private key which was generated in one of the previous steps:

    user@host:~$ cat <FQDN>.cert.pem <FQDN>.key.pem > <FQDN>.certkey.pem
    
  • Convert the combined certificate and the private key into the PKCS #12 format:

    user@host:~$ openssl pkcs12 -export -in <FQDN>.certkey.pem -out <FQDN>.certkey.pfx
    

    This command will request a password at the command line which will later on be needed again.

  • Convert the PKCS #12 file into a base64-encoded file:

    user@host:~$ openssl base64 -in <FQDN>.certkey.pfx -out <FQDN>.certkey.p12
    
  • Prepare the base64-encoded PKCS #12 file for the upload to the Dell iDRAC over its SOAP-interface:

    user@host:~$ echo "<p:SetCertificateAndPrivateKey_INPUT xmlns:p="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/root/dcim/DCIM_LCService"><p:Type>server</p:Type><p:PKCS12>" > <FQDN>.certkey.xml
    user@host:~$ cat <FQDN>.certkey.p12 >> <FQDN>.certkey.xml
    user@host:~$ echo "</p:PKCS12><p:PKCS12pin>PASSWORD_AT_PKCS12_EXPORT</p:PKCS12pin></p:SetCertificateAndPrivateKey_INPUT>" >> <FQDN>.certkey.xml
    

    Replace the string PASSWORD_AT_PKCS12_EXPORT with the password that was given in the previous step where the certificate was exported into the PKCS #12 format:

    user@host:~$ rpl PASSWORD_AT_PKCS12_EXPORT <Paßwort> <FQDN>.certkey.xml
    
  • Upload the base64-encoded PKCS #12 file to the Dell iDRAC over its SOAP-interface. Replace <IP> with the IP address of the Dell iDRAC and <USER> and <PASSWORD> with the appropriate credentials of a user allowed to access the Dell iDRAC:

    user@host:~$ wsman invoke -a SetCertificateAndPrivateKey http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/root/dcim/DCIM_LCService?SystemCreationClassName=DCIM_ComputerSystem,CreationClassName=DCIM_LCService,SystemName=DCIM:ComputerSystem,Name=DCIM:LCService -h <IP> -P 443 -u <USER> -p <PASSWORD> -c dummy.cert -y basic -V -v -J <FQDN>.certkey.xml -j utf-8
    

The last two steps are optional and only necessary if you don't want to or cannot install the Dell OpenManage DRAC Tools, which include the racadm command, or if the remote access via racadm is not or cannot be enabled.

This workaround can also be applied generally and independently of the mentioned bug in the Dell iDRAC in order to create a valid SSL certificate for Dell iDRAC and Dell Blade CMC systems.

// Integration of Dell EqualLogic PS-Series Storages with RANCID

Adding support for Dell EqualLogic PS-Series storage arrays to version 3.5.1 of the popular, open source switch and router configuration management tool RANCID.

For the impatient and TL;DR here are the extensions to RANCID for the management of Dell EqualLogic PS-Series storage arrays:

Login script for Dell EqualLogic PS-Series storage arrays
Perl module to generate, process and save the configuration of Dell EqualLogic PS-Series storage arrays

The sources are to be found in my RANCID repository on GitHub


RANCID has, in its current version 3.5.1, support for a large variety of network devices like routers, switches, load-balancers, etc. Unfortunately there is currently little or no support for the management of storage devices, even though a lot of them offer a command line interface which can be used by RANCID.

Although there probably are a couple of reasons for this, i suppose this is largely due to the fact that network and storage admins are – in most organizations – still in different groups, each with their own set of management and support tools. With RANCID originating from the realm of network administration, probably only few storage admins know about this very valuable tool to begin with. There is probably also very little transfer over from the position of network administrator into the area of storage administration and thus a limited amount of knowledge transfer between those two fields.

This blog post describes how to extend and configure RANCID in order to add support for Dell EqualLogic PS-Series storage arrays. The extensions are based on the – at the time of writing – current version 3.5.1 of RANCID. RANCID can either be build from source or be installed pre-packaged e.g. from the backports repository of Debian stable (jessie). Basically, the extension to RANCID consist of only two files:

Besides those two extensions only a small change to /etc/rancid/rancid.types.conf, as well as the standard RANCID configuration of a new device group and device is necessary. See the following full step-by-step configuration example for Dell EqualLogic PS-Series storage arrays:

  • Add the backports repository of Debian stable (jessie) to the APT configuration:

    root@host:~$ echo 'deb http://http.debian.net/debian jessie-backports main non-free contrib' >> /etc/apt/sources.list.d/jessie-backports.list
    root@host:~$ apt-get update
    
  • Install RANCID v3.5.1 from the backports repository of Debian stable (jessie):

    root@host:~$ apt-get -y install rancid/jessie-backports
    
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    Selected version '3.5.1-1~bpo8+1' (Debian Backports:jessie-backports [amd64]) for 'rancid'
    The following extra packages will be installed:
      expect tcl-expect
    The following NEW packages will be installed:
      expect rancid tcl-expect
    0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
    Need to get 511 kB of archives.
    After this operation, 2,178 kB of additional disk space will be used.
    [...]
    

    Optional: In case Subversion should be used as a revision control system (RCS) to store the switch configuration, install it:

    root@host:~$ apt-get -y install subversion
    
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    The following extra packages will be installed:
      libapr1 libaprutil1 libldap-2.4-2 libsasl2-2 libsasl2-modules libsasl2-modules-db libserf-1-1 libsvn1
    Suggested packages:
      libsasl2-modules-otp libsasl2-modules-ldap libsasl2-modules-sql libsasl2-modules-gssapi-mit libsasl2-modules-gssapi-heimdal subversion-tools db5.3-util patch
    The following NEW packages will be installed:
      libapr1 libaprutil1 libldap-2.4-2 libsasl2-2 libsasl2-modules libsasl2-modules-db libserf-1-1 libsvn1 subversion
    0 upgraded, 9 newly installed, 0 to remove and 0 not upgraded.
    Need to get 2,723 kB of archives.
    After this operation, 9,683 kB of additional disk space will be used.
    [...]
    
  • Download the login script for Dell EqualLogic PS-Series storage arrays and store it under the path /usr/lib/rancid/bin/.

  • Download the Perl module to process and save the configuration of Dell EqualLogic PS-Series storage arrays and store it under the path /usr/share/perl5/rancid/.

  • Edit the global RANCID configuration:

    root@host:~$ vi /etc/rancid/rancid.conf
    

    Select the RCS (CVS, SVN or Git) of your choice. In this example SVN is used:

    RCSSYS=svn; export RCSSYS

    Define a name for your Dell EqualLogic device group in the LIST_OF_GROUPS configuration variable. In this example we'll use the name dell-storage:

    LIST_OF_GROUPS="dell-storage"; export LIST_OF_GROUPS
  • Create the cloginrc configuration file, which will contain the login information for your Dell EqualLogic PS-Series devices and some default values:

    root@host:~$ touch /etc/rancid/cloginrc
    root@host:~$ chmod 660 /etc/rancid/cloginrc
    root@host:~$ chown root:rancid /etc/rancid/cloginrc
    root@host:~$ vi /etc/rancid/cloginrc
    

    Example:

    add user        dell-eql-1      dell-user
    add password    dell-eql-1      <login-passwort>
    
    [...]
    
    add user        *               <default-user>
    add password    *               <default-login-passwort>
    add method      *               ssh

    For the device named dell-eql-1 login as user dell-user with the password <login-passwort>.

    For all other systems, login as user <default-user> with the password <default-login-passwort>. The login method for all systems is via SSH.

    Since the cloginrc configuration file is parsed in a first-match fashion, the default values must always be at the bottom of the file.

  • Add a new device type for Dell EqualLogic PS-Series storage arrays to the RANCID configuration. See man router.db and /etc/rancid/rancid.types.conf. In this example and in the general case of Dell Dell EqualLogic PS-Series storage arrays the name of the device type is equallogic:

    root@host:~$ vi /etc/rancid/rancid.types.conf
    

    Here we set the login script to be used to the new eqllogin. The postprocessing script is set rancid -t equallogic in order to call the new Perl module equallogic, which will do the actual processing. The command to be issued on the Dell EqualLogic device is set to save-config -verbose. The -verbose part is essential here, otherwise the configuration of the device will only be saved to a file on the Dell EqualLogic device and not be printed to the terminal:

    equallogic;login;eqllogin
    equallogic;script;rancid -t equallogic
    equallogic;module;equallogic
    equallogic;inloop;equallogic::inloop
    equallogic;command;equallogic::SaveConfiguration;save-config -verbose
  • Change to the user rancid:

    root@host:~$ su - rancid
    
    • Create a symbolic link to the login configuration previously created in /etc/rancid/:

      rancid@host:~$ ln -s /etc/rancid/cloginrc /var/lib/rancid/.cloginrc
      
    • Initialize the directory structure for the RCS (CVS, SVN or Git) selected above. This will automatically be done for each device group configured in the LIST_OF_GROUPS configuration variable. The example shown here only creates the directory structure for the device group dell-storage defined above:

      rancid@host:~$ /usr/lib/rancid/bin/rancid-cvs
      Committed revision 1.
      Checked out revision 1.
      Updating '.':
      At revision 1.
      A         configs
      Adding         configs
      
      Committed revision 2.
      A         router.db
      Adding         router.db
      Transmitting file data .
      Committed revision 3.
      
      rancid@host:~$ find /var/lib/rancid/dell-storage/
      /var/lib/rancid/dell-storage
      /var/lib/rancid/dell-storage/configs
      /var/lib/rancid/dell-storage/router.db
      /var/lib/rancid/dell-storage/routers.all
      /var/lib/rancid/dell-storage/routers.down
      /var/lib/rancid/dell-storage/routers.up
      /var/lib/rancid/dell-storage/.svn
      /var/lib/rancid/dell-storage/.svn/entries
      /var/lib/rancid/dell-storage/.svn/format
      /var/lib/rancid/dell-storage/.svn/pristine
      /var/lib/rancid/dell-storage/.svn/pristine/da
      /var/lib/rancid/dell-storage/.svn/pristine/da/da39a3ee5e6b4b0d3255bfef95601890afd80709.svn-base
      /var/lib/rancid/dell-storage/.svn/tmp
      /var/lib/rancid/dell-storage/.svn/wc.db
      
    • Add Dell EqualLogic storage devices by their hostname to the configuration file router.db of the corresponding device group:

      rancid@host:~$ vi /var/lib/rancid/dell-storage/router.db
      

      In this example the device group dell-storage, the device type equallogic and the system dell-eql-1:

      dell-eql-1;equallogic;up;A comment describing the system dell-eql-1
    • Perform a login test with the previously configured new login script eqllogin for Dell EqualLogic devices on the newly defined system dell-eql-1. The following example output shows the steps that should automatically be performed by the eqllogin expect script. No manual intervention should be necessary.

      rancid@host:~$ /usr/lib/rancid/bin/eqllogin dell-eql-1
      spawn ssh -x -l grpadmin dell-eql-1
      The authenticity of host 'dell-eql-1 (<ip address>)' can't be established.
      RSA key fingerprint is <rsa key fingerprint>
      Are you sure you want to continue connecting (yes/no)?  yes
      Host dell-eql-1 added to the list of known hosts.
      Warning: Permanently added 'dell-eql-1,<ip address>' (RSA) to the list of known hosts.
      grpadmin@dell-eql-1's password: 
      Last login: Thu Dec  8 22:32:39 2016 from <ip address> on ttyp1
       
      
                 Welcome to Group Manager
      
              Copyright 2001-2015 Dell Inc.
      
      
      
      dell-eql-1-grp> 
      
    • Finish the login test by manually logging out of the system:

      dell-eql-1-grp> logout
      Do you really want to logout? (y/n) [n]y
      
      dell-eql-1-grp> Connection to dell-eql-1 closed.
      
    • Manually perform an initial RANCID run to make sure everything works as expected:

      rancid@host:~$ rancid-run
      

      If everything ran successfully, there should now be a file /var/lib/rancid/dell-storage/configs/dell-eql-1 containing the output of the command save-config -verbose for the system dell-eql-1.

  • Create the email aliases necessary for the proper delivery of the emails generated by RANCID. Again in this example for the device group dell-storage:

    root@host:~$ vi /etc/aliases
    
    rancid-dell-storage:       <email>@<domain>
    rancid-admin-dell-storage: <email>@<domain>

    Recreate your aliases DB. In case postfix is used as an MTA:

    root@host:~$ postalias /etc/aliases
    
  • Enable the RANCID cron jobs. Adjust the execution times and intervals according to your needs:

    root@host:~$ vi /etc/cron.d/rancid
    

Some final words: The contents of the directories /var/lib/rancid/<device group>/ and /var/lib/rancid/<device group>/configs/ are maintained in the RCS – CVS, SVN or Git – of your choice. You can operate on those directories with the usual commands of the selected RCS. There are also some really nice and intuitive web frontends to the RCS of choice. For me, the combination of SVN as RCS and WebSVN as a web frontend worked out very well.

// Backporting Open-iSCSI to Debian 8 "Jessie"

Starting with the Debian open-iscsi release 2.0.874-1, which is now available in the backports reprository for Debian 8 "Jessie", the manual backport of Open-iSCSI described below is no longer necessary.

The Debian Open-iSCSI package is now based on current upstream version of Open-iSCSI. Open-iSCSIs iscsiuio is now provided through its own Debian package. Several improvements (Git commit d05fe0e1, Git commit 6004a7e7) have been made in handling hardware initiator based iSCSI sessions.

Thanks to Christian Seiler for his work on bringing the Debian Open-iSCSI package up to a current upstream version and for helping to sort our some issues related to the use of hardware initiators!

In the previous article Debugging Segfaults in Open-iSCSIs iscsiuio on Intel Broadwell i mentioned using a backported version of Open-iSCSI on Debian 8 (“Jessie”). This new post describes the backport and the changes provided by it in greater detail. All the changes to the original Debian package from “unstable” (“Sid”) can be found in my Debian Open-iSCSI Packaging repository on GitHub.

Starting point was a clone of the Debian Open-iSCSI Packaging repository at Git commit df150d90. Mind though, that in the meantime between creating the backport and writing this, the Debian Open-iSCSI maintainers have been busy and a more recent version of the Debian Open-iSCSI package from “unstable” (“Sid”) is now available.

Within this particular version of the Debian Open-iSCSI package, i first enabled the build of Open-iSCSIs iscsiuio. On the one hand, this was done in order to ensure that the iscsiuio code would successfully build even at this old level of the Open-iSCSI code. On the other hand, this would be used as a differentiator for any issues surfacing later on, after the move to the more recent upstream Open-iSCSI sources, indicating the root cause of those would then solely be with the newer upstream version of Open-iSCSI. Some integration into the general system environment was also added at this point. In detail the changes were:

  • Git commit 32c96e6c removes the Debian patch 05-disable-iscsiuio.patch which disables the build of iscsiuio.

  • Git commit 984344a1 enables the build of iscsiuio, extends the cleanup build targets and adds iscsiuio to the dh_systemd build targets.

  • Git commit 89d845a9 adds the results from the successful build – the iscsiuio binary, the iscsiuio manual page, a readme file and a logrotate configuration file – to the Debian package. It also adds the kernel modules bnx2i and cnic to the list of kernel modules to be loaded at installation time.

  • Git commit 89195bbe adds the systemd service and socket unit files for iscsiuio. Those files have been taken from this discussion on the Open-iSCSI mailing list and have slightly been altered.

With the above changes a intermediary package was build for testing purposes. During the following tests sometimes all currently mounted filesystems – even those distinctly not based on iSCSI volumes – would suddenly be unmounted. For some filesystems this would succeed, for others, like e.g. the /var and the root filesystem, this would fail due to them being currently in use. The issue particularly occured while stopping the open-iscsi service either via its regular init script or via its systemd service. This is usually done at system shutdown or during uninstall of the Open-iSCSI package. Tracking down the root cause of this issue led to an unhandled case in the umountiscsi.sh script, which is called while stopping the open-iscsi service. Specifically, the following code section is responsible for the observed behaviour:

debian/extra/umountiscsi.sh
256    if [ $HAVE_LVM -eq 1 ] ; then
257        # Look for all LVM volume groups that have a backing store
258        # on any iSCSI device we found. Also, add $LVMGROUPS set in
259        # /etc/default/open-iscsi (for more complicated stacking
260        # configurations we don't automatically detect).
261        for _vg in $(cd /dev ; $PVS --noheadings -o vg_name $iscsi_disks $iscsi_partitions $iscsi_multipath_disks $iscsi_multipath_partitions 2>/dev/null) $LVMGROUPS ; do
262            add_to_set iscsi_lvm_vgs "$_vg"
263        done

The heuristic of the umountiscsi.sh script are trying to identify iSCSI based disk devices which are valid candidates for proper deactivation upon system shutdown. It turned out that in LVM based setups where there are currently no iSCSI based disk devices present, the variables $iscsi_disks, $iscsi_partitions, $iscsi_multipath_disks and $iscsi_multipath_partitions are left empty by the scripts logic. In line 261 in the above code snippet, this leads to a call to the pvs --noheadings -o vg_name command without any additional arguments limiting its output of volume groups. Hence, the returned output is instead a complete list of all volume groups currently present on the system. Based on this list, the associated logical volumes for each volume group are determined and added to the list of devices to be unmounted. Finally all devices in this list are actually unmounted.

Without making too invasive changes to the script logic of umountiscsi.sh a quick'n'dirty solution was to introduce a check before the call to pvs which would determine whether the variables $iscsi_disks, $iscsi_partitions, $iscsi_multipath_disks and $iscsi_multipath_partitions are all empty. If this is the case, the call to pvs is simply skipped. The following patch shows the necessary code changes which are also available in Git commit 5118af7f:

umountiscsi.sh.patch
diff --git a/debian/extra/umountiscsi.sh b/debian/extra/umountiscsi.sh
index 1206fa1..485069c 100755
--- a/debian/extra/umountiscsi.sh
+++ b/debian/extra/umountiscsi.sh
@@ -258,9 +258,11 @@ enumerate_iscsi_devices() {
                # on any iSCSI device we found. Also, add $LVMGROUPS set in
                # /etc/default/open-iscsi (for more complicated stacking
                # configurations we don't automatically detect).
-               for _vg in $(cd /dev ; $PVS --noheadings -o vg_name $iscsi_disks $iscsi_partitions $iscsi_multipath_disks $iscsi_multipath_partitions 2>/dev/null) $LVMGROUPS ; do
-                       add_to_set iscsi_lvm_vgs "$_vg"
-               done
+               if [ -n "$iscsi_disks" -o -n "$iscsi_partitions" -o -n "$iscsi_multipath_disks" -o -n "$iscsi_multipath_partitions" ]; then
+                   for _vg in $(cd /dev ; $PVS --noheadings -o vg_name $iscsi_disks $iscsi_partitions $iscsi_multipath_disks $iscsi_multipath_partitions 2>/dev/null) $LVMGROUPS ; do
+                           add_to_set iscsi_lvm_vgs "$_vg"
+                   done
+               fi
 
                # $iscsi_lvm_vgs is now unique list
                for _vg in $iscsi_lvm_vgs ; do

After this was fixed, the last step was to finally move to the more recent upstream Open-iSCSI sources. In detail the changes in this last step were:

  • Git commit f5ab51ff moves the code to version 2.0.873+git1.1dfb88a4 which is based upon the upstream Git commit 1dfb88a4. This is the last commit before the externalization of the Open-iSNS library. Since i didn't want to also backport the Open-iSNS packages from Debian “unstable” (“Sid”), i decided to just skip the next two upstream commits 76832662 and c6d1117b and stick with the locally delivered Open-iSNS library.

  • Git commit 8c1e6974 removes the local Debian patches 01_spelling-errors-and-manpage-hyphen-fixes.patch, 02_make-iscsistart-a-dynamic-binary.patch and 03_respect-build-flags.patch which have already been merged into the more recent upstream Open-iSCSI sources. The remaining local Debian patches were renamed and reordered to 01_fix_iscsi_path.patch, 02_var-lock_var-run_transition.patch and 03_makefile_reproducibility_issues.patch. A whole bunch of new patches named {04,05,06,07,08,09,10,11,12,13,14,15}_upstream_git_commit_<Git commit ID>.patch were added in order to bring the sources up to the – by then most recent – upstream Git commit 0fa43f29.

  • Git commit d051dece removes some files from the Debian package, which were dynamically generated during the build of iscsiuio.

  • Finally Git commit 0fabb948 deals with the issue described in Debugging Segfaults in Open-iSCSIs iscsiuio on Intel Broadwell.

With the steps and changes described above, a backported version of Open-iSCSI using its most recent sources was created as a package for Debian 8 (“Jessie”). This package also supports offloaded iSCSI connections via the Broadcom BCM577xx and BCM578xx iSOEs with the use of iscsiuio. The package has been in production use for over a month now and no major issues – neither with the newer upstream Open-iSCSI sources, nor with use of Broadcom BCM577xx and BCM578xx iSOEs through iscsiuio – have emerged so far.

// Debugging Segfaults in Open-iSCSIs iscsiuio on Intel Broadwell

Open-iSCSIs tool iscsiuio, which is used to configure and manage Broadcom BCM577xx and BCM578xx iSCSI offload engines (iSOE), currently crashes with a segmentation fault upon target login on Intel Broadwell based systems. Comparable system setups, based on the older Intel Haswell chips do not show this issue.

In the past i've been using QLogic 4000 and QLogic 8200 Series network adapters and iSCSI HBAs which provide a full iSCSI offload engine (iSOE) implementation in the adapters firmware. Unfortunately the QLogic 8200 Series network adapters are no longer available for Dell M-Series blade servers. The alternatives offered by Dell are the Intel X520 and Intel X540 series adapters, or the Broadcom BCM57810S series adapters. Instead of using the Intel X520/X540, which provide no iSOE at all, i decided to go with the Broadcom BCM57810S, which at least provide some kind of iSOE. According to the VMware terminology the Broadcom BCM57810S are dependent hardware iSCSI Initiators. Dependent in this context means, that the iSOE does not implement all the necessary features and thus cannot perform all the tasks (e.g. TCP/IP stack, configuration and session management, authentication, etc.) necessary for target handling by itself. Rather, some of these tasks are provided by a third party on which this kind of iSOE depends on. In case of the Broadcom BCM57810S this third party is the iscsiuio daemon, which has for some time been part of the Open-iSCSI project. Simply put, the iscsiuio daemon acts as an intermediary between the iscsid on the one side and the QLogic1) NetXtreme II driver (kernel module bnx2 or bnx2x) and the QLogic2) CNIC driver (kernel module cnic) on the other side, facilitating the creation and overall management of offloaded iSCSI sessions. Very simplified, the flow of information is as follows:

iscsiadm ←→ iscsid ←→ iscsiuio ←→ bnx2/bnx2x ←→ cnic ←→ Broadcom BCM57810S adapter ←→ Network ←→ Target

In my environment the Broadcom BCM57810S adapters are installed and used on six hosts (host1 and host{5,6,7,8,9}). They all connect to the same Dell EqualLogic storage systems in the backend, using the same network dedicated to iSCSI traffic. All hosts are Dell M630 blade servers with exactly the same firmware, operating system (Debian 8) and software versions. I'm using a backported version of Open-iSCSI, which is based on Git commit 0fa43f29, but excluding the commits 76832662 and c6d1117b which just implement the externalization of the Open-iSNS code. The systems originate from the same install image, so their configuration is – to a large extent – exactly the same. The only difference between the hosts is that host1 has Intel E5 v3 (aka Haswell) CPUs, while host{5,6,7,8,9} have Intel E5 v4 (aka Broadwell) CPUs.

On host1 everything works fine, iscsiuio runs as expected and access to targets via the Broadcom BCM57810S iSOEs is working flawlessly. On host{5,6,7,8,9} on the other hand, i was getting segmentation faults like the one in the example shown below, while trying to log in to any target.

host5:~# gdb /sbin/iscsiuio
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /sbin/iscsiuio...(no debugging symbols found)...done.


(gdb) # run -d 4 -f
Starting program: /sbin/iscsiuio -d 4 -f
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
INFO  [Wed Jul 27 10:01:45 2016]Initialize logger using log file: /var/log/iscsiuio.log
INFO  [Wed Jul 27 10:01:45 2016]Started iSCSI uio stack: Ver 0.7.8.2
INFO  [Wed Jul 27 10:01:45 2016]Build date: Fri Jul 22 15:40:04 CEST 2016
INFO  [Wed Jul 27 10:01:45 2016]Debug mode enabled
INFO  [Wed Jul 27 10:01:45 2016]Running on sysname: 'Linux', release: '3.16.0-4-amd64', version '#1 SMP Debian 3.16.7-ckt
25-2+deb8u3 (2016-07-02)' machine: 'x86_64'
DBG   [Wed Jul 27 10:01:45 2016]Loaded nic library 'bnx2' Version: '0.7.8.2' build on Fri Jul 22 15:40:04 CEST 2016'
DBG   [Wed Jul 27 10:01:45 2016]Added 'bnx2' nic library
DBG   [Wed Jul 27 10:01:45 2016]Loaded nic library 'bnx2x' Version: '0.7.8.2' build on Fri Jul 22 15:40:04 CEST 2016'
DBG   [Wed Jul 27 10:01:45 2016]Added 'bnx2x' nic library
[New Thread 0x7ffff760f700 (LWP 4942)]
INFO  [Wed Jul 27 10:01:45 2016]signal handling thread ready
INFO  [Wed Jul 27 10:01:45 2016]nic_utils Found host[11]: host11
INFO  [Wed Jul 27 10:01:45 2016]Done capturing /sys/class/iscsi_host/host11/netdev
INFO  [Wed Jul 27 10:01:45 2016]Done capturing /sys/class/iscsi_host/host11/netdev
INFO  [Wed Jul 27 10:01:45 2016]nic_utils looking for uio device for eth3
WARN  [Wed Jul 27 10:01:45 2016]Could not find assoicate uio device with eth3
ERR   [Wed Jul 27 10:01:45 2016]nic_utils Could not determine UIO name for eth3
INFO  [Wed Jul 27 10:01:45 2016]nic_utils Found host[12]: host12
INFO  [Wed Jul 27 10:01:45 2016]Done capturing /sys/class/iscsi_host/host12/netdev
INFO  [Wed Jul 27 10:01:45 2016]Done capturing /sys/class/iscsi_host/host12/netdev
INFO  [Wed Jul 27 10:01:45 2016]nic_utils looking for uio device for eth2
INFO  [Wed Jul 27 10:01:45 2016]nic_utils eth2 associated with uio0
INFO  [Wed Jul 27 10:01:45 2016]nic_utils NIC not found creating an instance for host_no: 12 eth2
DBG   [Wed Jul 27 10:01:45 2016]Could not increase process priority: Success
[New Thread 0x7ffff6e0e700 (LWP 4943)]
DBG   [Wed Jul 27 10:01:45 2016]iscsi_ipc Started iscsid listening thread
DBG   [Wed Jul 27 10:01:45 2016]iscsi_ipc Waiting for iscsid command
INFO  [Wed Jul 27 10:01:45 2016]NIC_NL Netlink to CNIC on pid 4938 is ready
DBG   [Wed Jul 27 10:01:57 2016]iscsi_ipc recv iscsid request: cmd: 1, payload_len: 11720
INFO  [Wed Jul 27 10:01:57 2016]iscsi_ipc Received request for 'eth2' to set IP address: '10.0.1.62' VLAN: '0'
INFO  [Wed Jul 27 10:01:57 2016]iscsi_ipc Using netmask: 0.0.0.0
INFO  [Wed Jul 27 10:01:57 2016]iscsi_ipc  eth2, using existing NIC
INFO  [Wed Jul 27 10:01:57 2016]nic_utils looking for uio device for eth2
INFO  [Wed Jul 27 10:01:57 2016]nic_utils eth2 associated with uio0
INFO  [Wed Jul 27 10:01:57 2016]Done capturing /sys/class/uio/uio0/name
INFO  [Wed Jul 27 10:01:57 2016]nic_utils eth2: Verified uio name bnx2x_cnic with library bnx2x
INFO  [Wed Jul 27 10:01:57 2016]eth2: found NIC with library 'bnx2x'
INFO  [Wed Jul 27 10:01:57 2016]iscsi_ipc eth2 library set using transport_name bnx2i
INFO  [Wed Jul 27 10:01:57 2016]iscsi_ipc eth2: requesting configuration using static IP address
DBG   [Wed Jul 27 10:01:57 2016]iscsi_ipc eth2 couldn't find interface with ip_type: 0x2 creating it
INFO  [Wed Jul 27 10:01:57 2016]nic eth2: Added nic interface for VLAN: 0, protocol: 2
INFO  [Wed Jul 27 10:01:57 2016]iscsi_ipc eth2: created network interface
[New Thread 0x7ffff660d700 (LWP 4947)]
WARN  [Wed Jul 27 10:01:57 2016]nic_utils eth2: device already disabled: flag: 0x1088 state: 0x1
INFO  [Wed Jul 27 10:01:57 2016]iscsi_ipc eth2: configuring using static IP IPv4 address :10.0.1.62
INFO  [Wed Jul 27 10:01:57 2016]iscsi_ipc  netmask: 255.255.255.0
[New Thread 0x7ffff5e0c700 (LWP 4948)]
INFO  [Wed Jul 27 10:01:57 2016]iscsi_ipc ISCSID_UIP_IPC_GET_IFACE: command: 1 name: bnx2i.d0:43:1e:51:98:53, netdev: eth2 ipaddr: 10.0.1.62 vlan: 0 transport_name:bnx2i
INFO  [Wed Jul 27 10:01:57 2016]nic_utils eth2: spinning up thread for nic
DBG   [Wed Jul 27 10:01:57 2016]iscsi_ipc Waiting for iscsid command
[New Thread 0x7ffff560b700 (LWP 4949)]
DBG   [Wed Jul 27 10:01:57 2016]nic eth2: Waiting to be enabled
INFO  [Wed Jul 27 10:01:57 2016]Created nic thread: eth2
INFO  [Wed Jul 27 10:01:57 2016]iscsi_ipc eth2: started NIC enable thread state: 0x1
DBG   [Wed Jul 27 10:01:57 2016]nic eth2: is now enabled
INFO  [Wed Jul 27 10:01:57 2016]bnx2x eth2: bnx2x driver using version 1.78.19
ERR   [Wed Jul 27 10:01:58 2016]bnx2x /dev/uio0: uio device has been brought up via pid: 4938 on fd: 7
INFO  [Wed Jul 27 10:01:58 2016]Done capturing /sys/class/uio/uio0/name
INFO  [Wed Jul 27 10:01:58 2016]bnx2x eth2: Verified is a cnic_uio device
DBG   [Wed Jul 27 10:01:58 2016]bnx2x eth2: using rx ring size: 15, rx buffer size: 1024
INFO  [Wed Jul 27 10:01:58 2016]Done capturing /sys/class/uio/uio0/event
DBG   [Wed Jul 27 10:01:58 2016]bnx2x Chip ID: 168e1000
INFO  [Wed Jul 27 10:01:58 2016]nic_id eth2: is found at 03:00.00
INFO  [Wed Jul 27 10:01:58 2016]bnx2x eth2: func 0x0, pfid 0x0, client_id 0x88, cid 0x1
DBG   [Wed Jul 27 10:01:58 2016]bnx2x eth2: mode = 0x100
INFO  [Wed Jul 27 10:01:58 2016]bnx2x eth2:  Using mac address: d0:43:1e:51:98:53
INFO  [Wed Jul 27 10:01:58 2016]eth2: bnx2x initialized
INFO  [Wed Jul 27 10:01:58 2016]nic eth2: Initialized ip stack: VLAN: 0
INFO  [Wed Jul 27 10:01:58 2016]nic eth2: mac: d0:43:1e:51:98:53
INFO  [Wed Jul 27 10:01:58 2016]nic eth2: Using IP address: 10.0.1.62
INFO  [Wed Jul 27 10:01:58 2016]nic eth2: Using netmask: 255.255.255.0
INFO  [Wed Jul 27 10:01:58 2016]nic eth2: enabled vlan 0 protocol: 2
INFO  [Wed Jul 27 10:01:58 2016]nic eth2: entering main nic loop
DBG   [Wed Jul 27 10:01:58 2016]nic_utils eth2: device enabled
[Thread 0x7ffff5e0c700 (LWP 4948) exited]
DBG   [Wed Jul 27 10:01:59 2016]iscsi_ipc recv iscsid request: cmd: 1, payload_len: 11720
INFO  [Wed Jul 27 10:01:59 2016]iscsi_ipc Received request for 'eth2' to set IP address: '10.0.1.62' VLAN: '0'
INFO  [Wed Jul 27 10:01:59 2016]iscsi_ipc Using netmask: 0.0.0.0
INFO  [Wed Jul 27 10:01:59 2016]iscsi_ipc  eth2, using existing NIC
INFO  [Wed Jul 27 10:01:59 2016]nic_utils looking for uio device for eth2
INFO  [Wed Jul 27 10:01:59 2016]nic_utils eth2 associated with uio0
INFO  [Wed Jul 27 10:01:59 2016]eth2: Have NIC library 'bnx2x'
INFO  [Wed Jul 27 10:01:59 2016]Done capturing /sys/class/uio/uio0/name
INFO  [Wed Jul 27 10:01:59 2016]nic_utils eth2: Verified uio name bnx2x_cnic with library bnx2x
INFO  [Wed Jul 27 10:01:59 2016]eth2: found NIC with library 'bnx2x'
INFO  [Wed Jul 27 10:01:59 2016]iscsi_ipc eth2 library set using transport_name bnx2i
INFO  [Wed Jul 27 10:01:59 2016]iscsi_ipc eth2: requesting configuration using static IP address
INFO  [Wed Jul 27 10:01:59 2016]iscsi_ipc eth2: using existing network interface
INFO  [Wed Jul 27 10:01:59 2016]iscsi_ipc eth2: IP configuration didn't change using 0x2
INFO  [Wed Jul 27 10:01:59 2016]iscsi_ipc eth2: NIC already enabled flags: 0x1084 state: 0x4

INFO  [Wed Jul 27 10:01:59 2016]iscsi_ipc ISCSID_UIP_IPC_GET_IFACE: command: 1 name: bnx2i.d0:43:1e:51:98:53, netdev: eth2 ipaddr: 10.0.1.62 vlan: 0 transport_name:bnx2i
DBG   [Wed Jul 27 10:01:59 2016]iscsi_ipc Waiting for iscsid command
INFO  [Wed Jul 27 10:02:00 2016]NIC_NL Received path_req for host 12
INFO  [Wed Jul 27 10:02:00 2016]Done capturing /sys/class/iscsi_host/host12/netdev
DBG   [Wed Jul 27 10:02:00 2016]NIC_NL Pulled nl event
INFO  [Wed Jul 27 10:02:00 2016]NIC_NL eth2: Processing 'path_req'
DBG   [Wed Jul 27 10:02:00 2016]NIC_NL eth2: PATH_REQ with iface_num -1 VLAN 32768
DBG   [Wed Jul 27 10:02:00 2016]CNIC eth2: Netlink message with VLAN ID: 0, path MTU: 9000 minor: 0 ip_addr_len: 4
DBG   [Wed Jul 27 10:02:00 2016]CNIC eth2: src=10.0.1.62
DBG   [Wed Jul 27 10:02:00 2016]CNIC eth2: dst=10.0.1.2
DBG   [Wed Jul 27 10:02:00 2016]CNIC eth2: nm=255.255.255.0
INFO  [Wed Jul 27 10:02:00 2016]CNIC eth2: Didn't find IPv4: '10.0.1.2' in ARP table
DBG   [Wed Jul 27 10:02:00 2016]CNIC eth2: Sent cnic arp request for IP: 10.0.1.2
INFO  [Wed Jul 27 10:02:00 2016]Found 10.0.1.2 at b0:83:fe:cc:57:bb
DBG   [Wed Jul 27 10:02:00 2016]CNIC neighbor reply sent back to kernel 10.0.1.62 at b0:83:fe:cc:57:bb with vlan 0
INFO  [Wed Jul 27 10:02:00 2016]NIC_NL eth2: 'path_req' operation finished

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff660d700 (LWP 4947)]
__lll_unlock_elision (lock=0x55555577fd40, private=0) at ../nptl/sysdeps/unix/sysv/linux/x86/elision-unlock.c:29
29      ../nptl/sysdeps/unix/sysv/linux/x86/elision-unlock.c: No such file or directory.


(gdb) # info threads
  Id   Target Id         Frame
  6    Thread 0x7ffff560b700 (LWP 4949) "iscsiuio" 0x00007ffff76f1ae3 in select () at ../sysdeps/unix/syscall-template.S:81
* 4    Thread 0x7ffff660d700 (LWP 4947) "iscsiuio" __lll_unlock_elision (lock=0x55555577fd40, private=0) at ../nptl/sysdeps/unix/sysv/linux/x86/elision-unlock.c:29
  3    Thread 0x7ffff6e0e700 (LWP 4943) "iscsiuio" 0x00007ffff79c9ccd in accept () at ../sysdeps/unix/syscall-template.S:81
  2    Thread 0x7ffff760f700 (LWP 4942) "iscsiuio" do_sigwait (set=<optimized out>, sig=0x7ffff760eeac) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwait.c:63
  1    Thread 0x7ffff7fea700 (LWP 4938) "iscsiuio" 0x00007ffff79c9e9d in recvmsg () at ../sysdeps/unix/syscall-template.S:81


(gdb) # thread apply all bt

Thread 6 (Thread 0x7ffff560b700 (LWP 4949)):
#0  0x00007ffff76f1ae3 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x000055555555ac06 in ?? ()
#2  0x000055555555d39e in nic_loop ()
#3  0x00007ffff79c30a4 in start_thread (arg=0x7ffff560b700) at pthread_create.c:309
#4  0x00007ffff76f887d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7ffff660d700 (LWP 4947)):
#0  __lll_unlock_elision (lock=0x55555577fd40, private=0) at ../nptl/sysdeps/unix/sysv/linux/x86/elision-unlock.c:29
#1  0x00007ffff79c7007 in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:94
#2  0x000055555555e803 in nl_process_handle_thread ()
#3  0x00007ffff79c30a4 in start_thread (arg=0x7ffff660d700) at pthread_create.c:309
#4  0x00007ffff76f887d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7ffff6e0e700 (LWP 4943)):
#0  0x00007ffff79c9ccd in accept () at ../sysdeps/unix/syscall-template.S:81
#1  0x00005555555641b0 in ?? ()
#2  0x00007ffff79c30a4 in start_thread (arg=0x7ffff6e0e700) at pthread_create.c:309
#3  0x00007ffff76f887d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7ffff760f700 (LWP 4942)):
#0  do_sigwait (set=<optimized out>, sig=0x7ffff760eeac) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwait.c:63
#1  0x00007ffff79ca693 in __sigwait (set=0x7ffff760eeb0, sig=0x0) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwait.c:97
#2  0x000055555555a49c in _start ()

Thread 1 (Thread 0x7ffff7fea700 (LWP 4938)):
#0  0x00007ffff79c9e9d in recvmsg () at ../sysdeps/unix/syscall-template.S:81
#1  0x000055555555e5e9 in ?? ()
#2  0x000055555555eea8 in nic_nl_open ()
#3  0x000055555555a1b8 in main ()

The Open-iSCSI command which lead to this segmentation fault was a simple login at a previously defined target node:

host5:~# iscsiadm -m node -T <target iqn> -I <interface> --login

Double-checking the configuration, the firmware and software versions as well as the general hardware setup didn't yield any usable indication as to where the root cause of this issue might be. Searching the web for __lll_unlock_elision in conjunction with the pthread_* function calls, led me to the following resources:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800574
https://lwn.net/Articles/534758/

Those are pointing towards a CPU (Broadwell and Skylake) specific problem when not carefully using mutexes. The general opinion from there and also other related bug reports seems to be, that the source of such issues is almost always an improper use of mutex locking, which – up to now – has either been tolerated or just by chance not lead to a failure. More recent CPUs and software implementations (e.g. the GNU libc) appear to be less forgiving in this regard. Thus the advice is to change the application behaviour towards a proper use of mutex locking, in order to address such an issue.

The article Intel's Broadwell Xeon E5-2600 v4 chips: So what's in it for you, smartie-pants coders offers a rather nice write-up of the new features introduced in the Intel Broadwell CPUs.

Tracking this issue further down in the Open-iSCSI sources, i ended up in the function nl_process_handle_thread() in iscsiuio/src/unix/nic_nl.c and specifically in the following code section:

iscsiuio/src/unix/nic_nl.c
474 /* NIC specific nl processing thread */
475 void *nl_process_handle_thread(void *arg)
476 {
[...]
483         while (!event_loop_stop) {
484                 char *data = NULL;
485
486                 rc = pthread_cond_wait(&nic->nl_process_cond,
487                                        &nic->nl_process_mutex);
488                 if (rc != 0) {
489                         LOG_ERR("Fatal error in NL processing thread "
490                                 "during wait[%s]", strerror(rc));
491                         break;
492                 }
[...]
499                 pthread_mutex_unlock(&nic->nl_process_mutex);
[...]

Debugging this revealed that the call to pthread_cond_wait() from the above GDB backtrace output of thread number 4 is the one from line 486 in the above code snippet.

Looking at the pthread_cond_wait() manpage showed the following constraint for its proper use:

[…]
The pthread_cond_timedwait() and pthread_cond_wait() functions shall
block on a condition variable. They shall be called with mutex locked
by the calling thread or undefined behavior results.
[…]

Although not shown in the above GDB output, this would on occasion – and again, probably just by chance – work on the first pass of the loop. At the end of the loop, at line 499 in the above code snippet, the mutex is then unlocked. Thus the cited constraint from the pthread_cond_wait() manpage is no longer met on the subsequent passes of the loop. On Intel E5 v3 (aka Haswell) CPUs, this seemed to be tolerated and without any impact. But on Intel E5 v4 (aka Broadwell) – and probably other CPUs implementing HLE and RTM – this causes the observed segmentation fault.

In order to verify my analysis and test this theory, i added a call to pthread_mutex_lock() right before the call to pthread_cond_wait() in line 486. The resulting change is available in the Git commit 9f770f9e of my Open-iSCSI fork on Github and also shown in the following patch:

nic_nl.c.patch
diff --git a/iscsiuio/src/unix/nic_nl.c b/iscsiuio/src/unix/nic_nl.c
index 391003f..581ddb0 100644
--- a/iscsiuio/src/unix/nic_nl.c
+++ b/iscsiuio/src/unix/nic_nl.c
@@ -483,6 +483,7 @@ void *nl_process_handle_thread(void *arg)
        while (!event_loop_stop) {
                char *data = NULL;
 
+               pthread_mutex_lock(&nic->nl_process_mutex);
                rc = pthread_cond_wait(&nic->nl_process_cond,
                                       &nic->nl_process_mutex);
                if (rc != 0) {

Posting this on the Open-iSCSI mailing list lead to this discussion. The suggestion from there was an additional change to the error handling code of nl_process_handle_thread() starting at line 488 in the above above code snippet. This adds proper handling of the locked mutex in case the loop is left due to an error returned from the call to pthread_cond_wait(). The resulting additional change is available in the Git commit 4191ca6b of my Open-iSCSI fork on Github. The following patch shows the summarised code changes:

nic_nl.c.patch
diff --git a/iscsiuio/src/unix/nic_nl.c b/iscsiuio/src/unix/nic_nl.c
index 391003f..1a920c7 100644
--- a/iscsiuio/src/unix/nic_nl.c
+++ b/iscsiuio/src/unix/nic_nl.c
@@ -483,9 +483,11 @@ void *nl_process_handle_thread(void *arg)
        while (!event_loop_stop) {
                char *data = NULL;
 
+               pthread_mutex_lock(&nic->nl_process_mutex);
                rc = pthread_cond_wait(&nic->nl_process_cond,
                                       &nic->nl_process_mutex);
                if (rc != 0) {
+                       pthread_mutex_unlock(&nic->nl_process_mutex);
                        LOG_ERR("Fatal error in NL processing thread "
                                "during wait[%s]", strerror(rc));
                        break;

With those two small changes to the sources of Open-iSCSIs iscsiuio, the iSCSI connections via the Broadcom BCM57810S iSOE do now work flawlessly even on newer Intel E5 v4 (aka Broadwell) based systems. Hopefully the original authors of the iscsiuio code at Broadcom/QLogic will also take part in the discussion and provide their feedback on the proposed code changes too.

1) , 2) formerly Broadcom

// Compatibility of DokuWiki "Detritus" and the Taratasy Template

The DokuWiki Taratasy Template offers a rather nice design, which i wanted to use for my new Wiki. This Wiki will be based upon the currently stable DokuWiki 2015-08-10a (aka “Detritus”) release. The Taratasy templates site currently shows the compatibility to the DokuWiki “Detritus” release as “unknown”, see the following screenshot:

Taratasy template compatibility status

Installing the Taratasy templates in an otherwise working DokuWiki 2015-08-10a instance lead to the rather broken appearance shown in the following example screenshot:

Screenshot of the Taratasy template in a DokuWiki 2015-08-10a (aka "Detritus") release

The colors and fonts are way off, the navigation bar at the head and bottom of the page as well as the tools menu are broken. Thus, my immediate suspicion for the root cause of the issues was in the direction of the CSS definitions provided by the Taratasy template.

Digging through the webserver and PHP logs as well as double checking the webservers rewrite rules showed no noticable indication of a problem with the DokuWiki instance itself. Switching to another template like e.g. Arctic also showed that the DokuWiki 2015-08-10a instance itself worked flawlessly. So this was not a general issue with my setup of the current DokuWiki release. The other way around, using the Taratasy template with an older DokuWiki 2014-09-29d (aka “Hrun”) release, also worked as expected and without any issues. So the problem was narrowed down to some change introduced between those two DokuWiki releases.

With the suspicion about the CSS definitions mentioned above, i tried to make some changes in the Taratasy templates style.ini file. Those changes immediately effected the layout of the re-rendered page. On the other hand, making some changes in the Taratasy templates style.local.ini file had no effect on the subsequently re-rendered page.

Browsing the commit history of the DokuWiki GitHub repository for the 2015-08-10a release and searching for commits with a reference to style.local.ini, i stubled upon the commit 656e584. The associated changes finally removed a feature which had already been marked as deprecated in previous DokuWiki releases. This feature was to include user defined CSS formatting definitions through the file style.local.ini in a templates main directory and merge these definitions with the ones from the templates style.ini file into the final CSS definitions. For several DokuWiki releases the user defined CSS formatting definitions are now supposed to be placed in the file DOKU_CONF.“tpl/$tpl/style.ini” instead.

With the source of the issue now known, my quick'n'dirty solution was to merge the CSS definitions from the templates style.local.ini into the templates style.ini and afterwards drop the now redundant style.local.ini. This was just easier to accomplish given my particular DokuWiki directory setup. Another solution would have been to create a directory DOKU_CONF/tpl/taratasy/ and move the style.local.ini into it as DOKU_CONF/tpl/taratasy/style.ini.

The modified template sources can be found in my Taratasy repository on GitHub

In conclusion, the following image shows an example screenshot of the same DokuWiki 2015-08-10a instance as above after the patch to the Taratasy template has been applied:

Screenshot of the patched Taratasy template in a DokuWiki 2015-08-10a (aka "Detritus") release

This website uses cookies for visitor traffic analysis. By using the website, you agree with storing the cookies on your computer.More information