In the previous article Debugging Segfaults in Open-iSCSIs iscsiuio on Intel Broadwell i mentioned using a backported version of Open-iSCSI on Debian 8 (“Jessie”). This new post describes the backport and the changes provided by it in greater detail. All the changes to the original Debian package from “unstable” (“Sid”) can be found in my Debian Open-iSCSI Packaging repository on GitHub.

Starting point was a clone of the Debian Open-iSCSI Packaging repository at Git commit df150d90. Mind though, that in the meantime between creating the backport and writing this, the Debian Open-iSCSI maintainers have been busy and a more recent version of the Debian Open-iSCSI package from “unstable” (“Sid”) is now available.

Within this particular version of the Debian Open-iSCSI package, i first enabled the build of Open-iSCSIs iscsiuio. On the one hand, this was done in order to ensure that the iscsiuio code would successfully build even at this old level of the Open-iSCSI code. On the other hand, this would be used as a differentiator for any issues surfacing later on, after the move to the more recent upstream Open-iSCSI sources, indicating the root cause of those would then solely be with the newer upstream version of Open-iSCSI. Some integration into the general system environment was also added at this point. In detail the changes were:

• Git commit 32c96e6c removes the Debian patch 05-disable-iscsiuio.patch which disables the build of iscsiuio.

• Git commit 984344a1 enables the build of iscsiuio, extends the cleanup build targets and adds iscsiuio to the dh_systemd build targets.

• Git commit 89d845a9 adds the results from the successful build – the iscsiuio binary, the iscsiuio manual page, a readme file and a logrotate configuration file – to the Debian package. It also adds the kernel modules bnx2i and cnic to the list of kernel modules to be loaded at installation time.

• Git commit 89195bbe adds the systemd service and socket unit files for iscsiuio. Those files have been taken from this discussion on the Open-iSCSI mailing list and have slightly been altered.

With the above changes a intermediary package was build for testing purposes. During the following tests sometimes all currently mounted filesystems – even those distinctly not based on iSCSI volumes – would suddenly be unmounted. For some filesystems this would succeed, for others, like e.g. the /var and the root filesystem, this would fail due to them being currently in use. The issue particularly occured while stopping the open-iscsi service either via its regular init script or via its systemd service. This is usually done at system shutdown or during uninstall of the Open-iSCSI package. Tracking down the root cause of this issue led to an unhandled case in the umountiscsi.sh script, which is called while stopping the open-iscsi service. Specifically, the following code section is responsible for the observed behaviour:

debian/extra/umountiscsi.sh

256    if [ $HAVE_LVM -eq 1 ] ; then
257        # Look for all LVM volume groups that have a backing store
258        # on any iSCSI device we found. Also, add $LVMGROUPS set in
259        # /etc/default/open-iscsi (for more complicated stacking
260        # configurations we don't automatically detect).
261        for _vg in $(cd /dev ; $PVS --noheadings -o vg_name $iscsi_disks $iscsi_partitions $iscsi_multipath_disks $iscsi_multipath_partitions 2>/dev/null) $LVMGROUPS ; do
262            add_to_set iscsi_lvm_vgs "$_vg"
263        done

The heuristic of the umountiscsi.sh script are trying to identify iSCSI based disk devices which are valid candidates for proper deactivation upon system shutdown. It turned out that in LVM based setups where there are currently no iSCSI based disk devices present, the variables $iscsi_disks, $iscsi_partitions, $iscsi_multipath_disks and $iscsi_multipath_partitions are left empty by the scripts logic. In line 261 in the above code snippet, this leads to a call to the pvs --noheadings -o vg_name command without any additional arguments limiting its output of volume groups. Hence, the returned output is instead a complete list of all volume groups currently present on the system. Based on this list, the associated logical volumes for each volume group are determined and added to the list of devices to be unmounted. Finally all devices in this list are actually unmounted.

Without making too invasive changes to the script logic of umountiscsi.sh a quick'n'dirty solution was to introduce a check before the call to pvs which would determine whether the variables $iscsi_disks, $iscsi_partitions, $iscsi_multipath_disks and $iscsi_multipath_partitions are all empty. If this is the case, the call to pvs is simply skipped. The following patch shows the necessary code changes which are also available in Git commit 5118af7f:

umountiscsi.sh.patch

diff --git a/debian/extra/umountiscsi.sh b/debian/extra/umountiscsi.sh
index 1206fa1..485069c 100755
--- a/debian/extra/umountiscsi.sh
+++ b/debian/extra/umountiscsi.sh
@@ -258,9 +258,11 @@ enumerate_iscsi_devices() {
                # on any iSCSI device we found. Also, add $LVMGROUPS set in
                # /etc/default/open-iscsi (for more complicated stacking
                # configurations we don't automatically detect).
-               for _vg in $(cd /dev ; $PVS --noheadings -o vg_name $iscsi_disks $iscsi_partitions $iscsi_multipath_disks $iscsi_multipath_partitions 2>/dev/null) $LVMGROUPS ; do
-                       add_to_set iscsi_lvm_vgs "$_vg"
-               done
+               if [ -n "$iscsi_disks" -o -n "$iscsi_partitions" -o -n "$iscsi_multipath_disks" -o -n "$iscsi_multipath_partitions" ]; then
+                   for _vg in $(cd /dev ; $PVS --noheadings -o vg_name $iscsi_disks $iscsi_partitions $iscsi_multipath_disks $iscsi_multipath_partitions 2>/dev/null) $LVMGROUPS ; do
+                           add_to_set iscsi_lvm_vgs "$_vg"
+                   done
+               fi
 
                # $iscsi_lvm_vgs is now unique list
                for _vg in $iscsi_lvm_vgs ; do

After this was fixed, the last step was to finally move to the more recent upstream Open-iSCSI sources. In detail the changes in this last step were:

• Git commit f5ab51ff moves the code to version 2.0.873+git1.1dfb88a4 which is based upon the upstream Git commit 1dfb88a4. This is the last commit before the externalization of the Open-iSNS library. Since i didn't want to also backport the Open-iSNS packages from Debian “unstable” (“Sid”), i decided to just skip the next two upstream commits 76832662 and c6d1117b and stick with the locally delivered Open-iSNS library.

• Git commit 8c1e6974 removes the local Debian patches 01_spelling-errors-and-manpage-hyphen-fixes.patch, 02_make-iscsistart-a-dynamic-binary.patch and 03_respect-build-flags.patch which have already been merged into the more recent upstream Open-iSCSI sources. The remaining local Debian patches were renamed and reordered to 01_fix_iscsi_path.patch, 02_var-lock_var-run_transition.patch and 03_makefile_reproducibility_issues.patch. A whole bunch of new patches named {04,05,06,07,08,09,10,11,12,13,14,15}_upstream_git_commit_<Git commit ID>.patch were added in order to bring the sources up to the – by then most recent – upstream Git commit 0fa43f29.

• Git commit d051dece removes some files from the Debian package, which were dynamically generated during the build of iscsiuio.

• Finally Git commit 0fabb948 deals with the issue described in Debugging Segfaults in Open-iSCSIs iscsiuio on Intel Broadwell.

With the steps and changes described above, a backported version of Open-iSCSI using its most recent sources was created as a package for Debian 8 (“Jessie”). This package also supports offloaded iSCSI connections via the Broadcom BCM577xx and BCM578xx iSOEs with the use of iscsiuio. The package has been in production use for over a month now and no major issues – neither with the newer upstream Open-iSCSI sources, nor with use of Broadcom BCM577xx and BCM578xx iSOEs through iscsiuio – have emerged so far.