Integration of Dell EqualLogic PS-Series Storages with RANCID

Adding support for Dell EqualLogic PS-Series storage arrays to version 3.5.1 of the popular, open source switch and router configuration management tool RANCID.

For the impatient and TL;DR here are the extensions to RANCID for the management of Dell EqualLogic PS-Series storage arrays:

Login script for Dell EqualLogic PS-Series storage arrays
Perl module to generate, process and save the configuration of Dell EqualLogic PS-Series storage arrays

The sources are to be found in my RANCID repository on GitHub

RANCID has, in its current version 3.5.1, support for a large variety of network devices like routers, switches, load-balancers, etc. Unfortunately there is currently little or no support for the management of storage devices, even though a lot of them offer a command line interface which can be used by RANCID.

Although there probably are a couple of reasons for this, i suppose this is largely due to the fact that network and storage admins are – in most organizations – still in different groups, each with their own set of management and support tools. With RANCID originating from the realm of network administration, probably only few storage admins know about this very valuable tool to begin with. There is probably also very little transfer over from the position of network administrator into the area of storage administration and thus a limited amount of knowledge transfer between those two fields.

This blog post describes how to extend and configure RANCID in order to add support for Dell EqualLogic PS-Series storage arrays. The extensions are based on the – at the time of writing – current version 3.5.1 of RANCID. RANCID can either be build from source or be installed pre-packaged e.g. from the backports repository of Debian stable (jessie). Basically, the extension to RANCID consist of only two files:

An expect login script for Dell EqualLogic PS-Series storage arrays (eqllogin in my RANCID repository on GitHub). This script provides the basic access to the command line interface of a EqualLogic system. It can be used for interactive sessions as well as scripted access. It is used by the following equallogic.pm RANCID perl module.
A perl module to generate, process and save the configuration of Dell EqualLogic PS-Series storage arrays (equallogic.pm in my RANCID repository on GitHub), with the use of the aforementioned expect login script. The save-config -verbose command is used on the Dell EqualLogic PS-Series storage arrays in order to generate and print the array configuration. The processing part currently just removes some informational messages from the output of the save-config -verbose command. If the environment variable FILTER_PWDS is set in rancid.conf, the iSCSI CHAP users as well as the iSCSI target authorization usernames and passwords will be removed before saving the configuration. If the environment variable NOCOMMSTR is set in rancid.conf, the SNMP Community strings will be removed before saving the configuration.

Besides those two extensions only a small change to /etc/rancid/rancid.types.conf, as well as the standard RANCID configuration of a new device group and device is necessary. See the following full step-by-step configuration example for Dell EqualLogic PS-Series storage arrays:

Add the backports repository of Debian stable (jessie) to the APT configuration:

root@host:~$ echo 'deb http://http.debian.net/debian jessie-backports main non-free contrib' >> /etc/apt/sources.list.d/jessie-backports.list
root@host:~$ apt-get update

Install RANCID v3.5.1 from the backports repository of Debian stable (jessie):

root@host:~$ apt-get -y install rancid/jessie-backports

Reading package lists... Done
Building dependency tree       
Reading state information... Done
Selected version '3.5.1-1~bpo8+1' (Debian Backports:jessie-backports [amd64]) for 'rancid'
The following extra packages will be installed:
  expect tcl-expect
The following NEW packages will be installed:
  expect rancid tcl-expect
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 511 kB of archives.
After this operation, 2,178 kB of additional disk space will be used.
[...]

Optional: In case Subversion should be used as a revision control system (RCS) to store the switch configuration, install it:

root@host:~$ apt-get -y install subversion

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libapr1 libaprutil1 libldap-2.4-2 libsasl2-2 libsasl2-modules libsasl2-modules-db libserf-1-1 libsvn1
Suggested packages:
  libsasl2-modules-otp libsasl2-modules-ldap libsasl2-modules-sql libsasl2-modules-gssapi-mit libsasl2-modules-gssapi-heimdal subversion-tools db5.3-util patch
The following NEW packages will be installed:
  libapr1 libaprutil1 libldap-2.4-2 libsasl2-2 libsasl2-modules libsasl2-modules-db libserf-1-1 libsvn1 subversion
0 upgraded, 9 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,723 kB of archives.
After this operation, 9,683 kB of additional disk space will be used.
[...]

Download the login script for Dell EqualLogic PS-Series storage arrays and store it under the path /usr/lib/rancid/bin/.
Download the Perl module to process and save the configuration of Dell EqualLogic PS-Series storage arrays and store it under the path /usr/share/perl5/rancid/.
Edit the global RANCID configuration:
```
root@host:~$ vi /etc/rancid/rancid.conf
```
Select the RCS (CVS, SVN or Git) of your choice. In this example SVN is used:
```
RCSSYS=svn; export RCSSYS
```
Define a name for your Dell EqualLogic device group in the LIST_OF_GROUPS configuration variable. In this example we'll use the name dell-storage:
```
LIST_OF_GROUPS="dell-storage"; export LIST_OF_GROUPS
```
Create the cloginrc configuration file, which will contain the login information for your Dell EqualLogic PS-Series devices and some default values:
```
root@host:~$ touch /etc/rancid/cloginrc
root@host:~$ chmod 660 /etc/rancid/cloginrc
root@host:~$ chown root:rancid /etc/rancid/cloginrc
root@host:~$ vi /etc/rancid/cloginrc
```
Example:
```
add user        dell-eql-1      dell-user
add password    dell-eql-1      <login-passwort>

[...]

add user        *               <default-user>
add password    *               <default-login-passwort>
add method      *               ssh
```
For the device named dell-eql-1 login as user dell-user with the password <login-passwort>.

For all other systems, login as user <default-user> with the password <default-login-passwort>. The login method for all systems is via SSH.

Since the cloginrc configuration file is parsed in a first-match fashion, the default values must always be at the bottom of the file.
Add a new device type for Dell EqualLogic PS-Series storage arrays to the RANCID configuration. See man router.db and /etc/rancid/rancid.types.conf. In this example and in the general case of Dell Dell EqualLogic PS-Series storage arrays the name of the device type is equallogic:
```
root@host:~$ vi /etc/rancid/rancid.types.conf
```
Here we set the login script to be used to the new eqllogin. The postprocessing script is set rancid -t equallogic in order to call the new Perl module equallogic, which will do the actual processing. The command to be issued on the Dell EqualLogic device is set to save-config -verbose. The -verbose part is essential here, otherwise the configuration of the device will only be saved to a file on the Dell EqualLogic device and not be printed to the terminal:
```
equallogic;login;eqllogin
equallogic;script;rancid -t equallogic
equallogic;module;equallogic
equallogic;inloop;equallogic::inloop
equallogic;command;equallogic::SaveConfiguration;save-config -verbose
```

Change to the user rancid:

root@host:~$ su - rancid

Create a symbolic link to the login configuration previously created in /etc/rancid/:
```
rancid@host:~$ ln -s /etc/rancid/cloginrc /var/lib/rancid/.cloginrc
```

Initialize the directory structure for the RCS (CVS, SVN or Git) selected above. This will automatically be done for each device group configured in the LIST_OF_GROUPS configuration variable. The example shown here only creates the directory structure for the device group dell-storage defined above:

rancid@host:~$ /usr/lib/rancid/bin/rancid-cvs
Committed revision 1.
Checked out revision 1.
Updating '.':
At revision 1.
A         configs
Adding         configs

Committed revision 2.
A         router.db
Adding         router.db
Transmitting file data .
Committed revision 3.

rancid@host:~$ find /var/lib/rancid/dell-storage/
/var/lib/rancid/dell-storage
/var/lib/rancid/dell-storage/configs
/var/lib/rancid/dell-storage/router.db
/var/lib/rancid/dell-storage/routers.all
/var/lib/rancid/dell-storage/routers.down
/var/lib/rancid/dell-storage/routers.up
/var/lib/rancid/dell-storage/.svn
/var/lib/rancid/dell-storage/.svn/entries
/var/lib/rancid/dell-storage/.svn/format
/var/lib/rancid/dell-storage/.svn/pristine
/var/lib/rancid/dell-storage/.svn/pristine/da
/var/lib/rancid/dell-storage/.svn/pristine/da/da39a3ee5e6b4b0d3255bfef95601890afd80709.svn-base
/var/lib/rancid/dell-storage/.svn/tmp
/var/lib/rancid/dell-storage/.svn/wc.db

Add Dell EqualLogic storage devices by their hostname to the configuration file router.db of the corresponding device group:
```
rancid@host:~$ vi /var/lib/rancid/dell-storage/router.db
```
In this example the device group dell-storage, the device type equallogic and the system dell-eql-1:
```
dell-eql-1;equallogic;up;A comment describing the system dell-eql-1
```

Perform a login test with the previously configured new login script eqllogin for Dell EqualLogic devices on the newly defined system dell-eql-1. The following example output shows the steps that should automatically be performed by the eqllogin expect script. No manual intervention should be necessary.

rancid@host:~$ /usr/lib/rancid/bin/eqllogin dell-eql-1
spawn ssh -x -l grpadmin dell-eql-1
The authenticity of host 'dell-eql-1 (<ip address>)' can't be established.
RSA key fingerprint is <rsa key fingerprint>
Are you sure you want to continue connecting (yes/no)?  yes
Host dell-eql-1 added to the list of known hosts.
Warning: Permanently added 'dell-eql-1,<ip address>' (RSA) to the list of known hosts.
grpadmin@dell-eql-1's password: 
Last login: Thu Dec  8 22:32:39 2016 from <ip address> on ttyp1
 

           Welcome to Group Manager

        Copyright 2001-2015 Dell Inc.



dell-eql-1-grp>

Finish the login test by manually logging out of the system:

dell-eql-1-grp> logout
Do you really want to logout? (y/n) [n]y

dell-eql-1-grp> Connection to dell-eql-1 closed.

Manually perform an initial RANCID run to make sure everything works as expected:
```
rancid@host:~$ rancid-run
```
If everything ran successfully, there should now be a file /var/lib/rancid/dell-storage/configs/dell-eql-1 containing the output of the command save-config -verbose for the system dell-eql-1.

Create the email aliases necessary for the proper delivery of the emails generated by RANCID. Again in this example for the device group dell-storage:
```
root@host:~$ vi /etc/aliases
```
```
rancid-dell-storage:       <email>@<domain>
rancid-admin-dell-storage: <email>@<domain>
```
Recreate your aliases DB. In case postfix is used as an MTA:
```
root@host:~$ postalias /etc/aliases
```
Enable the RANCID cron jobs. Adjust the execution times and intervals according to your needs:
```
root@host:~$ vi /etc/cron.d/rancid
```

Some final words: The contents of the directories /var/lib/rancid/<device group>/ and /var/lib/rancid/<device group>/configs/ are maintained in the RCS – CVS, SVN or Git – of your choice. You can operate on those directories with the usual commands of the selected RCS. There are also some really nice and intuitive web frontends to the RCS of choice. For me, the combination of SVN as RCS and WebSVN as a web frontend worked out very well.