bityard Blog

2016-12-13 // Integration of Dell EqualLogic PS-Series Storages with RANCID

Adding support for Dell EqualLogic PS-Series storage arrays to version 3.5.1 of the popular, open source switch and router configuration management tool RANCID.

For the impatient and TL;DR here are the extensions to RANCID for the management of Dell EqualLogic PS-Series storage arrays:

Login script for Dell EqualLogic PS-Series storage arrays
Perl module to generate, process and save the configuration of Dell EqualLogic PS-Series storage arrays

The sources are to be found in my RANCID repository on GitHub

RANCID has, in its current version 3.5.1, support for a large variety of network devices like routers, switches, load-balancers, etc. Unfortunately there is currently little or no support for the management of storage devices, even though a lot of them offer a command line interface which can be used by RANCID.

Although there probably are a couple of reasons for this, i suppose this is largely due to the fact that network and storage admins are – in most organizations – still in different groups, each with their own set of management and support tools. With RANCID originating from the realm of network administration, probably only few storage admins know about this very valuable tool to begin with. There is probably also very little transfer over from the position of network administrator into the area of storage administration and thus a limited amount of knowledge transfer between those two fields.

This blog post describes how to extend and configure RANCID in order to add support for Dell EqualLogic PS-Series storage arrays. The extensions are based on the – at the time of writing – current version 3.5.1 of RANCID. RANCID can either be build from source or be installed pre-packaged e.g. from the backports repository of Debian stable (jessie). Basically, the extension to RANCID consist of only two files:

Besides those two extensions only a small change to /etc/rancid/rancid.types.conf, as well as the standard RANCID configuration of a new device group and device is necessary. See the following full step-by-step configuration example for Dell EqualLogic PS-Series storage arrays:

  • Add the backports repository of Debian stable (jessie) to the APT configuration: 

    root@host:~$ echo 'deb http://http.debian.net/debian jessie-backports main non-free contrib' >> /etc/apt/sources.list.d/jessie-backports.list
root@host:~$ apt-get update

  • Install RANCID v3.5.1 from the backports repository of Debian stable (jessie): 

    root@host:~$ apt-get -y install rancid/jessie-backports

Reading package lists... Done
Building dependency tree       
Reading state information... Done
Selected version '3.5.1-1~bpo8+1' (Debian Backports:jessie-backports [amd64]) for 'rancid'
The following extra packages will be installed:
  expect tcl-expect
The following NEW packages will be installed:
  expect rancid tcl-expect
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 511 kB of archives.
After this operation, 2,178 kB of additional disk space will be used.
[...]

    Optional: In case Subversion should be used as a revision control system (RCS) to store the switch configuration, install it: 

    root@host:~$ apt-get -y install subversion

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libapr1 libaprutil1 libldap-2.4-2 libsasl2-2 libsasl2-modules libsasl2-modules-db libserf-1-1 libsvn1
Suggested packages:
  libsasl2-modules-otp libsasl2-modules-ldap libsasl2-modules-sql libsasl2-modules-gssapi-mit libsasl2-modules-gssapi-heimdal subversion-tools db5.3-util patch
The following NEW packages will be installed:
  libapr1 libaprutil1 libldap-2.4-2 libsasl2-2 libsasl2-modules libsasl2-modules-db libserf-1-1 libsvn1 subversion
0 upgraded, 9 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,723 kB of archives.
After this operation, 9,683 kB of additional disk space will be used.
[...]

  • Download the login script for Dell EqualLogic PS-Series storage arrays and store it under the path /usr/lib/rancid/bin/.

  • Download the Perl module to process and save the configuration of Dell EqualLogic PS-Series storage arrays and store it under the path /usr/share/perl5/rancid/.

  • Edit the global RANCID configuration: 

    root@host:~$ vi /etc/rancid/rancid.conf

    Select the RCS (CVS, SVN or Git) of your choice. In this example SVN is used: 

    RCSSYS=svn; export RCSSYS

    Define a name for your Dell EqualLogic device group in the LIST_OF_GROUPS configuration variable. In this example we'll use the name dell-storage: 

    LIST_OF_GROUPS="dell-storage"; export LIST_OF_GROUPS

  • Create the cloginrc configuration file, which will contain the login information for your Dell EqualLogic PS-Series devices and some default values: 

    root@host:~$ touch /etc/rancid/cloginrc
root@host:~$ chmod 660 /etc/rancid/cloginrc
root@host:~$ chown root:rancid /etc/rancid/cloginrc
root@host:~$ vi /etc/rancid/cloginrc

    Example: 

    add user        dell-eql-1      dell-user
add password    dell-eql-1      <login-passwort>

[...]

add user        *               <default-user>
add password    *               <default-login-passwort>
add method      *               ssh

    For the device named dell-eql-1 login as user dell-user with the password <login-passwort>.

    For all other systems, login as user <default-user> with the password <default-login-passwort>. The login method for all systems is via SSH.

    Since the cloginrc configuration file is parsed in a first-match fashion, the default values must always be at the bottom of the file.

  • Add a new device type for Dell EqualLogic PS-Series storage arrays to the RANCID configuration. See man router.db and /etc/rancid/rancid.types.conf. In this example and in the general case of Dell Dell EqualLogic PS-Series storage arrays the name of the device type is equallogic: 

    root@host:~$ vi /etc/rancid/rancid.types.conf

    Here we set the login script to be used to the new eqllogin. The postprocessing script is set rancid -t equallogic in order to call the new Perl module equallogic, which will do the actual processing. The command to be issued on the Dell EqualLogic device is set to save-config -verbose. The -verbose part is essential here, otherwise the configuration of the device will only be saved to a file on the Dell EqualLogic device and not be printed to the terminal: 

    equallogic;login;eqllogin
equallogic;script;rancid -t equallogic
equallogic;module;equallogic
equallogic;inloop;equallogic::inloop
equallogic;command;equallogic::SaveConfiguration;save-config -verbose

  • Change to the user rancid: 

    root@host:~$ su - rancid

    • Create a symbolic link to the login configuration previously created in /etc/rancid/: 

      rancid@host:~$ ln -s /etc/rancid/cloginrc /var/lib/rancid/.cloginrc

    • Initialize the directory structure for the RCS (CVS, SVN or Git) selected above. This will automatically be done for each device group configured in the LIST_OF_GROUPS configuration variable. The example shown here only creates the directory structure for the device group dell-storage defined above: 

      rancid@host:~$ /usr/lib/rancid/bin/rancid-cvs
Committed revision 1.
Checked out revision 1.
Updating '.':
At revision 1.
A         configs
Adding         configs

Committed revision 2.
A         router.db
Adding         router.db
Transmitting file data .
Committed revision 3.

rancid@host:~$ find /var/lib/rancid/dell-storage/
/var/lib/rancid/dell-storage
/var/lib/rancid/dell-storage/configs
/var/lib/rancid/dell-storage/router.db
/var/lib/rancid/dell-storage/routers.all
/var/lib/rancid/dell-storage/routers.down
/var/lib/rancid/dell-storage/routers.up
/var/lib/rancid/dell-storage/.svn
/var/lib/rancid/dell-storage/.svn/entries
/var/lib/rancid/dell-storage/.svn/format
/var/lib/rancid/dell-storage/.svn/pristine
/var/lib/rancid/dell-storage/.svn/pristine/da
/var/lib/rancid/dell-storage/.svn/pristine/da/da39a3ee5e6b4b0d3255bfef95601890afd80709.svn-base
/var/lib/rancid/dell-storage/.svn/tmp
/var/lib/rancid/dell-storage/.svn/wc.db

    • Add Dell EqualLogic storage devices by their hostname to the configuration file router.db of the corresponding device group: 

      rancid@host:~$ vi /var/lib/rancid/dell-storage/router.db

      In this example the device group dell-storage, the device type equallogic and the system dell-eql-1: 

      dell-eql-1;equallogic;up;A comment describing the system dell-eql-1

    • Perform a login test with the previously configured new login script eqllogin for Dell EqualLogic devices on the newly defined system dell-eql-1. The following example output shows the steps that should automatically be performed by the eqllogin expect script. No manual intervention should be necessary. 

      rancid@host:~$ /usr/lib/rancid/bin/eqllogin dell-eql-1
spawn ssh -x -l grpadmin dell-eql-1
The authenticity of host 'dell-eql-1 (<ip address>)' can't be established.
RSA key fingerprint is <rsa key fingerprint>
Are you sure you want to continue connecting (yes/no)?  yes
Host dell-eql-1 added to the list of known hosts.
Warning: Permanently added 'dell-eql-1,<ip address>' (RSA) to the list of known hosts.
grpadmin@dell-eql-1's password: 
Last login: Thu Dec  8 22:32:39 2016 from <ip address> on ttyp1
 

           Welcome to Group Manager

        Copyright 2001-2015 Dell Inc.



dell-eql-1-grp>

    • Finish the login test by manually logging out of the system: 

      dell-eql-1-grp> logout
Do you really want to logout? (y/n) [n]y

dell-eql-1-grp> Connection to dell-eql-1 closed.

    • Manually perform an initial RANCID run to make sure everything works as expected: 

      rancid@host:~$ rancid-run

      If everything ran successfully, there should now be a file /var/lib/rancid/dell-storage/configs/dell-eql-1 containing the output of the command save-config -verbose for the system dell-eql-1.

  • Create the email aliases necessary for the proper delivery of the emails generated by RANCID. Again in this example for the device group dell-storage: 

    root@host:~$ vi /etc/aliases
    rancid-dell-storage:       <email>@<domain>
rancid-admin-dell-storage: <email>@<domain>

    Recreate your aliases DB. In case postfix is used as an MTA: 

    root@host:~$ postalias /etc/aliases

  • Enable the RANCID cron jobs. Adjust the execution times and intervals according to your needs: 

    root@host:~$ vi /etc/cron.d/rancid

Some final words: The contents of the directories /var/lib/rancid/<device group>/ and /var/lib/rancid/<device group>/configs/ are maintained in the RCS – CVS, SVN or Git – of your choice. You can operate on those directories with the usual commands of the selected RCS. There are also some really nice and intuitive web frontends to the RCS of choice. For me, the combination of SVN as RCS and WebSVN as a web frontend worked out very well.

0 Comments | 2016-12-13 written by Frank Fegert (XING) | Permanentlink
Tags:

2015-10-01 // Integration of Dell PowerConnect M-Series Switches with RANCID

An – almost – out of the box integration of Dell PowerConnect M-Series switches (specifically M6348 and M8024-k) with version 3.2 of the popular, open source switch and router configuration management tool RANCID.

RANCID has, in the previous version 2.3, already been able to integrate Dell PowerConnect switches through the use of the custom ''dlogin'' and ''drancid'' script addons. There are already several howtos on the net on how to use those addons and also slightly tweaked versions of them here and here.

With version 3.2 of RANCID, either build from source or installed pre-packaged e.g. from Debian testing (stretch), everything has gotten much more straight forward. The only modification needed now is a small patch to the script ''/usr/lib/rancid/bin/srancid'', adapting its output postprocessing part to those particular switch models. This modification is necessary in order to remove constantly changing output – in this case uptime and temperature information – from the commands show version and show system, which are issued by srancid for those particular switch models. For the purpose of clarification, here are output samples from the show version: 

switch1# show version

System Description................ Dell Ethernet Switch
System Up Time.................... 90 days, 04h:48m:41s
System Contact.................... <contact email>
System Name....................... <system name>
System Location................... <location>
Burned In MAC Address............. F8B1.566E.4AFB
System Object ID.................. 1.3.6.1.4.1.674.10895.3041
System Model ID................... PCM8024-k
Machine Type...................... PowerConnect M8024-k

unit image1      image2      current-active next-active
---- ----------- ----------- -------------- --------------
1    5.1.3.7     5.1.8.2     image2         image2
2    5.1.3.7     5.1.8.2     image2         image2

and show system: 

switch1# show system

System Description: Dell Ethernet Switch
System Up Time: 90 days, 04h:48m:19s
System Contact: <contact email>
System Name: <system name>
System Location: <location>
Burned In MAC Address: F8B1.566E.4AFB
System Object ID: 1.3.6.1.4.1.674.10895.3041
System Model ID: PCM8024-k
Machine Type: PowerConnect M8024-k
Temperature Sensors:

Unit     Description       Temperature    Status
                            (Celsius)
----     -----------       -----------    ------
1        System            39             Good
2        System            39             Good

Power Supplies:

Unit  Description    Status

----  -----------  -----------
NA        NA            NA
NA        NA            NA

commands issued on a stacked pair of Dell PowerConnect M8024-k switches. Without the patch to srancid, the lines starting with System Up Time as well as the lines for each temperature sensor unit would always trigger a configuration change in RANCID, even if there was no real change in configuration. The provided patch adds handling and the subsequent removal of those ever-changing values from the configuration which is stored in the RCS used by RANCID.

Aside from this, the only pitfall with Dell PowerConnect switch models is to follow your intuition and use the RANCID device type dell (see man router.db). This device type is intended to be used with D-Link switches OEMed by Dell and will not work with Dell PowerConnect switch models. The correct device type for Dell PowerConnect switch models is smc, though.

For the sake of completeness, here a full step-by-step configuration example for Dell PowerConnect M6348 and M8024-k switches:

  • Add the Debian testing (stretch) packet sources to the APT configuration: 

    root@host:~$ echo 'deb http://ftp.debian.org:80/debian testing main contrib non-free' >> /etc/apt/sources.list.d/testing.list
root@host:~$ apt-get update

  • Install RANCID v3.2.x from Debian testing (stretch): 

    root@host:~$ apt-get -y install rancid/testing

    Optional: In case Subversion should be used as a revision control system (RCS) to store the switch configuration, install it: 

    root@host:~$ apt-get -y install subversion

  • Download and apply the patch to the script ''/usr/lib/rancid/bin/srancid'' for proper handling of the system information and configuration output of Dell PowerConnect M6348 and M8024-k switches:

    srancid.patch
    --- /usr/lib/rancid/bin/srancid.orig	2015-05-07 00:00:19.000000000 +0200
+++ /usr/lib/rancid/bin/srancid	2015-09-24 16:02:23.379156524 +0200
@@ -49,6 +49,8 @@
 #
 # Code tested and working fine on these models:
 #
+#	DELL PowerConnect M8024 / M8024-k
+#	DELL PowerConnect M6348
 #	DELL PowerConnect 62xx
 #	DELL 34xx (partially; configuration is incomplete)
 #
@@ -174,6 +176,7 @@
 sub Dir {
     print STDERR "    In Dir: $_" if ($debug);
 
+	ProcessHistory("COMMENTS","keysort","D1","!Directory contents:\n");
     while (<INPUT>) {
 	s/^\s+\015//g;
 	tr/\015//d;
@@ -184,6 +187,7 @@
 
 	ProcessHistory("COMMENTS","keysort","D1","! $_");
     }
+	ProcessHistory("COMMENTS","keysort","D1","! \n");
     return(0);
 }
 
@@ -198,6 +202,9 @@
 	# pager remnants like: ^H^H^H    ^H^H^H content
 	s/[\b]+\s*[\b]*//g;
 
+	# Remove Uptime
+	/ up time/i && next;
+
 	ProcessHistory("COMMENTS","keysort","B1","! $_");
     }
     return(0);
@@ -218,9 +225,12 @@
 	/ up time/i && next;
 
 	# filter temperature sensor info for Dell 6428 stacks
-	/Temperature Sensors:/ && next;
+	/Temperature Sensors:/ &&
+        ProcessHistory("COMMENTS","keysort","C1","\n! $_") &&
+        next;
 	if (/Temperature \(Celsius\)/ &&
-	    ProcessHistory("COMMENTS","keysort","C1","! Unit\tStatus\n")) {
+	    ProcessHistory("COMMENTS","keysort","C1","! Unit\tStatus\n") &&
+	    ProcessHistory("COMMENTS","keysort","C1","! ----\t------\n")) {
 	    while (<INPUT>) {
 		s/^\s+\015//g;
 		tr/\015//d;
@@ -228,6 +238,26 @@
 		ProcessHistory("COMMENTS","keysort","C1","! $1\t$2\n");
 		/^\s*$/ && last;
 	    }
+    } 
+    # Filter temperature sensor info for Dell M6348 and M8024 blade switches
+    #
+    # M6348 and M8024 sample lines:
+    #   Unit     Description       Temperature    Status
+    #                               (Celsius)
+    #   ----     -----------       -----------    ------
+    #   1        System            39             Good
+    #   2        System            39             Good
+    elsif (/Temperature/ &&
+	    ProcessHistory("COMMENTS","keysort","C1","! Unit\tDescription\tStatus\n") &&
+	    ProcessHistory("COMMENTS","keysort","C1","! ----\t-----------\t------\n")) {
+	    while (<INPUT>) {
+		/\(celsius\)/i && next;
+		s/^\s+\015//g;
+		tr/\015//d;
+		/(\d+)\s+(\w+)\s+\d+\s+(.*)$/ &&
+		ProcessHistory("COMMENTS","keysort","C1","! $1\t$2\t\t$3\n");
+		/^\s*$/ && last;
+	    }
 	}
 
 	/system description: (.*)/i &&
@@ -242,6 +272,7 @@
 sub ShowVlan {
     print STDERR "    In ShowVlan: $_" if ($debug);
 
+	ProcessHistory("COMMENTS","keysort","D1","!VLAN definitions:\n");
     while (<INPUT>) {
 	s/^\s+\015//g;
 	tr/\015//d;
@@ -254,6 +285,7 @@
 	/ up time/i && next;
 	ProcessHistory("COMMENTS","keysort","D1","! $_");
     }
+	ProcessHistory("COMMENTS","keysort","D1","! \n");
     return(0);
 }
 
    root@host:~$ patch < srancid.patch

  • Edit the global RANCID configuration: 

    root@host:~$ vi /etc/rancid/rancid.conf

    Select the RCS (CVS, SVN or Git) of your choice. In this example SVN is used: 

    RCSSYS=svn; export RCSSYS

    Define a name for your Dell PowerConnect device group in the LIST_OF_GROUPS configuration variable. In this example we'll use the name dell-sw: 

    LIST_OF_GROUPS="dell-sw"; export LIST_OF_GROUPS

  • Create the cloginrc configuration file, which will contain the login information for your Dell PowerConnect devices and some default values: 

    root@host:~$ touch /etc/rancid/cloginrc
root@host:~$ chmod 660 /etc/rancid/cloginrc
root@host:~$ chown root:rancid /etc/rancid/cloginrc
root@host:~$ vi /etc/rancid/cloginrc

    Example: 

    add user        dell-switch-1           dell-user
add password    dell-switch-1           <login-passwort> <enable-passwort>
add noenable    dell-switch-1           0

[...]

add user        *                       <default-user>
add password    *                       <default-login-passwort>
add noenable    *                       1
add method      *                       ssh

    For the device named dell-switch-1 login as user dell-user with the password <login-passwort>. Only for this system change into the enable mode (add noenable dell-switch-1 0) of the switch and use the password <enable-passwort> to do so.

    For all other systems, login as user <default-user> with the password <default-login-passwort>. Deactivate the use of the enable mode on all systems (noenable * 1). The login method for all systems is via SSH.

    Since the cloginrc configuration file is parsed in a first-match fashion, the default values must always be at the bottom of the file.

  • Determine the name of the device type. See man router.db and /etc/rancid/rancid.types.base. In this example and in the general case of Dell PowerConnect M6348 and M8024-k switches the name of the device type is smc: 

    root@host:~$ grep smc /etc/rancid/rancid.types.base
smc;script;srancid
smc;login;hlogin

    From this we can also determine the login script hlogin and the postprocessing script srancid, which will be used for this device type.

  • Change to the user rancid: 

    root@host:~$ su - rancid

    • Create a symbolic link to the login configuration previously created in /etc/rancid/: 

      rancid@host:~$ ln -s /etc/rancid/cloginrc /var/lib/rancid/.cloginrc

    • Initialize the directory structure for the RCS (CVS, SVN or Git) selected above. This will automatically be done for each device group configured in the LIST_OF_GROUPS configuration variable. The example shown here only creates the directory structure for the device group dell-sw defined above: 

      rancid@host:~$ /usr/lib/rancid/bin/rancid-cvs
Committed revision 1.
Checked out revision 1.
Updating '.':
At revision 1.
A         configs
Adding         configs

Committed revision 2.
A         router.db
Adding         router.db
Transmitting file data .
Committed revision 3.

rancid@host:~$ find /var/lib/rancid/dell-sw/
/var/lib/rancid/dell-sw
/var/lib/rancid/dell-sw/configs
/var/lib/rancid/dell-sw/router.db
/var/lib/rancid/dell-sw/routers.all
/var/lib/rancid/dell-sw/routers.down
/var/lib/rancid/dell-sw/routers.up
/var/lib/rancid/dell-sw/.svn
/var/lib/rancid/dell-sw/.svn/entries
/var/lib/rancid/dell-sw/.svn/format
/var/lib/rancid/dell-sw/.svn/pristine
/var/lib/rancid/dell-sw/.svn/pristine/da
/var/lib/rancid/dell-sw/.svn/pristine/da/da39a3ee5e6b4b0d3255bfef95601890afd80709.svn-base
/var/lib/rancid/dell-sw/.svn/tmp
/var/lib/rancid/dell-sw/.svn/wc.db

    • Add switch devices by their hostname to the configuration file router.db of the corresponding device group: 

      rancid@host:~$ vi /var/lib/rancid/dell-sw/router.db

      In this example the device group dell-sw, the device type smc and the system dell-switch-1: 

      dell-switch-1;smc;up;A comment describing the system dell-switch-1

    • Perform a login test with the previously determined login script hlogin on the newly defined system dell-switch-1. The following example output shows the steps that should automatically be performed by the hlogin expect script. No manual intervention should be necessary. 

      rancid@host:~$ /usr/lib/rancid/bin/hlogin dell-switch-1
dell-switch-1
spawn ssh -c 3des -x -l root dell-switch-1
The authenticity of host 'dell-switch-1 (<ip address>)' can't be established.
RSA key fingerprint is <rsa key fingerprint>
Are you sure you want to continue connecting (yes/no)?
Host dell-switch-1 added to the list of known hosts.
yes
Warning: Permanently added 'dell-switch-1,<ip address>' (RSA) to the list of known hosts.
root@dell-switch-1's password:

dell-switch-1>enable
Password:**********

dell-switch-1#

    • Finish the login test by manually logging out of the system: 

      dell-switch-1#exit
dell-switch-1>exit
rancid@host:~$

    • Manually perform a initial RANCID run to make sure everything works as expected: 

      rancid@host:~$ rancid-run

      If everything ran successfully, there should now be a file /var/lib/rancid/dell-sw/configs/dell-switch-1 containing the output of the commands show version, show system and show running-config for the system dell-switch-1.

  • Create the email aliases necessary for the proper delivery of the emails generated by RANCID. Again in this example for the device group dell-sw: 

    root@host:~$ vi /etc/aliases
    rancid-dell-sw:       <email>@<domain>
rancid-admin-dell-sw: <email>@<domain>

    Recreate your aliases DB. In case postfix is used as an MTA: 

    root@host:~$ postalias /etc/aliases

  • Enable the RANCID cron jobs. Adjust the execution times and intervals according to your needs: 

    root@host:~$ vi /etc/cron.d/rancid

Some final words: The contents of the directories /var/lib/rancid/<device group>/ and /var/lib/rancid/<device group>/configs/ are maintained in the RCS – CVS, SVN or Git – of your choice. You can operate on those directories with the usual commands of the selected RCS. There are also some really nice and intuitive web frontends to the RCS of choice. For me, the combination of SVN as RCS and WebSVN as a web frontend worked out very well.

0 Comments | 2015-10-01 written by Frank Fegert (XING) | Permanentlink
Tags:
This website uses cookies. By using the website, you agree with storing cookies on your computer. Also you acknowledge that you have read and understand our Privacy Policy. If you do not agree leave the website. More information about cookies