bityard Blog

2016-12-13 // Integration of Dell EqualLogic PS-Series Storages with RANCID

Adding support for Dell EqualLogic PS-Series storage arrays to version 3.5.1 of the popular, open source switch and router configuration management tool RANCID.

For the impatient and TL;DR here are the extensions to RANCID for the management of Dell EqualLogic PS-Series storage arrays:

Login script for Dell EqualLogic PS-Series storage arrays
Perl module to generate, process and save the configuration of Dell EqualLogic PS-Series storage arrays

The sources are to be found in my RANCID repository on GitHub

RANCID has, in its current version 3.5.1, support for a large variety of network devices like routers, switches, load-balancers, etc. Unfortunately there is currently little or no support for the management of storage devices, even though a lot of them offer a command line interface which can be used by RANCID.

Although there probably are a couple of reasons for this, i suppose this is largely due to the fact that network and storage admins are – in most organizations – still in different groups, each with their own set of management and support tools. With RANCID originating from the realm of network administration, probably only few storage admins know about this very valuable tool to begin with. There is probably also very little transfer over from the position of network administrator into the area of storage administration and thus a limited amount of knowledge transfer between those two fields.

This blog post describes how to extend and configure RANCID in order to add support for Dell EqualLogic PS-Series storage arrays. The extensions are based on the – at the time of writing – current version 3.5.1 of RANCID. RANCID can either be build from source or be installed pre-packaged e.g. from the backports repository of Debian stable (jessie). Basically, the extension to RANCID consist of only two files:

Besides those two extensions only a small change to /etc/rancid/rancid.types.conf, as well as the standard RANCID configuration of a new device group and device is necessary. See the following full step-by-step configuration example for Dell EqualLogic PS-Series storage arrays:

  • Add the backports repository of Debian stable (jessie) to the APT configuration: 

    root@host:~$ echo 'deb http://http.debian.net/debian jessie-backports main non-free contrib' >> /etc/apt/sources.list.d/jessie-backports.list
root@host:~$ apt-get update

  • Install RANCID v3.5.1 from the backports repository of Debian stable (jessie): 

    root@host:~$ apt-get -y install rancid/jessie-backports

Reading package lists... Done
Building dependency tree       
Reading state information... Done
Selected version '3.5.1-1~bpo8+1' (Debian Backports:jessie-backports [amd64]) for 'rancid'
The following extra packages will be installed:
  expect tcl-expect
The following NEW packages will be installed:
  expect rancid tcl-expect
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 511 kB of archives.
After this operation, 2,178 kB of additional disk space will be used.
[...]

    Optional: In case Subversion should be used as a revision control system (RCS) to store the switch configuration, install it: 

    root@host:~$ apt-get -y install subversion

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libapr1 libaprutil1 libldap-2.4-2 libsasl2-2 libsasl2-modules libsasl2-modules-db libserf-1-1 libsvn1
Suggested packages:
  libsasl2-modules-otp libsasl2-modules-ldap libsasl2-modules-sql libsasl2-modules-gssapi-mit libsasl2-modules-gssapi-heimdal subversion-tools db5.3-util patch
The following NEW packages will be installed:
  libapr1 libaprutil1 libldap-2.4-2 libsasl2-2 libsasl2-modules libsasl2-modules-db libserf-1-1 libsvn1 subversion
0 upgraded, 9 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,723 kB of archives.
After this operation, 9,683 kB of additional disk space will be used.
[...]

  • Download the login script for Dell EqualLogic PS-Series storage arrays and store it under the path /usr/lib/rancid/bin/.

  • Download the Perl module to process and save the configuration of Dell EqualLogic PS-Series storage arrays and store it under the path /usr/share/perl5/rancid/.

  • Edit the global RANCID configuration: 

    root@host:~$ vi /etc/rancid/rancid.conf

    Select the RCS (CVS, SVN or Git) of your choice. In this example SVN is used: 

    RCSSYS=svn; export RCSSYS

    Define a name for your Dell EqualLogic device group in the LIST_OF_GROUPS configuration variable. In this example we'll use the name dell-storage: 

    LIST_OF_GROUPS="dell-storage"; export LIST_OF_GROUPS

  • Create the cloginrc configuration file, which will contain the login information for your Dell EqualLogic PS-Series devices and some default values: 

    root@host:~$ touch /etc/rancid/cloginrc
root@host:~$ chmod 660 /etc/rancid/cloginrc
root@host:~$ chown root:rancid /etc/rancid/cloginrc
root@host:~$ vi /etc/rancid/cloginrc

    Example: 

    add user        dell-eql-1      dell-user
add password    dell-eql-1      <login-passwort>

[...]

add user        *               <default-user>
add password    *               <default-login-passwort>
add method      *               ssh

    For the device named dell-eql-1 login as user dell-user with the password <login-passwort>.

    For all other systems, login as user <default-user> with the password <default-login-passwort>. The login method for all systems is via SSH.

    Since the cloginrc configuration file is parsed in a first-match fashion, the default values must always be at the bottom of the file.

  • Add a new device type for Dell EqualLogic PS-Series storage arrays to the RANCID configuration. See man router.db and /etc/rancid/rancid.types.conf. In this example and in the general case of Dell Dell EqualLogic PS-Series storage arrays the name of the device type is equallogic: 

    root@host:~$ vi /etc/rancid/rancid.types.conf

    Here we set the login script to be used to the new eqllogin. The postprocessing script is set rancid -t equallogic in order to call the new Perl module equallogic, which will do the actual processing. The command to be issued on the Dell EqualLogic device is set to save-config -verbose. The -verbose part is essential here, otherwise the configuration of the device will only be saved to a file on the Dell EqualLogic device and not be printed to the terminal: 

    equallogic;login;eqllogin
equallogic;script;rancid -t equallogic
equallogic;module;equallogic
equallogic;inloop;equallogic::inloop
equallogic;command;equallogic::SaveConfiguration;save-config -verbose

  • Change to the user rancid: 

    root@host:~$ su - rancid

    • Create a symbolic link to the login configuration previously created in /etc/rancid/: 

      rancid@host:~$ ln -s /etc/rancid/cloginrc /var/lib/rancid/.cloginrc

    • Initialize the directory structure for the RCS (CVS, SVN or Git) selected above. This will automatically be done for each device group configured in the LIST_OF_GROUPS configuration variable. The example shown here only creates the directory structure for the device group dell-storage defined above: 

      rancid@host:~$ /usr/lib/rancid/bin/rancid-cvs
Committed revision 1.
Checked out revision 1.
Updating '.':
At revision 1.
A         configs
Adding         configs

Committed revision 2.
A         router.db
Adding         router.db
Transmitting file data .
Committed revision 3.

rancid@host:~$ find /var/lib/rancid/dell-storage/
/var/lib/rancid/dell-storage
/var/lib/rancid/dell-storage/configs
/var/lib/rancid/dell-storage/router.db
/var/lib/rancid/dell-storage/routers.all
/var/lib/rancid/dell-storage/routers.down
/var/lib/rancid/dell-storage/routers.up
/var/lib/rancid/dell-storage/.svn
/var/lib/rancid/dell-storage/.svn/entries
/var/lib/rancid/dell-storage/.svn/format
/var/lib/rancid/dell-storage/.svn/pristine
/var/lib/rancid/dell-storage/.svn/pristine/da
/var/lib/rancid/dell-storage/.svn/pristine/da/da39a3ee5e6b4b0d3255bfef95601890afd80709.svn-base
/var/lib/rancid/dell-storage/.svn/tmp
/var/lib/rancid/dell-storage/.svn/wc.db

    • Add Dell EqualLogic storage devices by their hostname to the configuration file router.db of the corresponding device group: 

      rancid@host:~$ vi /var/lib/rancid/dell-storage/router.db

      In this example the device group dell-storage, the device type equallogic and the system dell-eql-1: 

      dell-eql-1;equallogic;up;A comment describing the system dell-eql-1

    • Perform a login test with the previously configured new login script eqllogin for Dell EqualLogic devices on the newly defined system dell-eql-1. The following example output shows the steps that should automatically be performed by the eqllogin expect script. No manual intervention should be necessary. 

      rancid@host:~$ /usr/lib/rancid/bin/eqllogin dell-eql-1
spawn ssh -x -l grpadmin dell-eql-1
The authenticity of host 'dell-eql-1 (<ip address>)' can't be established.
RSA key fingerprint is <rsa key fingerprint>
Are you sure you want to continue connecting (yes/no)?  yes
Host dell-eql-1 added to the list of known hosts.
Warning: Permanently added 'dell-eql-1,<ip address>' (RSA) to the list of known hosts.
grpadmin@dell-eql-1's password: 
Last login: Thu Dec  8 22:32:39 2016 from <ip address> on ttyp1
 

           Welcome to Group Manager

        Copyright 2001-2015 Dell Inc.



dell-eql-1-grp>

    • Finish the login test by manually logging out of the system: 

      dell-eql-1-grp> logout
Do you really want to logout? (y/n) [n]y

dell-eql-1-grp> Connection to dell-eql-1 closed.

    • Manually perform an initial RANCID run to make sure everything works as expected: 

      rancid@host:~$ rancid-run

      If everything ran successfully, there should now be a file /var/lib/rancid/dell-storage/configs/dell-eql-1 containing the output of the command save-config -verbose for the system dell-eql-1.

  • Create the email aliases necessary for the proper delivery of the emails generated by RANCID. Again in this example for the device group dell-storage: 

    root@host:~$ vi /etc/aliases
    rancid-dell-storage:       <email>@<domain>
rancid-admin-dell-storage: <email>@<domain>

    Recreate your aliases DB. In case postfix is used as an MTA: 

    root@host:~$ postalias /etc/aliases

  • Enable the RANCID cron jobs. Adjust the execution times and intervals according to your needs: 

    root@host:~$ vi /etc/cron.d/rancid

Some final words: The contents of the directories /var/lib/rancid/<device group>/ and /var/lib/rancid/<device group>/configs/ are maintained in the RCS – CVS, SVN or Git – of your choice. You can operate on those directories with the usual commands of the selected RCS. There are also some really nice and intuitive web frontends to the RCS of choice. For me, the combination of SVN as RCS and WebSVN as a web frontend worked out very well.

0 Comments | 2016-12-13 written by Frank Fegert (XING) | Permanentlink
Tags:

2015-08-15 // Dell EqualLogic PS Series - Security

An initial setup is always a good opportunity to take a more in-depth look at systems, while they are not yet in production. In this case i'm looking at one of the advertised security measures of Dell EqualLogic PS Series systems.

We're currently in the process of implementing iSCSI-based Dell EqualLogic PS Series systems. Specifically we're using PS-M4110E and PS-M4110X models in several Dell M1000e blade chassis. The EqualLogic storages and the hosts they provide storage space for are being connected with PowerConnect M8024-K switches. The switches are located in the slots B1 and B2 of the blade chassis and are dedicated for iSCSI traffic with 10 gigabit ethernet.

Although the separation of SAN and LAN could be considered enough security, i was really pleased to read in the Dell EqualLogic Group Manager Administrator's Manual - PS Series Firmware Version 7.0 (110-6152-EN-R1) that the folks at EqualLogic thought otherwise and added further security measures in order to protect the EqualLogic PS Series systems from unauthorized access to its management functions over the iSCSI network. A quote from page 76 of said document reads:

[…]

About Dedicated Management Networks

For increased security, or if your environment requires the separation of management traffic and iSCSI traffic, you can configure a dedicated management network (DMN) that is used only for administrative access to the group. The management network is separate from the network that handles iSCSI traffic to the group.

- Without a dedicated management network (the default configuration), administrators connect to the group IP address for both administrative access to the group and iSCSI initiator access to iSCSI targets (volumes and snapshots).
- With a dedicated management network, administrators do not use the group IP address for administrative access to the group. Instead, administrators connect to the management network address. All iSCSI traffic, including traffic by replication partners, and access to Dell EqualLogic Auto-Snapshot Manager/Linux Edition (ASM/LE), Dell EqualLogic Auto-Snapshot Manager/Microsoft Edition (ASM/ME), and Dell EqualLogic Virtual Storage Manager for VMware (formerly ASM/VE), continues to use the group IP address. SAN Headquarters can connect to the group using either the management network address or the iSCSI address.

[…]

And further on page 78 of the same document:

[…]

When you complete the management network configuration, administrators cannot log in to the group using the group IP address. Instead, administrators must use the new management IP address. Any open GUI or CLI sessions using the group IP address eventually time out and close.
After configuring a dedicated management network, you might need to:
- Inform administrators of the new management network IP address.
- If you run the Group manager GUI as a standalone application and have a shortcut on the computer's desktop, the group address in the shortcut is not updated with the new management address. You must uninstall and then reinstall the GUI application.
- If you are running SAN Headquarters, you must update the group IP address in the application to the dedicated management address. For more information, see the SAN Headquarters documentation.

[…]

Judging from those two sections it would appear that the EqualLogic PS Series systems have no or at least a very small attack surface on the – potentially untrustworthy – host-facing network dedicated to iSCSI traffic. In open systems this is usually achieved by binding the services which are necessary for the management function to a specific network interface, instead of letting them listen on all available interfaces.

Using a DMN and thus separating the management traffic from the iSCSI traffic resulted in our case in the following configuration example: 

group1-grp(member_group1)> eth show
Name ifType          ifSpeed    Mtu  Ipaddress                     Status Errors DCB
---- --------------- ---------- ---- ----------------------------- ------ ------ ------
eth0 ethernet-csmacd 10 Gbps    9000 10.0.0.1                      up     0      off
eth1 ethernet-csmacd 100 Mbps   1500 123.123.123.123               up     0      off
group1-grp(member_group1)> eth select 0 show
_______________________________ Eth Information _______________________________
Name: eth0
Status: up
Changed: Mon Jul 20 12:23:15 2015
Type: ethernet-csmacd
DesiredStatus: up
Mtu: 9000
Speed: 10 Gbps
HardwareAddress: B0:83:FE:CC:52:C1
IPAddress: 10.0.0.1
NetMask: 255.255.0.0
IPv6Address:
Description: group1 iSCSI Interface
SupportsManagement: no
ManagementStatus: normal
DCB: off
group1-grp(member_group1)> eth select 1 show
_______________________________ Eth Information _______________________________
Name: eth1
Status: up
Changed: Wed Jul 29 08:33:35 2015
Type: ethernet-csmacd
DesiredStatus: up
Mtu: 1500
Speed: 100 Mbps
HardwareAddress: B0:83:FE:CC:52:C2
IPAddress: 123.123.123.123
NetMask: 255.255.255.0
IPv6Address:
Description: group1 Management Interface
SupportsManagement: only
ManagementStatus: mgmt
DCB: off

Here the lines with the SupportsManagement options could be construed in the way, that management access is not possible over the eth0 interface, which in this configuration is the connection to the host-facing iSCSI network.

From previous lessons learned not to be overly trusty of lofty vendor promises, i decided to double check this with the simple use of the nmap network and port scanner. The results of the TCP and UDP port scans against both the member and the group IP address are shown below.

  • Member IP - TCP scan: 

    root@host:~$ nmap -sS -p 0-65535 10.0.0.1

Starting Nmap 6.00 ( http://nmap.org ) at 2015-07-15 11:14 CEST
Nmap scan report for ******** (10.0.0.1)
Host is up (0.000097s latency).
Not shown: 9991 closed ports
PORT     STATE SERVICE
21/tcp   open  ftp
22/tcp   open  ssh
80/tcp   open  http
443/tcp  open  https
2606/tcp open  netmon
3002/tcp open  exlm-agent
3003/tcp open  cgms
3260/tcp open  iscsi
9876/tcp open  sd
20002/tcp open  commtact-http
20003/tcp open  unknown
25555/tcp open  unknown
MAC Address: B0:83:FE:CC:52:C1 (Unknown)

Nmap done: 1 IP address (1 host up) scanned in 107.66 seconds

  • Member IP - UDP scan: 

    root@host:~$ nmap -sU -p 0-65535 10.0.0.1

Starting Nmap 6.00 ( http://nmap.org ) at 2015-07-20 09:27 CEST
Nmap scan report for ******** (10.0.0.1)
Host is up (0.000086s latency).
Not shown: 65532 closed ports
PORT      STATE         SERVICE
0/udp     open|filtered unknown
123/udp   open          ntp
161/udp   open          snmp
65519/udp open|filtered unknown
MAC Address: B0:83:FE:CC:52:C1 (Unknown)

Nmap done: 1 IP address (1 host up) scanned in 2.55 seconds

  • Group IP - TCP scan: 

    root@host:~$ nmap -sS -p 0-65535 10.0.0.2

Starting Nmap 6.00 ( http://nmap.org ) at 2015-07-15 11:17 CEST
Nmap scan report for ******** (10.0.0.2)
Host is up (0.00010s latency).
Not shown: 9991 closed ports
PORT     STATE SERVICE
21/tcp   open  ftp
22/tcp   open  ssh
80/tcp   open  http
443/tcp  open  https
2606/tcp open  netmon
3002/tcp open  exlm-agent
3003/tcp open  cgms
3260/tcp open  iscsi
9876/tcp open  sd
20002/tcp open  commtact-http
20003/tcp open  unknown
25555/tcp open  unknown
MAC Address: B0:83:FE:CC:52:C1 (Unknown)

Nmap done: 1 IP address (1 host up) scanned in 103.71 seconds

  • Group IP - UDP scan: 

    root@host:~$ nmap -sU -p 0-65535 10.0.0.2

Starting Nmap 6.00 ( http://nmap.org ) at 2015-07-20 09:28 CEST
Nmap scan report for ******** (10.0.0.2)
Host is up (0.00013s latency).
Not shown: 65532 closed ports
PORT      STATE         SERVICE
0/udp     open|filtered unknown
123/udp   open          ntp
161/udp   open          snmp
65519/udp open|filtered unknown
MAC Address: B0:83:FE:CC:52:C1 (Unknown)

Nmap done: 1 IP address (1 host up) scanned in 3.07 seconds

These scan results are pretty disappointing with regard to the expected additional security measures mentioned above.

An actual login test via SSH and FTP was successful and i guess a login via Telnet would have been successful too, if the protocol hadn't already been disabled in our configuration. This means there is also no filter on the application layer preventing access to the management functions, which wouldn't have been recognized by the simple port scan above.

As described earlier it shouldn't be too hard binding the services which are necessary for the management function to a specific network interface. I can't help but wonder why EqualLogic didn't follow through with this, especially since it's already described in the product manual. Maybe there were too many feature requests from the customer side or even technical requirements to better integrate the EqualLogic PS Series systems with certain host systems. Even then i wonder why EqualLogic didn't at least provide some means of selectively enabling or disabling the access to the management function from the iSCSI network with some kind of configuration parameter. In any case, the situation at hand – describing one thing and implementing another – seems to be the least favorable one to me.

0 Comments | 2015-08-15 written by Frank Fegert (XING) | Permanentlink
Tags:
This website uses cookies. By using the website, you agree with storing cookies on your computer. Also you acknowledge that you have read and understand our Privacy Policy. If you do not agree leave the website. More information about cookies