As already mentioned in the article Experiences with Java and X.509 Certificates - Code Signing i recently had the task of researching two issues involving Java and X.509 certificates. While i'm familiar with X.509 certificates, i'm not too familiar with the inner workings of Java, the Java run-time environment or let alone programming in Java. So this was a good opportunity to familiarize myself with the inner workings of Java and also a great learning experience.

The second issue, which this blog post will be about, was in the area of TLS-secured HTTPS network connections, naturally involving a X.509 certificate on the server side. In particular the server processes of two Java application servers were connecting the application logic of two different systems via an API over a HTTPS based network connection. The source system would make RPC calls to an API on the target system in order to store data in and retrieve data from this system.

The communication between the two systems would work fine while using an unencrypted HTTP based network connection, but would fail while using a secured HTTPS network connections. The errors reported back to the server process on the source system which was initiating the communication, would point in the direction of an issue with the X.509 certificate that was being used on the target server. The error message would look something like this:

%% Invalidated:  [Session-1, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256]
main, SEND TLSv1.2 ALERT:  fatal, description = certificate_unknown
main, WRITE: TLSv1.2 Alert, length = 2
[Raw write]: length = 7
0000: 15 03 03 00 02 02 2E                               .......
main, called closeSocket()
main, handling exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException:
  PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find
  valid certification path to requested target
  Exception Failed to access the WSDL at: https://hostname:port/url/rpc-call. It failed with: 
        sun.security.validator.ValidatorException: PKIX path building failed:
            sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification
                path to requested target.
x.y.z.ExceptionHandler
        at x.y.z.method1(source1.java:line-number)
        at x.y.z.method2(source2.java:line-number)
        at x.y.z.method3(source3.java:line-number)

Checking the certificate used on the side of the target server would turn up nothing unusual. The certificate was issued by the internal CA of our infrastructure and like many other certificates looked valid. Verifiying this, by initiating a non-Java based HTTPS request (e.g. with a browser) to the target system worked fine. The certificate on the side of the target server was accepted and a secured HTTPS connection was successfully established. The next debugging step was to check the certstore on the side of the source server which was initiating the connection. It showed that a valid entry, containing the full chain of the root and the intermediate certificates of our internal CA, existed.

From an infrastructure point of view, everything looked reasonably good and should work just fine. Still, the connection between the two systems was failing.

Unfortunately, but inevitably the Java routines responsible for initiating the HTTPS connection on the source system were part of and thus embedded in a programming logic more complex than necessary or useful for the purpose of low-level debugging. For further debugging i wanted a Java-based test program which was stripped down to the bare minimum of initiating a HTTPS connection. Again, not being a seasoned Java programmer, i managed to cobble together the following test program from several examples on the internet:

CertChecker.java

import java.net.*;
import java.io.*;
import javax.net.ssl.*;
import java.security.GeneralSecurityException;
import java.security.KeyStore;
import java.security.SecureRandom;
import java.security.cert.*;
import java.util.*;
 
public class CertChecker implements X509TrustManager {
    public static void main(String[] args) throws Exception {
        try {
            SSLContext sc = SSLContext.getInstance("SSL");
            sc.init(null, new TrustManager[]{new CertChecker()}, new SecureRandom());
            SSLSocketFactory factory = (SSLSocketFactory)SSLSocketFactory.getDefault();
            SSLSocket socket = (SSLSocket)factory.createSocket("hostname", port);
            socket.startHandshake();
 
            PrintWriter out = new PrintWriter(
                                  new BufferedWriter(
                                  new OutputStreamWriter(
                                  socket.getOutputStream())));
 
            out.println("GET / HTTP/1.0");
            out.println();
            out.flush();
 
            if (out.checkError())
                System.out.println("SSLSocketClient:  java.io.PrintWriter error");
 
            BufferedReader in = new BufferedReader(
                                    new InputStreamReader(
                                    socket.getInputStream()));
 
            String inputLine;
            while ((inputLine = in.readLine()) != null)
                System.out.println(inputLine);
 
            in.close();
            out.close();
            socket.close();
 
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
 
    private final X509TrustManager defaultTM;
 
    public CertChecker() throws GeneralSecurityException {
        TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
        tmf.init((KeyStore) null);
        defaultTM = (X509TrustManager) tmf.getTrustManagers()[0];
    }
 
    public void checkServerTrusted(X509Certificate[] certs, String authType) {
        if (defaultTM != null) {
            try {
                defaultTM.checkServerTrusted(certs, authType);
                Set<TrustAnchor> trustAnchors = getTrustAnchors();
                System.out.println("Certificate valid");
            } catch (CertificateException ex) {
                System.out.println("Certificate invalid: " + ex.getMessage());
            }
        }
    }
 
    private Set<TrustAnchor> getTrustAnchors() {
        X509Certificate[] acceptedIssuers = defaultTM.getAcceptedIssuers();
        Set<TrustAnchor> trustAnchors = new HashSet<TrustAnchor>();
        for (X509Certificate acceptedIssuer : acceptedIssuers) {
            TrustAnchor trustAnchor = new TrustAnchor(acceptedIssuer, null);
            trustAnchors.add(trustAnchor);
        }
        return trustAnchors;
    }
 
    public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException {
    }
 
    public X509Certificate[] getAcceptedIssuers() {
        return null;
    }
}

In line 16 of the above Java program, the placeholders hostname and port were replaced by the appropriate values for the target server system. The code was then compiled and run against the target server system, with the following Java command line call:

user@host:$ java -Djava.security.debug=all -Djavax.net.debug=all CertChecker

The result from this was the following error message output:

[...]
certpath: BasicChecker.updateState issuer: CN=root-ca, DC=domain, DC=tld; subject: CN=intermediate-ca, DC=domain, DC=tld; serial#: ...
certpath: -checker6 validation succeeded
certpath: -Using checker7 ... [sun.security.provider.certpath.RevocationChecker]
certpath: RevocationChecker.check: checking cert
  SN:     26000000 0cff68dd 06c50fb0 4a000100 00000c
  Subject: CN=intermediate-ca, DC=domain, DC=tld
  Issuer: CN=root-ca, DC=domain, DC=tld
certpath: RevocationChecker.checkCRLs() ---checking revocation status ...
certpath: RevocationChecker.checkCRLs() possible crls.size() = 0
certpath: RevocationChecker.checkCRLs() approved crls.size() = 0
certpath: DistributionPointFetcher.getCRLs: Checking CRLDPs for CN=intermediate-ca, DC=domain, DC=tld
certpath: Trying to fetch CRL from DP http://crl-fqdn/filename.crl
certpath: CertStore URI:http://crl-fqdn/filename.crl
certpath: Downloading new CRL...
certpath: Exception fetching CRL:
java.security.cert.CRLException: Empty input
java.security.cert.CRLException: Empty input
        at sun.security.provider.X509Factory.engineGenerateCRL(X509Factory.java:397)
        at java.security.cert.CertificateFactory.generateCRL(CertificateFactory.java:497)
        at sun.security.provider.certpath.URICertStore.engineGetCRLs(URICertStore.java:419)
        at java.security.cert.CertStore.getCRLs(CertStore.java:181)
        at sun.security.provider.certpath.DistributionPointFetcher.getCRL(DistributionPointFetcher.java:245)
        at sun.security.provider.certpath.DistributionPointFetcher.getCRLs(DistributionPointFetcher.java:189)
        at sun.security.provider.certpath.DistributionPointFetcher.getCRLs(DistributionPointFetcher.java:121)
        at sun.security.provider.certpath.RevocationChecker.checkCRLs(RevocationChecker.java:552)
        at sun.security.provider.certpath.RevocationChecker.checkCRLs(RevocationChecker.java:465)
        at sun.security.provider.certpath.RevocationChecker.check(RevocationChecker.java:367)
        at sun.security.provider.certpath.RevocationChecker.check(RevocationChecker.java:337)
        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:219)
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:140)
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79)
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:347)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:249)
        at sun.security.validator.Validator.validate(Validator.java:260)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:961)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
        at CertChecker.main(CertChecker.java:31)
[...]
%% Invalidated:  [Session-1, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256]
main, SEND TLSv1.2 ALERT:  fatal, description = certificate_unknown
main, WRITE: TLSv1.2 Alert, length = 2
[Raw write]: length = 7
0000: 15 03 03 00 02 02 2E                               .......
main, called closeSocket()
main, handling exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: \
    PKIX path validation failed: java.security.cert.CertPathValidatorException: Could not determine revocation status
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: \
    java.security.cert.CertPathValidatorException: Could not determine revocation status
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:961)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
        at CertChecker.main(CertChecker.java:31)
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: \
    java.security.cert.CertPathValidatorException: Could not determine revocation status
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:352)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:249)
        at sun.security.validator.Validator.validate(Validator.java:260)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496)
        ... 8 more
Caused by: java.security.cert.CertPathValidatorException: Could not determine revocation status
        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:219)
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:140)
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79)
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:347)
        ... 14 more
Caused by: java.security.cert.CertPathValidatorException: Could not determine revocation status
        at sun.security.provider.certpath.RevocationChecker.buildToNewKey(RevocationChecker.java:1092)
        at sun.security.provider.certpath.RevocationChecker.verifyWithSeparateSigningKey(RevocationChecker.java:910)
        at sun.security.provider.certpath.RevocationChecker.checkCRLs(RevocationChecker.java:577)
        at sun.security.provider.certpath.RevocationChecker.checkCRLs(RevocationChecker.java:465)
        at sun.security.provider.certpath.RevocationChecker.check(RevocationChecker.java:367)
        at sun.security.provider.certpath.RevocationChecker.check(RevocationChecker.java:337)
        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
        ... 19 more

The Java exceptions shown above suggested an issue with the Certificate Revokation List (CRL) of the root CA. This was confusing, since the certificate worked when the target server was accessed with a browser. Manually downloading the CRL was also possibly. Verifying the downloaded CRL with the help of OpenSSL showed no anomalies to which this issue could be attributed.

The line:

java.security.cert.CRLException: Empty input

from the above error messages, lead the way to solve this issue. This line could be interpreted in two ways:

1. either there was a field in the CRL currently being parsed that had an empty or an unexpected value leading to a parse error
2. or the downloaded CRL data which was handed over to the CRL parsing code was an empty set.

I decided to first check option number 2 and – if this turned out to be not the case – move on from there to check option number 1. Since a manual download of the CRL worked per se, i decided to take a closer look at the download process with the curl command line utility:

user@host:$ curl -v http://crl-fqdn/filename.crl
* Hostname was NOT found in DNS cache
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.0.0.15...
* Connected to crl-fqdn (10.0.0.15) port 80 (#0)
> GET /filename.crl HTTP/1.1
> User-Agent: curl/7.38.0
> Host: crl-fqdn
> Accept: */*
> 
* HTTP 1.0, assume close after body
< HTTP/1.0 302 Found
< Location: https://crl-fqdn/filename.crl
< Server: loadbalancer-vendor
* HTTP/1.0 connection set to keep alive!
< Connection: Keep-Alive
< Content-Length: 0
< 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host crl-fqdn left intact

The HTTP request for the CRL file was in fact answered with a HTTP status code 302 and a new location for the CRL file, redirecting any client to a HTTPS secured location. What is apparently working for any regular browser or – with the “-L” command line option – even for the curl command line utility, seemed to be an issue for the part of the JRE that is handling the download of the CRL. Instead of following the HTTP redirect and requesting the CRL from the alternate location, the CRL-handling code within the JRE is given the content data that is returned from the initial HTTP request which is an empty set (see the Content-Length: 0 in the above curl output). This in turn leads to the failure of parsing the CRL, which leads to the failure verifying the certificate against the CRL, which leads to the failure of not establishing a TLS-secured HTTPS network connection between the source and the target server system.

The immediate and simple solution to this issue was to add an exception to the HTTP-to-HTTPS redirect for the URL of the CRL. This global HTTP-to-HTTPS redirect was implemented a while ago upon request by the information security department as an attempt make the site serving the CRL and other content more secure. Arguably serving CRLs with HTTPS is generally a bad idea, since it can cause a chicken-and-egg type of problem. In this case though it would have worked, since the host serving the CRL was protected by a certificate from a different CA than the one for which the CRL was being provided.

Recently i had the task of researching two issues involving Java and X.509 certificates. While i'm familiar with X.509 certificates, i'm not too familiar with the inner workings of Java, the Java run-time environment or let alone programming in Java. So this was a good opportunity to familiarize myself with the inner workings of Java and also a great learning experience.

The first issue, which this blog post will be about, was in the area of code signing of an additional third party component to Java. The second issue was in the area of HTTPS network connections and will be the subject of another blog post.

In our Windows client environment, we use the smartcard system ActivIdentity from HID in conjunction with the single sign-on software SecureLogin from NetIQ, now a part of Micro Focus. In our case primarily used for an inhouse Java based application, there is a Java extension which integrates an interface to the ActivIdentity software in the clients JRE. This aims to make the ActivIdentity smartcard system available for all Java based applications in order to provide a single sign-on feature for the users.

The Java extension consists of the files:

$JAVA_HOME/lib/ext/javasso.jar
$JAVA_HOME/lib/ext/xbean.jar

During the preliminary tests for a rollout of a current release of the JRE version v1.8.0 on the Windows clients, the following issue surfaced. Probably due to the more strict enforcement of security measures in the current JRE version, the single sign-on integration would not work reliably any more, sometimes even not at all. There have previously been issues with this and a – albeit ugly – workaround implemented by our Windows client team was to disable the certificate revokation checks for the entire JRE on the Windows clients. Now, with the new JRE to be rolled out, even this workaround wouldn't get the single sign-on to work any more.

From the console of the JRE the only clue was the following, but probably unrelated, Java exception:

java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at com.actividentity.sso.javasso.awt_swing.JavaSSOHook.addListenersRecursively(JavaSSOHook.java:356)
    at com.actividentity.sso.javasso.awt_swing.JavaSSOHook.addListenersRecursively(JavaSSOHook.java:455)
    at com.actividentity.sso.javasso.awt_swing.JavaSSOHook.addListenersRecursively(JavaSSOHook.java:455)
    at com.actividentity.sso.javasso.awt_swing.JavaSSOHook.addListenersRecursively(JavaSSOHook.java:455)
    at com.actividentity.sso.javasso.awt_swing.JavaSSOHook.addListenersRecursively(JavaSSOHook.java:455)
    at com.actividentity.sso.javasso.awt_swing.JavaSSOHook.addListenersRecursively(JavaSSOHook.java:455)
    at com.actividentity.sso.javasso.awt_swing.JavaSSOJob.refreshComponentTree(JavaSSOJob.java:168)
    at com.actividentity.sso.javasso.JavaSSOJobMgr.refreshComponentTrees(JavaSSOJobMgr.java:93)
    at com.actividentity.sso.javasso.JavaSSOJobMgr.run(JavaSSOJobMgr.java:190)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException
    at com.sun.proxy.$Proxy0.equals(Unknown Source)
    at java.util.Vector.indexOf(Unknown Source)
    at java.util.Vector.indexOf(Unknown Source)
    at java.util.Vector.removeElement(Unknown Source)
    at oracle.ewt.event.ListenerManager.removeListener(Unknown Source)
    at oracle.ewt.lwAWT.lwWindow.DesktopContainer.removeDesktopListener(Unknown Source)
    ... 14 more

After some searching it turned out that the file $JAVA_HOME/lib/ext/javasso.jar, which is part of the Java extension provided by ActivIdentity, was signed with a X.509 certificate which expired in 2016:

user@host:$ openssl pkcs7 -inform DER -print_certs -in SUNCODES.RSA -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0a:77:eb:6f:b1:d6:74:7c:f2:7d:4e:3d:43:fa:72:1c
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=Terms of use at https://www.verisign.com/rpa (c)10, CN=VeriSign Class 3 Code Signing 2010 CA
        Validity
            Not Before: Mar  6 00:00:00 2013 GMT
            Not After : Jun  4 23:59:59 2016 GMT
        Subject: C=US, ST=Utah, L=Provo, O=Novell, Inc., OU=Digital ID Class 3 - Java Object Signing, CN=Novell, Inc.
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:eb:e8:89:56:52:0f:be:7d:7a:90:8c:f6:a6:46:
                    c2:c5:d7:8d:de:ab:9d:44:79:b9:ca:be:d3:22:94:
                    58:a3:b9:49:b3:59:71:52:98:ec:30:48:c3:60:32:
                    13:19:ec:b0:19:f6:9c:4a:4b:89:6f:fd:cc:67:f1:
                    a4:c0:b6:37:b9:c7:3c:58:aa:0d:0e:cd:dc:06:ff:
                    17:64:ec:a9:9d:29:ef:ae:5b:49:ef:8c:ef:8c:38:
                    a4:1b:ec:b5:26:c2:65:80:c3:cf:b8:73:d5:e7:dc:
                    e2:54:3f:63:c8:c4:12:40:57:dd:9a:bc:56:ad:6a:
                    bc:65:a8:34:a0:df:d1:87:58:2c:06:65:74:a0:48:
                    0f:df:41:e4:6b:9b:d5:45:f2:3f:3a:c3:a9:c1:84:
                    bf:a0:d4:fa:ee:53:a3:09:51:b5:18:bf:98:aa:f0:
                    6e:77:8a:c1:fd:1c:4d:62:47:ca:2d:ae:93:4c:5a:
                    ae:32:39:eb:cc:4b:da:fe:cb:e7:5f:02:af:d1:c4:
                    5f:6b:d5:e0:3c:06:3c:3a:29:83:bc:c7:10:7a:4c:
                    9a:ff:ff:bd:84:62:a8:4c:bf:76:20:b8:d8:20:9c:
                    f7:86:3b:96:d4:30:52:30:66:f5:9f:48:59:e1:1c:
                    2d:10:e8:6b:67:be:8f:21:41:be:83:af:9f:e7:41:
                    10:73
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://csc3-2010-crl.verisign.com/CSC3-2010.crl

            X509v3 Certificate Policies:
                Policy: 2.16.840.1.113733.1.7.23.3
                  CPS: https://www.verisign.com/rpa

            X509v3 Extended Key Usage:
                Code Signing
            Authority Information Access:
                OCSP - URI:http://ocsp.verisign.com
                CA Issuers - URI:http://csc3-2010-aia.verisign.com/CSC3-2010.cer

            X509v3 Authority Key Identifier:
                keyid:CF:99:A9:EA:7B:26:F4:4B:C9:8E:8F:D7:F0:05:26:EF:E3:D2:A7:9D

            Netscape Cert Type:
                Object Signing
            1.3.6.1.4.1.311.2.1.27:
                0.......
    Signature Algorithm: sha1WithRSAEncryption
         40:18:43:e9:58:06:c5:3e:82:de:ec:8e:69:20:26:43:3f:0b:
         41:0f:1b:cf:ca:5d:f6:e2:f2:c3:31:e7:c3:d0:07:f4:ea:8e:
         d5:1f:72:de:1e:4c:d6:8a:d6:c5:87:5a:7b:d5:46:d1:18:1b:
         85:5c:d2:fe:62:76:ff:94:e9:7a:db:32:99:51:9a:36:55:c4:
         b1:5e:f0:9a:0b:42:07:2e:ce:b6:84:d7:20:b6:51:ef:f6:c7:
         20:fd:7d:95:68:52:f3:91:6f:5e:5f:25:3f:13:ee:f2:8d:75:
         2c:ef:b4:26:43:c5:dc:af:78:9c:45:b7:04:87:b8:a1:fd:c3:
         f4:84:7e:91:97:12:02:ad:d9:16:5a:45:62:56:85:03:71:90:
         a9:cf:61:01:9b:6d:8d:9e:59:bc:fc:8f:46:de:27:db:71:e2:
         58:13:d2:fb:1b:e0:58:f0:9f:2d:3a:bc:ca:12:78:33:d3:7a:
         76:95:7e:53:c2:2b:4d:fb:6d:bb:92:8f:c6:28:0f:15:1d:af:
         7d:60:b5:a3:21:b3:66:e1:44:ab:91:10:85:d2:20:44:45:96:
         2c:14:3e:c1:87:92:ae:a9:d6:a9:84:2a:5e:15:6c:d8:bf:37:
         f2:33:2e:cc:64:49:ce:2c:e8:30:84:22:2c:b6:a9:c1:fc:30:
         97:48:d1:fa

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            52:00:e5:aa:25:56:fc:1a:86:ed:96:c9:d4:4b:33:c7
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=(c) 2006 VeriSign, Inc. - For authorized use only, CN=VeriSign Class 3 Public Primary Certification Authority - G5
        Validity
            Not Before: Feb  8 00:00:00 2010 GMT
            Not After : Feb  7 23:59:59 2020 GMT
        Subject: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=Terms of use at https://www.verisign.com/rpa (c)10, CN=VeriSign Class 3 Code Signing 2010 CA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:f5:23:4b:5e:a5:d7:8a:bb:32:e9:d4:57:f7:ef:
                    e4:c7:26:7e:ad:19:98:fe:a8:9d:7d:94:f6:36:6b:
                    10:d7:75:81:30:7f:04:68:7f:cb:2b:75:1e:cd:1d:
                    08:8c:df:69:94:a7:37:a3:9c:7b:80:e0:99:e1:ee:
                    37:4d:5f:ce:3b:14:ee:86:d4:d0:f5:27:35:bc:25:
                    0b:38:a7:8c:63:9d:17:a3:08:a5:ab:b0:fb:cd:6a:
                    62:82:4c:d5:21:da:1b:d9:f1:e3:84:3b:8a:2a:4f:
                    85:5b:90:01:4f:c9:a7:76:10:7f:27:03:7c:be:ae:
                    7e:7d:c1:dd:f9:05:bc:1b:48:9c:69:e7:c0:a4:3c:
                    3c:41:00:3e:df:96:e5:c5:e4:94:71:d6:55:01:c7:
                    00:26:4a:40:3c:b5:a1:26:a9:0c:a7:6d:80:8e:90:
                    25:7b:cf:bf:3f:1c:eb:2f:96:fa:e5:87:77:c6:b5:
                    56:b2:7a:3b:54:30:53:1b:df:62:34:ff:1e:d1:f4:
                    5a:93:28:85:e5:4c:17:4e:7e:5b:fd:a4:93:99:7f:
                    df:cd:ef:a4:75:ef:ef:15:f6:47:e7:f8:19:72:d8:
                    2e:34:1a:a6:b4:a7:4c:7e:bd:bb:4f:0c:3d:57:f1:
                    30:d6:a6:36:8e:d6:80:76:d7:19:2e:a5:cd:7e:34:
                    2d:89
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Certificate Policies:
                Policy: 2.16.840.1.113733.1.7.23.3
                  CPS: https://www.verisign.com/cps
                  User Notice:
                    Explicit Text: https://www.verisign.com/rpa

            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            1.3.6.1.5.5.7.1.12:
                0_.].[0Y0W0U..image/gif0!0.0...+..............k...j.H.,{..0%.#http://logo.verisign.com/vslogo.gif
            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://crl.verisign.com/pca3-g5.crl

            Authority Information Access:
                OCSP - URI:http://ocsp.verisign.com

            X509v3 Extended Key Usage:
                TLS Web Client Authentication, Code Signing
            X509v3 Subject Alternative Name:
                DirName:/CN=VeriSignMPKI-2-8
            X509v3 Subject Key Identifier:
                CF:99:A9:EA:7B:26:F4:4B:C9:8E:8F:D7:F0:05:26:EF:E3:D2:A7:9D
            X509v3 Authority Key Identifier:
                keyid:7F:D3:65:A7:C2:DD:EC:BB:F0:30:09:F3:43:39:FA:02:AF:33:31:33

    Signature Algorithm: sha1WithRSAEncryption
         56:22:e6:34:a4:c4:61:cb:48:b9:01:ad:56:a8:64:0f:d9:8c:
         91:c4:bb:cc:0c:e5:ad:7a:a0:22:7f:df:47:38:4a:2d:6c:d1:
         7f:71:1a:7c:ec:70:a9:b1:f0:4f:e4:0f:0c:53:fa:15:5e:fe:
         74:98:49:24:85:81:26:1c:91:14:47:b0:4c:63:8c:bb:a1:34:
         d4:c6:45:e8:0d:85:26:73:03:d0:a9:8c:64:6d:dc:71:92:e6:
         45:05:60:15:59:51:39:fc:58:14:6b:fe:d4:a4:ed:79:6b:08:
         0c:41:72:e7:37:22:06:09:be:23:e9:3f:44:9a:1e:e9:61:9d:
         cc:b1:90:5c:fc:3d:d2:8d:ac:42:3d:65:36:d4:b4:3d:40:28:
         8f:9b:10:cf:23:26:cc:4b:20:cb:90:1f:5d:8c:4c:34:ca:3c:
         d8:e5:37:d6:6f:a5:20:bd:34:eb:26:d9:ae:0d:e7:c5:9a:f7:
         a1:b4:21:91:33:6f:86:e8:58:bb:25:7c:74:0e:58:fe:75:1b:
         63:3f:ce:31:7c:9b:8f:1b:96:9e:c5:53:76:84:5b:9c:ad:91:
         fa:ac:ed:93:ba:5d:c8:21:53:c2:82:53:63:af:12:0d:50:87:
         11:1b:3d:54:52:96:8a:2c:9c:3d:92:1a:08:9a:05:2e:c7:93:
         a5:48:91:d3

Due to a dependency between the smartcard readers installed at our Windows clients, farious driver and software versions, an update to the current and properly signed Java extension provided by ActivIdentity was not possible. The most immediate solution to this issue was to remove the original code signing information from the file $JAVA_HOME/lib/ext/javasso.jar and – for security purposes – to re-sign it with a still valid code signing certificate from our internal CA which is trusted by our Windows clients.

Current versions of the Dell iDRAC have limitations with regard to the field length for the SSL CSR. These limitations violate RFC 5280 and lead to an invalid CSR and possibly an invalid SSL certificate.

The Dell iDRAC allows the protection of browser based access via HTTPS. In order to facilitate this, SSL certificates are being used. Initially and by default there is a self-signed SSL certificate installed on the iDRAC which is signed by the Dell CA. For additional security this self-signed SSL certificate can be replaced with one provided by the systems administrator. There are two ways to create an own SSL certificate for the Dell iDRAC.

The first one is to use the iDRACs integrated facility to create a SSL CSR. The appropriate form can be found in the iDRAC WebUI under:

iDRAC Settings → Network → SSL → Generate Certificate Signing Request (CSR) → Next

After filling in the fields Common Name, Organization Name, Organization Unit, Locality, State Name, Country Code and Email with the appropriate values and clicking on Generate a CSR is generated in the background and after a short time offered for download. This CSR can now be fed into a CA – own or service provider operated – in order to generate the actual certificate. The generated certificate is then uploaded into the iDRAC via the WebUI.

So far so easy, and done countless times without any issues. Until the day, one of the values of the aforementioned fields Common Name, Organization Name, Organization Unit, Locality, State Name, Country Code or Email for the CSR changes to a significantly longer string. In my case the value of Organization Name changed from 11 characters to 60 characters. It appeares that unfortunately the developers of the iDRAC didn't expect values of that length, although those are absolutely in conformance with the appropriate standard RFC 5280 and well within the upper bounds mentioned there:

[...]
--  specifications of Upper Bounds MUST be regarded as mandatory
--  from Annex B of ITU-T X.411 Reference Definition of MTS Parameter
--  Upper Bounds

-- Upper Bounds
ub-name INTEGER ::= 32768
ub-common-name INTEGER ::= 64
ub-locality-name INTEGER ::= 128
ub-state-name INTEGER ::= 128
ub-organization-name INTEGER ::= 64
ub-organizational-unit-name INTEGER ::= 64
[...]
ub-emailaddress-length INTEGER ::= 255
ub-common-name-length INTEGER ::= 64
ub-country-name-alpha-length INTEGER ::= 2
[...]

-- Note - upper bounds on string types, such as TeletexString, are
-- measured in characters.  Excepting PrintableString or IA5String, a
-- significantly greater number of octets will be required to hold
-- such a value.  As a minimum, 16 octets, or twice the specified
-- upper bound, whichever is the larger, should be allowed for
-- TeletexString.  For UTF8String or UniversalString at least four
-- times the upper bound should be allowed.
[...]

Even worse, the iDRAC doesn't issue an error or a warning if it's given an exceedingly long value. On the contrary, it silently accepts the longer value and just cuts off everything after the 48th or 51st character.

I've checked various versions of the Dell iDRAC and encountered the issue on at least the following Dell PowerEdge hardware and iDRAC software versions:

Here are some examples of the Subject: from the CSRs generated on different Dell PowerEdge hardware and iDRAC software versions:

user@host:~$ for FL in *.csr; do echo -n "${FL}: "; openssl req -text -in ${FL} -noout | grep "Subject: "; done
host1.csr: Subject: C=CC, ST=SSSSSS, L=LLLLLL, O=OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO, OU=OUOUOUOUOUOUOUOU, CN=FQDN/emailAddress=EMAIL
host2.csr: Subject: C=CC, ST=SSSSSS, L=LLLLLL, O=OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO, OU=OUOUOUOUOUOUOUOU, CN=FQDN/emailAddress=EMAIL
host3.csr: Subject: C=CC, ST=SSSSSS, L=LLLLLL, O=OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO, OU=OUOUOUOUOUOUOUOU, FQDN/emailAddress=EMAIL
host4.csr: Subject: C=CC, ST=SSSSSS, L=LLLLLL, O=OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO, OU=OUOUOUOUOUOUOUOU, CN=FQDN/emailAddress=EMAIL
host5.csr: Subject: C=CC, ST=SSSSSS, L=LLLLLL, O=OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO, OU=OUOUOUOUOUOUOUOU, CN=FQDN/emailAddress=EMAIL
host6.csr: Subject: C=CC, ST=SSSSSS, L=LLLLLL, O=OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO, OU=OUOUOUOUOUOUOUOU, CN=FQDN/emailAddress=EMAIL

The values in the different fields have been anonymized with the characters of the field name, but have otherwise been preserved in the length they appeared in the CSR. The observed behaviour is mostly consistent, with only one exception where the field value is strangely cut off after the 51st character.

Needless to say, cropping the field values like this results in an invalid CSR which in turn leads to issues in the certificate generation process at the CA.

After raising a support case with Dell, the problem was verified and the issue was acknowledged as a defect. A future iDRAC update is supposed to fix this issue. Unfortunately i have no information what version that will be nor when it will be released.

In the meantime, until there is a proper fix for this issue, the second way can be used. This is to manually create all three components – an own SSL private key, an own CSR and a certificate – for the Dell iDRAC. Unfortunately this involves significantly more steps, which are described below.

• If necessary install some additional software packages. For Debian based systems:

  root@host:~# apt-get install openssl wsmancli rpl

• In the following steps and commands, replace <FQDN> and <IP> with the FQDN and IP address of the Dell iDRAC where necessary.

• Create a SSL private key:

  user@host:~$ openssl genrsa -out <FQDN>.key.pem 4096

• Generate a CSR for the FQDN and IP address of the Dell iDRAC. Replace <C>, <ST>, <L>, <O>, <OU>, and <EMAIL> with the country, state, locality, organization, organizational-unit and a valid email address respectively.

  user@host:~$ openssl req -new -sha256 -nodes -key <FQDN>.key.pem -out <FQDN>.csr.pem -subj '/C=<C>/ST=<ST>/L=<L>/O=<O>/OU=<OU>/CN=<FDQN>/emailAddress=<EMAIL>' -config <(
  cat <<-EOF
  [req]
  req_extensions = req_ext
  distinguished_name = dn
  [ dn ]
  [ req_ext ]
  subjectAltName = @alt_names
  [alt_names]
  DNS.1 = <FQDN>
  IP.1 = <IP>
  EOF
  )

  Additional DNS.X = <FQDN> or IP.X = <IP> lines can be added if necessary. Increment the value of X accordingly.

• With the generated CSR, request a certificate from your own CA or the one of your service provider. The range of different CAs and organizational processes involved is just too broad to cover this step in greater detail.

• Merge the received certificate and the private key which was generated in one of the previous steps:

  user@host:~$ cat <FQDN>.cert.pem <FQDN>.key.pem > <FQDN>.certkey.pem

• Convert the combined certificate and the private key into the PKCS #12 format:

  user@host:~$ openssl pkcs12 -export -in <FQDN>.certkey.pem -out <FQDN>.certkey.pfx

  This command will request a password at the command line which will later on be needed again.

• Convert the PKCS #12 file into a base64-encoded file:

  user@host:~$ openssl base64 -in <FQDN>.certkey.pfx -out <FQDN>.certkey.p12

• Prepare the base64-encoded PKCS #12 file for the upload to the Dell iDRAC over its SOAP-interface:

  user@host:~$ echo "<p:SetCertificateAndPrivateKey_INPUT xmlns:p="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/root/dcim/DCIM_LCService"><p:Type>server</p:Type><p:PKCS12>" > <FQDN>.certkey.xml
  user@host:~$ cat <FQDN>.certkey.p12 >> <FQDN>.certkey.xml
  user@host:~$ echo "</p:PKCS12><p:PKCS12pin>PASSWORD_AT_PKCS12_EXPORT</p:PKCS12pin></p:SetCertificateAndPrivateKey_INPUT>" >> <FQDN>.certkey.xml

  Replace the string PASSWORD_AT_PKCS12_EXPORT with the password that was given in the previous step where the certificate was exported into the PKCS #12 format:

  user@host:~$ rpl PASSWORD_AT_PKCS12_EXPORT <Paßwort> <FQDN>.certkey.xml

• Upload the base64-encoded PKCS #12 file to the Dell iDRAC over its SOAP-interface. Replace <IP> with the IP address of the Dell iDRAC and <USER> and <PASSWORD> with the appropriate credentials of a user allowed to access the Dell iDRAC:

  user@host:~$ wsman invoke -a SetCertificateAndPrivateKey http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/root/dcim/DCIM_LCService?SystemCreationClassName=DCIM_ComputerSystem,CreationClassName=DCIM_LCService,SystemName=DCIM:ComputerSystem,Name=DCIM:LCService -h <IP> -P 443 -u <USER> -p <PASSWORD> -c dummy.cert -y basic -V -v -J <FQDN>.certkey.xml -j utf-8

The last two steps are optional and only necessary if you don't want to or cannot install the Dell OpenManage DRAC Tools, which include the racadm command, or if the remote access via racadm is not or cannot be enabled.

This workaround can also be applied generally and independently of the mentioned bug in the Dell iDRAC in order to create a valid SSL certificate for Dell iDRAC and Dell Blade CMC systems.