TLS 1.2, Server Name Indication (SNI) and SOAP via CXF

It seems that I am destined to become a deep expert in the vagaries of TLS these days.  My most recent challenge was in figuring out why Server Name Indication (SNI) extensions weren't simply working in my BC-FIPS implementation that I talked about in the last few posts.

Background on SNI

For a brief moment, let's talk a little about SNI.  TLS is a lower layer session protocol on top of TCP that encrypts communication.  HTTP and HTTPS are higher layer (Application) protocols on top of TLS.  When you connect to an IP address over TCP, then initiate a TLS connection, the application layer hasn't yet seen the HTTP request, let alone the Host header.  SNI serves, in TLS, the same function as the HTTP Host header.  Effectively, this works in the same way that the HTTP Host header does.

In HTTP, the Host header allows one server to service multiple web sites or DNS endpoints, but unless SNI is used each endpoint must be served with the same certificate, either using a wildcare or multiple alternate names. SNI allows one host to service multiple sites with different certificates for each site.

Integrating SNI with Apache CXF and BCFIPS

Reading through BCFIPS documentation, you'd think at first that all you need to do is enable SNI extensions by setting jsse.enableSNIExtension=true.  Sadly, that's not quite enough, as section 3.5.1 Server Name Identification states.

"... Unfortunately, when using HttpsURLConnection SunJSSE uses some magic (reflection and/or internal API) to tell the socket about the "original hostname" used for the connection, and we cannot use that same magic as it is internal to the JVM. 

To allow the endpoint validation to work properly you need to make use of one of three workarounds:"

And then goes on further to suggest the recommended workaround as follows:

3. The third (and recommended) alternative is to set a customized SSLSocketFactory on the HttpsURLConnection, then intercept the socket creation call and manually set the SNI host_name on the created socket. We provide a utility class to make this simple, as shown in the example code below.  

// main code block 

{   SSLContext sslContext = ...;

     URL serverURL = ...;

     URLConnectionUtil util = new URLConnectionUtil();

     HttpsURLConnection conn =  

        (HttpsURLConnection)util.openConnection(serverURL);

}

That's pretty simple.  What URLConnectionUtil.openConnection does is wrap the socket factory provided by conn (see HttpsURLConnection.setSSLSocketFactory) with one that calls a method to set the server name extension in createSocket after calling the original createSocket method found in the connection.

So, looking at CXF, it's the HttpURLConnectionFactory class that calls url.openConnection.  We could simply override that class and replace with a call to util.openConnection, according the code in that class.  Here's the original.

    public HttpURLConnection createConnection(TLSClientParameters tlsClientParameters,
        Proxy proxy, URL url) throws IOException {
        HttpURLConnection connection =
            (HttpURLConnection) (proxy != null ? url.openConnection(proxy) : url.openConnection());
        if (HTTPS_URL_PROTOCOL_ID.equals(url.getProtocol())) {
            if (tlsClientParameters == null) {
                tlsClientParameters = new TLSClientParameters();
            }
            try {
                decorateWithTLS(tlsClientParameters, connection);
            } catch (Throwable ex) {
                throw new IOException("Error while initializing secure socket", ex);
            }
        }
        return connection;
    }

And my modest adjustment to the first two lines:

        URLConnectionUtil util = new URLConnectionUtil(
            tlsClientParameters == null ? null : tlsClientParameters.getSSLSocketFactory()
        );
        HttpURLConnection connection =
            (HttpURLConnection) (proxy != null ? util.openConnection(url, proxy) : util.openConnection(url));

But for some reason, that didn't work.

Debugging this, what I found was that the decorateWithTLS method also wraps connection's socket factory, but it fails to actually look at the server socket factory that may have already been set on the HttpsUrlConnection that was passed into it.

Here's a picture of that method.

It goes on for almost another 100 lines, doing all sorts of weird gyrations that low level code that needs to work with multiple libraries often to, including reflection and a bunch of other oddities.

What's missing here, is an initial check to see if connection is already an HttpsURLConnection, and if so, if it's already got an SSL Socket Factory set other than the default.  In that situation, that's the socket factory (created by URLConnectionUtil) that needs to be wrapped yet again.  Looking through everything this method does, I realized:

1. I don't care about other than JSSE implementations.
2. My socketFactory is always set when I enter this method, and that's the one to use.

So, I replaced the middle if statement in my overridden function with:

    if (HTTPS_URL_PROTOCOL_ID.equals(url.getProtocol())) {
        if (tlsClientParameters == null) {
            tlsClientParameters = new TLSClientParameters();
        }
        HostnameVerifier verifier = SSLUtils.getHostnameVerifier(tlsClientParameters);
        connection.setHostnameVerifier(verifier);
    }

Which very much simplifies everything, as all the decorateWithTLS does of interest for me is to set the host name verifier.

So, that is how I enabled SNI with BCFIPS in an older version of Apache CXF.  There's other code you will need as well, because you'll have to get that subclass that creates the connection into the factory used by the Conduit.  That's outlined below.

public class HTTPConduit extends URLConnectionHTTPConduit {
    public static class Factory implements HTTPConduitFactory {
        @Override
        public org.apache.cxf.transport.http.HTTPConduit createConduit(HTTPTransportFactory f, Bus b,
            EndpointInfo localInfo, EndpointReferenceType target) throws IOException {

            HTTPConduit conduit = new HTTPConduit(b, localInfo, target);
            // Perform any other conduit configuration here
            return conduit;
        }
    }
    public HTTPConduit(Bus b, EndpointInfo ei, EndpointReferenceType t) throws IOException {
        super(b, ei, t);
        // Override the default connectionFactory.
        connectionFactory = new ConnectionFactory();
    }
}

Elsewhere in your application, you should include an @Bean declaration to create that bean in one of your configuration classes.

@Configuration class MyAppConfig {

    // ...

   @Bean HTTPConduitFactory httpConduitFactory() {

      return new HTTPConduit.Factory();

   }

    ...

}

Debugging TLS Protocol Failures in BC-FIPS and Spring Applications

Debugging TLS protocol failures can be a nightmare.  With JSSE, you can use the old standby java JVM option: 
    -Djavax.net.debug=ssl,handshake,
data,trustmanager,help
 
to get detailed reporting of what is happening.  Usually that provides more than enough (in fact too much) information to debug the protocol problem, but when using BCFIPS, guess what, it doesn't work anymore.  Why? Well, while these command line arguments make debugging easier, they also transmit decrypted information to the console, which is a huge leak of encrypted information.

So, what's a developer to do?

BCFIPS uses java.util.logging to provide reports on protocol failures.  These reports do NOT include decrypted information, but do include enough information to tell you exactly where the protocol failure happened.  But to enable java.util.logging to work with a SpringBoot application using Logback as its logging agent you have to jump through just a few small hoops.

First, you need to include jul-to-slf4j in your dependencies.  This is a bridge between java.util.logging and SLF4J implementations.

        <dependency>
          <groupId>org.slf4j</groupId>
          <artifactId>jul-to-slf4j</artifactId>
        </dependency>

Next you'll need to activate the bridge during application startup.  It's a good idea to do this as early as possible (before bean loading even).

      import org.slf4j.bridge.SLF4JBridgeHandler;

        ...

      public static void main(String ... args) {
          SLF4JBridgeHandler.removeHandlersForRootLogger();
          SLF4JBridgeHandler.install();

          ...

Once you've done all of the above, you will start getting BCFIPS logs reported via Logback.  But Logback and the SLF4J Bridge has a cost, so you want to add a bit of optimization.  You'll want to avoid some of the extra cost by implementing the LevelChangePropagator to propogate LogBack configuration back to JUL so that you don't have to worry about some of the overhead for disabled logging methods. 

<configuration>
  <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
    <!-- reset all previous level configurations of all j.u.l. loggers -->
    <resetJUL>true</resetJUL>
  </contextListener>

To enable reporting on protocol errors, 

  <!-- Enable BC Debug Logging by setting level to DEBUG or TRACE -->
  <logger name="org.bouncycastle.jsse.provider" level="INFO"/>

Once you've done all of the above, you will start getting your logs reported to Logback.

I tracked down my problem to an issue with TLS 1.2 Renegotiation, where my client was trying to connect to a server that first allowed the connection, and then renegotiated with client authentication to get to my client certificate.  BCFIPS disables renegotiation by default, to enable it under a limited set of circumstances (that are secure) you can add:
    -Dorg.bouncycastle.jsse.client.acceptRenegotiation=true
to your java command line, or set it in System properties at application startup.

Dynamically Reloading TLS Trust and Identity Material

Wouldn't it be nice if you didn't have to restart your server to dynamically update keys, certificates or trust stores?  I've spend a good bit of time on this across both client and server implementations and so I have a few pointers.  If you've read the last two posts, you know I've been working through requirements and implementation.  Now I'm going to add this auto-renewal of trust and key material to that effort.

Most folks will just need to deal with setting up trust and key managers for their web application.  That's fairly straightforward.  The challenge that I face with this particular application is that there are at least three different ways that trust and key material is provided to the underlying application, depending on how the connection is handled.

The basic idea is to set up a polling thread that periodically checks for changes in trust material, and then when that happens, go off and single anyone that has registered to those change events to update trust material in whatever way they need.

For my uses, inbound connections go through the server, which is what most will have to deal with.  But I also have two different types of outbound connections which are configured in different ways.  Some are SOAP using Apache CXF, others are RESTful API calls made through the HttpsURLConnection class (those APIs aren't that difficult to work with, and so don't need much more).  But each requires a different way to communicate trust and identity material to the system.

Let's start with the first, and most common:

Since Apache Tomcat 8.5 there is an API that enables you to reload key and trust material through the protocol handler for the connection.  During Embedded Tomcat setup (if you do it programatically), you create a Connector and add it to the service.  This connector is where you will add the SSLHostConfig and setup the protocol parameters (e.g., connection timeout, max connections), et cetera through a protocol handler derived from AbstractHttp11Protocol.

Somewhere in this process you will eventually wind up with three things:

1. The Connector connector.
2. The SSLHostConfig configuration.
3. The protocol handler nioProtocol.

    // Configure SSL

    connector.addSslHostConfig(configuration);

    // Get the protocol handler

    Http11NioProtocol nioProtocol = (Http11NioProtocol) connector.getProtocolHandler();

    // Do any configuration to it to the protocol handler.

        ...

After all of this is where you add the magic.  What you are doing here is calling a method to add a runnable to a list of methods to call when trust or key material needs to be updated.  I use this model because three different components need to do something to update trust and key material in the system I'm working with.

    // set up to reload configuration.

    addSslTrustChangedListener(() -> nioProtocol.reloadSslHostConfigs());

My actual implementation of the runnable is a little more complex, because I reuse portions of code that access key and trust stores, but generally, the main idea is to call reloadSslHostConfigs() to force a refresh of key and trust material.

CXF is a bit easier.  I'm still using XML configuration for the HTTPConduit that is used, but the for the bean containing the TLSClientParameters on that conduit, I set up a runnable to refresh the socket factory thus:

    @Bean(name="tlsParamsClientWs")
    public TLSClientParameters getTLSClientParameters() {
        TLSClientParameters p = new TLSClientParameters();
        // Force reload of Socket Factory
        p.setSSLSocketFactory(getSocketFactory());
        // Add listener to update the factory.
        addSslTrustChangedListener(() -> p.setSSLSocketFactory(getSocketFactory(true)));
        return p;
    }

This method constructs the bean that contains the client parameters, and the adds a listener that forces an update of the SSLSocketFactory.  You may be able to just update the parameters and let the factory be created for you, I need a bit more control for my application.  Note: getSocketFactory() and getSocketFactory(boolean forceReload) methods aren't shown here.

For my outbound restful connections which for now use HttpUrlConnection since they aren't that complicated, I have one last method which relies on bean that that eventually calls the getSocketFactory() method referenced above.

This enables all of my inbound and outbound connections to dynamically response to updates in trust material with the addition of a scheduled executor that checks for changes to files every 10 seconds (configurable), and then calls each trust changed listener (catching exceptions inside the loop so that an exception thrown by any single listener doesn't break the next one.

I'm not going to reproduce all of the code, it's fairly straightforward.  You can use something like the Java WatchService (see https://dzone.com/articles/how-watch-file-system-changes) or working with commons.io.monitor classes.

This is the basic idea though:

public void startMonitoring() {

            ScheduledExecutorService s = Executors.newSingleThreadScheduledExecutor();

            s.scheduleAtFixedRate(this::updateTrust, 10, 10, TimeUnit.SECONDS);

        }

public void updateTrust() {

try {

if (checkForUpdates()) {

for (Runnable trustChangedListener  : trustChangedListeners) {

try {

trustChangedListener.run();

} catch (Exception e) {

LOGGER.error("Failed to update trust material", e);

}

}

reloadCount = getReloadCount() + 1;

clientStoreOutOfDate = serverStoreOutOfDate = false;

LOGGER.info("Key and Trust stores updated.");

}

} catch (IOException e) {

LOGGER.error("Could not determine trust material update status", e);

}

}

 

You will probably have to do a bit of work to make this operate in your own environment, but now you can see how to integrate it with both server and client endpoints in several different ways.