Wednesday, May 1, 2024

Running BCFIPS in SpringBoot 3.2

I'm rewriting wrappers for a Spring Boot 1.5 application using Spring Boot 3.2 to help me eliminate some older dependencies I cannot even get patches for anymore.  So now I have to make my new Spring Boot application work with the latest Bouncy Castle FIPS code.

I've previously mentioned the NIST Certified Bouncy Castle FIPS TLS libraries in other posts.  SOAP is tremendously complicated to configure and manage.  REST is so much easier, and when you don't need all of the power of a SOAP stack, you can certainly send SOAP messages in XML quite readily using a normal REST transport layer.  You may need to write some extra marshalling code, but for standardized interfaces, that doesn't tend to be a big lift and there are plenty of existing tools to help (e.g., Faster Jackson XML).

There are a few challenges. One of these stems from the need for the BC-FIPS modules to validate themselves on startup.  This uses jar validation processes, but the code running the validation needs to be able to find itself.  That gets complicated by a minor change in Spring Boot 3.2 URL format when it repackages the application into an uber-jar. The URL conflicts with what the BC modules expect, and then they won't validate.  While the fix is a one-liner in BC-FIPS, it's a change to a certified software module, which means that it has to go through processes which make it take longer than anyone would like.

Spring Boot uber class loading has the ability to target classes which need to be unpacked from the Uber-jar before loading. The JAR files will then be unpacked from the uber-jar into your TEMP folder before being loaded, and those will be accessible from a different kind of URL, that of the JAR file, which BCFIPS knows how to resolve.

To make this work, you will need to update your configuration for the spring-boot-maven-plugin to tell it which jar files need to be unpacked.  The necessary lines of code are below:

Rewriting my wrapper to ditch CXF and modernize to the latest Spring Boot will save a lot on future maintenance efforts and is already making the application much simpler.  There's plenty of glue code needed to run inside the CXF framework that can just be done away with.
As you now know, I had to go back and rework for SNI again too.  It turns out the same fix was basically needed, calling URLConnectionUtil.openConnection().

import org.bouncycastle.jsse.util.URLConnectionUtil;


   * This method obtains an SNI enabled Server Socket Factory
   * @param location The URL to obtain a connection for
   * @return An HttpURLConnection that supports SNI if the URL starts with HTTPS or has no URL Scheme.
   * @throws IOException
  public HttpURLConnection getSNIEnabledConnection(URL location) throws IOException {
    String protocol = location.getProtocol();
    if (protocol == null || !protocol.startsWith("https")) {
      // Not HTTPS
      return (HttpURLConnection) location.openConnection();
    URLConnectionUtil util = new URLConnectionUtil(getSSLContext().getSocketFactory());
    HttpsURLConnection conx = (HttpsURLConnection) util.openConnection(location);
    return conx;

More fun with SNI and TLS with Akamai edge servers and Bouncy Castle internal_error(80)

Adam Savage "Well, there's your problem"Recently, endpoints that one of the systems I maintain frequently connects to underwent a change in how they hosted their servers.  They were moved into an Akamai edge server framework, which requires use of the Server Name Indicator (SNI) extension in TLS communications.  This isn't routinely enabled in a straight Java JSSE client connection, especially when using Bouncy Castle FIPS.  As I previously stated, you have to configure it in your code.

My guess is that when a request is made without the SNI TLS extension, the Akamai edge environment reports a TLS error.  Sadly, Akamai reports this TLS error using the TLS internal_error(80) fatal alert code instead of the more descriptive missing_extension(109).  That's because missing_extension(109) wasn't defined until TLS 1.3 (which is much more expressive of the reason for failure).

On my side, I had half-enabled SNI, where it was used for the main protocol request and response, but wasn't being used for the requests sent to the underlying authorization protocol (OAuth 2.0).  The reason that was missed was simply because it is a completely independent module.

If you've got WireShark, and you see something like this in your capture, look at the list of extensions.

If server_name extension is missing, as in the above capture, you need to fix your code to use SNI.  Once you've fixed it, it should look like this: 

In any case, if you are using BC FIPS, and get an internal_error(80) while trying to connect to an Akamai Edge Server, and your TLS Client Hello doesn't contain the SNI Extension: Well, there's your problem.

Wednesday, April 10, 2024

The nastiest code I ever wrote

Creating a mock that injects random errors into the expected responses is necessary to ensure that your server resiliency features work as intended. Doing so can be very challenging, especially if your mock has to simulate complex behavior.  FWIW, I call it a mock, because it is, but the endpoint is a test endpoint used by a production environment to verify that the server is operating correctly.

What appears below all-out wins the award for the nastiest, most evil code I've ever written that appears in a production system.

resp = ReflectionUtils.unwrapResponse(resp);
if (resp instanceof ResponseFacade rf) {
  try {
   // Drill through Tomcat internals using reflection to get to the underlying socket and SHUT it down hard.
    NioEndpoint.NioSocketWrapper w
      ReflectionUtils.getField(rf, "response.coyoteResponse.hook.socketWrapper", NioEndpoint.NioSocketWrapper.class);
    ReflectionUtils.attempt(() -> w.getSocket().getIOChannel().shutdownInput());  
    ReflectionUtils.attempt(() -> w.getSocket().getIOChannel().shutdownOutput());
  } catch (Exception e) {
    log.warn("Cannot access socket: {}", e);
} else {
  throw new RuntimeException("Simulated Socket Reset", new SocketException("Connection Reset [simulated]"));

It is used for a mock endpoint developed to performance test an application that sends messages to other systems.  The purpose of this is to force a reset on the server socket being used to response to a request.  The mock endpoint receives a request, and randomly shuts down the socket to help test the resilience and performance of the main application.

This delves deeply into Tomcat internals and uses a few reflection utility functions that I wrote.

The unwrapResponse() function just gets to the inner most HttpServletResponse assuming that is the beast that is closed to Tomcat internals (in my case, it is).

public static HttpServletResponse unwrapResponse(HttpServletResponse servletResp) {
  return ((servletResp instanceof HttpServletResponseWrapper w) ?
    unwrapResponse((HttpServletResponse)w.getResponse()) : servletResp);

The getField() method breaks the string given as the second argument into parts around the period (.) character, and then recursively extracts the object at that field, returning it.  The last argument is the type of the object.  I'll leave this one as an exercise for the reader.  These lines of code First get the socket from socket wrapper of the action hook in the CoyoteResponse of the Response object that's backing the outermost HttpServletResponse object given in resp.

The attempt() method just calls the callable (which is allowed to throw an exception), and returns, regardless of whether or not the attempt succeeds.  This is because I don't want this code to fail.

What's inside that callable is the critical code.  That uses the socketWrapper  to get the NIOChannel, and force a shutdown on both sides of the socket.  This is the hardest I can crash an inbound connection without messing with operating system calls, and it works in Tomcat 8.5, and I'm pretty certain it will work in Tomcat 10 as well.  I'll know in a couple of days once I get done migrating.

This is truly evil code; however, it wouldn't exist without a purpose.  That is to test how the system responds when a socket goes randomly belly-up during a performance test.  Yes, this code is randomly triggered during performance testing whilst the application is attempting to make requests.  As much as possible, I've basically installed a robot walking through the server room and randomly pulling cables from the endpoint the application is trying to get to, without having any access to that room or the cables.

This is basically the code that it getting invoked by this ugliness (that link is for Java 8, I'm actually using JDK 17 and previously used JDK11).  You might find this technique useful yourself.  I find it's a lot better than having to pull my wired connection a dozen or more times for my weekly performance test.  Yes, this runs in a scheduled automated test at least once a week.

Thursday, July 27, 2023

TLS 1.2, Server Name Indication (SNI) and SOAP via CXF

It seems that I am destined to become a deep expert in the vagaries of TLS these days.  My most recent challenge was in figuring out why Server Name Indication (SNI) extensions weren't simply working in my BC-FIPS implementation that I talked about in the last few posts.

Background on SNI

For a brief moment, let's talk a little about SNI.  TLS is a lower layer session protocol on top of TCP that encrypts communication.  HTTP and HTTPS are higher layer (Application) protocols on top of TLS.  When you connect to an IP address over TCP, then initiate a TLS connection, the application layer hasn't yet seen the HTTP request, let alone the Host header.  SNI serves, in TLS, the same function as the HTTP Host header.  Effectively, this works in the same way that the HTTP Host header does.

In HTTP, the Host header allows one server to service multiple web sites or DNS endpoints, but unless SNI is used each endpoint must be served with the same certificate, either using a wildcare or multiple alternate names. SNI allows one host to service multiple sites with different certificates for each site.

Integrating SNI with Apache CXF and BCFIPS

Reading through BCFIPS documentation, you'd think at first that all you need to do is enable SNI extensions by setting jsse.enableSNIExtension=true.  Sadly, that's not quite enough, as section 3.5.1 Server Name Identification states.

"... Unfortunately, when using HttpsURLConnection SunJSSE uses some magic (reflection and/or internal API) to tell the socket about the "original hostname" used for the connection, and we cannot use that same magic as it is internal to the JVM. 

To allow the endpoint validation to work properly you need to make use of one of three workarounds:"
And then goes on further to suggest the recommended workaround as follows:

3. The third (and recommended) alternative is to set a customized SSLSocketFactory on the HttpsURLConnection, then intercept the socket creation call and manually set the SNI host_name on the created socket. We provide a utility class to make this simple, as shown in the example code below.  
// main code block 
{   SSLContext sslContext = ...;
     URL serverURL = ...;
     URLConnectionUtil util = new URLConnectionUtil();
     HttpsURLConnection conn =  
That's pretty simple.  What URLConnectionUtil.openConnection does is wrap the socket factory provided by conn (see HttpsURLConnection.setSSLSocketFactory) with one that calls a method to set the server name extension in createSocket after calling the original createSocket method found in the connection.

So, looking at CXF, it's the HttpURLConnectionFactory class that calls url.openConnection.  We could simply override that class and replace with a call to util.openConnection, according the code in that class.  Here's the original.

    public HttpURLConnection createConnection(TLSClientParameters tlsClientParameters,
        Proxy proxy, URL url) throws IOException {
        HttpURLConnection connection =
            (HttpURLConnection) (proxy != null ? url.openConnection(proxy: url.openConnection());
        if (HTTPS_URL_PROTOCOL_ID.equals(url.getProtocol())) {
            if (tlsClientParameters == null) {
                tlsClientParameters = new TLSClientParameters();
            try {
                decorateWithTLS(tlsClientParameters, connection);
            } catch (Throwable ex) {
                throw new IOException("Error while initializing secure socket", ex);
        return connection;

And my modest adjustment to the first two lines:

        URLConnectionUtil util = new URLConnectionUtil(
            tlsClientParameters == null ? null : tlsClientParameters.getSSLSocketFactory()
        HttpURLConnection connection =
            (HttpURLConnection) (proxy != null util.openConnection(url, proxyutil.openConnection(url));

But for some reason, that didn't work.

Debugging this, what I found was that the decorateWithTLS method also wraps connection's socket factory, but it fails to actually look at the server socket factory that may have already been set on the HttpsUrlConnection that was passed into it.

Here's a picture of that method.

It goes on for almost another 100 lines, doing all sorts of weird gyrations that low level code that needs to work with multiple libraries often to, including reflection and a bunch of other oddities.

What's missing here, is an initial check to see if connection is already an HttpsURLConnection, and if so, if it's already got an SSL Socket Factory set other than the default.  In that situation, that's the socket factory (created by URLConnectionUtil) that needs to be wrapped yet again.  Looking through everything this method does, I realized:
  1. I don't care about other than JSSE implementations.
  2. My socketFactory is always set when I enter this method, and that's the one to use.
So, I replaced the middle if statement in my overridden function with:

    if (HTTPS_URL_PROTOCOL_ID.equals(url.getProtocol())) {
        if (tlsClientParameters == null) {
            tlsClientParameters = new TLSClientParameters();
        HostnameVerifier verifier = SSLUtils.getHostnameVerifier(tlsClientParameters);

Which very much simplifies everything, as all the decorateWithTLS does of interest for me is to set the host name verifier.

So, that is how I enabled SNI with BCFIPS in an older version of Apache CXF.  There's other code you will need as well, because you'll have to get that subclass that creates the connection into the factory used by the Conduit.  That's outlined below.

public class HTTPConduit extends URLConnectionHTTPConduit {
    public static class Factory implements HTTPConduitFactory {
        public org.apache.cxf.transport.http.HTTPConduit createConduit(HTTPTransportFactory f, Bus b,
            EndpointInfo localInfo, EndpointReferenceType target) throws IOException {

         HTTPConduit conduit = new HTTPConduit(b, localInfo, target);
            // Perform any other conduit configuration here
            return conduit;
    public HTTPConduit(Bus b, EndpointInfo ei, EndpointReferenceType t) throws IOException {
        super(b, ei, t);
        // Override the default connectionFactory.
        connectionFactory = new ConnectionFactory();

Elsewhere in your application, you should include an @Bean declaration to create that bean in one of your configuration classes.

@Configuration class MyAppConfig {
    // ...
   @Bean HTTPConduitFactory httpConduitFactory() {
      return new HTTPConduit.Factory();

Thursday, July 13, 2023

Debugging TLS Protocol Failures in BC-FIPS and Spring Applications

Debugging TLS protocol failures can be a nightmare.  With JSSE, you can use the old standby java JVM option:,handshake,
to get detailed reporting of what is happening.  Usually that provides more than enough (in fact too much) information to debug the protocol problem, but when using BCFIPS, guess what, it doesn't work anymore.  Why? Well, while these command line arguments make debugging easier, they also transmit decrypted information to the console, which is a huge leak of encrypted information.

So, what's a developer to do?

BCFIPS uses java.util.logging to provide reports on protocol failures.  These reports do NOT include decrypted information, but do include enough information to tell you exactly where the protocol failure happened.  But to enable java.util.logging to work with a SpringBoot application using Logback as its logging agent you have to jump through just a few small hoops.

First, you need to include jul-to-slf4j in your dependencies.  This is a bridge between java.util.logging and SLF4J implementations.


Next you'll need to activate the bridge during application startup.  It's a good idea to do this as early as possible (before bean loading even).

      import org.slf4j.bridge.SLF4JBridgeHandler;


      public static void main(String ... args) {


Once you've done all of the above, you will start getting BCFIPS logs reported via Logback.  But Logback and the SLF4J Bridge has a cost, so you want to add a bit of optimization.  You'll want to avoid some of the extra cost by implementing the LevelChangePropagator to propogate LogBack configuration back to JUL so that you don't have to worry about some of the overhead for disabled logging methods. 

  <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
    <!-- reset all previous level configurations of all j.u.l. loggers -->

To enable reporting on protocol errors, 

  <!-- Enable BC Debug Logging by setting level to DEBUG or TRACE -->
  <logger name="org.bouncycastle.jsse.provider" level="INFO"/>

Once you've done all of the above, you will start getting your logs reported to Logback.

I tracked down my problem to an issue with TLS 1.2 Renegotiation, where my client was trying to connect to a server that first allowed the connection, and then renegotiated with client authentication to get to my client certificate.  BCFIPS disables renegotiation by default, to enable it under a limited set of circumstances (that are secure) you can add:
to your java command line, or set it in System properties at application startup.

Monday, July 10, 2023

Dynamically Reloading TLS Trust and Identity Material

Wouldn't it be nice if you didn't have to restart your server to dynamically update keys, certificates or trust stores?  I've spend a good bit of time on this across both client and server implementations and so I have a few pointers.  If you've read the last two posts, you know I've been working through requirements and implementation.  Now I'm going to add this auto-renewal of trust and key material to that effort.

Most folks will just need to deal with setting up trust and key managers for their web application.  That's fairly straightforward.  The challenge that I face with this particular application is that there are at least three different ways that trust and key material is provided to the underlying application, depending on how the connection is handled.

The basic idea is to set up a polling thread that periodically checks for changes in trust material, and then when that happens, go off and single anyone that has registered to those change events to update trust material in whatever way they need.

For my uses, inbound connections go through the server, which is what most will have to deal with.  But I also have two different types of outbound connections which are configured in different ways.  Some are SOAP using Apache CXF, others are RESTful API calls made through the HttpsURLConnection class (those APIs aren't that difficult to work with, and so don't need much more).  But each requires a different way to communicate trust and identity material to the system.

Let's start with the first, and most common:

Since Apache Tomcat 8.5 there is an API that enables you to reload key and trust material through the protocol handler for the connection.  During Embedded Tomcat setup (if you do it programatically), you create a Connector and add it to the service.  This connector is where you will add the SSLHostConfig and setup the protocol parameters (e.g., connection timeout, max connections), et cetera through a protocol handler derived from AbstractHttp11Protocol.

Somewhere in this process you will eventually wind up with three things:

  1. The Connector connector.
  2. The SSLHostConfig configuration.
  3. The protocol handler nioProtocol.
    // Configure SSL
    // Get the protocol handler
    Http11NioProtocol nioProtocol = (Http11NioProtocol) connector.getProtocolHandler();
    // Do any configuration to it to the protocol handler.

After all of this is where you add the magic.  What you are doing here is calling a method to add a runnable to a list of methods to call when trust or key material needs to be updated.  I use this model because three different components need to do something to update trust and key material in the system I'm working with.

    // set up to reload configuration.
    addSslTrustChangedListener(() -> nioProtocol.reloadSslHostConfigs());

My actual implementation of the runnable is a little more complex, because I reuse portions of code that access key and trust stores, but generally, the main idea is to call reloadSslHostConfigs() to force a refresh of key and trust material.

CXF is a bit easier.  I'm still using XML configuration for the HTTPConduit that is used, but the for the bean containing the TLSClientParameters on that conduit, I set up a runnable to refresh the socket factory thus:

    public TLSClientParameters getTLSClientParameters() {
        TLSClientParameters p = new TLSClientParameters();
        // Force reload of Socket Factory
        // Add listener to update the factory.
        addSslTrustChangedListener(() -> p.setSSLSocketFactory(getSocketFactory(true)));
        return p;

This method constructs the bean that contains the client parameters, and the adds a listener that forces an update of the SSLSocketFactory.  You may be able to just update the parameters and let the factory be created for you, I need a bit more control for my application.  Note: getSocketFactory() and getSocketFactory(boolean forceReload) methods aren't shown here.

For my outbound restful connections which for now use HttpUrlConnection since they aren't that complicated, I have one last method which relies on bean that that eventually calls the getSocketFactory() method referenced above.

This enables all of my inbound and outbound connections to dynamically response to updates in trust material with the addition of a scheduled executor that checks for changes to files every 10 seconds (configurable), and then calls each trust changed listener (catching exceptions inside the loop so that an exception thrown by any single listener doesn't break the next one.

I'm not going to reproduce all of the code, it's fairly straightforward.  You can use something like the Java WatchService (see or working with classes.

This is the basic idea though:

public void startMonitoring() {
            ScheduledExecutorService s = Executors.newSingleThreadScheduledExecutor();
            s.scheduleAtFixedRate(this::updateTrust, 10, 10, TimeUnit.SECONDS);

public void updateTrust() {
try {
if (checkForUpdates()) {
for (Runnable trustChangedListener  : trustChangedListeners) {
try {;
} catch (Exception e) {
LOGGER.error("Failed to update trust material", e);
reloadCount = getReloadCount() + 1;
clientStoreOutOfDate = serverStoreOutOfDate = false;"Key and Trust stores updated.");
} catch (IOException e) {
LOGGER.error("Could not determine trust material update status", e);

You will probably have to do a bit of work to make this operate in your own environment, but now you can see how to integrate it with both server and client endpoints in several different ways.

Friday, June 30, 2023

Addressing technical challenges with BC-FIPS

Last week I talked about the requirements for implementing TLS and a certified encryption module (specifically Bouncy Castle FIPS or BC-FIPS).  Today I'm going to tell you a bit more about technically how one my go about this, and the specific technical challenges that you may run into.

First of all, BC-FIPS provides some installation instructions that a) no longer work with JDK-11, and b) also don't play well with Spring Boot uber-jar class loading using standard Classpath override mechanisms.  I never found root cause for this problem, all I wound up doing was simply dynamically loaded the BC-FIPS security modules at application startup.

These (non-working) instructions include modifications needed to the JDK, specifically the file and lib/ext folders.

There are three aspects of this configuration:

  1. Creating a compliant SecureRandom (this is described in the BC-FIPS documentation).
  2. Installing the BC FIPS Security Provider
  3. Installing the BC JSSE Security Provider

I do this in a static method BEFORE database initialization.  The reason for this is that DB initialization code needs to be able to get a FIPS compliant socket factory to initialize the connection pool.

private static void init() {
    // This is necessary initialization to use BCFKS module
    Security.insertProviderAt(new BouncyCastleFipsProvider(), 1);
    Security.insertProviderAt(new BouncyCastleJsseProvider(), 2);

 * Generate a a NIST SP 800-90A compliant secure random number
 * generator.
 * @return A compliant generator.
private static SecureRandom getSecureRandom() {

     * According to NIST Special Publication 800-90A, a Nonce is
     * A time-varying value that has at most a negligible chance of
     * repeating, e.g., a random value that is generated anew for each    
     * use, a timestamp, a sequence number, or some combination of
     * these.
     * The nonce is combined with the entropy input to create the initial
     * DRBG seed.
    byte [] nonce = ByteBuffer.allocate(8).putLong(System.nanoTime()).array();
    EntropySourceProvider entSource = new BasicEntropySourceProvider(new SecureRandom(), true);
    FipsDRBG.Builder drgbBldr = FipsDRBG.SHA512
    return, true);

The code above effectively does what making changes to the JDK's (as recommended by BC-FIPS documentation).  I make all the recommended changes except the ones that initialize the security providers, because I cannot configure the JDK to load the BC classes from the lib/ext folder since that is no longer supported in JDK-11.  The alternative suggested is to put the location of those classes on your classpath during application startup.  However, I also discovered that doesn't work, likely due to conflicts with how uber-jar classloading works (as in fact, those classes technically are on the classpath in the uber-jar).  I also swap out the default keystore format from JKS to BCFKS to ensure compliance with BC-FIPS KeyStore requirements.  Technically JKS is fine for Certificate stores, but frankly, I didn't even want to enable JKS support in case something broke somewhere else.

If your database is in the cloud (e.g., AWS or Azure), you may need to add a certificate to cacerts to enable the database connection using JSSE (BC-FIPS or native Java JSSE code).  I just do this to the cacerts file in the deployed JDK.

    keytool -keystore cacerts -storepass SECRET -noprompt -trustcacerts -importcert -alias awscert -file certificate.der

Next, to convert cacerts to BCFIPS format, this is what you will need to do:

    keytool -importkeystore -srckeystore cacerts -srcstoretype JKS -srcstorepass changeit \

        -destkeystore jssecacerts -deststorepass changeit -deststoretype BCFKS -providername BCFIPS \

        -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath lib/bc-fips-

This does the conversion, and will create a new file "jssecacerts" in the BCFKS format.  The JDK looks for jssecacerts before cacerts, and so now I have both formats still hanging around in case I need them.

A simpler way to do this conversion is with KeyStore Explorer, I tool I often use to inspect/modify key and trust store content.  This tool already has BCFKS support built in, even if it may not be BCFIPS compliant (straight BC also supports the BCFKS format, it's just not a certified component).

Finally, you'll have to change how you configure SSL/TLS for your server and/or client components.  Our system programatically configures using beans for KeyStore, TrustStore, et cetera, but other servers may just use property or configuration values (e.g., server.xml for Tomcat).

Anywhere the default keystore type is present, you'll need to change the type of keystore to BCFKS, and if the provider type is specified, you'd use BCFIPS (as for keytool commands above).

If you want to get a KeyManagerFactory, TrustManagerFactory, or SSLContext programatically, here's how you'd get those:

KeyManagerFactory keyMgrFact = KeyManagerFactory.getInstance("PKIX", "BCJSSE");
TrustManagerFactory trustMgrFact = TrustManagerFactory.getInstance("PKIX", "BCJSSE");
SSLContext sslContext = SSLContext.getInstance("TLS""BCJSSE");

Anywhere the default keystore type is present, you'll need to change the type of keystore to BCFKS, and if you need to specify the KeyStore provider, you'd specify BCFIPS as the provider.

This ensures that all encryption used to protect key and trust material is FIPS compliantly encrypted.  Sadly, the encryption used for JKS nor PKCS12 formats are themselves compliant.

In the continuing saga of this effort, I just recently completed another set of code changes that ensures that I can just drop in new key and trust stores on a shared file system, and all my servers will automatically reconfigure themselves with the latest and greatest.  This greatly simplifies updating certificates for annual renewals or for other reasons, with zero downtime.  More on that later.