Running BCFIPS in SpringBoot 3.2

I'm rewriting wrappers for a Spring Boot 1.5 application using Spring Boot 3.2 to help me eliminate some older dependencies I cannot even get patches for anymore.  So now I have to make my new Spring Boot application work with the latest Bouncy Castle FIPS code.

I've previously mentioned the NIST Certified Bouncy Castle FIPS TLS libraries in other posts.  SOAP is tremendously complicated to configure and manage.  REST is so much easier, and when you don't need all of the power of a SOAP stack, you can certainly send SOAP messages in XML quite readily using a normal REST transport layer.  You may need to write some extra marshalling code, but for standardized interfaces, that doesn't tend to be a big lift and there are plenty of existing tools to help (e.g., Faster Jackson XML).

There are a few challenges. One of these stems from the need for the BC-FIPS modules to validate themselves on startup.  This uses jar validation processes, but the code running the validation needs to be able to find itself.  That gets complicated by a minor change in Spring Boot 3.2 URL format when it repackages the application into an uber-jar. The URL conflicts with what the BC modules expect, and then they won't validate.  While the fix is a one-liner in BC-FIPS, it's a change to a certified software module, which means that it has to go through processes which make it take longer than anyone would like.

Spring Boot uber class loading has the ability to target classes which need to be unpacked from the Uber-jar before loading. The JAR files will then be unpacked from the uber-jar into your TEMP folder before being loaded, and those will be accessible from a different kind of URL, that of the JAR file, which BCFIPS knows how to resolve.

To make this work, you will need to update your configuration for the spring-boot-maven-plugin to tell it which jar files need to be unpacked.  The necessary lines of code are below:

            <plugin>
                <groupid>org.springframework.boot</groupid>
                <artifactid>spring-boot-maven-plugin</artifactid>
                <configuration>
                    <mainclass><!--your-application-main-class--></mainclass>
                    <requiresunpack>
                        <dependency>
                            <groupid>org.bouncycastle</groupid>
                            <artifactid>bcpkix-fips</artifactid>
                        </dependency>
                        <dependency>
                            <groupid>org.bouncycastle</groupid>
                            <artifactid>bc-fips</artifactid>
                        </dependency>
                        <dependency>
                            <groupid>org.bouncycastle</groupid>
                            <artifactid>bctls-fips</artifactid>
                        </dependency>
                    </requiresunpack>
                </configuration>
                <executions>
                    <execution>
                        <id>default-jar</id>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

Rewriting my wrapper to ditch CXF and modernize to the latest Spring Boot will save a lot on future maintenance efforts and is already making the application much simpler.  There's plenty of glue code needed to run inside the CXF framework that can just be done away with.

As you now know, I had to go back and rework for SNI again too.  It turns out the same fix was basically needed, calling URLConnectionUtil.openConnection().

	import org.bouncycastle.jsse.util.URLConnectionUtil;
  ...
  /**
   * This method obtains an SNI enabled Server Socket Factory
   * @param location	The URL to obtain a connection for
   * @return	An HttpURLConnection that supports SNI if the URL starts with HTTPS or has no URL Scheme.
   * @throws IOException
   */
  public HttpURLConnection getSNIEnabledConnection(URL location) throws IOException {
    String protocol = location.getProtocol();
    if (protocol == null || !protocol.startsWith("https")) {
      // Not HTTPS
      return (HttpURLConnection) location.openConnection();
    }
    URLConnectionUtil util = new URLConnectionUtil(getSSLContext().getSocketFactory());
    HttpsURLConnection conx = (HttpsURLConnection) util.openConnection(location);
    conx.setHostnameVerifier(ClientTlsSupport::verifyHostname);
    return conx;
  }

More fun with SNI and TLS with Akamai edge servers and Bouncy Castle internal_error(80)

Recently, endpoints that one of the systems I maintain frequently connects to underwent a change in how they hosted their servers.  They were moved into an Akamai edge server framework, which requires use of the Server Name Indicator (SNI) extension in TLS communications.  This isn't routinely enabled in a straight Java JSSE client connection, especially when using Bouncy Castle FIPS.  As I previously stated, you have to configure it in your code.

My guess is that when a request is made without the SNI TLS extension, the Akamai edge environment reports a TLS error.  Sadly, Akamai reports this TLS error using the TLS internal_error(80) fatal alert code instead of the more descriptive missing_extension(109).  That's because missing_extension(109) wasn't defined until TLS 1.3 (which is much more expressive of the reason for failure).

On my side, I had half-enabled SNI, where it was used for the main protocol request and response, but wasn't being used for the requests sent to the underlying authorization protocol (OAuth 2.0).  The reason that was missed was simply because it is a completely independent module.

If you've got WireShark, and you see something like this in your capture, look at the list of extensions.

If server_name extension is missing, as in the above capture, you need to fix your code to use SNI.  Once you've fixed it, it should look like this: 

In any case, if you are using BC FIPS, and get an internal_error(80) while trying to connect to an Akamai Edge Server, and your TLS Client Hello doesn't contain the SNI Extension: Well, there's your problem.

The nastiest code I ever wrote

Creating a mock that injects random errors into the expected responses is necessary to ensure that your server resiliency features work as intended. Doing so can be very challenging, especially if your mock has to simulate complex behavior.  FWIW, I call it a mock, because it is, but the endpoint is a test endpoint used by a production environment to verify that the server is operating correctly.

What appears below all-out wins the award for the nastiest, most evil code I've ever written that appears in a production system.

resp = ReflectionUtils.unwrapResponse(resp);
if (resp instanceof ResponseFacade rf) {
  try {
   // Drill through Tomcat internals using reflection to get to the underlying socket and SHUT it down hard.
    NioEndpoint.NioSocketWrapper w = 
      ReflectionUtils.getField(rf, "response.coyoteResponse.hook.socketWrapper", NioEndpoint.NioSocketWrapper.class);
    ReflectionUtils.attempt(() -> w.getSocket().getIOChannel().shutdownInput());  
    ReflectionUtils.attempt(() -> w.getSocket().getIOChannel().shutdownOutput());
  } catch (Exception e) {
    log.warn("Cannot access socket: {}", e);
  }
} else {
  throw new RuntimeException("Simulated Socket Reset", new SocketException("Connection Reset [simulated]"));
}

It is used for a mock endpoint developed to performance test an application that sends messages to other systems.  The purpose of this is to force a reset on the server socket being used to response to a request.  The mock endpoint receives a request, and randomly shuts down the socket to help test the resilience and performance of the main application.

This delves deeply into Tomcat internals and uses a few reflection utility functions that I wrote.

The unwrapResponse() function just gets to the inner most HttpServletResponse assuming that is the beast that is closed to Tomcat internals (in my case, it is).

public static HttpServletResponse unwrapResponse(HttpServletResponse servletResp) {
  return ((servletResp instanceof HttpServletResponseWrapper w) ?
    unwrapResponse((HttpServletResponse)w.getResponse()) : servletResp);
}

The getField() method breaks the string given as the second argument into parts around the period (.) character, and then recursively extracts the object at that field, returning it.  The last argument is the type of the object.  I'll leave this one as an exercise for the reader.  These lines of code First get the socket from socket wrapper of the action hook in the CoyoteResponse of the Response object that's backing the outermost HttpServletResponse object given in resp.

The attempt() method just calls the callable (which is allowed to throw an exception), and returns, regardless of whether or not the attempt succeeds.  This is because I don't want this code to fail.

What's inside that callable is the critical code.  That uses the socketWrapper  to get the NIOChannel, and force a shutdown on both sides of the socket.  This is the hardest I can crash an inbound connection without messing with operating system calls, and it works in Tomcat 8.5, and I'm pretty certain it will work in Tomcat 10 as well.  I'll know in a couple of days once I get done migrating.

This is truly evil code; however, it wouldn't exist without a purpose.  That is to test how the system responds when a socket goes randomly belly-up during a performance test.  Yes, this code is randomly triggered during performance testing whilst the application is attempting to make requests.  As much as possible, I've basically installed a robot walking through the server room and randomly pulling cables from the endpoint the application is trying to get to, without having any access to that room or the cables.

This is basically the code that it getting invoked by this ugliness (that link is for Java 8, I'm actually using JDK 17 and previously used JDK11).  You might find this technique useful yourself.  I find it's a lot better than having to pull my wired connection a dozen or more times for my weekly performance test.  Yes, this runs in a scheduled automated test at least once a week.