Healthcare Standards

Friday, February 25, 2022

Debugging Network Communications - For the Rest of the World

In one of my very early forays into IHE standards implementation, I had to implement TLS version 1.0 shortly after it became an IETF RFC, in Java, using Tomcat. Thus began the ATNA FAQ. Thiat document is now 15 years old, and is still being used (at least by me). I started writing these kinds of things down so I don't forget them, which eventually resulted in this blog.

I'm still debugging TLS connections, and I have to admit that it's become a particular art form, but not so much as in finding out what when wrong, and figuring out the solution, but rather in explaining it to someone who is NOT versed in the ingredients of TLS connection sausages so that can a) find the right person to fix it, and b) communicate what needs to be fixed without c) further intervention from me. That's high art.

I have a highly evolved tool set including:

DNS, Ping and TraceRt
WireShark
OpenSsl
JVM debugging arguments

Here's the problems these tools help me identify. Each requires appropriate communication..

My computer cannot get a good TCP/IP address for the supplied hostname.

Either DNS is down somewhere, or the new DNS record is still propagating, or it simply does not exist.

My computer cannot connect to the TCP/IP address that DNS gave it.

The DNS change for your server may still be propagating
That IP address is not connected to the Internet, is not within my reachable networks

The sending computer sends a SYN, but no ACK is ever returned.

Ping returns no response to that IP (may be blocked by a firewall).

The firewall doesn't like the computer it is coming from.

Same symptoms as above. Firewalls and application gateways (Firewalls on steroids) and other interfacing technologies (e.g., AWS Lambdas or Azure Functions, gateways that have been weaned from Steroids) can mimic just about anything, but most often, they just hide your computer from mine.

The computer is actively NOT listening to connections from that address.

SYN goes out, but a RST comes back. In other words, I'm listening to communications, just not to THAT port.

The computers can connect (three way TCP SYN, ACK, SYN-ACK completed), but we cannot make a TLS connection.

My client sent your computer a TLS version it didn't like. You server needs to enable support for TLS Version (1.2 or higher).
My client sent your computer a set of encryption protocols it really doesn't care for. Your server needs to enable the following encryption protocols ... (RSA or DCHE Galois Cipher Mode[GCM]). Cyclic block cyphers (CBC) are no longer considered to be secure.
Your server sent my system a certificate it didn't like, it was:

Expired, please renew it
Revoked, please replace it with a new certificate
Not secure enough ... please use a certificate supporting at cipher strength of at least (2048 bits).
Not signed by an entity trusted on my end

Our system doesn't accept self-signed certificates, please get your certificate signed by (a commonly trusted CA, or a CA of my choosing)

Prepared using an algorithm my server doesn't support. Please use an certificate signed using (the RSA algorithm).
For a server going by a different name (this one is rare. Hostname matching is important in many circumstances, but in cases where the server that is supplying the certificate is three layers deep behind other network infrastructure, it has no clue, and so is often disabled.

Your server required a client certificate from my system that it doesn't have.

Here's our certificate chain, please trust someone in this chain - OR -
Please supply me with the necessary client certificate and the instructions for the obtaining one (the administrative hoops that I have to jump through to get them).

The problem with this are:

The tools are complicated and require either good training or documentation for folks who haven't needed to learn to use them.
You actually have to understand how the Internet standards work to determine what tool to use and when.
I don't scale. We need me in a box (or more specifically, in a service).

So, I've resolved to work on a tool to do this sort of testing for me, with the details about what is wrong written in very user friendly language about what it finds about the problem. It's an interesting bit of interface engineering.

Monday, February 21, 2022

Automating FHIR IG Generation from Non-FHIR Sources by cloning myself

An awesome lot of my FHIR IG development involves taking data in one structure and converting it to other structured formats required by the FHIR IG Development process.

That includes creating .fsh and .md files from structured data.

I do most of my FHIR IG development in Eclipse (because if it involves code or transforms, that's where my XML Editors plug in, or my Java transform code for other formats). I've narrowed down my processes so that I can run Sushi, the FHIR Build or my "custom build script" from within eclipse with one of three external tool configuration scripts:

Here's my normal "Build" script, which just launches build.bat in the project root folder:

Sometimes I just want to rebuild the IG, for that I can just call the IG Publisher:

And sometimes I just want to rerun Sushi alone, without running FHIR:

Save any of these snipped to a file named your-file-name.launch and you should be able to import it as an external tool configuration.

I've even gone so far (finally) as to make this build process an automated part of my check-ins to the FHIR Repository (that way I don't need to be present to run the scripts when source material changes). What I usually do is take the command line (Windows) script and convert it to Unix and run it in GitHub CI/CD. For V2-to-FHIR, I just recently created a CI/CD workflow that updates the Sushi from CSV files downloaded from Google Sheets. This is all for publication purposes. For SANER, I had other tools to create the SANER IG from an XML file I've been using for IG creation for IHE profiles. Effectively what I'm doing is cloning myself using GitHub CICD. I'm not even the first person in the community to do this. It's basically how build.fhir.org runs, essentially as Grahame in-a-box, but it scales and neither I nor Grahame do.

We were running that up until a few weeks before final publication, after which we did all edits to the files created by the automated process, and removed all the code that supported that process. That's because the FSH and FHIR IG content in GitHub are the official sources for the IG, not the process inputs used to generate them, and I don't want to be on the hook for maintenance for life for my own tooling that I'm using MOSTLY to make my life easier.

I've build now about 5 different guides using automated FHIR IG (and FSH generation tools). It's the only way to fly when you have to turn dozens of structured inputs into FHIR IG outputs.

Wednesday, January 12, 2022

Omicron: Questions but few Answers

As a member of the Health IT industry, I follow a lot of what is going on related to Healthcare, and as a result, many of my family members ask me for my opinions, advice, and guidance about healthcare issues. That includes where to go and who to talk to in order to resolve an issue in our Healthcare System [sic].

With Omicron, what I know well is fairly limited:

Get Immunized / Boosted
Stay Masked when you leave your house/car or enter any public space.
Avoid Crowds / Stay in your bubble

But here are some of the questions I have:

You can be asymptomatic with COVID (even more likely with Omicron). Should I be testing on a regular basis to be sure I don't have an asymptomatic + breakthrough infection? If so, how frequently, and what kind of test (e.g., PCR or Antigen, and if the latter which brand or brands) should I be using?

If I become symptomatic, what should I do? Test? Isolate? Both? If I test positive, what's my exit criteria from isolation and what are the risk factors I need to consider? If I test negative, how likely is that to be a false negative?

If one needs to travel within the US, what should we know about that destination and how should it affect our behavior. Are there destinations we should avoid right now? Why?

Because the information I get is coming in so quickly, often has yet to be peer reviewed, or may be poorly reported, and is also often conflicting even from well known and respected authorities, who should I be listening to and trust?

I have a ton of Surgical Masks, a more limited supply of KN95 respirators. I use the latter in places where I feel more exposed, the former in places with less exposure. What kind of exposure criteria should I be considering? Should I toss all my surgical masks and simply replace them with KN95 masks?

For me personally, I honestly think I'm at low risk of experiencing a serious adverse event due to COVID, because of the precautions I am taking. But if I include in that "serious adverse event" that I also want to avoid being the cause of a serious adverse event for some other person (whether a member of my family, bubble or community), then I quickly become clueless.

It's not that I don't know where to look or who to ask for myself, but rather, I have no good way to communicate to those I interact with what I know (or think I know).

And oh, by the way, don't ask me anything about the next variant. I have no clue what that will bring. Does anyone?

If any of you out there reading this can do better, please provide feedback in comments below.

Friday, December 3, 2021

New Years Resolutions for 2022

It's been a while since I've posted anything related to Health IT Standards here. I have three New Years Resolutions for 2022:

Don't schedule or accept any meetings on Friday afternoon.
Focus attention on Patient Administrative Burden (a phrase I read in a response to this recent ~~rant~~ thread from @HITPolicyWonk).
Do more Deep work and report on it here.

I've been doing a bit of reading (of the non-fiction philosophical variety) lately.

Algorithms to Live By by Brian Christian and Tom Griffiths
This is an awesome collection essays about the application of computer algorithms to various aspects of life, and what they have to teach us about the tasks we don't always involve computers in. I think what I enjoyed most about this book was the creative links the authors made between algorithsm (most of which I know well), and everyday tasks and processes.
Thinking, Fast and Slow by Daniel Kahneman
This book endeavors to explain how decisions get made and are influenced by the conscious and unconscious processes in our brains, and the various tendencies that are influenced by our intuitions, the biases that they introduce, and to some degree, methods by which one can eliminate some of those biases from our thinking.
The Most Human Human by Brian Christian
I picked this one up because I've been interested in Artificial Intelligence since early college years, and the Turing test always amused me, and because I enjoyed an earlier book by the same author. There are some interesting musings here that again cross between computing and everyday life. Compared to Algorithms to Live By, this one wasn't quite AS interesting, but a fun read none-the-less.
Deep Work by Cal Newport
I'm still ploughing through this one, but after 3 chapters, I'd heartily recommend it. That's saying something because it's VERY hard to find a book I cannot finish over a weekend, but this one involves that kind of reading. It's deep work to understand deep work. I haven't been doing enough deep work lately, and it makes me sad. Hopefully, I'll learn enough to fix that.
Clean Code by Robert C. Martin
A classic, and one that I recently re-read, a decade + later. I don't fully agree with everything said in this book b/c Robert misses a key point. There's concepts and nuance and a whole language associated with so many programming frameworks, and in many of these, the frameworks are so large that even the best engineer can't keep them all available to them. As part of a team that has a common approach, following the guidelines in this book are valuable. But what the book won't do for teams is enable a new person to join the team without prior experience in the framework and be able to understand the code as written. I recently read through several thousand lines of an application built by a very skilled engineer following the principles in this book, and because I know where to find the right stuff, was eventually able to understand it, but having never used one aspect of that framework in a production environment, found myself lost until I could go do some useful reading. A simple comment describing the design pattern or framework in use would have made the code much easier to understand. Do NOT underestimate the longevity of your code. All too many times in my life I've encountered code where the person who wrote is no longer living, the person who took it over is retired, and basically, there's some grunt who can compile, build and occasionally fix bugs in the code, and the ideas, frameworks, thoughts and standards of two decades ago are completely unknown to the originators. As developers, we live on internet time. What is obvious to us today is forgotten 5 years from now because someone figured out a better way. Be kind to them who have forgotten everything that is in the front of you mind and obvious (to you). Still useful reading, but do so with a critical eye. Like any other concept that is at least a decade old, there's some new thinking that has replaced some of these ideas.

Saturday, November 27, 2021

On Air: Part Three

I'm finally finished with my On Air lamp prototype. The goal of this project was to build something that would indicate when I have the camera and mic rolling in my office without any intervention on my part (basically, all I need to do is start a meeting and it goes on, or end it and it goes off). I've managed that.

It took me quite some time to figure out how to get Windows 10 to tell me when the microphone is in use. The key is in the registry keys located in the registry hive for the current used found in SOFTWARE\Microsoft\Windows\CurrentVersion\CapabilityAccessManager\ConsentStore\microphone.

In this hive there is at the top level, a hive of keys for each Microsoft Store app, and beneath it, another hive of keys found in the NonPackaged hive.

If you inspect the registry with RegEdit, you will see that each hive under that key looks something like this:

The key (pun intended) key is LatUsedTimeStop. When this value is 0 for any application, that application is presently using the microphone. When it's done, that value will be a Windows timestamp (the number of 100 nanosecond intervals since 1/1/1601 UTC).

I wrote a little Java program called OnAir that scans the registry, checks for the Microphone being in use, and sends the appropriate messages to the wireless controller to adjust the state of the lamp.

I tested it with almost all of my headsets (I have four in my office of various types), and it works with any of them, and with any application that uses a microphone.

To make this code work I needed to be able to access the registry from Java. There's a lovely little Java package that does this called JNA. The dependencies to include it are below.

<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna</artifactId>
<version>5.10.0</version>
</dependency>
<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna-platform</artifactId> <version>5.10.0</version>
</dependency>

I suppose I could have used this as an opportunity to write in .Net, but realistically, I was just looking to get the lamp off my desk, and to do that, I needed to finish the prototype.

There's so much more I could do to finish this. The application could use a UI to indicate the state of the lamp, and to activate/deactivate it for certain applications, and indicate which application is using the microphone. The app itself should run as a service. The electronics could be cleaned up so that I don't need two USB cables, and the wireless component could be mounted inside the lamp. For now, it's good enough for what I wanted, so I'm calling it done, and declaring "Ship it!".

Thursday, October 14, 2021

Responding before reading @alissaknight's: Playing with FHIR: Hacking and Securing FHIR APIs

If you've been sitting under a rock, you missed the latest big explosion in the FHIR World, Alissa Knight's most recent report: Playing with FHIR: Hacking and Securing FHIR APIs. I've downloaded it, and looked at the fantastic graphics (e.g., image right), but I've not really read it yet. This is the kind of report that will require as deep a read I suspect as some Federal regulations. If I get the time (perhaps this weekend), I may even do a tweet-through-read-through.

I have some initial thoughts based on my reading of the tweets in this stream:

“(FHIR app ecosystem) contained “pervasive authorization vulnerabilities” that enabled Knight to access > 4M patient and clinician records with just a single patient login account” This is truly frightening. Would love your perspective @motorcycle_guy @amalec @aneeshchopra https://t.co/EOHKNvzQT0
— claudiawilliams (@claudiawilliams) October 13, 2021

Here's what I expect to find, noting that this is only my guess as to what is happening:

Various apps rely on a FHIR backend which will allow a replay attack to be performed whereby:

The attacker obtains access to the authorization token used by the FHIR API call to make other API calls on a different patient.

NOTE: There are a number of ways to obtain this authorization token depending on how the application is constructed and the level of access one has to developer tools, and application hardware and software. Assume that the hostile attacker is one in a million that has access to all of that, not the common high-school student (but don't count them out either, some of them are that good). Is ia a Java or .Net app? There's a debugger for that, and I can almost assuredly final all of your assemblies or jar files on my device emulator, and debug code running in a device. Did you obfuscate your code? If not, I can reverse compile it, there are tools for that too, and even obfuscation is not enough when you consider that platform calls are still going to be obvious, and all the important ones are well known, so I can work back the stack to the code I need to manually investigate.
The attacker constructs new API calls to make request using the same authorization token.
The call succeeds because the only check that is performed on the authorization token by the back end server is that it is a valid token issued by the appropriate authorizer.

The fix is not simple. Basically, the back end developer needs to bind the access control restrictions associated with the original access request to obtain the authorization token to the subset of data that it authorizes access to (this is the easy part), and enforce them on every request (this is the hard part), incoming and outgoing (unless you can prove the outgoing data will match the authorization based on how the query works). JWT tokens provide more than adequate capability to bind exactly those details in an easily accessible way. Essentially, the claims in the JWT should indicate what is allowed to be accessed, when, how, and by whom. Sadly for my first FHIR Server, JWT was still a work in progress, but I simply encoded an index in the token which pointed to memorialized claims in a database, constructed when the token was first created.

Once you can do that much, fine-grained access control is actually quite simple. That was originally considered to be out of scope in my first effort for reasons of cost and complexity, but because we had to build the right security infrastructure even to support coarse-grained access control, fine-grained control simply became just a little bit harder, but also a worthwhile differentiator from other offerings.

As I said earlier, my guess as to what is wrong is only supposition, an experienced, wild-assed guess at what I will find upon reading. But if I can guess it before reading, then arguably, so can someone else as smart as I without barely even an incentive. Consider also that intelligence and quality of character are not necessarily correlated, nor do they even need to exist in the same body for a vulnerability such as I described above to be readily and easily exploited by someone with enough motive, and knowledge about how to use tools that people smarter than them wrote.

In the world of the great world wide web, FHIR API servers, Java, .net or other platform, the basic concepts of servlets, intercepters, filters, hooks, cut points or similar concepts by other names in non-Java platforms all exist and are able to perform these checks and validations both before and after the call completes, so that you can:

Verify that what is being asked for is allowed to be asked for (never assume that the querant of your back end can only be your application)
The data that is being returned also matches what the authorization allows the end user to see, and either filter, or simply reject the request (after having performed some work that you wish you hadn't).

I led teams building (writing some of the code myself) commercial FHIR servers on top of readily available open source frameworks a number of times in my life (most are still in use), and I've also implemented other API interfaces pre-FHIR. When I built in security controls, I bound the security tokens to the specifically authorized purpose and sources, and I didn't trust that token to be used only for that purpose, I made sure it was verified every time.

I have to tell you that was also a moderate pain point, because it cost about 10% of my processing time budget. I could however, justify it by citing the specific regulatory text regarding fines for breach, and that I think made a huge difference in my ability to argue for that kind of security (this was perhaps, the hardest part).

I expect to be reading a wakeup call, we'll see how my prediction turns out. Remember too, APIs are still a work in progress.

Security is hard.
Good Security is harder.
Great Security is an ongoing work in progress.
Perfect security does not exist.

On Air: Part Two

Since my initial foray into developing my own little IoT device that will be controlled by my headset that started in On Air: Part 1, I've managed to accomplish three things:

Discovered that the battery drain from the light is too much for as often as I'm on a headset so that batteries would bankrupt me (three batteries lasted two days). Fortunately, there's a USB C power adapter for this fixture.
Rewired the power connection so that it was driven by the power to thee on-off switch, rather than power supply from the battery.
Written a couple of node.js scripts to turn the beasty on and off from the command line.

In part 3 or 4 I'll open this thing up to show you my (crappy) wiring.

Step 3 was not nearly as hard as I expected, and that's because everything I need to control the device is right here.

The two scripts are dead simple. Here's the one to turn it on. The one to turn it off is left as an exercise for the reader.

const ewelink = require('ewelink-api');

/* instantiate class */
const connection = new ewelink({
email: '█████████@██████',
password: '███████████',
region: 'us',
});

async function f1() {
/* turn on/off device */
const device = await connection.setDevicePowerState('██████████', 'on');
console.log(device);
}

f1();

First, create a connection class, and give it your username (which is your e-mail address), your password, and your region. Mine happens to be 'us'.

Secondly, create an async function that will turn the device on and report its status. You'll need the device id, fortunately, it is written on a white sticky attached the controller chip of the device, so get your phone or a magnifying glass to read it, and enter it (in quotes) as the first argument to the call to setDevicePowerState().

Finally, call that function.

To run the script, just enter:

node turnOn.js

at the command line, and walla, the light goes on. Run

node turnOff.js

and it turns off.

Now, to detect whether my headset is in use, and call the right script. This will also be remarkably easy. But the devil is in the details.

I suppose I could simply this script so that it took a command line argument, but again, that's an exercise for the reader.

Pages