It’s Time

For nearly thirteen years, I’ve been employed by GE Healthcare, or it’s successor,  Virence Health, working on or implementing Health IT Standards.  Today I turned off the last device that connected me to that organizations electronic infrastructure, and synced my email with my newest employer starting tomorrow.

Over that time, my skills, my influence, and all that I am on this blog has been shaped and influenced by the passionate people I have worked with, who truly care about what they do.  As much as I have taught, I have also learned.  In that time, many people have turned a vision I had into reality for me, for themselves, and for our customers.

From the manager who placed a bet on me and found the loophole that could bring me on board, to the strategist who threatened to quit if I wasn’t, to the genomicist who taught me most of what I know, to the budding and passionate architect with a single name (like Madonna), to the brilliant engineer and architect who should have been an architect years before I ever met him, to the young lady whose name I remember simply because of her divinely festival decorated hands who accepted and excelled at evry challenge I set before her, to the woman who project managed three teams with members across three time zones, five cities and two continents, to the services leader who crossed all the t’s I missed, and the integration specialist who passionately drove our engineering teams to do the right things for us, them and our customers, to the guy who taught me almost every thing I know about risk assessment, and the guy who made it impossible for me to say “I know nothing about DICOM”, and the other fellow who made it necessary for me to prove it, and to the person I called regularly to listen to my challenges and help me work them out, to the guy who took SMART and FHIR and built a tool that others said couldn’t be done, the woman who became her products CDA expert and then applied those skills to another key product, and to the best boss I ever had who pushed me up harder and faster than any before him, and perhaps any since, I thank you.

For those, and all the teams I’ve worked with over the years, at Connectathons and showcases, through two different ONC certification regimens (pre and post-HITECH), and beyond, you deserve recognition.

For your continued dedication through all challenges, I’m awarding you all the Ad Hoc Activa award.  It’s the workhorse of a nation, pushing its people to work every day to make their lives better, just as you are.

—  Keith

P.S. If you don’t already know how to find me, don’t worry, I’ll still be around, and a later tweet will link to the details.  If you follow me on LinkedIn, you’ll likely find out tomorrow.

Resolving Notworking Woes

Networking woes are the bane of any interface engineer, service representative, or IT help desk person.  Let's talk about the various ways our network connections go down:

1. We screw something up in the URL we are requesting and get back an error we didn't expect.
2. Someone fat fingers the hostname or port address somewhere and we cannot find the network endpoint.
3. The hostname doesn't match the certificate associated with it.
4. The certificate has expired.
5. We don't like any of the certificates that have been offered because we don't trust one of the root CAs.
6. The host is down.
7. The website on that host is down (e.g., the host can be reached, but the port isn't being listened to).
8. The proxy server is down.
9. The system isn't configured with the correct proxy server.
10. We cannot resolve the IP address of the proxy server.
11. DNS used to resolve the proxy server address is down.
12. DNS used to resolve the server hostname is down.
13. We should/shouldn't be using a proxy.
14. We haven't entered the proper credentials to authenticate to the proxy.
15. The firewall doesn't like our URL for some reason (someone once reported to me that since a  URL contained the word "sex" it was rejected by an overly sensitive firewall).
16. The VPN is down.
17. DNS registration expired.
    .
    .
    .

I could go on for quite a bit longer.  

The set of diagnostic tools we have is vast: ping, tracert, nslookup, ipconfig, wireshark, openssl, ... (I could go on with many more) but most of these are run from the command line, have lots of options and require human interpretation.  

DNS, Proxy, host, web server, firewall.  By the time  you have everything correct, at least 10 things have to be working.  If every one of them is running at 5 nines, you are now down to 4.  If you have 1000 customers, you now have about a 1 in 10 chance that for one of them, something is wrong in the notwork [sic].  

Why do we do this to ourselves?  Wouldn't it be good if our platform software could tell us IN DETAIL exactly what is wrong when something doesn't work the way we expect it to?  Wouldn't it be even better if the platform told you what could be done to fix it?  Wouldn't it be absolutely awesome if the platform could actually take its own advice and fix the problem?

     Keith

Stages of the Software Legacy Lifecycle

Software has a life cycle just like everything else, and sometimes even goes beyond into the after-life (in keeping with this season, that's zombie-hood).  We've all had to deal with legacy code, either as developers or as users.  I've developed a 13 point scale to measure legacy-ness of software from inception (1) to death (12) and beyond into zombie-hood (13).  The following guidelines to help you place where the code you are working on or using might fit into.

The form of this is "Stage / What They Say / What They are Thinking", followed by what it might really mean when talking to someone who knows something (perhaps not much) about the code in question.

1. First Date / May be Coming Soon / Never heard of that one before.

• The product manager hasn't heard that idea before, but it sounds intriguing.  Could you say more?

2. Talking about It / In a Future Major Release / Heard of that, still thinking about it.

• There's an open requisition for the architect who can design that.
• It's not in the plans.

3. Engaged / In a Future Release / Heard of that, think it might be useful.

• The architect knows what needs to happen, but needs to document it.
• There's an open requisition for the engineers who can write that.
• It's in not in the plans yet.

4. Twinkle / In the Next Major Release / Heard of that, pretty sure it's useful.

• There's an open requisition for the architect who can design that.
• It's in the plans.

5. Pregnancy / Coming Soon / Know what that is, planning on it.

• The architect knows what needs to happen, but may need to document it.
• There's an open requisition for engineers who can write that.
• It's in the plans.

6. Birthing / In the Next Release / Know what that is, implementing it.

• The architect knows what needs to happen, and has documented it.
• There are engineers who know how to write that code.
• It's in the plans.

7. Infancy / Piloting / Still working the details out.

• The architect knows what needs to happen, and has documented it.
• There are engineers wrote that code.
• There are still some bugs preventing full shipping.

8. Childhood / It's in Production Today / Implemented it.

• The architect who designed it is still around.
• The engineers who wrote it are still assigned to the project.
• It shipped.

9. Adult / In Maintenance / Been there, done that, losing interest.

• The last engineer who knows how that works is leaving the company for another opportunity.
• An open requisition exists for an engineer to maintain that which is expected to be filled soon (hopefully before the last engineer who knows how that works leaves).

10. Middle-Age / Reaching the End of Life / Losing interest.

• The last engineer who knows how all of that works left the company for another opportunity.
• There's an intern who can compile it and can fix the occasional bug.

11. Elderly / Legacy / Lost interest but the customer's haven't.

• The person who wrote that retired.
• The engineer who took it over from left the company for another opportunity.
• This is an engineer or intern who maintains that and understands how some of it works.

12. Dead / End of Life / Customers have lost interest.

• The intern who maintains that goes back to school full-time next week (or at least that's what they said).

13. Zombie-Hood / Extreme Legacy / Everybody else BUT the customers have lost interest.

For that mission critical software that just can't go to end of life.

• The guy who wrote that is dead, 
• The person who took it over from him has retired, 
• The engineer hired to maintain it quit,
• There's an intern who can run the legacy compiler used for it and who can fix the occasional bug.
• Nobody else knows how that any of that works anymore.

Just like real life (in the movied), death and zombie-hood can happen to software at just about any time, and death isn't really a prerequisite stage before zombie-hood.

   -- Keith

Challenges using SMART on FHIR for Multi-Vendor Authorization

Challenge by Nick Youngson CC BY-SA 3.0

One of the challenges for application developers in implementing SMART on FHIR is that of metadata endpoint discovery.  In order to initiate the SMART on FHIR authorization flow, the application developer needs to know the metadata or Conformance endpoint for the FHIR Server it is going to communicate with.  However, this endpoint is going to vary based on the vendor supplying the FHIR Server, the version of FHIR that is supported, and possibly even the healthcare organization deploying it.

The problem here is in the use of metadata as a way to discover the authorization endpoint, when in fact, the challenge for application vendors is configuring to support multiple endpoints with the same application.  If you use MyChart or other apps to access your health records (as I do), you can see how this plays out when you go to login as a patient.  The first step of your login process has you identify your state and healthcare practice (this shows up in other applications as well).  I'm certain that somewhere in that application and others like it, that selection process is doing something to resolve the practice specific endpoint details.

In a single vendor environment, it's pretty easy to address this, but when developing an application to support multiple environments, this can be quite challenging for the application developer.

NO, this isn't a claim that we need a global SMART on FHIR endpoint directory (although that is one way to resolve this issue).  It's more a statement that we've combined the process of conformance inspection with the problem of endpoint discovery.

Think about it: If your authorization endpoints are different, it is also likely that FHIR conformance associated with those endpoints could be different.  Why should they both use the same mechanism.

This creates a challenge for patients, because it means that App developers are likely unable to support as broad a variety of endpoints as they'd like simply due to the configuration challenges presented to them.

Smart SMART developers will obviously work around this.  One possibility is to modify the Launch sequence so that the application first asks the patient where their practice is located through some sort of internally managed directory, so that the application first selects the practice, and then the directory resolves the authorization endpoints AND the actual FHIR conformance endpoint the application can use to customize its operations based one what capabilities are available.

This would allow patient facing SMART applications to support multiple versions of FHIR from multiple provider organizations.

There are a lot of other challenges.  Like anyone else, you have to get your product into the vendor's App store, and there's a lot more vendors than one has to typically deal with for smart phone applications (two covers a lot of territory here, whereas you need 5 or 6 for EHRs to get the same coverage).  And of course, you also have to get your app into the smart phone stores too.  Those aren't problems I'm going to try to solve here.

   Keith

But We're Different

The number of times I hear this phrase no longer astounds me. In making this statement the speaker rejects an offered solution because of a perceived difference based on a special need. I've often seen that the special need is similar to other special needs where the proposed solution is already in use elsewhere (Healthcare people sometimes act as if they are the only ones operating in a regulated industry).

Some years ago I led a diverse workgroup across three quite distinct stakeholders trying to solve (what appeared to me to be) the same problem. By "lead", I mean cajoled, bugged, spanked (verbally), herded, out-waited, out-witted, listened, learned, fumed, and eventually rejoiced. Over the course of a year I watched this group evolve three completely separate white papers and approaches into one, and after that evolve into an IHE workgroup (QRPH). That workgroup now looks more deeply into their commonalities than they do their differences.

In my most recent dive into medication management I see a similar opportunity for CDS Vendors, EDI vendors, PDMPs, eRX & CPOE developers, payers and pharmacies to come together around a singular solution for improving medication orders.

The challenge for this group is quite different though. Unlike QRPH, which faced a lack of solutions, attention and funding, medication management has a plethora of all of the above. The poverty in what we have is commonality, dare I say it "standards".

"Oh, yes we do too have those." it will be argued. And I will agree, the solutions have standards in the same way that an organization with 10 priorities has any priorities.  And the challenge we face in replacing 10 with 1 is best summarized by this (de-facto) standard response.

Uses with Permission from XKCD

HAPI FHIR BDD Testing using Serenity and RESTAssured for DSTU2

One of the challenges I've had with using Serenity and REST Assured for testing with the HAPI Server is related to the Content-Type header used with STU2 implementations.

In STU2, that content type is application/xml+fhir and application/json+fhir respectively.
In STU3 and after, these changed to application/fhir+xml and application/fhir+json.
This is discussed in more detail in HAPI FHIR Issue 445.

I finally figured out (I think) where to make the changes to address this in Serenity to handle appropriate reporting.  The key challenge is that I wasn't able to track down the content body of requests in error (or even those that were successful).  That detail can be found in Serenity-core Issue 448.

Unfortunately, what this requires is a hack around the given() method in which one intercepts the given() call, gets the response back, checks to see if it is of an appropriate type, and changes the RestResponseHelper method to appropriately format the response body (an exercise left to the reader).

I'm testing this now (because I have nothing better to do at 3 am).

   Keith

Deep, Significant and Detailed HL7 Contributions

One of the points of the Ad Hoc Harley award is to recognize unsung heroes in heath IT standards. Almost all past awardees have been people I've worked with directly in standards development. While I have met the current awardee in person we've not spent much time collaborating on anything together, even though he's a fellow New Englander. Even so, I've still benefited from his work in standards, and his open source contributions.

According to two very smart people I know (one a past awardee), he's one of the smartest people they know. In addition to being a brilliant computer scientist [an assessment I heartily agree with having read, used and tweaked his code], he is also an accomplished singer who can harmonize with anything, a person of great humility, and a devoted family man.

While he was unable to be at the HL7 Working group this week, his work was demonstrated in several committee meetings.

His contributions include:

• CQL
• CIMPL
• CDS Connect
• Maintenance of the Quality Data Model
• Invaluable feedback on the CDC Opioid prescribing guidelines.
• Synthea’s Generic Module Framework

Without further ado:

This certifies that

Chris Moesel of MITRE

has hereby been recognized for deep, significant and detailed contributions to HL7 standards.