The Challenge of Measuring Interoperability

Many years ago, I created this map showing where in the world IHE XD* profiles and HL7 CDA Implementation guides were used.

The point of the map was to show how popular the specifications were, and basically to provide enough evidence to move the needle forward on these specifications with regard to how they were seen in the industry.

This is a crowd sourced map, but most of the data came from two sources: What I knew of North America, and what another colleague know about Europe and Asia.  Keeping this map up to date is hard.  Authoritative information is difficult to come by for a number of reasons:

1. It's about what is implemented in the field. 
1. The organizations that implement these specifications often don't know which specifications they are using, nor do they care to publicize this information.  
2. The companies that sell the products that implement these specifications also implement other specifications, and aren't necessarily aware of which specifications their customers are using (some of the products I work on have dozens of different interfaces).
3. Many of these implementations combine the work of several different products from different vendors, so none save the implementers have complete detail.
2. The information is valuable intellectual property
   This idea is going to be challenging for some folks who think all data about Health IT should be transparent. I'm not going to get into that argument here.

Let's look at two different examples of interoperability: Exchanging transfers of care using some form of CCDA Document, and ePrescribing.  Both of these must be supported by Meaningful Users of Health IT in the US.

We can fairly state that those organizations implementing either are using the standards: CCDA for transfers of care, and NCPDP for ePrescribing.  But when we get down to the underlying exchange mechanisms, we cannot say a whole lot about how the data moves.  For ePrescribing, the transport standards are fixed by regulation for ePrescribing (NCPDP Telecommunications protocol), and we know most of the data moves over one network, but we don't know how it gets there (without exploring the ePrescribing solution market). For Transfers of care, there are multiple choices for transport: the CCDA can move over an HIE, it can move through the Direct protocol, or it can move over some other web services protocol (and in fact, it can move through several protocols before it gets to its final destination).  We've got some pretty good data on ePrescribing (see the ONC Data Brief on Prescribing increases from 2006 to 2014), but I've not found a similar report on transfers of care (perhaps because we haven't been tracking it as long or as well).

I think the crucial point here is that in measuring interoperability, we'd like, as an industry to be able to report successes or failures as was done for ePrescribing. Having organizations (either vendors or providers) report what is moving and how creates an additional burden for them to manage the reporting, and additional burdens are disincentives.  It also exposes information relevant to that organization potentially including processes, market share (e.g., number of patients treated/seen, number of providers using), or vendors.  There is a delicate balance that has to be maintained.  The information gathered needs to be a) easily captured and reported, b) meaningful in terms of measurement, and c) sensitively managed and reported.