Questioning the Question

Some days things come out of my mouth that make me stop and want to say: Why didn’t I think of that? The internal conversation continues something like this: But you just did. Okay, why didn’t I think of that sooner? Sometimes it takes the right question at the right time to cause my brain to synthesize a good (and sometimes even really good) response.

The question at this particular point in time was: How can I query for a document containing certain information I need. I stopped the questioner to provide me with more context, as I usually do. Essentially, I’m trying to get at the real problem. The problem in this case was a process where a patient is sent for referral, and certain clinical data is sent with the referral. The clinical data being sent is specified by an implementation guide that indicates required and optional data. In some number of cases though, the optional data is more important than others. It depends on the clinical context, and it’s hard to judge when it will be valuable.

The questioner didn’t have all the details on how often this would occur, how likely it would be to delay care. The proposed process simply mirrored an existing paper referral process, but put more interoperability requirements around it.

There are two ways to solve this person’s problem. Solution 1 would have been to simply answer his question. I could have done that using at least three different standards or implementation guides or IHE profiles, some of which are more mature than others.

The problem can also be simply resolved using solution 2. Solution 2 makes all of the data necessary to make any referral decision required in the original content sent with the referral. This has a related implementation cost, which is unknown, but can be estimated. The other solution, send data that is easy to get to and in exceptional cases perform at least one, and possibly more round trip communications to get additional data when needed also has costs, also unknown but also estimable.

The right answer in this case is to do a bit more digging before coming up with a solution.

Let’s look at some examples using imaginary data. I’ll use $5,000 / year as the generic annual maintenance cost of a single interface, and multiply that value by a factor of 10 for initial implementation of each interface. These are imaginary values. They could be higher or lower, and the purpose is not to scare anyone, just to show an example of the analysis.

Solution 1 has 6 interfaces, two outbound on the referrer side, one inbound on the referrer side, and two inbound in the referred to side, and one outbound on that side. For the sake of argument, let’s assume that the two outbound and inbound interfaces are identical, it’s just the addition of a query / response pair. That gives us a total of four interfaces that have to be implemented and maintained, with some interrelated moving parts. That adds up to an implementation cost of $200K for the four interfaces, plus $20K per year to maintain them.

Solution 2 is a bit more complicated to implement for the outbound and inbound interfaces, call it a 50% increase in implementation cost, but the same costs to maintain them. Solution 2 has an outbound interface on the referring side, and an inbound interface on the referred to side. The total implementation cost is $150K for these two interfaces, and annual maintenance is $10K / year.

Taking the analysis that far would indicate an immediate win for solution 2, but let’s assume that it wasn’t an immediate win and take the analysis further. Let’s say it was three times harder to implement the more complex interface. You’d be at $300K to implement, and $10K / year to maintain, which would take 10 years to provide sufficient ROI for this solution to become equal in cost to solution 1 (300K + 10K * 10 = 200K + 10 * 20K = 400K).

So far, so good, we understand the analysis. But against implementation costs we also have to weigh other possible costs and risks to see what could be tolerated. Solution 1 incorporates a potential workflow delay. Anytime a delay is introduced into a workflow, we have to ask ourselves what the impact would be on patient care, and whether it introduces a risk to the patient. If 1 referral out of 20 needs more data, and the delays introduced in getting that data result in an impact in patient care again, 1 time out of 20, is that an acceptable risk? Is one chance in 400 an acceptable likelihood for this risk? I cannot answer this question without knowing the possible impact on the patient (it could even be me). If life threatening, then I certainly wouldn’t accept those chances, but if this is simply a minor inconvenience, I might.

If the patient is being referred for consultation on a potential cancer finding, a delay in the referral process could introduce a life threatening risk, but for a routine and non-life-threatening condition, it might not.

Now, let me take this one step forward into implementation guide development. We have two different use cases, and in those, two different potential solutions, each with associated costs, and different risk profiles depending upon the context. The referral process is a common use case that can be as simple as a referral to an ENT for an ear infection, or as complex as a consult for a patient with a possible cancer diagnosis. The choices we make for the content to be exchanged is broad because the use case is broad, and without more specifics, we cannot require more data because it would introduce an implementation cost that might not be acceptable for a wide variety of common use cases. That has to be weighed against other use cases that require more stringent controls on data exchange, because these address rarer, but also, disproportionately more complex and costly care.

How do you weigh the value of these two use cases against each other? What choices do you make about the data that is required and optional to cover both these cases (assuming you even know about these two cases in the first place). What is the value of the solution for the common but not life-threatening solution? That value of the solution for rarer but life-threatening cases?

I don’t have a good process to explain to others on how to decide this sort of thing. For one, I’m not clinical, so I have to rely on the clinical judgment of others to make many of these assessments. Secondly, there isn’t really a way to objectively measure these risks. To do so would be to put a value on the cost of human life, which may be technically feasible, but is arguably infeasible in so many other ways.

There are some tools that we can use. Cost / benefit calculations, even using fictional data can be very revealing. If you are right even on the order of magnitude here, the results to the real world are often comparable^{[citation needed]}. Risk assessments are also extremely valuable, and should be done even before cost / benefit calculations. Even with these tools though, expanding a use case to provide broader coverage can make things too complicated to completely analyze using these tools. Then, you just have to go with your gut.

In some ways, standards development is still an art. Someday it will become a science, and when we get there, I’ll know it, because there will be nothing else left for me to do but enjoy life. I hope that day comes when or after I decide to retire.

^{[citation needed]} I wish I could find the research that backs this up. When I do get into a Masters program it would be an interesting topic for further study.