Changing the Way Standards are Developed

ONC has been arguing for some time that we need to change the way that standards are developed.  From their perspective, the process takes too long, and doesn't result in sufficient implementation.  I have to agree that standards development processes that JUST focus on producing the standards aren't as successful as those which produce working code.

Most standards bodies recognize this.  For example, W3C requires at least one and prefers two implementations of each feature to advance to publication as a W3C Recommendation (standard).  OASIS requires three statements of use before advancement to an OASIS standard.  IETF requires two implementations to move to draft standard level.  IHE requires successful Connectathon testing of three separate implementations, in two different regions, with tests covering all actors at at least one Connectathon.  HL7 has a requirement to identify at least two implementers for projects on the Standards track, but doesn't have formal evaluation of implementations process that is consistently applied by work groups before advancing to DSTU or Standard.

The Direct Project started by ONC was intended to change that model, and became the pattern for what the S&I Framework does today.  There's a lot more focus on working code, pilots and implementation at the same time as specifications are being developed. But there are some limitations to these initiatives as well.

Neither the Direct Project, nor the S&I Framework is a Standards Development Organization.  These are simply ONC funded and coordinated projects.  To be successful, they need to work with organizations like HL7, IHE, IETF, WEDI and X12 to further develop the specifications as Voluntary Consensus Standards.  While Direct worked with IHE (see Support for Metadata Limited Document Sources) and used some IHE specifications (XDR and XDM), the core Direct Applicability Statement still needs to go through the IETF process (which it was targeted for) to ensure continued maintenance, and validate the consensus achieved.  At present, the "owner" of that document is still the "Direct Project" as far as I can tell, and I've heard nothing about advancement of it through IETF as was intended.

In S&I Framework, there has been more cooperation with SDO's, and key specifications like the Consolidated CDA (which was rolled into the Transitions of Care initiative), HQMF Release 2 and QRDA Release 2 (used by Query Health), Health eDecisions, Laboratory Ordering and Reporting and others being coordinated with an HL7 Ballot.  Other initiatives, such as the ABBI project, have yet to develop any sort of formalized relationship with IHE or HL7.  Members of both of those organizations have done significant work and are planning more which could advance the goals of the ABBI project.

The much celebrated success of the Direct Project is often overstated, both in terms of time and number of implementations.  Overall, the Direct Project was "completed" in the same time frame as an IHE profile, or an HL7 DSTU.  What it did result in that IHE and HL7 don't always succeed in is the development several implementations very quickly.  There's at least one open-source, and several commercial offerings.

What is missing from the celebrated successes are two factors.  The first is the time ONC spent in advance of the project setting things up.  Because most of us never saw it, ONC gets away with not reporting it when they talk about timelines.  Add 3-6 months to each ONC project before it ever gets off the ground in a public way, and you'll have a better idea about time spent.  This same kind of time is also spent in HL7 and IHE advancing new projects.  We have a much better record of what happens, because it is all done openly and transparently.  Yes, you do have to be a member to see all of the detail, but much of it is freely available to the public in both IHE and HL7.  To be part of starting a project in S&I, you need to be invited to the table (or a White House Meeting) , and even then, the agenda is often pretty well established before you arrive.

With regard to implementations, well, ONC has a few levers that SDOs don't.  The first of these is that it is a regulatory agency.  Anyone who is paying attention is aware that standards that go through the S&I Process are likely to be cited in regulation.  And once cited, well, that pretty much creates the demand and a market for implementations.  The other lever is money.  ONC had plenty of money invested in State HIEs (more than $500M).  Through these they were able to control WHAT the state would implement with respect to HIE technology.  Several Directors of State HIE programs told me about the pressure that ONC was placing on them to implement the Direct Project specifications, to the exclusion even of other plans, some of which had substantial development investments.

The S&I Framework itself evolved out of something like 11 different contracts.  Given the time and staffing involved, I estimate that ONC spent between $10M and 20M over two years, and I've probably undershot.  Some of that was spent on development, implementation and testing resources.  When you have that kind of leverage, it makes for a very responsive market with respect to implementations.

The "big money" [the initial $2B given to ONC] ran out in September of this year, so any continuing work on S&I Framework comes from ONC's operating budget.  We'll see how well S&I succeeds in the coming year since it's no longer possible for ONC to trade money for time.  There has been lots of great work developed through S&I, and I even include Direct in that, but there are some things where it could be greatly improved.

1. More transparency and openness in the governance of what projects are done, and how they are selected and initiated.  This is ONE of the key failings of S&I (and Direct) in openness.  SDOs have an open and transparent process NOT just to create standards, but also in selecting what standards to go forward with.
2. Greater collaboration with MORE standards bodies.  I love some of the activity that is going on with HL7, but ONC has yet to establish a relationship with IHE International, which has the lowest cost membership model of any of they SDOs they've worked with thus far (it's free).  My hope is that ONC will join us next week (they've been invited) to discuss profiling of OAuth 2.0 for Healthcare uses.
3. Documentation  of project procedures and greater consistency across initiatives.  There's too little documentation of process, and too much inconsistency across projects.  As someone who's been involved in numerous S&I Projects, I'm still confused about process when I join a new project.

As a final note, just in case you think that outputs of the S&I Framework are standards (without the assistance of an SDO), or that it is itself an SDO, the Federal Government wouldn't.  See OMB Circular A-119 on the definition of Voluntary, Consensus standards.  ONC is a Federal agency, not a private sector organization.  And the process they use, while it includes consensus in some parts, doesn't in the selection of projects to move forward.

S&I Framework will need the ongoing assistance of SDOs like HL7, IHE, WEDI and others to continue to move forward in creating standards.  Without them, what ONC will create is what Direct is today, "A Government Unique Standard".

  -- Keith