Breaking: HL7 Opens Ballot to confirm FreeIP Policy Changes in its Bylaws

You may recall a few months ago that I reported about HL7's policy change to make its IP free to non-members.  Today, HL7 announced to its members the opening of an administrative ballot to change the organization's bylaws to enable that policy to be made effective.  This is but one of the many things the organization has been working on between then and now to achieve that goal.

The ballot itself is open now to HL7 Members only, and will close on March 31st.  The announcement to members was accompanied by a three page summary of changes, which I will further summarize below:

1. Separates HL7 IP Policy into a separate, board approved document, from the Bylaws, to enable future adjustments of IP Policy without requiring a membership-wide vote.
2. Removes prohibitions on use of HL7 IP for certain classes of members from the current bylaws.
3. Clarifies some procedures around revision of bylaws.
4. Clarifies the rights of student and health professional members.

The new IP Policy proposes that:

1. Affiliate rights are governed under their affiliate agreement.
2. Grants of rights on "specified material" as designated by the board is as follows:
1. Organizational members are granted the right to use and incorporate "specified material" into products and services without additional license fees, and to incorporate portions of the protocol specification (defined in the policy document) into implementation guides and/or product documentation, and to share the specifications within the organization.
2. Other members (individuals, students, and providers) and non-members are granted the right (on agreeing to abide by the HL7 IP Policy) to use and incorporate the "specified material" into products and services without additional license fees, but may not incorporate portions of the specification text itself into their product documentation or to share the specifications with others.

At present, the information that has been made publicly available about "specified material" is that HL7 standards and other IP will be freely available.  I can probably safely tell you that the intent is to include implementation guides (e.g., CCDA) in other IP.  Don't ask me for more than I can say, as I've probably already said too much.

NOTE: This isn't a done deal.  The membership has to agree to these changes before they can be made effective.  If you are an HL7 member, I urge you to vote FOR these changes.

  -- Keith

How should we pay for standards development in HealthIT

I know of a few different models:

1. Organizations pay $$$ to be members.  Then the SDO gives away the standards for free.  This includes organizations like OASIS, IETF and the W3C.
2. Organizations pay $$-$$$ to be members, and the SDO makes the material available to members for free, and non-members for a fee $/$$ (often the cost of membership).  ASTM and HL7 are like this.
3. Organizations that use the standards pay $-$$$ for them.  CPT is like this.
4. Governments whose populations use the standards pay $$$$ for them, and the standards are available to those populations for free.  IHTSDO (SNOMED CT) is like this.
5. Governments develop them $$$-$$$$ and given them away for free.  RxNORM and UMLS are like this.
6. Some organizations pay $$$ to participate in related testing events, and the guides are given away for free.  Other development costs are paid for by benfactors.  IHE is like this.

$ - tens to hundreds
$$ - hundreds to thousands
$$$  - tens of thousands or more
$$$$ - Millions or more

There are challenges to each model. 

Model #1 doesn't seem to work for vertical markets like healthcare because the $$ available are simply not enough to support the work enough to give away the IP.

Model #2 can cut out "little guys" and open source initiatives who don't have resources to purchase the standard or become members.

Model #3 annoys the people who have to pay, but at least puts the onus on the organization benefitting from their use.  Collecting payment is a challenge.  It's not clear how this funding is distributed when payment is for an individual product/standard.

Model #4 is interesting, except when it means that governments exert too much control over what they are paying for.

Model #5 lacks balance in the development side.  I like it for dealing with large aggregations of data like RxNORM or UMLS, but not so good when it's other content that needs consensus development.

Model #6 doesn't seem to scale up and relies on largess from organizations that want to influence what is going on.

Models 1, 4, 5 and 6 rely the largess from somewhere to make standards freely available.  Models 2 and 3 make users of the standards pay for them.  In all cases, the payers recieve some benefit.  In case 1, it seems to be prestige and influence in the development.  Case 2 is sort of an amalgamation of case 1 and 3.  Case 3 is pretty clear that the users benefit, and pay for the standard.  But I know of very few cases where this model is used elsewhere.  Case 4 and 5 seem to argue for public good as the reason for investing by governments.  Case 6 is interesting, as the benefit that comes from testing is certainly worth the investment.

The reason this is interesting to me has to do with my post on HL7 Strategic Initiatives.  It has the most to do with HL7's business model.  It should be no secret that HL7 is considering a new model.  One of the strategic initiatives is "Align HL7's business and revenue models to be responsive to national bodies while supporting global standards development" and the two measures of sucess are:

• A new HL7 International business/revenue model has been developed
• A revised HL7 International membership/governance has been developed

HL7 would love to be able to make its IP more accessible to the users.  The key question is what is the best way to do that.  I have my own opinions, and would love to hear yours.

The Jig is Up, the News is Out

If you've been following the right people on twitter, the news is out.  HL7 is planning on making it's IP available free-of-charge.

Here's the tweet heard round the world this morning:

newsflash #HL7 decides to provide standards and other HL7 IP available free-of-charge FAQ at hl7.org/about/faqs/Fre… Press release to follow
— Rene Spronk (@Ringholm) September 4, 2012

It's true.  And the organization has been working towards this for at least two years.  Sometimes it takes a while to turn an organization this size around, but it can happen.

It wouldn't have happened without the efforts of many people.  Stan Huff insisted that the Business Model task force look at alternatives where IP would be freely available almost two years ago (and I supported him with a few maneuvers through Robert's rules).  Graham Grieve insisted that his IP which became FHIR, wouldn't be given to HL7 unless certain stipulations were agreed to about its availability.  Bob Dolin suggested to the board that the HL7 Domain Analysis Models and Functional Profiles be freely available almost a year ago, and that happened.  And about a month ago, Members of the advisory council strongly supported the idea, and Don Mon (current board chair) and the HL7 Board strongly agreed that we need to make the standards available to everyone, which results in the news you see this morning.

The press release should hit you shortly (12 EDT I'm thinking), and it will contain more details.  The FAQ on the announcement was apparently published early this morning in support of a communication that went to key stakeholders.  It seems that someone found that a few hours before the press release.

This is the kind of thing that can only happen in an organization where key stakeholders believe in doing the right thing, and then do it.  It will take a bit of time to fully implement this decision, but we are now talking months, not years.

You know what my biggest problem is now?  Working on the next big change.  This stuff takes time, and dodging the sniping from the sidelines is precious little help.  It's a lot easier to change a member driven organization when the members are the ones doing the changing.  Consider what you might do from inside HL7, and give it a try.  You might even like it.

---

See the announcement to HL7 Members

Forwards on ABBI

In this, I'm going to respond to some comments by Alan Viars on use of CCDA in the ABBI Project:

HL7 is fine for HIE, provider-to-provider, and provider-to-payer communication. For Blue Button however, the goal is to break from business as usual.

I was there when ABBI was first brainstormed, I get the vision.  If we really want to talk about breaking from business as usual, well, that's a whole 'nother post.  ABBI is very much like any other S&I Framework project in the last two years.

It seems that his main concern you have is that CCDA is hard for patients (and developers) to manage or deal with.  I won't even argue that point.  FHIR would be much easier.  The reality for me, is that my doctor today, could produce a CCD, and will in the very near future, be able to produce a CCDA, and I even have an App on my iPhone that understands CCD today enough to produce it.  I want it to be able to consume it, and I think they could do it.  And I know HealthVault can deal, and so can others.  There are at least three pieces of open source software available to you today to parse a CCD, and at least one supporting CCDA. There's even a stack you can use to get the content from CCD into JSON.  And all of that is freely available and unencumbered by HL7 licenses.

I understand the intent of the ABBI project is aimed at being something new and something very simple for people to manage their own data. I'm not sure what the answer is just yet, but I'm not of the option that CCDA is a good starting point. I say that because it is a overly-complex, closed standard and is the intellectual property of HL7. Its specification is not in the public domain.

Very few standards are in the public domain, and there are good reasons for organizations who publish them to maintain copyright.  We talked about overly complex, and I agree with that point.  Closed vs. Open is a struggle with definitions.  I think he means open as free of cost and redistributable to others, as he describes below:

I'd define an "open data standard" with the following meaning:
The standard is in the public domain. The specification, in its entirety, is free and public. There is no license or restriction to redistribute the standard's text in whole or in part. Any related schema, such as an XSD, is also open in the same way.

That's not the same thing as open in my book.  Open has to do with access.  Access to the standard is one part of this, but another part is access to governance over it (e.g., voting).  But if you don't assert copyright, you've lost a key tool in governance.  But he may mean public domain in the full sense, which implies that there is no governance over the standard.  If so, we'll have to agree to disagree.

Somewhere along the way, HL7 and many other organizations have lost the distinction between innovation, and standards, and so now we use standards to innovate.  It's kind of funny, because the whole point of standards is to pick one way to do things that everyone does, and has already figured out, and innovation is about picking the best way that nobody else has figured out yet.  But, I diverge.

Nothing wrong with clubs with memberships. I used to have a gym membership. But some people want to go just go to the park instead. It wouldn't be very nice if the people from the gym all got together and tried to prevent folks from opening a park as an alternative. That's kind of what it sounds like when we are at the beginning of a new effort and the tone is "it must be CCDA and DIRECT". 

Battles over content are a HUGE distraction from the real need for ABBI.  Give me My DAMN Data. You can use a CCR, but I'd prefer a CCD.  I wrote that song at the point in time when CCR and CCD were the two options.  I want what providers can give me, with as little work as possible.  That's a CCR or CCD (in fact, both are CCRs), and in the near future that's going to be CCDA.  And hopefully after that, something like FHIR will be available.

One of the missing content pieces that is still important to discuss, is the fact that we don't have financial data accessible in the same way.  I want an EOB statement that can be parsed by an App.  I could actually care less now, but later, well, after a few million of these are available to patients, we can surely do something about the pricing transparency issue.  We don't have ANY content standards for that.

I can see how open source standards groups can work with memberships, but their work products should be open using the definition I gave above. HL7 does not meet the definition. If HTTP was like Hl7, I'd have to join and pay a bunch of money just to know that 404 means "not found" and 200 means "OK". I'd agree W3C and OASIS might not be good models either. I'd offer GeoJSON, as a good example.

I think we agree in principle in this.  We'd all like HL7 to make their IP free.  Open Source isn't an option for their business model.  That's an over beer discussion, but the short version is that open source works for one, but not for many, and HL7 has (Too) MANY standards.

I do not work for a company with HL7-based solutions. My vested interest and/or personal mission is to lower the bar for developers, reduce complexity, and increase market competition. I believe this is the best thing for me and my family as a patient and for the American taxpayer.

For this, I need to ask what else other than IP makes people avoid collaborating with, and helping to fix what is wrong in HL7?  If IP weren't an issue, would you work with someone like Grahame Grieve to help HL7 reduce its own complexity?  Or is it simple a need to compete with HL7?  I've been down both roads with SDOs, and I can tell you that I'd rather work with than against them when I can.

I've already shared my deepest, most personal and most selfish motive with Alan, here it is for the rest of you:

My doctor has a system that can produce a CCD today.  In a year or so, he'll have a system that can produce CCDA.  And it will be able to send it out via Direct, and through the NwHIN Exchange.  I want that data.  Personally. On my own computer.  Updated automatically after every visit.

If we don't use the standards that are mandated now, who will connect me to my doctor using those standards, and what will incent my doctor to use that technology?  That is what ABBI is supposed to about, connecting us to our data.  It's what I want for me and for my kids.  If it was CCR that he was going to have, and not CCDA, then I'd be a little bit upset about the standard being used, but still calling for it.  I'm not a Direct fan-boy by any stretch of the imagination.  But you better believe that if ABBI PUSH doesn't support Direct, I'm going to be hugely upset.  If ABBI doesn't support CCDA, I'm also going to be hugely upset.  Why?  Because doctors will have Direct, and ABBI should make it possible for Doctors to cc: me using it, with very little in-between.  Maybe even nothing.  Wouldn't that be awesome?

I sincerely believe we are on to something with this project.  I think content is mostly a distraction right now.  ABBI should support ANY content Alan, Grahame, myself, or anyone else can dream up.  All we should have to do, ever, is just Ask for it.

-- Keith

P.S. If you liked the short video in that last link the director's cut is a three minute ad for ABBI

The Future of HL7

... is looking pretty rosy right now, at least from a leadership perspective.  There are three excellent (in my opinion) candidates from which to choose the next HL7 Chair,

• Calvin Beebe of Mayo, long time co-chair of Structured Documents, past board member and current treasurer.
• Dough Fridsma of ONC, Chief Scientist at ONC and former Director of the Office of Standards and Interoperability.
• Me

I'd be hard pressed to choose between Doug and Calvin if I weren't running myself, but I am, so my decision is easy.  Yours is perhaps a bit more challenging.

Elsewhere you can read my profile, and see my work experience.  Here, I'd like to talk a little bit more about my vision for HL7.

HL7 as an organization needs to change.  It needs to become more Internationally focused so as to streamline efforts globally, and at the same time, it needs to find a way to become responsive to national initiatives which need HL7 standards.  It's been more than a decade since I joined and first heard the phrase "one member one vote", and while we have made progress on that front, we still haven't achieved that goal.

At the same time, we've taken a tremendous shift towards implementers of our standards, with initiatives such as freeing HL7 IP which I've been a part of, and in the development of implementer specifications like FHIR, which I've also been a strong supporter of.

So, why would you vote for me?

HL7 will be a different organization in four years.  That isn't a promise, nor a prediction, it's a simple fact.  I think the organization needs leadership that recognizes the need for change, and that is willing to act in bringing about that change.  We've made some pretty good changes in the past couple of years, but the momentum needs to increase.  We haven't stopped changing, and to grow, we need even more.  We need to complete the journey of one-member one-vote that we started a decade ago, and we need to complete the major transformation of the business model of this organization that started with our free IP initiative.  In some ways, we also need to return to our roots of developing standards for those who implement them, and focus on our customers.  You should vote for me if you think I'm the right person to lead those changes.

   Keith

P.S.  And if you feel like Doug or Calvin is the right person, you should vote for them.  Like I said at the beginning of the post, there are a lot of great choices available.

Freely Available Standards

One of the discussions that I routinely have in HITSP meetings lately is about the cost of the standards to implementers. The assumption seems to be that standards HITSP selects should be available for free to the implementers, and this has been raised as an issue for a number of standards that HITSP has selected. When HITSP reviews the standards it selects (in what we call a Tier 2 evaluation), one of the factors is indeed cost for use of the standard. HITSP does recognize that standards aren't always free, but that reasonable costs shouldn't be a barrier to selection. I reproduce the guidance given for tier 2 evaluation below:

"6.7 Favorable intellectual property and licensing terms

The standard produced by the organization should be readily available to
entities for implementation and usage. The standard should be available to most
stakeholders at no or reasonable cost. Code sets or terminologies should have
licensing arrangements, which do not pose a barrier to usage. This
availability should include a willingness to share necessary standards with the
HITSP and its Technical Committees for use in evaluation and, based on further
agreement, in HITSP Interoperability Specifications. When necessary,
evaluation of favorable IP and licensing terms may be applied to individual
standards if the standards organization maintains different terms for different
standards (see 3.0)"

I can see the point being made by those arguing for free access to the standards. This is especially important those in the open source community who do not have a revenue stream to support purchase of standards. However, I also understand why standards development organizations need to charge for their standards. Developing standards is not without cost, and somehow that cost needs to be recovered, or their simply won't be any new standards to use.

One of the proposals that has been suggested is that the Federal government purchase the rights for implementors to use standards that it mandates for use within electronic transactions through various regulations (e.g, ARRA, eRX and HIPAA). The government has already done this for SNOMED CT. This would make the standards freely accessible to implementors and users of the standards, and would also do something to reduce the costs of implementation of Healthcare IT.

Perhaps ONC should consider using some of the funds that it hasn't allocated yet to ensure that the necessary standards are available. Another important aspect of this would be that it would also allow the American public free access to the technology that is being used to support the provision of healthcare. This brings up several questions, including how each license would be negotiated, how value would be fairly apportioned to the copyright holders, and how access to the standards would be provided.

I think these are all interesting topics, and I frankly don't have an answer to any of them, but I do have another related idea that might be part of the framework for resolving these issues. A lot of what we wind up doing within HITSP which naturally flows outwards to the various SDOs is similar to what Canada InfoWay does with their Standards Collaborative. It might be interesting to review the Canadian model, and to think about what a public-private collaborative organization serving a similar purpose here in the US might do to make the standards more available.

It's just a thought...

Wouldn't it be interesting if ...

"Wouldn't it be interesting if ..." the tweet starts.  And continues: "... there were a standards development organization founded/ran by patients?"

There is an idea in here that is absolutely right, and an implementation that isn't ideal.

Let's start with what is right.  What is right is that patients are the ultimate consumer of what we do in Health IT, and they absolutely need to be at the table and well represented.

What is wrong is simply that standards are about consensus among all stakeholders.  Any standards process that gives one body more representation or control than another is broken, even if it does so with the best of intentions.

I've been involved with standards efforts where one stakeholder group had more power than others (even today this is still true in S&I Framework -- just ask yourself who sets the agenda), and can tell you that it can be challenging.  Been there, done that (on both sides).  Certainly it is "comfortable" to be with the stakeholder group in power, but it doesn't lead to the best outcomes for all.

The harder thing, but probably more useful to do is to become influential in an existing community. Been there, done that. If you can manage it, it results in more success.  Rather than attempting to compete on dramatically unequal footing, what you wind up doing is co-opting the existing community on your own terms.  You have to start softer.  You aren't trying to change the world all at once (see reboot or re-boot).  Just trying to get the direction shifted a little bit, then a bit more.  Until eventually, well, you get the idea.

This is happening to some degree inside HL7.  There are people who are seeking change, and making it happen (e.g., Mobile Health, Quality, Free IP and other initiatives), bringing in other stakeholders to help.  The balance of power is shifting.  Directions are changing.

The biggest challenge for patients in all of this is understanding how they/we can participate. Some are techno-geeky, like me, but with non-healthcare backgrounds (also how I started).  Others are non-techno geeky.  Others have a healthcare background.  And others, are simply just frustrated with the way things work today.

The biggest challenge for organizations wanting to engage patients is figuring out what it means to be a representative of that stakeholder group.  The usual classifications here don't necessarily work.  The minimal requirements for classification of stakeholders (according to section 2.3 of ANSI Essential Requirements) is:

1. Producer
2. User
3. General Interest

A fourth category that often shows up is "government", from the perspective of regulators or imposers of the standard. Producers and users (usually purchasers) often have an obvious financial stake that makes it possible for them to obtain funding to participate in the development of standards.  Government too has a way to fund its participation.  But patients most often fall into the "general interest" category, and from that perspective, often don't have a funding source for participation.

More often than not, other non-profit organizations devoted to representation of patients or consumers will often show up, and do have funding.  But these organizations aren't the same as the stakeholders (and in fact, there are SOME I would NOT have represent me, because I don't agree with their perspective).

I like the idea of patient scholarships, but it isn't clear how that would work, or what the proper governance is.  

The final challenge is how to address the fact that we can all fit ourselves (as I did in this post on Patient-Centric Health IT) into the "General Interest" category.  While each of us can fit into that category, we have different perspectives based on our involvements as producers or users or other categories as well.  And we all balance those differently.  General interest in this list is almost an "other" category, as in Not Otherwise Specified.

What may be important are the various "declarations" that a participant can make:

1. Do you spend money or resources to implement or conform to the standard?
2. Do you receive money or resources to implement or conform to the standard?
3. Are you in a position to require use of those standards in a particular market?

I find it difficult for many who would argue that they are representatives of patients who can say yes to either 1 or 2 as being strictly representative of the "patient" stakeholder group.

So, wouldn't it be interesting if, we were to find a way to enable patients more participation in the development of standards?  Certainly it would be.  But, I think the journey will possibly be more interesting than the destination.

Understanding and addressing technical debt

Architects and accountants have something in common, which is that they need to understand their organizations assets and liabilities.  For an accountant, these are fairly understood.  For an architect, one might think that they are as well.  Your assets are you IP and processes that add value, that enable your organization to out-pace its competition.  And your liabilities are those that don't.  We have a special word in architecture for IP liabilities: it's call technical debt.

Technical debt is a great opportunity for architects to benefit their organization, and here's why: it's something that is already costing your organization in terms of resources and credibility.  You can probably count the defects in the package, the tech support calls raised, the number of open customer issues that  are due to technical debt. You can put a very clear value on it, which makes it a great candidate for reducing cost.  It isn't free, but it is often quite worthwhile.

How do you do it? It's pretty simple -- pick a mess and clean it up.  I don't just mean pick the stuff up off the floor either, like your teen would clean their room.  At the very least, polish it like a fourth year recruit at West Point.  At best, remodel, and I mean completely remodel or rebuild -- like Grahame Grieve did for HL7 Version 3 creating FHIR.  The corollary to "if it ain't broke don't fix it" should be "if it keeps breaking, stop fixing it and replace it."  When car repairs exceed the cost of payments, it makes sense to get a new car (unless you are talking about something like a 69 Pontiac LeMans*).

It's painstaking work.  Usually messes like this accrue because code becomes fragile, knowledge gets lost, nobody knows quite how that works (or doesn't).  And yet there is still some underlying value to the code because it does something important and cannot be otherwise expunged, so some extra effort is needed.  It's like the antique in the attic that just needs the right refinishing to become an awesome heirloom.  This is frustrating work, often risky, and sometimes it's downright boring  and tedious (ever read through nearly a thousand different logging messages).  On the other hand, the value of the work can be made very clear and well defined.

The biggest challenge you will run into in trying to take on work like this are people who are concerned about the risks you are taking on. The biggest tool you have to combat risk is knowledge, and sometimes that means making the time to obtain more. The most fragile software components are usually the ones where the least is known about them. Go learn it. In the end, you'll be glad you did, even though getting it finished wasn't the most glorious thing you've ever done.

  -- Keith

P.S. As a teen, I spent the better part of a winter replacing an engine in a 69 Pontiac. It was cold, it was hard, it sucked.  It was my ride to school, and I learned a ton.  It looked something like the picture below, but was black.

Growing Pains

I think the September Working Group Meeting is when I'll have a new ribbon on my HL7 badge.  I joined HL7 as an individual member either late summer or early fall in 2003, and became an organizational member later that year.  That means I'll get to wear the 10-year member badge if they are still using those.  I guess that makes me an HL7 old-timer, although I must say, that I still feel quite young.  HL7 has a few 30-year members, which still makes me feel like the baby in the room.

Even so, I've been through a lot of HL7's history.  When I joined, HL7 was considering how to become a more international organization.  I seem to recall they even had a grant to work on that, I think it was from RWJF. One of the outcomes from that effort was a board decision to have one working group meeting every other year in an international setting.  Up until that time, all working group meetings for the 19 years prior had been held on the North American continent if my understanding is correct.

In 2005 I worked on my first HL7 specification, called the Care Record Summary (CRS).  It was clear at that time that HL7 V3 was troubled and delayed.  Ballots were taking forever to pass and we needed to clean up that process. The CRS was conceived of in February, launched in March 3, and went out for ballot in the beginning of April. I spent most of March working on, and writing the text of that document, with a copy of another specification sitting open beside me to use as a model.  That was the e-Medical Summary created by Vancouver Island Health Authority, that they had freely shared with HL7 members.  Schematron became the way of doing things because I learned it first from there.

The CRS has now been superseded thrice in HL7, once by CCD, a second time by CRS Release 2, and most recently by C-CDA. However, the CRS is one of the intellectual ancestors of the C-CDA, and for that I am quite proud.  It took almost another year to get the CRS ballot approved.  We went for three more ballots, and had a bit of a bother with ASTM over alleged IP misappropiration.  I may never get over that accusation, but while I freely admit reusing ideas from eMS (with permission), and harmonization (before we called it that) in CRS of Claims Attachments with the then notional sections of CCR, there was no "theft".  But perhaps few believed that someone in HL7 could write a hundred page specification and have it ready for ballot in a month without copying from somewhere, given it's challenges with V3 at the time. It used to be that it could take years to get something through committee.  Today, a project that takes longer than 18 months is the exception rather than the rule.

In May of 2005, my wife and two children joined me in the Netherlands for the first European Working group meeting, where we reviewed that ballot.  Not only was that HL7's first experience off the North American continent it was also my first and my children's.  My wife and children had a blast on that trip, and we've tried to do something international every few years since.  CRS was the first step in a two step collaboration with IHE on what became XDS-MS, and I wrote that specification in short order as well.  The first edition of XDS-MS was produced for Trial Implementation in August, having started sometime in July, also quite a record.

In 2006, HL7 held its 20th annual plenary meeting, and the organization was still changing.   In the January WGM I'd been handing out resumes to a few select folk. At the May WGM I was gainfully unemployed but starting my new job doing standards full time the week after.  It was fun being the last person in the Claims Attachments workgroup, standing up, identifying myself, and indicating a complete lack of affiliation with any company.

That was the year I'd finally put CRS to bed, became a cochair in Structured Documents, and later kicked off the Continuity of Care Document later.  It took most of that year for HL7 and ASTM to bury the virtual hatchet.  Over the next year, the two organizations worked together to produce what is now the most widely implemented CDA Implementation Guide in the world.  And at the same time IHE PCC and HL7 did their second collaboration, which resulted in the development of XPHR.  And the Bush-era ONC was born.  Back then, they had the short-lived acronym ONCHIT (say it with a soft CH and you'll understand why).

The CCD finished in January of 2007, HITSP adopted it shortly thereafter, and IHE had to play quite a bit of catch-up on use of the CCD standard in the C32. Oh, and I had to rewrite XDS-MS to support CCD now instead of CRS.  HL7 standards had taken on several of the lead roles in the US National program.

At the same time as HL7, ASTM, IHE and HITSP were all working together on what would eventually become the C32, I participated in another collaboration with IHE, HL7 and HITSP on the XD-LAB specification.  The former was chaotic, filled with personalities, and quite hectic.  The latter was chaotic, filled with personalities, and also quite hectic, but fortunately I wasn't leading that charge, so I think I remember it more fondly.  In the XD-LAB work, I recall having a T-Con with Francois Macary (located in France), the HL7 O&O Workgroup, and I think we listened to it in a hotel in Washington DC while at HITSP meeting.  Now that was a collaboration.  We would follow each other from one organizational meeting to the next polishing off the next chunk of the XD-LAB specification.  I think I was the only person who was involved as a member of all three contributing organizations.  It was a grand collaboration, and I recall how impressed I was with HL7 O&O for being so open to it.

As April 2007 rolled around, I found myself in Germany, in the City of Cologne, at HL7's second international WGM.  The international meetings piled on after that, as HL7 moved into an every year plan. There was Vancouver (2008), then Kyoto (2009) and Brazil (2010).  HL7 truly had become much more international in flavor.

Along the way, the balloting process got much cleaned up, and the HL7 Governance process changed, new intermediate steering divisions were formed, and the TSC took on a much more aggressive role in organizational governance.  At the same time, ballots got a lot shorter.  Instead of taking three years or more, we were averaging less than 2, and there were some special cases where work was run through the process in 6 months.

Brazil was my first international meeting as a member of the HL7 Board, and I was about to learn what little progress we had made on the business planning we had made since the original grant back when I started.  I joined the Internationalization Workgroup, and then moved on to the new business planning workgroup as a member the board.

Through two years of effort on the board, we wavered from one extreme to another, first considering the IHTSDO model of selling HL7 to national governments as a way of supporting the organization (starting with the US), and subsequently thinking about alternative structures.  Somewhere in the just past middle of all that (in the summer of 2012), we realized that to remain competitive in both the US and international markets, HL7 standards would have to become free for non-members to implement.  Now business restructuring was critical.  With IP not being the dominant reason to join HL7, the board was now in a position where we needed to put up or shut up.  We needed to deliver new value to members, or could expect the membership growth we had been experiencing as a result of organizations needing to acquire access to the CCD specifications as a result of Meaningful Use to dry up.

Here we are a year later, and tonight I got to retweet the following tweet from HL7:

Today the #HL7 Board approved new membership benefits to be announced at the September Plenary meeting. More details to follow. #HealthIT

We've been working on this since September of last year.  I won't steal @HL7's thunder by telling you the details of the new benefits, but what I can say is that they do create more value for existing members.  And they provide a reason for new members to join the organization.  There are a number of new programs we'll be piloting or initiating as part of this benefit package, and we'll be working hot and heavy to roll many of those out in time for the 27th September Plenary.  Keep your eyes on @HL7 in the coming weeks to learn a bit about what's in the package, and join me and the HL7 leadership at the Wednesday morning business meeting of the working group to learn the full details.

It's been a long haul from my first HL7 activity to today, and I expect I still have many more miles to go.  But I think HL7's going to be headed for a better future.  We aren't all grown up yet, but we've certainly made quite a bit of progress in the last two years.  There are still issues to address (e.g., one member one vote, a US Affiliate, and still more challenges to come), but I think we have the will to pursue these.

The year in review

If I had to sum up this year quickly, I'd say for me it was about Quality Measurement, Patient Engagement, and Meaningful Use Stage 2.  Most of my posts have been related to these topics.  This was also the year I took serial writing seriously.

January started year 2 of Meaningful Use.  In January, I didn't go to the IHE Connectathon for the first time in nine years.  I'll be back again this year.  For fun, I wrote The Characters in a HealthIT Standards Meeting.  In January, I was deep into planning and designing transforms for HQMF and Query Health.

February started the month that Consolidated CDA 1.1 was finalized by HL7, after several months of work with IHE, HL7 and S&I Framework.  Early in the month was an IHE meeting in Paris.  My family came along for the ride, and we learned quite a bit about communicating with French speakers.  Of course there was HIMSS, and the almost anti-climactic announcement on the Meaningful Use regulation the day after most everybody has already gone home.  ONC and HHS, stuck in their process, couldn't say much until too late.  I coded, and subsequently demonstrated transforms for HQMF and Query Health at HIMSS.  I spent a good bit of time reviewing the rules.

In March, I started a four-part post which starts here and is completed here about having too many definitions of summary care record in the rules.  It got fixed in the final rule, but there is still yet room for improvement (maybe in the 2016 edition).  In case you weren't aware, the data required for "data portability", and for transfers of care is defined identically in the rule.  HL7 began forming its Mobile Health Workgroup. 

I began my series on Health IT Standards 101 in April (and still have to finish it, although I expect it to be a never ending story).  For fun I wrote Four Career Lessons from the Incredible Hulk.  I issued an Ad Hoc Harley to Jeff Klann for his ability to pick up where I left off on Query Health.

May is the month for one of three regularly scheduled HL7 Working Group Meetings.  We spent a lot of time talking about FHIR, CDA, Templates and the NwHIN RFI at that meeting.  The NwHIN RFI resulted in nothing significant^* happening with respect to governance out of ONC in 2012, but there were some good discussions going on here and elsewhere.  It remains to be seen what will happen in the US.

June is another good month to read some more posts about HQMF and Query Health.  At the beginning of the month, I got to take my daughter to a not so secret White House meeting called the Patient Access Summit.  And my daughter joined The Walking Gallery.  She also wrote her first post for me for this blog.  June begins the summer of standards work that I typically engage in.

July was a slow month for the blog, which usually means a heavy one for other activities.  July is when we meet to deal with the public comments and produce the trial implementation content for IHE Profiles.  I finally got around to starting my series on moving from C32 to C-CDA at the end of the month.  A good friend and colleague, Glen Marshall, retired this month.  He got the first lifetime achievement Ad Hoc Harley.

In August, ONC finally kicked off the ABBI Project, which has become one of my bigger "volunteer" efforts.  Of course, I love the acronym, and so does my daughter.  We often joke that ONC gave it that name to suck us in.  She and her younger sister even did a 180 second ad for it.  I continued my series on moving from C32 to C-CDA in this month.  And of course, we got the final rules this month, which led to another series of posts.  In August, I attended the HL7 board meeting where we approved making the IP free.  Only the rest of the world wasn't to know until nearly a month later.

September was kicked off by this huge announcement from HL7.  I've been working towards that since I was elected to the board, and will continue working on it in my second term.  I began playing quite a bit with OAuth in this month, even implementing an OAuth 1.0 Provider.  The test methods for Meaningful Use Stage 2 began to show up this month in waves.

October is the month that my ABBI prototype was moved to the cloud.  I spent a lot of time on ABBI in this month. Just for fun, I built a go-cart in a couple of days.

In November, continuing with one of this year's themes, I created a slide called Hashtag Soup: Relating QDM, HQMF, eMeasures, QueryHealth, QRDA, SIFramework and MeaningfulUse Stage2.  That slide got picked up an used by an ONC staffer for an educational slide deck.  Elsewhere, I spent half a day listening to, and responding to testimony to a House subcommittee on Meaningful Use and Meaningful Results.

December isn't over yet, but I've already been to an IHE Meeting, the mHealth Summit, and the ONC Stakeholder Meeting this month, including the ABBI Town Hall Meeting.  IHE USA announced it's new certification program, and well, there's just a lot going on.

^{* Just days before the year ended, ONC announced a grant on Governance.}

Implementation is always the real test

When HL7 implemented the board decision at the beginning of the week to make selected IP free, many of you noted that DSTUs were still marked as being available to members only.  The board intention somehow was not implemented as some (including me) thought it had been clearly specified.   I'm told that will be corrected shortly now that the board has clarified its intentions.

I certainly never expected DSTUs (including CCDA) to be treated any different from anything else.

It just goes to show you that what seems to be obvious isn't always.

I don't think even RFC 6919 would have helped here.

Open Source Software as a way forward for HealthIT

I'm a big fan of open source.  A lot of the software I use is open source, and much of the software that isn't relies on open source.  I've given a wee bit to the open source community, but not a whole lot.  I think my largest contribution has been publicizing it on this blog.

But I've been reading a number of posts and tweets lately that tout "Open Source" as the way forward for Healthcare IT.  Some of the recently posted links appear below.

• http://blogs.hbr.org/cs/2011/03/open_source_software_hits_a_st.html
• http://www.medicalnewstoday.com/articles/218627.php
• http://www.jmir.org/2011/1/e24/

In the interest of full disclosure, I work for a software vendor (see my profile).  You might think my views are skewed against open source.  I don't think they are, but I'll let you make your own decisions.

There's nothing inherently better or worse about the quality, features or capabilities of Open Source Software than of commercial offerings. In fact, for some things, the Open Source implementation is THE industry leader for the task.  I challenge you to find Java based applications that doesn't support Xerces for XML parsing or Xalan for XSLT transforms.  It's just hard enough of a task to show how these open source solutions dominate that space.

The two major benefits of open source seem to be cost, and the opportunity to modify the product to suit your own needs.  For the small physician office or hospital, the cost of the software seems especially attractive. For larger organizations with a skilled programming staff, the opportunity to modify the product may be even more attractive.  One of my own favorite benefits of open source is how close it brings product developers to their customers, because I think it has incredible value.  That may be the one reason why open source is viewed as a way forward.

The true costs of software ownership only start with the licensing or purchase price of the software.  The Total cost of ownership can be much higher, and many open source offerings don't include the same services that commercial offerings do.  Smaller providers need to be aware of the gaps, because organizations using open source software often have to make up for what is missing from open source.  This can include documentation, education and training, service, support, maintenance, and in the era of Meaningful Use: Certification.   Larger organizations will often have their own staff who can often fill those gaps, but that just shifts the cost from one place to another (it may in fact be cheaper, but can still require a substantial investment).

Because open source efforts are often volunteer supported, they don't always have funding to support things like Certification.  Getting an EHR ONC-ATCB certified is no small task.  It requires weeks of preparation, and person days committed to testing, and can include fees¹  from around $20,000 to as high as $33,000 dollars to certify a complete ambulatory EHR.  Few Open source efforts have the funds to cover that kind of cost.  That responsibility would become a cost needing to be borne by the implementers of the product.  Smart implementers would be we well advised to partner up with other users in their community to share the cost of product certification in this case (see ONC FAQ 4).

Many Open Source efforts are supported by large software and/or hardware vendors.  For example, Intel has a fairly large commitment to open source supporting their processors.  If you take a look at the membership of Open Health Tools, you will see a number of large IT vendors, and several EHR vendors who participate.  Many of these have contributed substantial amounts of intellectual property, and development resources to these efforts.  Their staff are often very active in, or even leading open source efforts.  These vendors often provide services to fill in the gaps in conjunction with their involvement in the development of the software.  Other vendors or consultants may also offer services to backfill without necessarily being directly involved in the open source development efforts.  Some  vendors have created their own open source product offerings:  The code is completely free and downloadable, and you can contribute to it, but the open source effort is managed by the vendor (and in some but not all cases, the IP is retained by the vendor). These vendors often generate revenue by providing services, support, education, consulting, et cetera, around the implementation and use of the product.

The Federal Government has event supported a couple of Open Source efforts.  One of these is the CONNECT Open Source project which supports connection to the Nationwide Health Information Network.  Another appears to be the popHealth project designed to support Meaningful Use quality reporting.  In that particular case it appears (see Project Milestones on the project home page) that the project will also be ONC-ATCB certified for that purpose.  Federally supported projects have their own set of challenges.

Federal projects live as long as there is funding support from the agency, and even when the agency (or agencies) would like to renew the funding, it's not always the case that it does.  When it does, there is no guarantee that the supporting contractors will remain the same, or that they will do things the same way.  The contract for the CONNECT project was awarded to CGI/Stanley, but was protested by the original contract holder Harris.  The lights are currently on for CONNECT, but little work is happening until the dispute is resolved. This is very frustrating on two counts:  Some Federal agencies were relying on this activity to meet some of their objectives.  There are others who have invested time and effort into the activity, and in other intellectual property in the hopes of providing services that they won't be able to capitalize on as much as they had hoped to until the protest is addressed.  I don't know of any studies on risks for Federally supported open source activities.  Given my own experience with Federally funded initiatives, I would expect that there's about a two-year risk cycle to consider at the very least.

So, my advice:  Be aware of your needs.  Go into your software implementation project with your eyes open, and understand the total cost of the software you will be using. Do look at open source efforts (here is a list to consider).  If the open source efforts meet your needs, then by all means, consider them.  Don't assume that because the cost of the software license is 0, that it will be cheaper in the long run.  Do the analysis, and be aware of potential risks.  Take the same effort you would vetting an open source project as you would a software vendor.  If, in the end, you decide that open source is the way to go, then it will be the way forward for you.

  -- Keith

¹ Not all ONC-ATCBs publish their fee structures for certification. These figures are from publicly available information.

Optimizing Inter-Microservice Communications using GZip Compression in HAPI on FHIR

It's widely known that XML and JSON both compress really well.  It's also pretty widely known that one should enable GZip compression on Server responses to improve server performance.  Not quite as widely known , you can also compress content being sent to the server (for POST or PUT requests).  Also, most people can tell you: JSON is smaller that XML.

And size matters when communicating over a network.

So it should be obvious that one should always GZip compress data whenever possible when connecting between two servers, right?

Uhmm, not so much, but you could already see that coming, because what would I write a blog post about if if were true.

Here's the thing.  Compress saves time for two reasons:

1. It takes less time to transmit less data.
2. There's less packet overhead with less data.

But it also takes CPU time to compress the data.  So long as the CPU time taken to compress the data on one size, and uncompress it on the other side, is LESS than the savings in transmission and packet overhead, it's a net win for performance.  

Let's look at data transmission:

A maximum transmission unit (MTU) is about 1400 bytes.   This takes a certain amount of time to transmit over the network.  Here are some values based on different networking speeds:

Bandwidth (Mbps)	Time (ms)
5	2.24
10	1.12
20	0.56
100	0.112
200	0.056
300	0.037
1000	0.012

Depending on network speeds, time saving on sending a single packet can save anywhere from  12 µs to 2.2ms.  This isn't very much, but if you have to send more than one packet, then you have interpacket latency, which is basically dealing with round-trip times from client to server for acknowledgements.  ACKs don't need to be immediate in TCP, a certain number of ACKs can be outstanding at once, there's not latency introduced on every packet) sent.  But your network latency also has an impact (network latency is generally measured on the order of 10s of ms) on the throughput.

I ran an experiment to see which method was fastest when sending data in a POST/PUT Request, using GZip or not using GZip, and the results were interesting.  I send 200 create requests in which I controlled for the size of the resource being sent in terms of the number of packets it would required to be sent over, from 1 to 10 packets of data (where I mean packet, the size of a single TCP segment transmission, controlled by maximum MTU size).  I sent the request in two different formats (XML and JSON), over three different networks.

For a control network, I used localhost, which actually involves no transmission time or effective latency.  Also also did the transmission over my local network, so that it actually went from my system, to my router, and then back to my system.  And then finally, I transmitted from my system, to my external IP address (so it left the router, went to my cable model and came back through it).

I burned the first batch of 200 requests to initialize the code through the JIT compiler.

Here's what I found out:

1. Don't bother compressing on localhost, you are just wasting about 2ms of compute on a fast machine. 
2. Don't bother compressing within your local network (i.e., to a switch and back).  Again, about 2ms loss in compute on a fast machine.
3. Going across a network boundary, compress JSON after 3 packets, and XML always*.
4. Use JSON rather than XML if you are using a HAPI server.  JSON is ALWAYS faster for the same content.  For smaller resources, the savings is about 20%, which is fairly significant.

What does this mean for your microservices running in your cloud cluster?  If they are talking to each other over a fast network in the same cluster (e.g., running on the same VM, or within the same zone with a fast network), compression isn't warranted.  If they are communicating across regions (or perhaps even different zones within the same region), then it might be worth it if your content is > 4.5K, but otherwise not.  A single resource will generally fit within that range, so generally, if what you are compressing is a single resource, you probably don't need to do it.

It won't hurt, you'll lose a very little bit of performance (less than 5% for a single request if it doesn't take much work), and much less if you do something like a database store or something like that [all my work was just dumping the resource to a hash table].

That very limited savings you get for turning outbound compression on in the client when making an interservice request is swapping compute time (what you pay for) for network time (which is generally free within your cloud), and saves you precious little in performance of a single transaction.  So any savings you get actually comes at a financial cost, and provides very little performance benefit.  Should your cloud service be spending money compressing results? Or delivering customer functionality?

Remember also when you compress, you pay in compute to compress on one end, and decompress on the other.

    Keith

* My results say the compression is faster, but that the difference in results (< 2%) isn't statistically significant for less than 2 packets for XML.  I had to bump the number of tests run from 20 to 200 to get consistent results in the comparison, so it probably IS a tiny bit faster, I'd just have to run a lot more iterations to prove it.