Evaluating Standards

I've been involved in several national and international efforts where at some point, a standards selection needs to be made.  This happens in IHE, has occurred in the past in HITSP, and frequently occurs in the S&I Framework projects.  It also occurs in the HIT Standards Committee, often based on recommendations of various sub-committees.  And in the past, when I was active in Continua, we spent time on it there as well.

Overall, the process worked pretty much the same way, the first step was to investigate the possibly relevant standards, then they would be evaluated, and finally we would propose which would be used.  Each organization that did the work had different criteria that they evaluated the standard against, but they often came with the same general set of criteria:

• Fairness
• Appropriateness
• Cost to Use
• Maturity

They all approached it somewhat differently, some with more process and some with less process.  

Fairness

One criterion that is sometimes overlooked has to do with the development process and availability of the standard.  In all the processes I've been involved in, there is some time spent on availability, but not a lot on other aspects of development.  The need for a standard to be freely accessible to the public has emerged as an important criteria in various initiatives worldwide (and it helped sell the idea of freely accessible IP to the HL7 board).  The other issue is who can participate in standards development and whether that process is fair.  

There are a couple of ways in which this criterion is approached, but for the most part, there are few cases where it has ever become hugely relevant.  The real issue that is never addressed or put on the table is dominance of participants in the standards making process.  You can usually find multiple standards supporting the same capability, and different players and 800-lb gorillas in different places.  

I've rarely seen a standard be dropped from the selection process because it was unfairly produced.  Usually what happens is that it doesn't make it on the list because it does not get past the first stage of identifying standards.  "Oh, that's not a standard, that's ____'s proprietary way of doing it."   If that is not really a questionable statement, it's not worth spending a lot of time on.

I have seen some initiatives fail to list appropriate standards in the initial list, and that usually has to do with dominance of one group in the identification process.  It's usually an easy problem to spot and fix, but it often indicates a sign of struggles to come in the selection process. 

Appropriateness

Is the standard appropriate to the task at hand?  Does it meet the needs of the use case?  Was it designed for this use case, or a closely related one? Will it work?  Does it add value?  Is it technically appropriate?

More often that not there will be multiple choices to do the job, and both are equally capable as far as real-world engineers are concerned.  I'm not talking about uber-geeks here. If the average engineer cannot see a difference, then the difference is for the most part, not worth discussing.  It isn't what the experts think, but rather what the engineers in the field are going to do with it that matters.  Ubergeeks specialize and know all the ins and outs and details of the standard.  We can chest-thump our favorite standards with the best, and often do.  Ignore the chest-thumping gorillas and go look at what the rest of the geeks are doing.

I've watched a lot of standards selection processes go into deep rat-holes on appropriateness, when in fact, what it really happening is that other battles are being fought (e.g., cost).  Focus on what is relevant.

Cost

This has two aspects, the second of which is always difficult to evaluate.  The first aspect is how much it costs to acquire or license the standard for use.  This is pretty easy to work out.  The second aspect has to do with what it costs to deploy the standard in product.  This is rarely investigated deeply because few really have the time or inclination to develop a full-on project schedule to do a cost comparisons, and those schedules are only initial battle plans that will change immediately upon trying to execute them.  Cost and maturity can be quite intertwined.  

One really good way to address cost is to look at ease of use.  Evidence of that can be found by looking for maturity.

Maturity

The question of maturity really isn't about how long the standard has been around, but rather about its prevalence and support for it in the market.  A standard like XML was already rather mature shortly after it arrived simply because it was a refinement of the existing SGML standard, and could be supported by SGML processors.  A ten-year-old standard that nobody has implemented is a lot less relevant that a specification that isn't even a standard yet that is being implemented all over the place.

There are a lot of different ways you can get at maturity: How many open source implementations are there?  How compatible is it with existing technology?  How much technology uses the standard?  These all get at what the standard's position is in the market.

You don't need to ask an uber-geek what the market position is, in fact, that's not the best idea, because each uber-geek has his or her own set of tools they like to use, and they will definitely be influenced by their preferences.  That's what market research companies do.  Can't find the research?  That in and of itself is an indication, and usually not a good sign.

A couple of good substitutes for maturity:  How many books on Amazon can you find on the standard?  How many classes or education or certification programs can you find on the standard?  What does Google (or Bing) tell you?  Look not only and number of hits, but also diversity of sources (1000 hits from one source just means you've found ONE authority).  Where are the Internet communities discussing it?  If the standard is being discussed on Stack Overflow, that's probably a good sign.

Putting it all Together

Different people and projects will have different concerns about fairness, appropriateness, cost and maturity, and so will apply different weights to each of the various aspects.  And proponents of one standard over another will often weight the various factors to favor their preferences.  Weights are subjective inputs and will influence selections.  There is no single right answer here.  It depends on what you are trying to do.

Don't expect the amount of process that you put into the evaluation to be all that influential in the credibility of the assessment.  That is a lot more dependent on how deep you really dig into the facts vs. how much you rely on opinion and rhetoric.  The IHE process is fairly lightweight, but the assessments are pretty quick and well informed.  Other processes are fairly rigorous (e.g., HITSP), but the rigor of that process didn't substantially improve the credibility of the assessment.  If you are assessing standards on these axes on a scale of 1-10, you are probably already too fine-grained.

A three or five point scale will provide a great deal more reliability, and will also help you identify cases where there truly are equally capable competing standards.  It's important to recognized when you have that situation, and to make a conscious decision.  Even if the reason that you select standard A over standard B is simply that you like or are more familiar with Standard A, that is still a good reason for choosing standard A when all else is equal.  After all, you have to live with those choices.