Planning a Certification Project

I'd so like for John Halamka to be right about certification costs, but I don't think he is based on a number of different factors that he didn't accounted for in his post:

Before I go any futher, the project management knowledge that goes into this post is based on my work experiences as as software engineer/architect before I started with my current (or even my prior) employer.  I am not responsible for planning certification projects today.

1. Syndromic surveillance may be optional menu-set criteria under the incentives rule, but it is required under the certification rule (and we still DON"T KNOW what the implementation guide will be if using HL7 V2.5.1).

2. Prior CCHIT certification only means that an EHR has the features found in prior CCHIT requirements.  New test plans and tests will need to be developed and internally executed on the product before an EHR is certified. Even if the product is CCHIT 2011 preliminary certified the whole certification process will need to be re-done based on APPROVED test methods.

3. The certification process needs a project manager, regardless of how many features you have ready to be certified. That's going to be nearly a full-time job.

4. Not included in John's posts are the costs comprehending and responding to the regulation.  The regulations include about 750 pages of preliminary text, and changes described in about 1000 pages of final text. That's no small task.  I'd put a couple or three people on that alone for a week.

5. There's a 135 page externally developed test plan that must be reviewed (the NIST Approved Tests Methods), as well as all the materials from the certifier.  This job may be smaller than understanding the regs, but still big enough. And if you are an EHR vendor, you need to do it twice because products are already under development as customers cannot wait given tight time frames for meaningful use, so that means you already reviewed the preliminary work.

6. Because the EHR System is likely already under development, this means that any new features have to go through a engineering change order (ECO) process. Essentially, you are changing requirements in the middle of development. That's at least twice and possibly three times more costly than going through normal development processes.  And because we are still waiting on the surveillance guide, another ECO will have to be filed later, and we still don't know when that will drop yet!

7. Based on this EMR and EHR post, you can see that the fees for certification range from a low of around $15K to a high of $33K. That's also a few days where the entire team does nothing else but deal with testing.

8. Features have to be documented. Assuming about 5-10 pages per "new" feature, thats about 50-100 pages of documentation according to ONC estimates on the "gaps" (about 25%). That's two months of one person right there, or more depending on experience.

9. Features that have to be certified have to be demonstrated. A lot of EHR systems support integrations using a variety of techniques, including the use of integration engines and report generation tools. Report generation tools especially come into play in developing quality metrics, because these may also need to be customized based on the implementing organization's workflow and policies. So, that means that a sample implementation needs to demonstrate the ability to generate the various metrics. That implementation will also likely become a starter implementation for providers to use.

10. Saying you have to compute the umpteen quality metrics as a single feature is misleading. Each needs to be evaluated separately. Yes, there is some reuse, but individual Quality metrics are NOT a single feature, each one needs separate evaluation.   BTW: Please keep that in mind when time comes to add new ones in the future.

11. In addition, there is another burden that EHR vendors have to pay attention to that John's project will not (at least under the same regulatory scrutiny). EHR vendors have many QA/RA processes equal to FDA requirements for medical devices.  There are a large number of regulatory requirements that simply aren't present in normal software development activities. Some have estimated that this increases development costs by as much as 500% [citation needed].  Note that EMR systems developed internally by healthcare providers don't go through the same scrutiny because they are not products for sale that the FDA can regulate.

Based on just this much data alone, and prior experience as a software developer / project manager in non-medical IT, I can tell you that while I'd like for the ONC estimates to be lower, they already look pretty good. 

I'd need a certification project plan to evaluate them further, and to put costs to the development efforts themselves, which is how John and his team will be looking at the problem over time. So let's try building a project plan as an experiment:

Given what I said about quality measures alone, I'm going to start with ONC's estimates, and not what John assumed of three to four new features. ONC estimates about 10 features will need to be completed (25% * 40 = 10) in a previously CCHIT Certified product. I guestimate a feature to be about 3-9 months of effort depending on size, just based on past software devepment experience with no evaluation of the gaps between CCHIT and the Certification rule, or the size of those gaps.

I'm going to use three-point estimating, but simplify by saying all features are the same size.  With a low estimate of 3 months, a high estimate of 9 months, and the most likely right in the middle at 6 months.

I read through Average IT staff salaries from Computerworld and picked 80K / developer as a ballpark.

I'll esimate the total cost (including benefits, taxes, workspace, and equipment) as being about 2.5 to 3 times salary, as this is what I was trained with years ago as a software developer.  I have no clue if this figure is right for healthcare software.

Nicely, the average effort is simply going to be 10 * the mid range estimate, or 60 months, and the variance is 10 months², for a standard deviation of ~ 3 months.
That makes the 95% completion time for this project about ~65 months or ~5.5 person years.

Average yearly developer salary: $80,000
Fully Burdened yearly cost: 200,000 - 240,000
Timesthe length of the project (~5.5 years)
Total certification labor cost: $1.1M - $1.32M

My example project plan is pretty much in the mid to high range of ONC's estimate of beween $500K to $1.5M (remember, John did say $1M or more).

If I use the estimate of 4 features (very generous based on my comment about quality metrics above) with the same three point effort estimates, it still works out to $450K - 540K

You can run your own estimates with your own figures.  I think if you put together a plan considering all of the factors above, you'll see that the ONC estimates are not terribly out of line.

Note, I won't be asking certification bodies for their opinions or product development cost estimates by any stretch, because I recognized they have a vested interest in the answer being lower than reality.

My guess is that a really good process can shave points off of the ONC estimate, but not by much more than 10%.