I'm in Portland Oregon for most of this week for a workout with one of our development teams. Joining me are my boss and one of our other Standards Geeks. I've long known the importance of face to face contact in developing relationships, and the benefits of my being out here for just one day are already apparent. For one, I've learned that my boss is a better pool player than I am, but not by much ;-)
Our discussions today were far ranging, but one of them centered around validating CDA constructs. There's a great tool that's been developed by NIST called the CDA Guideline Validator. This tool is based on collaborative work from NIST, Alschuler Associates, LLC, Integrating the Healthcare Enterprise (IHE) and the CCHIT Health IT Collaboration Effort "LAIKA" project. While it is a great tool, there are still some process issues that need to be worked out for it to be of greater service to the healthcare IT industry. Don't get me wrong, I love these tools, and point people to them several times a month, but I'd like to see a little bit more.
One of the issues that I run into is that we have to revalidate this tool every time it gets updated to use as part of our testing processes. Here are a few suggestions I'd like to make that I think will improve the use of this tool as part of certification testing and vendor implementation.
1. I'd like to see a design document that describes the overall design of the validation tools. The schematron source code for the validator is great, and frankly, I know a good bit about how that was built, but others need access to that information as well. This need not be a long document, it could be as short as 3 - 5 pages.
2. I'd like to see some explanations about how to read the output documented somewhere.
3. I'd really like to see a validation plan that shows how the rules implemented by the tool are tested, and a validation suite that tests the rules (both positively and negatively).
4. Having Andrew's and Mary's e-mail contact information available is terrific, but a little hard for people to find when they need to report problems. Also, not having an active bug list that people can view and track makes keeping up with the issues a bit of a struggle. I'd like to see a real bug tracking system installed and accessible from the main page.
5. Coordinating feedback from bugs to the organizations responsible for interpretation of the various templates is also difficult (I'm involved in three of them, and I hear from Andrew regularly with all three hats on).
6. I'd love to see a way to directly link each of the errors reported to the appropriate place in the document from which rule is derived. For HITSP and HL7 this is fairly straightforward (it's just a link to the appropriate constrain ID in the document), but for IHE is a little bit more difficult. The IHE profiles need to be a little bit more formal about their constraints in the profiles, now that it (we) have removed the Schematrons themselves from the technical framework (note to self, add as a discussion item at the PCC face to face in two weeks).
7. We need a little bit more formal governance model for how to deal with interpretation of the standards and implementation guides that this tool is supporting. Somewhere along the way I'd like to see a way to verify that the various tests do in fact meet the requirements specified by each of the various guides, and a process for resolving issues that require input from the appropriate authorities.
8. I think this tool could be taken further, and I'd like to see it run as an open source project(*) that we could all participate in. I think structuring it that way would provide more capacity for improvement.
We've already got a terrific bunch of players that has developed this tool, let's put them all on the same team.
* LAIKA is already an open source project, but only supports C32. I'd like to see the entire set of Schematron validation tools be part of a separate open source project. Oh, and it'd be really nice if they supported the IHE SVS profile for value set validation (ITI changed the profile during public comment last year to support an HTTP GET retrieval of the value set in XML that is suited for that purpose for that very reason). And furthermore..., oh just start the thing and I'll chime in.
Keith,
ReplyDeleteVery good points, an important, if not the most critical one, I only see hinted at in number 7, let me offer an expansion of it, or you can add to the list:
Ownership of "testing the test" by spec developers, e.g., HL7, IHE, HITSP, et al. If any organization is going to develop a profile or IG or whatever, and expect or rely on others to create the Schematron, then they have an obligation, IMO, to provide the necessary test data files that will be used to make sure the Schematron that is written is accurate. Further, they should be actively involved in the test development process so the whole operation moves forward in a timely and well coordinated fashion, resulting in something everyone can point to and say "you can rely on this."
This, to me, is a gating factor in moving forward on any of the other points you raise. Without this commitment and cooperation from spec developers, then we have no end-to-end process that results in a sense of veracity and utility of the tools in question, and they will remain useful-to-a-point, and that point would be far removed from what any production-oriented developer would require. At best, they would be better experimental tools, but still experimental.
By the way, I think the work done by the NIST folks and others to date is nothing short of Herculean and heroic, given that they have made such significant strides in the current environment. I can only imagine what we can accomplish by following through on all of these suggestions for improvement.
Bob,
ReplyDeleteI happen to be thrilled with the work that NIST has done, and it's become the de facto standard for testing.
I agree, we need to have the collaboration between NIST and the specification developers to address these issues.
-- Keith
Are you still looking for that open source repo for schematrons?
ReplyDelete