Stupid XSLT Tricks for OID and UUID recognition

I'm building a FHIR to CDA translator to convert a FHIR Composition in a Bundle to a CDA Document.  One of my challenges is recognizing identifiers that are already in OID or UUID form.

This is a simplified token matching problem.
A UUID is in the form ########-####-####-############, where each # is one of the hexidecimal digits in [0-9a-fA-F].  To test for this, I can take the string, translate all hex digits into # characters and then test for a match to the form.  This test can be used in a choice as follows:

<xsl:when test="translate($value,'0123456789abcdefABCDEF','######################')                   = '########-####-####-############'">
  <!-- ... stuff to do when $value is an UUID -->
</xsl:when>

Handling OIDs is a little more difficult.  The pattern there is number[.number]*, where number matches the pattern 0|[1-9][0-9]* (ensuring no leading zeros in the number).

First off, we can reject anything that is not solely made up of digits or the . character.  That's an easy task for translate again.  The expression translate($value,'0123456789.','') will turn any string in $value to the empty string if it is made up of the specified characters.

We also need to make sure that the OID neither starts with, nor ends with a . character.  The first just uses not(starts-with($value,'.')).  It would be nice if XSLT Version 1.0 supported ends-with, but it doesn't.  So we have to find the last character using substring, and check to see that it isn't a . character.  That expression is substring($value,string-length($value))!='.'.

Next, we need to make sure than no sequence of digits starts with 0 except the single digit sequence containing 0.  Let's create a new string called testValue as follows:

<xsl:variable name='testValue' select='translate($value,'123456789','#########')/>

If testValue contains a .0#, then we have a problem, because it contains a number with a leading 0. But we need to go a bit further than that, because two leading zeros are also a problem, so we need to check to see if it contains .00.  That also catches three or more leading zeros, so we've solved that case.  Oh, and we need to check for the case where the first number contains leading zeros, as it won't have a preceding '.'.  We could either check that one separately, or we could force testValue to contain a leading ., and that would let us reuse the previous test.

Leading to this test for OIDs:
<xsl:variable name='testValue'
  select='translate(concat('.',$value),'123456789','#########')/>
<xsl:when test="
    string-length(translate($value,'0123456789.',''))=0 and
    not(contains($value,'..')) and
    not(contains($testValue,'.0#') or contains($testValue,'.00')) and
    not(starts-with($value,'.')) and
        substring($value,string-length($value))!='.'">
  <!-- ... stuff to do when $value is an UUID -->
</xsl:when>

Using translate to match character classes can also help with other test patterns, for example matching dates, phone numbers,  etc., without needing to rely on an external regular expression library (such do exist though, see EXSLT).

You have to be careful to get this kind of matching right.  You can see the evolution of my OID pattern, which, if I hadn't written it out, might very well have let patterns like 00.1 through incorrectly.

When I use patterns like these in code targeted for production use, I'm very careful to document what the code is doing, because it sure as hell isn't obvious.  If you use these tricks, do the same for the poor slob who has to maintain your code after you have moved on.

     Keith

P.S.  Why is recognizing OID or UUID important in FHIR translations?  I'll leave that to your imagination until I cover the bigger challenge (FHIR to CDA) in detail.

Agile Development

I've been improving my template comparison tool over various iterations, as I have been prone to do most of my life as a software developer.  It never really occured to me that I use agile without really thinking about it in projects like these.  Since I'm both the audience, and the developer, I try something out, and see where it goes.  I start with something that minimally works, or is at least good enough to start with.  Then I look at it, and see what I want to make better or improve upon.

The tool has evolved in various stages.  I started with basic comparisons over the narrative text of the constraints.  I made some changes to remove some non-essentials from the text comparison when I first built it, e.g., conformance identifiers.  Subsequently (after implementing a few other tweaks), I realized that I could further improve that comparison by simply removing parenthetical text.

In between those tweaks, I fixed my context "bug", where I hadn't counted on the fact that that same context could be constrained in different ways.  Originally, I just iterated through all the variations, but found that two different contexts that should have lined up (entryRelationships where typeCode='COMP'] didn't, simply because of ordering issues.  So I then sorted those contexts differently, using typeCode.  I later realized that wasn't good enough, because I was comparing an entry of type Act to another of type Procedure.  So, now I look at both @typeCode and @class.

But adding typeCode and class to the context makes the output ugly, and I really don't need to see this stuff, just organize by it.  So I'll have to change my output to eliminate text between [] in my rewritten contexts.

It's still not perfect.  One of my current challenges is that while I want to keep my comparison table columns aligned between templates, I also want to support indenting of nested constraints in some way.

outer context constraint 1		Old stuff	New Stuff
outer context constraint 2		Old stuff	New Stuff
	inner context constraint 2a	Old Stuff	New Stuff
	inner context constraint 2b	Old Stuff	New Stuff

The problem is that I don't know how many rows of inner constraints will be produced until I produce the output.  So I cannot compute how many rows I need to span value for the empty cells in the inner context.

There are at least three ways I can think of to resolve this.  The simplest of these is to apply a second pass transform over the output to clean up the context breaks.  I can simply output a tag in the recursing section of the table, and then count rows and compute how to clean it up.  See there?  I've done it again.

Well, it's about bedtime (I'm in Budapest for the next three days), so I better finish up and get some sleep.

Localizing Time in HL7 CDA Rendering with XSLT

One of the questions that has come my way from several different sources is how to display times that appears in a CDA document in a way that is locally relevant.  Since the document narrative does very nothing with time that is under the viewer's control, there are really only two places where date time values might appear in CDA:

1. The Document Header
2. CDA Entries

While CDA entries do include date and time, rendering them as part of the display of document sections is rarely done.  The same techniques I mention for the header could also be applied to entries in the document sections though, if you happen to need something like that.

Date and Time values in the Header

There are numerous places within the document header where date and time could appear.  Perhaps the most visible one is the /ClinicalDocument/effectiveTime element, which indicates when the document was created.  Others include:

• author/time (the date and time the author wrote the document)
• legalAuthenticator/time (the date and time the document was legally signed)
• documentationOf/serviceEvent/effectiveTime (the date and time associated with the documented service)
• componentOf/encompassingEncounter/effectiveTime (the date and time associated with the encounter)

Time Zone or Locally Relevant Time?

Time zone is a function of politics, not math.  When the author writes a document at 201408041452-0700, what is the time zone?  Well, it depends upon where they are.  They could be in California, in which case the time zone is PDT.  But they could also be in Flagstaff, Arizona (where I will be next week for the HL7 Board Retreat), or in Zona Noroeste in the state of Baja California, in Mexico.  Time zone then, is a function of where you are, not when you are, and even if you have the when, it doesn't narrow the where down sufficiently.  A few years back, the US changed when it entered daylight savings time, so when you are, is also not just a function of time of day, but also of day, month and year.  Give up yet?  Good.  Don't try to show the time zone in these cases.  Just use the time offset.

Locally Relevant Time Zones

But people don't know what time 1200-0700 is, you say?  OK, so what you need is to convert that to a locally relevant time in your stylesheet.  So, how would you do that?  I'm not going to go into a lot of detail here, but I will make some design recommendations:

• If possible, use XSLT 2.0 and the fn:adjust-dateTime-to-timeZone() function. In your transform.  Get the locale from the user agent.
• If XSLT 2.0 is not available to you, try using EXSL date-time extensions with your XSLT processer.
• To determine the local time zone offset from UTC (which is not the same as the time zone, but will serve for this purpose), try using the getTimezoneOffset() function on a JavaScript Date object.

XSLT 2.0

Let's look at how you might do this with XSLT 2.0 first.  You need to get the time zone from the browser. I'll assume you have a form somewhere, and that you submit that form at some point to the server.  Here is how you might pass the time zone as offset in minutes from UTC (note that getTimezoneOffset() returns offset of UTC from the current time zone, and we want the inverse, the offset of the time zone from UTC, so we just negate it).

<input type='hidden' id='TZ' name='TZ' value=''/>

<script type='text/javascript'>

  Date d = new Date();

  document.getElementById('tz').value = -d.getTimezoneOffset();

</script>

   

On the server size, you'd simply pass TZ as a parameter to your transform (you ARE executing this CDA transform server side, aren't you?  There are some very good reasons why you should do it that way).

In your stylesheet, you might do something like this to turn the offset in minutes into a time zone:

<xsl:variable name='tzString'>

   <xsl:if test='fn:number($TZ) &lt; 0'>-</xsl:if>

   <xsl:text>PT</xsl:text><xsl:value-of select='fn:abs(fn:number($TZ))'/>

   <xsl:text>M</xsl:text>

</xsl::variable>

<xsl:variable name='tzDuration' select='xs:dayTimeDuration($tzString)'/>

Then, when generating a time, you'd do something like this:

<xsl:variable name='timeValue'>

  <xsl:call-template name='reformatTimeFromHL7toXML'>

    <xsl:with-param name='time' select='...'/>

  </xsl:call-template/>

<xsl:variable>

<xsl:value-of select='format-time(xs:adjust-dateTime-to-timeZone($timeValue,$tzDuration),$myTimeFormat)'/>

Without XSLT 2.0

If you don't have an XSLT 2.0 parser, it gets a bit tricky.  There are a couple of different ways to handle date/time formatting, but you really don't want to write any of that code yourself in XSLT.  My favored way of handling it is by calling out to Java code to handle this sort of mess.  You can use standard date/time functions in Java in this case.  The java.text.SimpleDateFormat and java.util.GregorianCalendar classes provide just about everything you need to parse date/time values, and format them.

Below are a pair of templates using Java that are based on some work I used to convert CDA documents to XDS submission sets.  With a little bit of tweaking, you could use this to format date time values however you wanted.

  <!-- 

    Take a V3 date/time stamp with or without milliseconds, and with or 

    without timezone specification and convert it to a date/time stamp in 

    the specified zone precise to seconds, or less

  -->

  <xsl:template name="toZone">

    <xsl:param name="time"/>

    <xsl:param name="precision" select="14"/>

    <xsl:param name="length"

      select="string-length(substring-before(concat(

        translate($time, '-', '+'),'+'), '+'))"/>

    <xsl:param name="zone"/>

    <xsl:variable name="fmtString">

      <xsl:value-of select="substring('yyyyMMddHHmmss',1,$length)"/>

      <xsl:if test="contains($time, '+') or contains($time, '-')">Z</xsl:if>

    </xsl:variable>

    <xsl:variable name="inputFormatter" 

      select="java:java.text.SimpleDateFormat.new($fmtString)"/>

    <xsl:variable name="parsedDate" select="java:parse($inputFormatter, $time)"/>

    <xsl:call-template name="dateInZone">

      <xsl:with-param name="time" select="$parsedDate"/>

      <xsl:with-param name="precision" select="$precision"/>

      <xsl:with-param name="zone" select="$zone"/>

    </xsl:call-template>

  </xsl:template>

  <!-- 

    Take a Java Date and convert it to a date/time stamp in zone, precise

    to seconds, or less

  -->

  <xsl:template name="dateInZone">

    <xsl:param name="time"/>

    <xsl:param name="precision" select="14"/>

    <xsl:variable name="outputFormatter"

      select="java:java.text.SimpleDateFormat.new('yyyyMMddHHmmss')"/>

    <xsl:variable name="MyZone"

      select="java:java.util.TimeZone.getTimeZone($zone)"/>

    <xsl:variable name="void" 

      select="java:setTimeZone($outputFormatter, $MyZone)"/>

    <xsl:value-of 

      select="substring(java:format($outputFormatter, $time), 1, $precision)"/>

  </xsl:template>

Of course, this assumes your platform is Java.  What about all those .Net folk who are stuck with C#? The same principles apply, as C# has similar capabilities, I just don't know what they are ;-)

There is also a platform independent way to handle this, relying on EXSLT date and time extensions.  But, as mentioned there, no XSLT processors support that natively.  So you might just as well use a platform dependent implementation.

Deduplicating Lists in XSLT

I'm in Saudi Arabia for a couple of days before I head off to a 10-day vacation in England with my family.  I'm not sure how much I'll be writing on my vacation, we'll see how it goes.

I have a few projects to finish up before I get to take some well-deserved time off.  In one of those projects I needed to generate a set of lab results, ordered by the date and type of test performed.  However, the XML I was presented with was not normalized in a way that would make that easy.  Instead of each result being organized into separate panels with the panel reflecting the test performed (a complete blood count), with each result in the panel, it was instead organized into a table where each result included the panel, the result and the date performed.

That can be pretty challenging to handle in XSLT.  What I wanted to do was loop over each separate panel, which could be identified by the panel type and date performed, and then within that list, iterate over the separate results.  I could do it using the EXSLT set:distinct function, but this was one of those cases where the code I'm writing doesn't allow me to use EXSLT.  I suppose I could have changed the rules, seeing as how I was the one who made them, but I had gotten pretty far into the code without needing EXSLT and I didn't want to add third party dependencies.

I've done this before, but it always relied on some rather tricky code using the preceding and following axes. I knew there had to be a better way, so I started searching and found the solution. It shows up in the XSLT: Programmer's Reference by Michael Kay, but you have to know where to look for it.

The key as it were, is in the key element and key() functions.  The key element allows you to define an index on a set of elements that you want to find.  It's syntax is:

<xsl:key name="name" match="match pattern" use="key expression">

The name specifies the name of the key and is used later in the key function.  The match pattern provides the list of elements for which you want to generate a key for.  The key expression defines the expression that generates one or more keys in the context of the matched node.

Later, you use the key function, giving it the name of the key that you are looking things up from, and an expression that generates one (or more) keys to locate.  The function returns the list of nodes matching the match pattern that have one or more of the specified keys.

For my example, we'll pretend I had a list of items like this (it was more complex than this, but this is sufficient to show you the technique:

<test test="name" date="date" result="result" value="value"/>

I created a key like the following:
<xsl:key name="myKey" match="test" use="concat(@date,@name)"/>

Thus, each row was indexed by name and date.  

The next step was to select all the test and deduplicate them based on their keys, producing a list of elements with unique key values.  Here is the code that does that:

  <xsl:variable name="tests" select="./test"/>
  <xsl:variable name="distinctTests" 
    select="$tests[generate-id() = 
                   generate-id(key('myKey', concat(@date,@name)]))[1])]"/>


The tests variable defines a list of tests that I want to deduplicate.  The distinctTests variable iterates over each test element in $tests, selecting it if the unique id of the node matches the unique id of the first matching test that has the same key identifier.

One problem with this technique is that it fails when your selection context and your key contexts aren't aligned.  I didn't run into that issue with the problem I was working on.  I'm sure there is a way around it, but I do need to get some sleep.

An XSLT Breadth-First Search for Processing XML Schema (TL;DR)

Last night after spending several hours working with the HL7 HQMF ballot content, I started getting frustrated with the content structure.  The HL7 Publication DTD (yes, I know) is rather roughly documented.  It's based on the W3C Publication DTD, and as in many things in IT, "nobody knows how that works anymore".  Actually, I know just the person who does, but he's retired into his second career, taking pictures of birds.

One of the limitations (and a not unreasonable one at that) is the level of nesting allowed.  They only number sections to the fourth level.  After that, it starts getting funky (and it looks bad too).  So, I want to navigate through the R-MIM Walkthrough, and I just can't organize things the way I want.  One of the problems is that the document has been edited by several people, and the outline wasn't consistent.  Since this section was basically a walk of the DAG (directed acyclic graph) that is the RMIM, I figured I could probably automate generating the Table of Contents.

So I got someone to give me the latest schema generated by the HL7 tools, and started writing a stylesheet to traverse the schema and generate the outline.  Well, that was daring.  After all, here we are in the 11th hour (or perhaps at this point the 35th hour), and I want to completely reorganize a major chunk of the documentation.  Well, needless to say, I didn't get any sleep last night, but that's not because I reorganized things, but rather because of what I discovered when I did finally succeed.  I'll talk about that later.  What I really want to discuss today is this cool stylesheet that I wound up building.

You see, it walks the Schema, starting with the complexType that is used for the root of the HQMF document schema.  It then identifies each Participation, Role, Entity and ActRelationship in the tree, and generates my outline.

Here's the general structure I worked out.  It isn't ideal, but it works and people can navigate it.

1.1 Act
1.1.1 Act Attributes
1.1.1.1 Act.attribute1
1.1.1.1 Act.attribute2
1.1.2 Act Participations
1.1.2.1 Act.participation1
1.1.2.2 participation1.attribute1
1.1.2.3 participation1.attribute2
1.1.2.4 participation1.role1
1.1.2.5 role1.attribute1
1.1.2.6 role1.attribute2
1.1.2.7 role1.player1
1.1.2.8 player1.attribute1
1.1.2.9 role1.scoper1
1.1.2.10 scoper1.attribute1
1.1.3 Act Relationships
1.1.3.1 Act.relationship1
1.1.3.2 relationship1.attribute1
1.1.3.3 relationship1.attribute2
1.1.3.4 relationship1.act2
1.2 Act2
 ... and so on

The focus in the HL7 RIM is on the acts being documented, and this documentation structure keeps the attention on the acts, and doesn't exceed a nesting depth of 4, so I was pretty happy.

So I began writing the stylesheet.  After about 45 lines, I had everything by the recursion back to Act all working.  There were 6 templates in the whole thing.  One called Act, another Participation, another Role, another Entity and a final one called ActRelationship.   I knew I was going to have to break cycles in my traversal, because sections can have sections and so on.  So I wrote a little bit of code to detect the cycles, and the first thing that happened was a stack crash.  After a few moments adding some <xsl:message> elements, I was able to see that my cycle detection only worked for cycles of length two (after all, I knew there weren't any longer cycles in the R-MIM).

As it turns out, I was wrong.  The R-MIM designed allows for something called a Choice box in which you can include multiple model elements.  Then you can attach relationships from the choice box back to itself.  This is such a common thing in HL7 V3 that there's even a special shape for it in the tools.  The problem there is that if you have such a choice box that can go back to itself, the cycle length increases.  Say you had two items, A and B, and a link between them through C.  Now instead of having to detect this loop:  ACA, you also have to detect ACBCA and BCACB.  The more items you put in the choice box, the longer your loop detection has to detect loops for.

OK, so I tried the next best thing which was, as I iterated over things and went down the stack, to pass a list of where I'd been.  Well, the problem with that (besides being clunky), was that it just didn't work.  There were too many pathways through the cycles, and you could get into a real mess.  Now I was really stuck and it was around 10pm.  I recalled a post by Michael Kay (author of the XSLT Programmers Reference) somewhere about loop detection in XSLT, so I dug out my second edition.  No good.  It's described in the third edition, and he finally got the code right in the fourth edition (neither of which I have or needed until last night).  And he wrote it in XSLT 2.0, which is no good for me, because I'm still working in XSLT 1.0 (I know, I'm a glutton for punishment).

So after an hour of digging around trying to find a DFS or BFS (depth- or breadth-first search) implementation in XSLT I gave up, made some coffee and took a walk.  Then I went back to my desk, wrote the recursive BFS code down a couple of different ways, and then made it tail recursive.  Tail recursion is a good thing to do when you are writing in XSLT, because you can overcome a lot of limitations that way.  XSLT doesn't let you change the values of variables once you set them, but a tail recursive call lets you change the input parameters, and it can be optimized by a good XSLT parser into loop.

Here's the basic algorithm:

process(node)
{ ... whatever you want to do ... }


BFS(list todo, list done)
{
    if (!empty(todo)) {
      head = car(todo);
      tail = cdr(todo);
      newNodes = nodesReachableFrom(head);
      needsDoing = (newNodes - todo) - done;
      process(head);
      BFS(todo + tail, done + head);
   }
}


BFS(root, null);

You will note that I never modify a variable after setting it.

This is how it looks translated into XSLT.

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  version="1.0" xmlns:xs="http://www.w3.org/2001/XMLSchema"
>
  <xsl:key name="node" use="." match="xs:complexType/@name"/>
  <xsl:key name="neighbors" 
    use="//xs:complexType[.//xs:element/@type = current()]/@name"
    match="xs:complexType/@name"/>

  <xsl:template match="/">

    <xsl:call-template name="BFS">

      <xsl:with-param name="todo"

         select="key('node','POQM_MT000001UV.QualityMeasureDocument')"/>

    </xsl:call-template>

  </xsl:template>

  

  <xsl:template name="process">

    <p><xsl:value-of select="."/></p>

  </xsl:template>

  

  <!-- BFS generates an XML fragment containing the keys

    of elements which need to be processed in the order they

    should be handled based on a breadth-first search of the 

    tree represented 

  -->

  <xsl:template name="BFS">

    <!-- todo is a node-set() containing a list of all nodes that have 

      yet to be processed -->

    <xsl:param name="todo" select="/.."/>

    <!-- done is a node-set() containing a list of all nodes that have

      already been processed -->

    <xsl:param name="done" select="/.."/>

    <!-- If todo is empty, we don't do anything -->

    <xsl:if test='count($todo)!=0'>

      <!-- head is the first node in todo in document order -->

      <xsl:variable name="head" select="$todo[1]"/>

      <!-- tail is the rest of todo in document order -->

      <xsl:variable name="tail" select="$todo[position() != 1]"/>

      

      <xsl:variable name="reachable" select="key('neighbors',$head)"/>

      <xsl:variable name="needsDoing" 

        select="$reachable[not(. = ($todo|$done))]"/>

      <xsl:for-each select="$head">

        <xsl:call-template name="process"/>

      </xsl:for-each>

      <xsl:call-template name="BFS">

        <xsl:with-param name="todo" select="$tail|$needsDoing"/>

        <xsl:with-param name="done" select="$done|$head"/>

      </xsl:call-template>

    </xsl:if>

  </xsl:template>

</xsl:stylesheet>

This is pretty generalizable to any problem where you need to traverse a DAG without crossing links.  A couple of notes are in order:
<xsl:key> and key(N, V) are XSLT elements and functions that are designed to do element lookup by key values.  These are worthy tools to have in your kit if you do a lot of XSLT programming.  Larn em!
I've set up two keys, one called "node" which returns the @name attribute of the xs:complexType element that has the name you specify.  That's a very straight-forward use of key in XSLT.  However, the second one called "neighbors" is much trickier.  It uses the key mechanism to return the set of xs:complexType/@name values that are needed by the elements in the named complexType.  For what I'm doing, I'm only interested in elements, so this isn't as hard as it could have been.

The reason that these return the @name attributes instead of the xs:complexType element is because the node-set returned can then be used as an argument to the key() function.  I won't go into all the details. You can use other problem-specific logic to find the "neighbors" of your node.

The next bit is also tricky.  The todo variable is a node-set, that I'm treating as a list.  $todo[1] is the head of the list.  $todo[position() != 1] is everything but the head.  So I have a built in CAR/CDR functions (remember LISP?).

Finally, given that you have two lists of items, how do you select items in the first list that aren't also in the second list.  This is how you do that: $first[not(. = $second)].
Where most people go wrong is in writing: $first[. != $second].  Since . and $second are node-sets, they use node-set comparision logic.  X = Y is true for node-sets X and Y if the string values of any to nodes in X and Y are the same.  X != Y is true if there is a node in X whose string value is not = to the string value of a node in Y.  If you don't believe me, read the spec.  Anyway, I showed you this little trick a few weeks ago.

This works, but the output wasn't in the order I expected.  My list is really a priority queue.  The nodes in the node-set are processed in document order.  So, when I "remove" the first node from the list, I'm actually getting the first @name attribute that appears in the document.  That is NOT the order I'm adding them in however.  They get added in whatever order the Schema follows.

It's important to me that I process this stuff in BFS order, because it makes it easier to follow the documentation that way.  To fix that, I had to use another XSLT trick, and that was to keep the lists in document fragments, which I then converted to node-sets using the EXSLT node-set() extension.

BTW: It isn't clear to me yet whether BFS or DFS produces a better order for documentation, but either one can be made to work.

The final XSLT for creating my table of contents can be found here.

This took me about four hours to figure out.  It created more work for me because I was able to see what content was missing.  It also vastly improved the HQMF result because the content is generated from the HQMF artifacts, so I cleaned up a lot of naming errors introduced by a new version of the HL7 RIM and Datatypes (we were using an much older RIM and Datatypes R1.1 for HQMF Release 1.0).  And because I was able to fix those errors, I'm a lot happier about the ballot quality (although still not satisfied).  I'll probably feel better after I get some sleep.

Matching Items in one list against another list in XSLT

This is an overdue explanation for how to do something in XSLT that I was supposed to write up a couple of weeks ago for the HL7 CCD Bluebutton project.

The challenge is that we have two lists:  One of observations (particular lab results) for a particular patient, and the other a set of lab results that mean something (e.g., blood type).  The challenge is to find the set of results from the first list that match any of the codes in the second list.

The key is setting up a test of an item against a node list.

If you have in one file:

<observation>
   ...
   <code code='interestingCode1' codeSystem='...'/> 
   ...
</observation>
...
<observation>

   ...
   <code code='interestingCode2' codeSystem='...'/> 
   ...
</observation>

And in another file you have a list of codes:

<RetrieveValueSetResponse ... >
  <ValueSet id="..." 
    displayName="An Interesting Code Set" version="20061023">
    <ConceptList xml:lang="en-US">
      <Concept code="interestingCode1" displayName="Code 1"
        codeSystem="..."/>
      <Concept code="interestingCode2" displayName="Code 2"
        codeSystem="..."/>
    </ConceptList>
  </ValueSet>
</RetrieveValueSet>



You can make the second file available through the document function:

<xsl:variable name='InterestingCodes' 
              select='document("interestingCodes.xml")'/>

And then select the observations using the following expression:

<xsl:for-each select='//cda:observation[cda:code/@code = $InterestingCodes//svs:Concept/@code]'>
  ...
</xsl:for-each>

Inside the for-each, you'd do something interesting with the matched coded.  If you just needed the first on, you could add a [1] to the end of the expression.

According to XSLT and XPath, this will work because:

If one object to be compared is a node-set and the other is a string, then the comparison will be true if and only if there is a node in the node-set such that the result of performing the comparison on the string-value of the node and the other string is true.

So, that's how you'd do it.

The XSLT document() function

Yesterday, someone asked a question about how to address issues of translating a code to a display name on one of the the Structured Documents workgroup's e-mail lists.  There's a technique that I've been using in XSLT for quite some time that allows me to access look-up tables very easily without having to embed translation logic in the XSLT stylesheet.  Before I describe the technique, I thought I'd share some of the various uses for it:

1. Code translation.  Often you will have codes in a one code system that need to be translated into codes from another code system.  This technique allows you to look up the translation.  I've used this to translate local codes to codes from standard vocabularies for:
1. Unit translation from ANSI+ to UCUM
2. Local codes for problem severity to SNOMED codes for severity used in the HITSP C32
3. Local codes for problem status to SNOMED codes for problem status.
4. Local codes for problem type to SNOMED codes for problem type.
5. Local codes for vital signs to LOINC codes for vital signs.
2. Display name lookup.  Closely related to #1 above.  Often times, I have a standard code, but not the display name associated with it.  I can use this technique on small value sets (less that 1000 codes) to look up the display name (this is the use case for the problem presented on the list).
3. Mapping from an identifier to a web service end point.  You can use this technique to map from:
1. The home community ID to an XCA Web Service address
2. A DICOM AE Title to a WADO Web Service endpoint.
4. Validating against a dynamically changing rule, such as the validation of a code element against the current version of a vocabulary or value set.

The basic technique is to create (or have access to) an XML document resource which you will use in your stylesheet.  To declare this resource, you do something like the following:

<xsl:variable name="myDocument" select="document('mydocument.xml')"/>

This creates a variable which can be used in an XPath expression subsequently in your XML.  In the use case the querant posed, the issue was how to get a display name for a language code, to that the patients preferred language (expressed as a code) could be displayed in the UI.  The patient's language preferences are stored in the patient/languageCommunication/languageCode/@code attribute.

The following XSLT fragment shows a template that will return the display name of the patient language by looking it up through an XML document.

<xsl:variable name="langs" select="document('lang.xml')"/>
   ...
<xsl:template name='patientLanguage'>
  <!-- get the code -->
  <xsl:variable name='lang'
    select='//patient/languageCommunication/languageCode/@code'/>
  <xsl:variable name='mappedLang' select='$langs//language[@code=$lang]'/>
  <xsl:choose>
    <xsl:when test='$mappedLang'>
      <xsl:value-of select='$mappedLang/@displayName'/>
    </xsl:when>
    <xsl:otherwise>Unknown</xsl:otherwise>
  </xsl:choose>
</xsl:template>


The same technique can also be used to access a resource that is created dynamically through a RESTful web-server end-point.  I demonstrate one use of this technique in the post on Values Sets and Query Health.  


Another use for this technique is to check value-set conformance inside Schematron rules. If you have a requirement that code/@code come from a particular value set, you can write a rule that accesses a web resource based on the value set, as in the following example:


<rule context='*/cda:templateId[@root = templateIdentifier]'>
  ...
  <let name='code' value='cda:code/@code'/>
  <let name='valueSetDoc' value='document("https://example.com/RetrieveValueSet?id=1.2.840.10008.6.1.308")'/>
  <assert test='$valueSetDoc//ihe:Concept[@code = $code]'>
    The code/@code element must come from the XXX Value Set (OID: 1.2.840.10008.6.1.308)
  </assert>
</rule>


The use of external XML data files is a very powerful feature of XSLT.  Combining that use with dynamically created XML resources through web services makes it even more capable.

Integrating Schematron Rules and Clinical Terminology Services

On one of the discussions on the Structured Document Work's CCD mailing list, a member notes that Schematron validation such as that found in the NIST Validator doesn't support validation of content using restricted value sets well.  He's right in that Schematron doesn't include an explicit mechanism designed to perform validation against value sets.  But there is a way to integrate a clinical terminology service into a Schematron rule set using the XSLT document() function and Schematron <let> statement along with appropriately structured rules.  The IHE SVS profile was designed to support this sort of validation, as I mention in a previous post.

For validation, their are two practical classifications of value sets: Those that can be enumerated fully in single XML document, and those which cannot practically be enumerated.  The dividing line is based on the available system memory for the Validator.  I would expect that value sets containing hundreds of elements could practically be enumerated in full, but those containing thousands or more terms would not be practically enumerable.  Note that I do not address issues of "static" vs. "dynamic" value sets here, because the value sets can be enumerated dynamically through a web service call.

The mechanism for validating the smaller value sets in Schematron is to create a variable that contains the content of an XML document.  This is done at the top of the Schematron using the <let> statement:

<sch:let name='ValueSet' value='document("https://example.com/RetrieveValueSet?id=1.2.840.10008.6.1.308")'/>

Later in the Schematron rule set, you'd have a rule context where you would use that variable as follows:

<sch:rule context='cda:manufacturedMaterial/cda:code'>
  <sch:assert test='$ValueSet//svs:Concept[@code = current()/@code'>
   ... report error if concept is not found ...
  </sch:assert>
</sch:rule>

This idea was used in CDA Implementation guide Schematrons as far back as 2005, in the Schematron use for the Care Record Summary release 1.0.  In that Schematron, an external file was used (so the URL would have been file:voc.xml), rather than an HTTP URL.

Now, when the value set is large (such as a list of LOINC Lab Results, SNOMED CT problems, or RxNORM Drugs), it's not practical to enumerate every term because the resulting document would be very large.  In these cases, you could enhance the SVS defined Web Service to support a code parameter.  When this parameter was present, the service would return the entry for the single code in that value set when present, or an empty ConceptList if it didn't exist.  The rule context remains the same in this case, but the assertion changes:


  <sch:assert test='document(concat("https://example.com/RetrieveValueSet?id=1.2.840.10008.6.1.308&code=",@code)//svs:Concept'>
   ... report error if concept is not found ...
  </sch:assert>

In this example, the HTTP Web Service request is dynamically created.  If it returns an empty code list, the rule fails, but if it finds the code, the rule succeeds.

So, while Schematron itself does not support "validation against value sets", appropriate integration with a Clinical Terminology Service and a very simple RESTful API does enable it.  I hope that the NIST Validator takes advantage of this approach in the future.

Accessing Positional XML Elements in XSLT

One of the most frustrating things about XHTML and some other document XML formats is that they don't deal with document structure very well.  An H1 tag simply creates a level 1 heading, and so on for H2 through H6.  These tags only "introduce" a new section, they don't really create the appropriate section structure in the XML document.  This creates nightmare for organizations trying to manage structured documentation because it is rather difficult to deal with section structures declaratively in languages like XSLT.  Even the Apple PLIST format is frustrating (prompting this tweet) because the key and value (or dictionary) are arranged positionally rather than through containment.

The same sort of problem shows up when processing HL7 Version 2 messages (say to convert them to HL7 CDA) when manipulating often proprietary XML translations of HL7 Version 2 supported by different interface engines.  HL7 does have a standard Version 2 XML format (see Version 2.x Schemas on this page), but it is not widely supported in products.  So, if you want to process a Version 2 ORU message , you will often find XML that contains one or more OBR tags followed by several OBX tags without proper containment.

The general structure of the ORU includes the following definition:

{ [ORC] Order common
   OBR Observations Report ID
  { [NTE] } Notes and comments
  { [OBX] Observation/Result
    {[NTE]} Notes and comments
  }
}

But most commonly when this is translated into XML, instead of the OBX being contained within the preceding OBR as would be expected, it follows it. So you wind up with this:

OBR
OBX
OBX

Instead of this:

OBR
  OBX
  OBX

Just like in XHTML where you wind up with this:

H1
P
P
H2
P
P

Instead of:

H1
 P
 P
 H2
  P
  P

To process either of these in XSLT can be very challenging, because often you want to be able to relate the processing of each OBX (or P) to its OBR (or H#) in the hierarchy.

So how do you process this sort of XML using XSLT?  And how can you make the processing efficient?

There are several different tricks you can use:
The first trick is a little dicey sometimes but can be pretty efficient:  Use a two-pass transform where the first pass creates the appropriate structure and the second pass can do the real work.  It makes use of the xsl:text element with the disable-output-escaping attribute set to yes.  The basic details of this trick are:

1. For the first heading, you generate some sort of section XML tag, inside the xsl:text element
2. For every heading thereafter, you close the previous section tag, and open a new one using the same mechanims.
3. At the end of processing, you close the last open section tag. 

This works just fine for the OBR/OBX example, because there is one level of nesting.  It doesn't work very well for the H1/H2/P example because of multiple nesting levels.  Two pass processing is OK, but I like to keep it inside one XSLT.  There is a way to do that using the EXSLT node-set() function:

1. Create a variable containing the XML generated during the first pass.
2. Convert it to a node-set using the EXSLT node-set() function.
3. Apply templates to a selection from that node-set.

The skeleton below shows how you would do this:

‹xsl:variable name='pass1xml'› .. Generate the first pass XML ... ‹/xsl:variable›
‹xsl:apply-templates select='exslt:node-set($pass1xml)'/›

This will work with SOME XSLT processors, but not others.  The way that disable-output-escaping often works is by inserting an XML processing instruction in the XML output that will write the appropriate text when it is output to a file or stream.  This processing instruction is understood internally by the XSLT processor as a special processing instruction, and the XML output is not usually reparsed. So, normally you need a two-stage pipeline instead of being able to handle it all in one stage.  [Note: This technique isstill very useful for creating a two stage processing pipeline inside one XSLT when the disable-output-escaping feature is not used, and almost all XSLT processors support the node-set() function.]

Another way to deal with this problem is by using the sibling axes in XSLT.  This is more complex, and requires a good deal more explanation of how it works.  I'll save that discussion for next week.

Another XSLT Trick

Today I had to process an XML document that was made up of several sequences of elements that needed to be grouped together in an outer element.  This is a pretty common task for me when converting between different formats.  Usually I use EXSLT sets capability to deal with list intersections but this time the XSLT parser I deployed to didn't support it, and I didn't want to have to test another implementation.

So, looking at this XML:

‹items›
‹item›Outer Item 1‹/item› 
‹item›Inner Item 1.1‹/item›
‹item›Inner Item 1.2‹/item› 
‹item›Inner Item 1.3‹/item›
‹item›Outer Item 2‹/item› 
‹item›Inner Item 2.1‹/item›
‹item›Inner Item 2.2‹/item›
‹item›Outer Item 3‹/item› 
‹item›Outer Item 4‹/item›
‹item›Inner Item 4.1‹/item›
‹item›Outer Item 5‹/item›
‹/items›

Assume you want to produce this:

‹items›
 ‹outer›‹title›Outer Item 1‹/title› 
  ‹item›Inner Item 1.1‹/item›
  ‹item›Inner Item 1.2‹/item› 
  ‹item›Inner Item 1.3‹/item›
 ‹/outer›
 ‹outer›‹title›Outer Item 2‹/title›
  ‹item›Inner Item 2.1‹/item›
  ‹item›Inner Item 2.2‹/item›
 ‹/outer›
 ‹outer›‹title›Outer Item 3‹/title›‹/outer›
 ‹outer›‹title›Outer Item 4‹/title›
  ‹item›Inner Item 4.1‹/item›
 ‹/outer›
 ‹outer›‹title›Outer Item 5‹/title›
 ‹/outer›
‹/items›

1. So the outer items wrapper is the same.  Inside it you have more work to do.
2. The first step is to process all item children of items that contain the text "Outer Item" (or whatever other matching criteria signals the start of a new list. 
3. The next step is to find the end of the list of inner components.  That's simply the next "Outer Item" that follows this one in sequence.
4. Now, the trick.  You process each following sibling of the Outer Item that precedes the end point.

Now for the XSLT that does the work.

‹xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0"›
    ‹xsl:template match="items"›
        ‹xsl:copy›‹!-- 1 --›
            ‹xsl:apply-templates 
              select="item[contains(.,'Outer')]"/›‹!-- 2 --›
        ‹/xsl:copy›
    ‹/xsl:template›
    ‹xsl:template match="item"›
        ‹outer›‹title›‹xsl:value-of select="."/›‹/title›
            ‹xsl:variable name="endPoint" 
                select="following-sibling::item[contains(.,'Outer')][1]"
                /›‹!-- 3 --›
            ‹xsl:for-each 
                select="following-sibling::item[
                  . = $endPoint/preceding-sibling::item
                ]"›‹!-- 4 --›
                ‹inner›‹xsl:value-of select="."/›‹/inner›
            ‹/xsl:for-each›
        ‹/outer›
    ‹/xsl:template›
‹/xsl:stylesheet›

Now, if you think about what this is doing, it doesn't seem to be the MOST efficient way to process because you are creating two lists using the preceding-sibling and following-sibling axes in XPath, and then intersecting them.  But:
1) these lists are likely to be delayed in their complete evaluation until needed, and
2) A smart XSLT processor can and SHOULD recognize this idiom and use a more efficient evaluation

It's a handy thing to know how to do when you have to process lists of stuff that doen't use list-type XML markup to indicate list boundaries, and you don't have access to Java or EXSLT extensions to support procedural programming.

Now, can you guess what I was doing?  It's related to a previous post on some other blog...