Automating V2toFHIR conversion and Testing

Yesterday morning morning I talked about automating my V2 to FHIR testing.  Now I'm going to talk about automating conversions.  I had to write a segment converter (more like four) before I finally found the right conversion pattern to base the automation on.

It sort of looks like this:

public class PIDParser extends AbstractSegmentParser {
    @Override
    public void parse(Segment pid) throws HL7Exception {
        if (isEmpty(pid)) {
            return;
        }

        Patient patient = this.createResource(Patient.class);
        // Patient identifier
        addField(pid, 2, Identifier.class, patient::addIdentifier);
        // Patient identifier list
        addFields(pid, 3, Identifier.class, patient::addIdentifier);
        // Patient alternate identifier
        addField(pid, 4, Identifier.class, patient::addIdentifier);
        // Patient name
        addFields(pid, 5, HumanName.class, patient::addName);
        // Mother's maiden name
        addFields(pid, 6, patient, HumanName.class, this::addMothersMaidenName);
        // Patient Birth Time
        addField(pid, 7, patient, DateTimeType.class, this::addBirthDateTime);
    
        /* ... and so on for another 30 fields */
    }

Let met help you interpret that.  The addField() method takes a segment and field number, gets the first occurrence of the field.  If it exists, it converts it to the type specified in the third argument (e.g., Identifier.class, HumanName.class, DateTimeType.class).  If the conversion is successful, it then sends the result to the Consumer<Type> or BiConsumer<Patient, Type> lambda which adds it to the resource.

The BiConsumer lambdas are needed to deal with special cases, such as adding Extensions to a resource as in the case for mother's maiden name, or birthDate when the converted result is more precise than to the day.

I want to make a parser annotation driven, so I've created a little annotation that serves both as documentation on the parser, as well as can drive the operation of the parse function.  So I came up with an annotation that is used like this on the PIDParser:

@ComesFrom(path="Patient.identifier", source = "PID-2")
@ComesFrom(path="Patient.identifier", source = "PID-3")
@ComesFrom(path="Patient.identifier", source = "PID-4")
@ComesFrom(path="Patient.name", source = "PID-5")
@ComesFrom(path=
    "Patient.extension('http://hl7.org/fhir/StructureDefinition/patient-mothersMaidenName')",
    source = "PID-6")
@ComesFrom(path="Patient.birthDate", source = "PID-7")
@ComesFrom(path=
    "Patient.extension('http://hl7.org/fhir/StructureDefinition/patient-birthTime')",
    source = "PID-7")

The path part of this annotation is written using the Simplified FHIRPath grammar defined in FHIR.  These paths can easily be traversed and set using the getNamedProperty() and setNamedProperty() of Element types.  A simple parse and traverse gets to the property, from which you can determine the type, and then call the appropriate converter on the field, which you can extract from the message by either parsing or using an HAPI V2 Terser on the source property.

I anticipate that ComesFrom will introduce new attributes as I determine patterns for them, for example to set a field to a fixed value, or translate a code from one code system to another before setting it via the destination consumer.  Another attribute may specific the name of a method in the parser to use as a lambda (accessed as a method via reflection on the parser itself).

ComesFrom annotations could obviously be written in CSV format, something like this used by the V2-to-FHIR project team to create the mapping spreadsheets.  In fact, what I'm doing right now for parsers is translating those spreadsheets by hand.  That will change VERY soon.

So, if I have ComesFrom annotations on my parsers, and a bunch of test messages, the next obvious (or maybe not-so-obvious) step is to parse the messages, and create the (almost) appropriate annotations to match those messages.  I did this later that day for the 60+ test messages in my initial test collection (which I expect to grow to by at least one if not two orders of magnitude).  Instead of the 100 assertions I started with that morning, I now have over 2775 that are almost complete.

Here's a small snippet of one:

MSH|^~\&|TEST|MOCK|DEST|DAPP|20240618083101-0400||RSP^K11^RSP_K11|RESULT-01|P|\
  2.5.1|||ER|AL|||||Z32^CDCPHINVS
  @MessageHeader.event.code.exists($this = "K11")
  @MessageHeader.meta.tag.code.exists($this = "RSP")
  @MessageHeader.definition.exists($this.endsWith("RSP_K11"))
  // ComesFrom(path="MessageHeader.source.sender", source = { "MSH-22", "MSH-4", "MSH-3"})
  @MessageHeader.source.sender.exists( \
    /* MSH-22 = null */ \
    /* MSH-4 = MOCK */ \
    /* MSH-3 = TEST */ \
  )
  // ComesFrom(path="MessageHeader.destination.receiver", source = { "MSH-23", "MSH-6", "MSH-5"})
  @MessageHeader.destination.receiver.exists( \
    /* MSH-23 = null */ \
    /* MSH-6 = DAPP */ \
    /* MSH-5 = DEST */ \
  ) 
  // ComesFrom(path="MessageHeader.source.sender.name", source = "MSH-4")
  @MessageHeader.source.sender.name.exists( \
    /* MSH-4 = MOCK */ \
  ) 
  // ComesFrom(path="MessageHeader.source.sender.endpoint", source = "MSH-3") 
  @MessageHeader.source.sender.endpoint.exists( \
    /* MSH-3 = TEST */ \
  )
  // ComesFrom(path="MessageHeader.destination.reciever.name", source = "MSH-6")
  @MessageHeader.destination.reciever.name.exists( \
    /* MSH-6 = DAPP */ \
  )
  // ComesFrom(path="MessageHeader.destination.reciever.endpoint", source = "MSH-5") 
  @MessageHeader.destination.reciever.endpoint.exists( \
    /* MSH-5 = DEST */ \
  )
  // ComesFrom(path="Bundle.timestamp", source = "MSH-7")
  @Bundle.timestamp.exists( 
    /* MSH-7 = 20240618083101-0400 */ 
  )

The stuff in comments (e.g., MSH-7 = 20240618083101-0400) lets the user know what message content is relevant for this assertion test.  Right now they (they = I) would have to manually change those values.  The corrected expression looks like this:

  @Bundle.timestamp.exists( 
    $this = "2024-06-18T08:31:01-04:00" 
  )

I'll get around to fixing that soon.  I have to be careful not to let the converter do the work of writing the test cases, because then they'll just be self-fulfilling prophecies instead of real tests.