Testing Co-occurrence constraints in FHIRPath

Recently I wrote about one of the limitations of FHIRPath.  The relevant text is below:

group.population	consisting of a set of population elements, one for each criteria defined in each group.	%def.group.population.all(   $this.code ~     (%resource.group.population)[$index].code )
group.stratifier	In addition, each group will contain stratifiers with a value stratum for each value defined by the stratifier criteria, for each criteria defined in the measure.	%def.group.stratifier.all(   $this.code ~      (%resource.group.stratifier)[$index].code ) and %def.group.stratifier.component.all(   $this.code ~      (%resource.group.stratifier.stratum        .component)[$index].code )

What was really challenging in this was a limitation of FHIRPath.  FHIRPath provides the $this and $index variables to define the current position within the evaluated context.  It works very much like "." does within XPath expressions.  Unfortunately, FHIR has no way to assign $this or $index to a variable so that it can be used with the same values in subexpressions contained inside the initial expression.

These two sets of constraints say both MORE and LESS than I actually want to say.  What they say is for each Population in %def.group.population, the code associated with population in the Measure has to match the corresponding population in the MeasureReport by numeric index.  That's a bit of BS.  Index has no semantics in the slice, code is what supplies the semantics, but there's no way to compare features of two lists that have matching semantics.

To put it simply, I'm trying to write a test that says the children of every 2 node in the structure to the left below needs to match up to with the children of the 2 node on the on the right, and the same for every other numbered node (code) that I test, without regard to the order of the codes. It's a simple graph isomorphism problem with trees (as opposed to a hard one with connected subgraphs).

Given: %def.group represents groups sliced by code in Measure, and %resource.group represents groups sliced by code in MeasureReport.group, how would you match up other attribute values in def.group A to concepts in report.group B where A.code ~ B.code (these kind of constraints are called co-occurence constraints).

Take %def.group as the source of codes to work from.  The following simply won't work:

%def.group.all(
    $this.somethingElse = 
        %resource.group.select(code ~ $this.code).somethingElse
) 

The reason is that select() takes expression as an argument, meaning that inside the where(), $this means something different than it means in %def.group.all().  To resolve this, what I'd need would be some sort of value assignment:

%def.group.all(
   let that = $this
   $this.somethingElse = 
      %resource.group.select(code ~ $that.code).somethingElse 
)

There are other ways to achieve a similar effect, for example, introducing a syntax that escapes to the outer level.  If $this means the context of this expression, $$this could mean the context of the outer expression, and $$$this could go the next level up, and I would write:

%def.group.all(
   $this.somethingElse = 
      %resource.group.select(code ~ $$this.code).somethingElse 
)

However, I'm going to lose track of the difference in levels quite quickly, so I'd rather give these things a name.  I've not found any other way to accomplish this sort of test in FHIRPath.

The point of testing in this way is that I want to ensure that the graph on the right matches the graph on the left, even if it happens to have MORE stuff than is required by the graph on the left.  Because different local jurisdictions may have MORE requirements than the base requirements of a Measure needed for broader national reporting.

Validating Constraints imposed by Definition Resources

One of my outstanding action items related to SANER is to propose a mechanism by which constraints imposed by definition resources can be applied to a FHIR Resource for validation.

A definition resource is one which defines content allowed in a resource referencing that definition.

For example, for MeasureReport, the definition is Measure, for QuestionnaireResponse, the definition is Questionnaire, and for CarePlan, the definition is CarePlanDefinition.

The challenge here is that the invariants imposed by a definition resource require access to that resource in order to be implemented, and they aren't stated in the specifications either on the resource, or on the definition resource in a machine readable format.

Since the definition resource is what actually applies them, and creators of the definition resource are who's basically creating the rules, what is needed here is a different kind of invariant statement.  The invariant is stated in the definition resource, but applies to the defined resource.

Typically invariants are written in the context of the resource to which they apply. However, the invariants implied by the use of a definition need to be written in the context of the defined resource, and the context of that resource's definition need to be available.

Let's have a look at Measure and MeasureReport (this is my first take):

Field	Description	Constraint with respect to %def = resolve(MeasureReport.measure)
subject	The subject is compatible with %def.subjectCodeableConcept	not(%def.subjectCodeableConcept.exists()) or    subject.resolve().exists(     ofType(       %def.subjectCodeableConcept.where(         system =          'http://hl7.org/fhir/resource-types'       ).code     )   )
subject	The subject is compatible with %def.subjectReference.type	not(%def.subjectReference) or  resolve(%def.subjectReference).exists(   (type = 'person' and      %resource.subject.where(        ofType('Patient') or ofType('Practitioner') or ofType('RelatedPerson')      )   ) or   (type = 'animal' and       %resource.subject.where(ofType('Patient'))   ) or   (type = 'practitioner' and       %resource.subject.where(ofType('Practitioner')   ) or   (type = 'device' and       %resource.subject.where(ofType('Device'))   )   // Don't need to worry about medication or substance, as Measure doesn't allow these )
period	Period is in the effectivePeriod for the measure.	not(%def.effectivePeriod.exists()) or (   period.start >= %def.effectivePeriod.start and   period.end <= %def.effectivePeriod.end )
improvementNotation	improvementNotation is compatible with %def.improvementNotation	not(%def.improvementNotation) or %def.improvementNotation ~ improvementNotation
group	A MeasureReport will contain one group of data for each group specified in the corresponding Measure	%def.group.all(   $this.code ~ %resource.group[$index].code )
group.population	consisting of a set of population elements, one for each criteria defined in each group.	%def.group.population.all(   $this.code ~     (%resource.group.population)[$index].code )
group.stratifier	In addition, each group will contain stratifiers with a value stratum for each value defined by the stratifier criteria, for each criteria defined in the measure.	%def.group.stratifier.all(   $this.code ~      (%resource.group.stratifier)[$index].code ) and %def.group.stratifier.component.all(   $this.code ~      (%resource.group.stratifier.stratum        .component)[$index].code )

What was really challenging in this was a limitation of FHIRPath.  FHIRPath provides the $this and $index variables to define the current position within the evaluated context.  It works very much like "." does within XPath expressions.  Unfortunately, FHIR has no way to assign $this or $index to a variable so that it can be used with the same values in subexpressions contained inside the initial expression.

Fortunately, [index: Integer] is defined as an operation, which means we can use [$index] to select the $index^th item of a collection, but figuring that out was a royal pain.

What seems to be needed in FHIRPath is a way to define a variable, e.g.: let that = $this (which would resolve to this), so that one could write some more complex correlation validations.

Eco-Theater Footers

You know what I'm talking about, those silly e-mail footers that read something like:

P Please consider the environment before printing this email  

I did some back of the napkin analysis on these silly buggers:

Data	Quantity		Notes	Source
Emails Per Day	40	Emails/Day		https://review42.com/how-many-emails-are-sent-per-day/
Length of Please Consider text	64	Bytes	Assumes UTF-8 encoded text
Working Days Per Year	200	Days/Year
Emails Per Year	8000	Emails/Year
Extra bytes sent per year	512000	Email bytes/Year
Transmission Cost	9.31E-07	Wh/Byte	Adjusted downwards for a decade of efficiency improvements, units converted to Wh	https://onlinelibrary.wiley.com/doi/full/10.1111/jiec.12630
Storage Cost	2.78E-11	Wh/Byte	Units converted to Wh	http://large.stanford.edu/courses/2018/ph240/jiang2/
Total Energy/Year	0.477	Wh/Year
Paper	0.032	Wh/Page	Assumes an average size of an e-mail is ~ 1 page, and that the energy costs for 20lb paper are larger enough than that of paper used in journal production to make up the difference.	https://scholarlykitchen.sspnet.org/2012/01/19/the-hidden-expense-of-energy-costs-print-is-costly-online-isnt-free/
Break-even point	14.9	Emails Not-Printed
	0.2%
Grams of Carbon/Wh	0.2252252	g/Wh
Carbon Weight	0.107399	g

It turns out, that the carbon cost in sending these works about to about 0.1g / year for a person sending 40 emails a day for a working year of 200 days (8000 e-mails).  Oh, and if you use Outlook, and it sends multi-part mime because you prettify your e-mails, then double it.  In order for this to balance the energy cost (w/o the doubling), it has to prevent someone from printing about 15 emails.  Considering that I've not printed a single e-mail for more than a decade, and almost all of my colleagues operate in a similar fashion, it seems unlikely that it would even have this efficacy.

It's stupid, it's not really saving the world, and it's just more stuff to not read.  So, as I say, eco-theater.

    Keith

When National Informatics Infrastructure Fails

We had an interesting discussion on The SANER Project call today.  One of the issues that came up was how to get the trigger codes from RCKMS (the Reportable Condition Knowledge Management System).

As a former employee of an EHR Vendor, I'm well aware of VSAC (the Value Set Authority Center) as a source value sets (I also played a role in writing the HIT Standards Committee recommendation for the creation of VSAC).  

As an Informaticist, I'm also well aware of LOINC Value Sets and SNOMED CT Value Sets for COVID that many have published.

As an Interoperability policy geek, I am also familiar with the ONC Interoperability Standards Advisory COVID-19 content.

As a highly educated expert, I CANNOT, for the life of me tell you which of these is the most authoritative content to use for COVID-19 reporting, either for what had been reported to NHSN, or presently for HHS Protect, or for eCase Reporting.  I do know enough to build a pretty damn good value set of my own for a system that I'm responsible for developing, but that's NOT what I want.  Developers don't need more work, we need experts to do their work, and make it readily available for others, and not just to publish it, but also to Market it for their jurisdictions so that it can be found.

I can also tell you that there's simply FAR too much data available to developers to let this situation continue.

Here are some rules for organizations responsible for these value sets to consider:

1. Figure out where the developers who have build systems get their information, and publish to those sources where they can get more.
2. Consider cognitive load on developers. Don't give them hard to remember web site names with 80 character URLs to get to the data they need.  Register a reasonable and memorable domain name.  Make the information easy to find. Remember that not every developer works for a hospital or an EHR vendor.
3. Remember that data distribution and publication needs different governance for access that data creation. If you have a system that supports creation, don't force a developer to get credentials for that system just to access data, just give them a URL.  Sure, give them a way to share an e-mail address for updates, but don't force them to create ONE more login just to learn what they heck they need to do.
4. Make sure that the distribution system has a way of reporting that data (especially value sets) using standards. Sure, CSV files are good, but come on, we're trying to work on modernization and APIs.  Put the same effort into your distribution mechanisms with respect to APIs and publication that you expect developers to put into the systems that will use the standards you are promoting.
   <rant>There's absolutely NO EXCUSE for a system designed to support FHIR to have an easily accessible CSV distribution mechanism for value sets, BUT not to have a FHIR ValueSet distribution mechanism.</rant>
5. Curation of a value set is a responsibility that steps up as demand increases. If you are responsible for curating a value set and the demand for updating it steps up due to an emergency.  Quarterly may be fine for things that change annually, but when the situation is changing week by week, changes are also needed week by week.  Yeah, I know, funding and all that ... figure it out, that's part of what being responsible means.

Of course all my International friends are simply going to tell me that the fundamental issue is that the US does not have a coordinated national infrastructure.  I can't argue with that.  Every agency is a plural of services, centers or institutes, I just wish ...

A model for SANER Measure Automation

By Birger Eriksson CC BY-SA 3.0

One of the principles of FHIR is to be able to provide the essential data about the standard, the same data that is necessary to support automatic generation of implementations.  This also shows up in definitional resources, resources that describe the essential structure of other resources, such as Measure, Questionnaire, and CarePlan, in which these resources can be given a computer friendly name to distinguish them from others of the same type.

While it likely won't be the subject of anything appearing in The SANER Implementation Guide, I was considering (more like, the idea popped into my head when I was writing) how one would define a DOM-like binding for a Measure in a language like JavaScript.

In SANER Measures, the measure has a name, a period, a subject, a reporter, and groups of populations.  Each group has a code, and so does each population.  I won't get into strata, but the same princples would apply to these.

If you've got a measure with the Name X, Groups Y1, Y2, and Y3, and populations within those groups of X1a, X1b, X1c, ... X3a... X3c.

Then a MeasureReport that is defined by that measure has Type X (with supertype MeasureReport).  MeasureReport has some common features, such as as subject, reporter, period.  So, you might reference X.subject, X.reporter, et cetera.  But group is kind of an intrisic collection associated with a Measure, so you might reference data in groups as X.Y1, X.Y2, X.Y3.  To get to the score, reference X.Y1.score (rather than the more laborious measureScore).

Within a group, X.Y1.X1a would reference the population "named" X1a, where X1a is simply the code value.

For the CDC Measures we had been working from, that would give us something like 

CDCPatientImpactAndHospitalCapacity as the type, and for an instance X of that type X.Beds would refer to the Beds group, and X.Beds.numTotBeds as the first population of that group.

It's a pretty simple translation between these from and to the components of the Measure.

This is a pretty helpful observation, in that it relates definitional resources in FHIR to user defined types (or structures or classes) in programming environments, and variables of those types to instances of the resource defined by the definitional resource.

Principles for the SANER Project

Principles by Nick Youngson CC BY-SA 3.0 Alpha Stock Images

One of the things I try to understand for every project I work on are what are the fundamental principles of the project.  Often I even attempt to get these principles formally adopted by a project (when I can remember this important step).  That's because these principles facilitate and simplify decision making.  When the principles are clearly stated and agreed to, then when a contentious decision comes up, it can be evaluated according to these already agreed upon principles, and the decision making can move forward on that basis, rather than the sometimes deeply emotional arguments that can result. 

Often times, one can understand the principles of a project by looking at definitions of the project scope or approach (i.e., methods).  We never actually established the principles of The SANER Project, but that never stopped me from thinking about them and at least developing a mental model of what I think they are.  So, here they are, enumerated and explained in modest detail:

1. The work needs to be done quickly.  As a result, it must
1. Work with existing systems where possible, minimizing new development.
2. Utilize off the shelf components and FHIR Servers where feasible.
3. Use existing FHIR capabilities and operations where feasible.
4. Minimize efforts needed to integrate external systems.
2. The work needs to be aligned with current US national and international initiatives, thus it will use FHIR Release 4.
3. While the end-goal is automation of the supporting solution, there must be intermediate features which still enable value for users who are not able to provide more advanced capabilities.  This is really just another way of saying "minimize new development", but more clearly describes that partial implementation must both be possible, and add value.
4. Finally, in the extreme situation impacting facilities who might use this solution, a great deal of attention must be paid to verification and validation of the specification.  There is absolutely no room for implementers to break a system that is otherwise functional and vital to operations.

Decision points in The SANER Project have been constantly evaluated against this mental model as we tracked the work.

Who do you trust?

If you've been involved in software for any period of time, you've heard the terms validation and verification.

If you've been involved in standards for any period of time, you've also heard the terms certified and accredited.

By non-pedants, these two pairs of terms are often slightly confusing, because one term of each pair is often used incorrectly to mean the other. 

The differences are subtle:

• Validation is the process of ensuring that a specification meets the customer requirements.
• Verification is the process of ensuring that a product meets the specifications.

• Accreditation is a third party process of ensuring that an organization has the capacity (skill set and processes) to create validated and verified products.
• Certification is a third party process of ensuring that a product meets a specification, and may include verification that a particular set of skills and processes were used during development.

In the end, Validation and Verification are everyday engineering processes in an organization. Accreditation is something that occurs periodically, and which ensures that organizations are following those everyday processes.  And Certification happens sometime around product delivery to ensure that a product is verified.

The word accreditation includes the same stem as credible, and is about establishing trust in an organization.

In software development, these words are all meaningful when you think about the acquisition of third party software products for use in your own software development.  When you acquire that software (either through purchase, or via open source), you need to validate that it meets your needs, and may also need to verify it.  It depends on your organizations process requirements.

In my past, one of the ways to acquire new software included a step where we basically "accredited" an organization, essentially convinced ourselves (or our leaders) that the organization that developed the materials had good processes and followed them, which mean that we had less effort to go through when we verified that the software met our needs.  You basically do the same when you look at a brand to make a decision (and yes, Apache is a brand)

This can be very helpful, because a full-blown verification of something like an XSLT processor is a pretty extensive (and expensive) task.

As I look at the recent discussions about the exchange of Situation Awareness data in the media recently, what we face is this challenge of trust.  The question of who you trust is important.  CDC has a trusted brand, HSS Protect is brand new, it has yet to establish such a brand, and the consequent trust.

The last thing any Software Engineering manager in the non-governmental world of software engineering is going to allow in the development of a system is the replacement of a trusted branded system with a novel system in the middle of a product release.  It's just NOT the way we've learned to do things successfully.  Yet, there have been times when it has been necessary (I'm not saying it is at this time).  What has to happen then, is that the NEW system has to be validated and verified.  It has to be thoroughly tested, and that also means that there's a lot about how it works that has to be made transparent to people who are going to rely on that system. That's how trust works.

The challenge of validation is a real one.  I've worked on software projects where we built a fully verified system, but failed to validate one of the requirements (that it was sufficient to be as accurate as a human), and that caused the product to fail.  Humans can explain their rationale (right or wrong), but the product we build (and verified was as accurate as a human), could NOT explain it's rationale, and so failed to meet a fundamental requirement of its users, which was to be something that could be trusted.  And since trust failed, the product failed.  

I don't know if HHS Protect is going to be successful in the long run. HHS Protect's  primary requirement is to be a trusted system reporting the data about what's going on with COVID. It won't succeed if it cannot be trusted.  That's not a statement of opinion, or about politics.  That's a statement of experience.