Why computers should manage combinatorial explosion in test cases

 In A Test Case Generator for FHIR and SUSHI (and SANER) I wrote about how I'm working on generating test cases, and a little language for test case generation.

Here's one thing (among many) that I encountered.  Data for individual test cases should be dealt with independent from other test cases, so that tests don't interfere with each other.  That's why unit tests have setup and teardown.  But the test case generator is creating data for test cases that will be stored in a FHIR Server, and the FHIR Server cannot necessarily do set up and tear down between each test.

So, I cannot use the same patient for each test case, but rather, each test case must refer to patients created specifically for it, so that all the test data can be loaded into a FHIR Server for use at the same time.

It's a lovely little nuance about integration testing that you don't really have to deal with for unit testing.  I've accounted for it in data production for test cases, but it's made for some pretty interesting challenges, as I now have about four phases for parsing generating the data.

Parsing of the test model happens in the first phase. I have this working.

The generation step has at least three phases:

1. Generating the essential resources and their variants.  I should probably talk about test cases and variants, and so will in more detail below.  This step has to be done in a particular order, because encounter cannot talk about patient or location until these two are defined (on purpose, so that I make the test case author deal with ordering, and I don't have to deal with forward references to stuff that doesn't exist).  I have this working.
2. Generating Sushi code for each variant needed.  I have this working.
3. Packaging a set of resources into a bundle for each variant of the test case.  I'm working on this now.

Test Cases and Variants

For my purposes, a test case is a package of data needed to test a measure groups: A bundle of resources.  I have a test case with an encounter, patient and location with the following linkages:

Patient patient (stands alone)

Location location (stands alone)

Encounter.patient  references Patient/patient

Encounter.location references Location/location

I have multiple measure groups, and I want to test the different facets of inclusion/exclusion criteria for the group.  So, an encounter in a test case might be "in-progress" or "entered-in-error".  These are two variations for the test case for one of the encounter measure groups.  If the test case is TestCase1, these variations will be labeled something like TestCase1a and TestCase1b, to distinguish them.  The bundles will be different for each variation.

Also, while each bundle might contain multiple linked resources (e.g., patient, location, and encounter),

the patient, location and encounter in each bundle must be distinct from the patient, location and encounter resources in other variants.  

So, with three possible variations on location, two in encounter, and one on patient, we'll see six (3 x 2 x 1) different cases for Encounter in the bundles.

Bundle1: TestCase1a

Encounter11.patient references Patient/patient11

Encounter11.location references Location/location/11

Bundle2: TestCase1b

Encounter12.patient references Patient/patient12

Encounter12.location references Location/location12

Bundle3: TestCase1c

Encounter13.patient references Patient/patient13

Encounter13.location references Location/location13

Bundle4: TestCase1d

Encounter21.patient references Patient/patient21

Encounter21.location references Location/location21

Bundle5: TestCase1e

Encounter22.patient references Patient/patient22

Encounter22.location references Location/location22

Bundle6: TestCase1f

Encounter23.patient references Patient/patient23

Encounter23.location references Location/location23