ML and AI in HealthIT ... things to think about

A very long time ago (at the beginning of my career in HealthIT), I got do do some really cool work on the front end and processing infrastructure for a set of machine learning and linguistic services that would automatically extract problems, medications, allergies and procedures and for problems, even code the diseases into a subset of SNOMED CT.  This was before most people had even heard about SNOMED CT, so some significant effort there.  We also did some work in the ICD-9-CM coding space as well, based on a software product that the company I worked for had purchased that was essentially the life's work of a physican / informaticist.

The product worked remarkably well from a technical perspective, and had a pretty decent precision/recall curve.  In fact, as configured, the system was shown to do as well as professional ICD-9-CM coders.

The biggest challenge it had was basically two-fold:

1. It needed to be incorporate expert feedback to refine future results (we simply hadn't had the time to develop that feature).
2. The original product couldn't explain how it got to a particular result, although subsequent ones could do a bit better.

To simplify, it couldn't argue for itself, or accept any corrections.

ML and AI are often "black boxes".  Most of what people are talking about with regard to AI today are implementations of some form of neural network.  Can anyone really explain what the weights and connections in a neural net mean?  This is a hard AI problem.  Machine learning algorithms have there own set of "hidden parameters" that drive their outputs, but that cannot always be easily explained.

Yet, we're expect to trust these things.  And they applied to hard problems that humans only solve correctly 90% of the time, and do slightly better.  How do you develop trust for something that's wrong for 1 out of 20 cases?  And yet we can trust a human because they can explain their reasoning, even when the result of it turned out wrong.

Interestingly enough, even when computers do as well as humans, the two together often do better than individually b/c the humans may understand nuance that the computer misses and vice versa.

If you are trying to implement AI (or ML), consider: 

1. How do you work in user feedback about the goodness of the proposed solutions, 
2. How will you explain to the user why this result is good.

Reporting on FHIR Structures

In yesterday's post, I mentioned that I had put together a report on FHIR resource use of code, Coding and CodeableConcept data types.  The stylesheet I used is provided below, and it was run over the profile-resources.xml file found in this download from the FHIR R4 website.

This is one of the truly valuable pieces of FHIR, that the specification itself is machine readable, which makes it possible to perform a variety of analyses and code generation tasks.

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    xmlns:f="http://hl7.org/fhir"
    exclude-result-prefixes="xs"
    version="1.0">
    <xsl:output method="text"/>
    <xsl:template match="/">
        <xsl:text>Name,Type,Binding,Cardinality&#xA;</xsl:text>
        <xsl:apply-templates select="//f:snapshot/f:element"></xsl:apply-templates>
    </xsl:template>
    <xsl:template match="f:snapshot/f:element">
        <xsl:if test="f:binding">
            <xsl:value-of select="@id"/>
            <xsl:text>,</xsl:text>
            <xsl:apply-templates select="." mode='type'/>
            <xsl:text>,</xsl:text>
            <xsl:value-of select="f:binding/f:strength/@value"/>
            <xsl:text>,</xsl:text>
            <xsl:value-of select="f:min/@value"/>
            <xsl:text>..</xsl:text>
            <xsl:value-of select="f:max/@value"/>
            <xsl:text>&#xA;</xsl:text>
        </xsl:if>
    </xsl:template>
    <xsl:template match="f:element" mode='type'>
        <xsl:choose>
            <xsl:when test='f:type/f:code[@value = "code"]'>code</xsl:when>
            <xsl:when test='f:type/f:code[@value = "Coding"]'>Coding</xsl:when>
            <xsl:when test='f:type/f:code[@value = "CodeableConcept"]'>CodeableConcept</xsl:when>
            <xsl:when test='f:type/f:code[@value = "string"]'>string</xsl:when>
            <xsl:otherwise>
                <xsl:value-of select="f:type/f:code/@value"/>
            </xsl:otherwise>
        </xsl:choose>
    </xsl:template>
        
</xsl:stylesheet>

FHIR Terminology Datatype Usage Patterns

For the V2toFHIR project, I recently took up a task to write up my preferred approach for recording the original V2 code in FHIR Resources when mapping from a V2 message to FHIR.  To do that, I wanted to make a list of all the possible things that one might need to consider, so I ran a quick report of the variations in how code, Coding, and CodeableConcept are used with terminology in FHIR with respect to Binding Strength and Cardinality.

The report can be found at the bottom of the post.

Some interesting points:

code ~ required

The code datatype always uses a required vocabulary EXCEPT in one case in which it is preferred in Resource.language (which of course propagates to all resources in R4).  Now, why Resource.language says preferred instead of extensible is beyond me.  Who couldn't work with it as extensible?

max(cardinality(code)) is almost always 1

Instructive cases where there can be more than one code: AllergyIntolerance.category, Questionnaire.subjectType, CoverageEligibilityRequest.purpose and CoverageEligibilityResponse.purpose, anything dealing with daysOfWeek, and definition or conformance resources.  

The complete list where more than one can be used is AllergyIntolerance.category, CapabilityStatement.patchFormat, CapabilityStatement.rest.resource.referencePolicy, Endpoint.payloadMimeType, HealthcareService.availableTime.daysOfWeek, ImplementationGuide.definition.resource.fhirVersion

Location.hoursOfOperation.daysOfWeek, ObservationDefinition.permittedDataType, OperationDefinition.resource, PractitionerRole.availableTime.daysOfWeek, Questionnaire.subjectType, SearchParameter.comparator, SearchParameter.modifier, SearchParameter.target, StructureMap.group.rule.target.listMode, CapabilityStatement.format, 

CodeSystem.filter.operator, CoverageEligibilityRequest.purpose, CoverageEligibilityResponse.purpose, ImplementationGuide.fhirVersion, SearchParameter.base

Coding never uses required

The Coding data type is never bound with a required vocabulary, and rarely with a preferred one.  

Coding is not often used

In general, Coding is used 42 times, while CodableConcept is used 614 times. Hmm, do we really need Coding?  There are a few cases where I think the answer might be yes, but generally, I'd consider that most cases could probably survive quite well with just CodeableConcept.  In fact, in HL7 V2, CWE replaced ID/IS in the version 2.7 time frame throughout much of the specification.  There seem to be a small number of cases where Coding might be essential (QuestionnaireResponse.item.answer.value[x] could be one of these).

CodeableConcept has some interesting required cases

AdverseEvent.outcome, AdverseEvent.severity, AllergyIntolerance.clinicalStatus, AllergyIntolerance.verificationStatus, Condition.clinicalStatus, Condition.verificationStatus

These fields look mostly categorical and generally I have no complaints.  Seeing AdverseEvent.severity as a CodeableConcept makes me wonder though if AllergyIntolerance.criticality should also be CodeableConcept, or if the former should be code.

Measure.improvementNotation, MeasureReport.improvementNotation

The various improvementNotation attributes (on Measure and MeasureReport) could arguably be code.  I'm not sure what the use case is for CodableConcept in these.

EffectEvidenceSynthesis.effectEstimate.unitOfMeasure, 

ResearchElementDefinition.characteristic.unitOfMeasure, 

RiskEvidenceSynthesis.riskEstimate.unitOfMeasure

The unitOfMeasure fields above, when used alongside a decimal value argue that this information could have been modeled using SimpleQuantity to keep FHIR resources aligned around data types.

SupplyDelivery.type, InsurancePlan.plan.specificCost.benefit.cost.applicability, MolecularSequence.structureVariant.variantType

I don't have any feedback on these, just listing them for completeness.

Row Labels	Count of Name
code	319
preferred	1
0..1	1
Resource.language	1
required	318
0..*	15
AllergyIntolerance.category	1
CapabilityStatement.patchFormat	1
CapabilityStatement.rest.resource.referencePolicy	1
Endpoint.payloadMimeType	1
HealthcareService.availableTime.daysOfWeek	1
ImplementationGuide.definition.resource.fhirVersion	1
Location.hoursOfOperation.daysOfWeek	1
ObservationDefinition.permittedDataType	1
OperationDefinition.resource	1
PractitionerRole.availableTime.daysOfWeek	1
Questionnaire.subjectType	1
SearchParameter.comparator	1
SearchParameter.modifier	1
SearchParameter.target	1
StructureMap.group.rule.target.listMode	1
0..1	112
ActivityDefinition.intent	1
ActivityDefinition.kind	1
ActivityDefinition.priority	1
AllergyIntolerance.criticality	1
AllergyIntolerance.reaction.severity	1
AllergyIntolerance.type	1
Appointment.participant.required	1
AuditEvent.action	1
AuditEvent.agent.network.type	1
AuditEvent.outcome	1
BiologicallyDerivedProduct.productCategory	1
BiologicallyDerivedProduct.status	1
BiologicallyDerivedProduct.storage.scale	1
Bundle.entry.search.mode	1
CapabilityStatement.rest.resource.conditionalDelete	1
CapabilityStatement.rest.resource.conditionalRead	1
CapabilityStatement.rest.resource.versioning	1
CarePlan.activity.detail.kind	1
CareTeam.status	1
CatalogEntry.status	1
ClaimResponse.processNote.type	1
CodeSystem.hierarchyMeaning	1
Communication.priority	1
CommunicationRequest.priority	1
Composition.confidentiality	1
Composition.section.mode	1
Consent.provision.type	1
Contract.status	1
DetectedIssue.severity	1
Device.status	1
Device.udiCarrier.entryType	1
DeviceMetric.calibration.state	1
DeviceMetric.calibration.type	1
DeviceMetric.color	1
DeviceMetric.operationalStatus	1
DeviceRequest.priority	1
DeviceRequest.status	1
DocumentReference.docStatus	1
EffectEvidenceSynthesis.resultsByExposure.exposureState	1
Encounter.location.status	1
EnrollmentRequest.status	1
EnrollmentResponse.outcome	1
EnrollmentResponse.status	1
EvidenceVariable.characteristic.groupMeasure	1
EvidenceVariable.type	1
ExplanationOfBenefit.processNote.type	1
ImplementationGuide.license	1
InsurancePlan.status	1
Location.mode	1
Location.status	1
Medication.status	1
MedicationKnowledge.status	1
MedicationRequest.priority	1
MessageDefinition.category	1
MessageDefinition.responseRequired	1
MolecularSequence.referenceSeq.orientation	1
MolecularSequence.referenceSeq.strand	1
MolecularSequence.type	1
ObservationDefinition.qualifiedInterval.category	1
ObservationDefinition.qualifiedInterval.gender	1
OperationDefinition.parameter.searchType	1
OperationDefinition.parameter.type	1
Patient.contact.gender	1
Patient.gender	1
PaymentReconciliation.outcome	1
PaymentReconciliation.processNote.type	1
Person.gender	1
Person.link.assurance	1
PlanDefinition.action.cardinalityBehavior	1
PlanDefinition.action.groupingBehavior	1
PlanDefinition.action.precheckBehavior	1
PlanDefinition.action.priority	1
PlanDefinition.action.requiredBehavior	1
PlanDefinition.action.selectionBehavior	1
Practitioner.gender	1
Questionnaire.item.enableBehavior	1
RelatedPerson.gender	1
RequestGroup.action.cardinalityBehavior	1
RequestGroup.action.groupingBehavior	1
RequestGroup.action.precheckBehavior	1
RequestGroup.action.priority	1
RequestGroup.action.requiredBehavior	1
RequestGroup.action.selectionBehavior	1
RequestGroup.priority	1
ResearchElementDefinition.characteristic.participantEffectiveGroupMeasure	1
ResearchElementDefinition.characteristic.studyEffectiveGroupMeasure	1
ResearchElementDefinition.variableType	1
SearchParameter.xpathUsage	1
ServiceRequest.priority	1
Specimen.status	1
StructureDefinition.derivation	1
StructureDefinition.fhirVersion	1
StructureMap.group.rule.source.listMode	1
StructureMap.group.rule.target.contextType	1
StructureMap.group.rule.target.transform	1
Subscription.channel.payload	1
Substance.status	1
SupplyDelivery.status	1
SupplyRequest.priority	1
SupplyRequest.status	1
Task.priority	1
TerminologyCapabilities.codeSearch	1
TestScript.setup.action.assert.contentType	1
TestScript.setup.action.assert.direction	1
TestScript.setup.action.assert.operator	1
TestScript.setup.action.assert.requestMethod	1
TestScript.setup.action.assert.resource	1
TestScript.setup.action.assert.response	1
TestScript.setup.action.operation.accept	1
TestScript.setup.action.operation.contentType	1
TestScript.setup.action.operation.method	1
TestScript.setup.action.operation.resource	1
1..*	6
CapabilityStatement.format	1
CodeSystem.filter.operator	1
CoverageEligibilityRequest.purpose	1
CoverageEligibilityResponse.purpose	1
ImplementationGuide.fhirVersion	1
SearchParameter.base	1
1..1	185
Account.status	1
ActivityDefinition.participant.type	1
ActivityDefinition.status	1
AdverseEvent.actuality	1
Appointment.participant.status	1
Appointment.status	1
AppointmentResponse.participantStatus	1
Binary.contentType	1
Bundle.entry.request.method	1
Bundle.type	1
CapabilityStatement.document.mode	1
CapabilityStatement.fhirVersion	1
CapabilityStatement.kind	1
CapabilityStatement.messaging.supportedMessage.mode	1
CapabilityStatement.rest.interaction.code	1
CapabilityStatement.rest.mode	1
CapabilityStatement.rest.resource.interaction.code	1
CapabilityStatement.rest.resource.searchParam.type	1
CapabilityStatement.rest.resource.type	1
CapabilityStatement.status	1
CarePlan.activity.detail.status	1
CarePlan.intent	1
CarePlan.status	1
CatalogEntry.relatedEntry.relationtype	1
ChargeItem.status	1
ChargeItemDefinition.propertyGroup.priceComponent.type	1
ChargeItemDefinition.status	1
Claim.status	1
Claim.use	1
ClaimResponse.outcome	1
ClaimResponse.status	1
ClaimResponse.use	1
ClinicalImpression.status	1
CodeSystem.content	1
CodeSystem.property.type	1
CodeSystem.status	1
Communication.status	1
CommunicationRequest.status	1
CompartmentDefinition.code	1
CompartmentDefinition.resource.code	1
CompartmentDefinition.status	1
Composition.attester.mode	1
Composition.relatesTo.code	1
Composition.status	1
ConceptMap.group.element.target.equivalence	1
ConceptMap.group.unmapped.mode	1
ConceptMap.status	1
Consent.provision.data.meaning	1
Consent.status	1
Contract.contentDefinition.publicationStatus	1
Coverage.status	1
CoverageEligibilityRequest.status	1
CoverageEligibilityResponse.outcome	1
CoverageEligibilityResponse.status	1
DetectedIssue.status	1
Device.deviceName.type	1
DeviceDefinition.deviceName.type	1
DeviceMetric.category	1
DeviceRequest.intent	1
DeviceUseStatement.status	1
DiagnosticReport.status	1
DocumentManifest.status	1
DocumentReference.relatesTo.code	1
DocumentReference.status	1
EffectEvidenceSynthesis.status	1
Encounter.status	1
Encounter.statusHistory.status	1
Endpoint.status	1
EpisodeOfCare.status	1
EpisodeOfCare.statusHistory.status	1
EventDefinition.status	1
Evidence.status	1
EvidenceVariable.status	1
ExampleScenario.actor.type	1
ExampleScenario.instance.resourceType	1
ExampleScenario.status	1
ExplanationOfBenefit.outcome	1
ExplanationOfBenefit.status	1
ExplanationOfBenefit.use	1
FamilyMemberHistory.status	1
Flag.status	1
Goal.lifecycleStatus	1
GraphDefinition.link.target.compartment.code	1
GraphDefinition.link.target.compartment.rule	1
GraphDefinition.link.target.compartment.use	1
GraphDefinition.link.target.type	1
GraphDefinition.start	1
GraphDefinition.status	1
Group.type	1
GuidanceResponse.status	1
ImagingStudy.status	1
Immunization.status	1
ImmunizationEvaluation.status	1
ImplementationGuide.definition.page.generation	1
ImplementationGuide.definition.parameter.code	1
ImplementationGuide.global.type	1
ImplementationGuide.status	1
Invoice.lineItem.priceComponent.type	1
Invoice.status	1
Library.status	1
Linkage.item.type	1
List.mode	1
List.status	1
Measure.status	1
MeasureReport.status	1
MeasureReport.type	1
Media.status	1
MedicationAdministration.status	1
MedicationDispense.status	1
MedicationRequest.intent	1
MedicationRequest.status	1
MedicationStatement.status	1
MessageDefinition.focus.code	1
MessageDefinition.status	1
MessageHeader.response.code	1
MetadataResource.status	1
MolecularSequence.quality.type	1
MolecularSequence.repository.type	1
NamingSystem.kind	1
NamingSystem.status	1
NamingSystem.uniqueId.type	1
NutritionOrder.intent	1
NutritionOrder.status	1
Observation.status	1
OperationDefinition.kind	1
OperationDefinition.parameter.binding.strength	1
OperationDefinition.parameter.use	1
OperationDefinition.status	1
OperationOutcome.issue.code	1
OperationOutcome.issue.severity	1
Patient.link.type	1
PaymentNotice.status	1
PaymentReconciliation.status	1
PlanDefinition.action.condition.kind	1
PlanDefinition.action.participant.type	1
PlanDefinition.action.relatedAction.relationship	1
PlanDefinition.status	1
Procedure.status	1
Provenance.entity.role	1
Questionnaire.item.enableWhen.operator	1
Questionnaire.item.type	1
Questionnaire.status	1
QuestionnaireResponse.status	1
RequestGroup.action.condition.kind	1
RequestGroup.action.relatedAction.relationship	1
RequestGroup.intent	1
RequestGroup.status	1
ResearchDefinition.status	1
ResearchElementDefinition.status	1
ResearchElementDefinition.type	1
ResearchStudy.status	1
ResearchSubject.status	1
RiskAssessment.status	1
RiskEvidenceSynthesis.status	1
SearchParameter.status	1
SearchParameter.type	1
ServiceRequest.intent	1
ServiceRequest.status	1
Slot.status	1
SpecimenDefinition.typeTested.preference	1
StructureDefinition.context.type	1
StructureDefinition.kind	1
StructureDefinition.status	1
StructureMap.group.input.mode	1
StructureMap.group.typeMode	1
StructureMap.status	1
StructureMap.structure.mode	1
Subscription.channel.type	1
Subscription.status	1
Task.intent	1
Task.status	1
TerminologyCapabilities.kind	1
TerminologyCapabilities.status	1
TestReport.participant.type	1
TestReport.result	1
TestReport.setup.action.assert.result	1
TestReport.setup.action.operation.result	1
TestReport.status	1
TestScript.status	1
ValueSet.compose.include.filter.op	1
ValueSet.status	1
VerificationResult.status	1
VisionPrescription.lensSpecification.eye	1
VisionPrescription.lensSpecification.prism.base	1
VisionPrescription.status	1
CodeableConcept	614
example	432
0..*	152
ActivityDefinition.bodySite	1
ActivityDefinition.topic	1
Appointment.serviceCategory	1
Appointment.serviceType	1
AuditEvent.agent.role	1
BodyStructure.locationQualifier	1
CarePlan.activity.detail.reasonCode	1
CarePlan.activity.outcomeCodeableConcept	1
CarePlan.category	1
CareTeam.category	1
CareTeam.participant.role	1
CareTeam.reasonCode	1
ChargeItem.bodysite	1
ChargeItem.reason	1
Claim.diagnosis.type	1
Claim.item.detail.modifier	1
Claim.item.detail.programCode	1
Claim.item.detail.subDetail.modifier	1
Claim.item.detail.subDetail.programCode	1
Claim.item.modifier	1
Claim.item.programCode	1
Claim.item.subSite	1
Claim.procedure.type	1
ClaimResponse.addItem.detail.modifier	1
ClaimResponse.addItem.detail.subDetail.modifier	1
ClaimResponse.addItem.modifier	1
ClaimResponse.addItem.programCode	1
ClaimResponse.addItem.subSite	1
ClinicalImpression.prognosisCodeableConcept	1
Communication.category	1
Communication.medium	1
Communication.reasonCode	1
CommunicationRequest.category	1
CommunicationRequest.medium	1
CommunicationRequest.reasonCode	1
Composition.category	1
Composition.event.code	1
Condition.bodySite	1
Condition.evidence.code	1
Consent.provision.action	1
Consent.provision.code	1
Contract.subType	1
Contract.term.action.performerType	1
Contract.term.action.reasonCode	1
Contract.term.asset.context.code	1
Contract.term.asset.periodType	1
Contract.term.asset.subtype	1
Contract.term.asset.type	1
Contract.term.offer.decisionMode	1
CoverageEligibilityRequest.item.modifier	1
CoverageEligibilityResponse.insurance.item.authorizationSupporting	1
CoverageEligibilityResponse.insurance.item.modifier	1
DetectedIssue.evidence.code	1
DeviceDefinition.safety	1
DeviceRequest.reasonCode	1
DiagnosticReport.category	1
DiagnosticReport.conclusionCode	1
DocumentReference.category	1
DocumentReference.context.event	1
EffectEvidenceSynthesis.topic	1
Encounter.hospitalization.dietPreference	1
Encounter.type	1
EpisodeOfCare.type	1
EventDefinition.topic	1
Evidence.topic	1
EvidenceVariable.topic	1
ExplanationOfBenefit.addItem.detail.modifier	1
ExplanationOfBenefit.addItem.detail.subDetail.modifier	1
ExplanationOfBenefit.addItem.modifier	1
ExplanationOfBenefit.addItem.programCode	1
ExplanationOfBenefit.addItem.subSite	1
ExplanationOfBenefit.diagnosis.type	1
ExplanationOfBenefit.item.detail.modifier	1
ExplanationOfBenefit.item.detail.programCode	1
ExplanationOfBenefit.item.detail.subDetail.modifier	1
ExplanationOfBenefit.item.detail.subDetail.programCode	1
ExplanationOfBenefit.item.modifier	1
ExplanationOfBenefit.item.programCode	1
ExplanationOfBenefit.item.subSite	1
ExplanationOfBenefit.procedure.type	1
FamilyMemberHistory.reasonCode	1
Flag.category	1
Goal.category	1
Goal.outcomeCode	1
HealthcareService.category	1
HealthcareService.characteristic	1
HealthcareService.program	1
HealthcareService.referralMethod	1
HealthcareService.serviceProvisionCode	1
HealthcareService.type	1
ImagingStudy.reasonCode	1
Immunization.programEligibility	1
Immunization.protocolApplied.targetDisease	1
Immunization.reasonCode	1
Immunization.subpotentReason	1
ImmunizationEvaluation.doseStatusReason	1
ImmunizationRecommendation.recommendation.contraindicatedVaccineCode	1
ImmunizationRecommendation.recommendation.forecastReason	1
ImmunizationRecommendation.recommendation.vaccineCode	1
InsurancePlan.type	1
Library.topic	1
Measure.topic	1
Media.reasonCode	1
MedicationAdministration.reasonCode	1
MedicationAdministration.statusReason	1
MedicationDispense.substitution.reason	1
MedicationKnowledge.intendedRoute	1
MedicationRequest.category	1
MedicationRequest.reasonCode	1
MedicationStatement.reasonCode	1
MedicationStatement.statusReason	1
NutritionOrder.excludeFoodModifier	1
NutritionOrder.foodPreferenceModifier	1
NutritionOrder.oralDiet.fluidConsistencyType	1
NutritionOrder.oralDiet.type	1
Observation.referenceRange.appliesTo	1
ObservationDefinition.category	1
ObservationDefinition.qualifiedInterval.appliesTo	1
Organization.type	1
OrganizationAffiliation.code	1
PlanDefinition.goal.addresses	1
PlanDefinition.topic	1
PractitionerRole.code	1
Procedure.bodySite	1
Procedure.complication	1
Procedure.followUp	1
Procedure.reasonCode	1
Procedure.usedCode	1
Provenance.agent.role	1
ResearchDefinition.topic	1
ResearchElementDefinition.topic	1
ResearchStudy.category	1
ResearchStudy.condition	1
ResearchStudy.focus	1
ResearchStudy.keyword	1
RiskEvidenceSynthesis.topic	1
Schedule.serviceCategory	1
Schedule.serviceType	1
ServiceRequest.bodySite	1
ServiceRequest.category	1
ServiceRequest.locationCode	1
ServiceRequest.orderDetail	1
ServiceRequest.reasonCode	1
Slot.serviceCategory	1
Slot.serviceType	1
SpecimenDefinition.collection	1
SpecimenDefinition.patientPreparation	1
SpecimenDefinition.typeTested.rejectionCriterion	1
SupplyRequest.reasonCode	1
VerificationResult.primarySource.communicationMethod	1
VerificationResult.primarySource.type	1
VerificationResult.validationProcess	1
0..1	209
Account.type	1
ActivityDefinition.code	1
ActivityDefinition.participant.role	1
ActivityDefinition.product[x]	1
AdverseEvent.event	1
AdverseEvent.seriousness	1
AdverseEvent.suspectEntity.causality.assessment	1
AdverseEvent.suspectEntity.causality.method	1
AllergyIntolerance.code	1
AllergyIntolerance.reaction.exposureRoute	1
AllergyIntolerance.reaction.substance	1
Appointment.cancelationReason	1
BiologicallyDerivedProduct.processing.procedure	1
BiologicallyDerivedProduct.productCode	1
BodyStructure.location	1
BodyStructure.morphology	1
CarePlan.activity.detail.code	1
CarePlan.activity.detail.product[x]	1
ChargeItem.performer.function	1
ChargeItem.product[x]	1
ChargeItemDefinition.code	1
Claim.careTeam.qualification	1
Claim.careTeam.role	1
Claim.diagnosis.onAdmission	1
Claim.diagnosis.packageCode	1
Claim.fundsReserve	1
Claim.item.bodySite	1
Claim.item.category	1
Claim.item.detail.category	1
Claim.item.detail.revenue	1
Claim.item.detail.subDetail.category	1
Claim.item.detail.subDetail.revenue	1
Claim.item.location[x]	1
Claim.item.revenue	1
Claim.related.relationship	1
Claim.subType	1
Claim.supportingInfo.code	1
Claim.supportingInfo.reason	1
ClaimResponse.addItem.bodySite	1
ClaimResponse.addItem.location[x]	1
ClaimResponse.formCode	1
ClaimResponse.fundsReserve	1
ClaimResponse.item.adjudication.reason	1
ClaimResponse.payeeType	1
ClaimResponse.payment.adjustmentReason	1
ClaimResponse.subType	1
ClinicalImpression.code	1
ClinicalImpression.finding.itemCodeableConcept	1
ClinicalImpression.statusReason	1
Communication.statusReason	1
Communication.topic	1
CommunicationRequest.statusReason	1
Composition.section.code	1
Condition.code	1
Condition.stage.summary	1
Condition.stage.type	1
Contract.contentDefinition.subType	1
Contract.contentDerivative	1
Contract.expirationType	1
Contract.scope	1
Contract.term.action.performerRole	1
Contract.term.action.subject.role	1
Contract.term.asset.scope	1
Contract.term.offer.type	1
Contract.term.subType	1
Contract.term.type	1
Contract.type	1
CoverageEligibilityRequest.item.category	1
CoverageEligibilityRequest.item.diagnosis.diagnosis[x]	1
CoverageEligibilityRequest.item.productOrService	1
CoverageEligibilityRequest.priority	1
CoverageEligibilityResponse.form	1
CoverageEligibilityResponse.insurance.item.category	1
CoverageEligibilityResponse.insurance.item.network	1
CoverageEligibilityResponse.insurance.item.productOrService	1
CoverageEligibilityResponse.insurance.item.term	1
CoverageEligibilityResponse.insurance.item.unit	1
Device.type	1
DeviceDefinition.type	1
DeviceRequest.parameter.code	1
DeviceRequest.performerType	1
DeviceUseStatement.bodySite	1
DocumentManifest.type	1
DocumentReference.context.facilityType	1
DocumentReference.context.practiceSetting	1
Encounter.hospitalization.dischargeDisposition	1
Encounter.hospitalization.reAdmission	1
Encounter.location.physicalType	1
Encounter.priority	1
Encounter.serviceType	1
ExplanationOfBenefit.addItem.bodySite	1
ExplanationOfBenefit.addItem.location[x]	1
ExplanationOfBenefit.benefitBalance.network	1
ExplanationOfBenefit.benefitBalance.term	1
ExplanationOfBenefit.benefitBalance.unit	1
ExplanationOfBenefit.careTeam.qualification	1
ExplanationOfBenefit.careTeam.role	1
ExplanationOfBenefit.diagnosis.onAdmission	1
ExplanationOfBenefit.diagnosis.packageCode	1
ExplanationOfBenefit.formCode	1
ExplanationOfBenefit.fundsReserve	1
ExplanationOfBenefit.fundsReserveRequested	1
ExplanationOfBenefit.item.adjudication.reason	1
ExplanationOfBenefit.item.bodySite	1
ExplanationOfBenefit.item.category	1
ExplanationOfBenefit.item.detail.category	1
ExplanationOfBenefit.item.detail.revenue	1
ExplanationOfBenefit.item.detail.subDetail.category	1
ExplanationOfBenefit.item.detail.subDetail.revenue	1
ExplanationOfBenefit.item.location[x]	1
ExplanationOfBenefit.item.revenue	1
ExplanationOfBenefit.payee.type	1
ExplanationOfBenefit.payment.adjustmentReason	1
ExplanationOfBenefit.payment.type	1
ExplanationOfBenefit.priority	1
ExplanationOfBenefit.related.relationship	1
ExplanationOfBenefit.subType	1
ExplanationOfBenefit.supportingInfo.code	1
FamilyMemberHistory.condition.outcome	1
FamilyMemberHistory.dataAbsentReason	1
Goal.start[x]	1
Goal.target.detail[x]	1
Goal.target.measure	1
Group.code	1
HealthcareService.eligibility.code	1
Immunization.fundingSource	1
Immunization.reportOrigin	1
Immunization.route	1
Immunization.site	1
Immunization.statusReason	1
ImmunizationRecommendation.recommendation.targetDisease	1
List.code	1
List.entry.flag	1
Location.physicalType	1
Media.bodySite	1
Media.modality	1
Media.view	1
Medication.code	1
Medication.form	1
MedicationAdministration.dosage.method	1
MedicationAdministration.dosage.route	1
MedicationAdministration.dosage.site	1
MedicationAdministration.performer.function	1
MedicationDispense.performer.function	1
MedicationDispense.statusReason[x]	1
MedicationDispense.substitution.type	1
MedicationDispense.type	1
MedicationKnowledge.code	1
MedicationKnowledge.doseForm	1
MedicationKnowledge.drugCharacteristic.type	1
MedicationKnowledge.packaging.type	1
MedicationRequest.courseOfTherapyType	1
MedicationRequest.performerType	1
MedicationRequest.statusReason	1
MedicationRequest.substitution.reason	1
MessageHeader.reason	1
MolecularSequence.quality.method	1
MolecularSequence.quality.standardSequence	1
MolecularSequence.referenceSeq.chromosome	1
MolecularSequence.referenceSeq.referenceSeqId	1
NutritionOrder.enteralFormula.additiveType	1
NutritionOrder.enteralFormula.baseFormulaType	1
NutritionOrder.oralDiet.nutrient.modifier	1
NutritionOrder.oralDiet.texture.foodType	1
NutritionOrder.oralDiet.texture.modifier	1
NutritionOrder.supplement.type	1
Observation.bodySite	1
Observation.method	1
ObservationDefinition.method	1
OperationOutcome.issue.details	1
PaymentNotice.paymentStatus	1
PaymentReconciliation.formCode	1
PlanDefinition.action.participant.role	1
PlanDefinition.goal.category	1
PlanDefinition.goal.start	1
PlanDefinition.goal.target.measure	1
Procedure.category	1
Procedure.code	1
Procedure.outcome	1
Procedure.performer.function	1
Procedure.statusReason	1
ResearchStudy.phase	1
ResearchStudy.reasonStopped	1
RiskAssessment.method	1
RiskAssessment.prediction.outcome	1
RiskAssessment.prediction.qualitativeRisk	1
ServiceRequest.asNeeded[x]	1
ServiceRequest.code	1
ServiceRequest.performerType	1
Specimen.collection.bodySite	1
Specimen.collection.method	1
Specimen.container.additive[x]	1
Specimen.container.type	1
Specimen.processing.procedure	1
Specimen.type	1
SpecimenDefinition.typeCollected	1
SpecimenDefinition.typeTested.container.cap	1
SpecimenDefinition.typeTested.container.material	1
SpecimenDefinition.typeTested.container.type	1
SpecimenDefinition.typeTested.handling.temperatureQualifier	1
SpecimenDefinition.typeTested.type	1
SupplyDelivery.suppliedItem.item[x]	1
SupplyRequest.category	1
SupplyRequest.parameter.code	1
Task.businessStatus	1
Task.code	1
Task.reasonCode	1
Task.statusReason	1
VerificationResult.attestation.communicationMethod	1
1..*	2
AllergyIntolerance.reaction.manifestation	1
Endpoint.payloadType	1
1..1	69
Basic.code	1
ChargeItem.code	1
Claim.diagnosis.diagnosis[x]	1
Claim.item.detail.productOrService	1
Claim.item.detail.subDetail.productOrService	1
Claim.item.productOrService	1
Claim.payee.type	1
Claim.priority	1
Claim.procedure.procedure[x]	1
Claim.supportingInfo.category	1
ClaimResponse.addItem.detail.productOrService	1
ClaimResponse.addItem.detail.subDetail.productOrService	1
ClaimResponse.addItem.productOrService	1
ClaimResponse.error.code	1
ClaimResponse.item.adjudication.category	1
ClaimResponse.payment.type	1
ClaimResponse.total.category	1
ClinicalImpression.investigation.code	1
Contract.contentDefinition.type	1
Contract.term.action.intent	1
Contract.term.action.status	1
Contract.term.action.type	1
Contract.term.offer.party.role	1
Coverage.costToBeneficiary.exception.type	1
CoverageEligibilityResponse.error.code	1
CoverageEligibilityResponse.insurance.item.benefit.type	1
DeviceRequest.code[x]	1
ExplanationOfBenefit.addItem.detail.productOrService	1
ExplanationOfBenefit.addItem.detail.subDetail.productOrService	1
ExplanationOfBenefit.addItem.productOrService	1
ExplanationOfBenefit.benefitBalance.category	1
ExplanationOfBenefit.benefitBalance.financial.type	1
ExplanationOfBenefit.diagnosis.diagnosis[x]	1
ExplanationOfBenefit.item.adjudication.category	1
ExplanationOfBenefit.item.detail.productOrService	1
ExplanationOfBenefit.item.detail.subDetail.productOrService	1
ExplanationOfBenefit.item.productOrService	1
ExplanationOfBenefit.procedure.procedure[x]	1
ExplanationOfBenefit.supportingInfo.category	1
ExplanationOfBenefit.total.category	1
FamilyMemberHistory.condition.code	1
FamilyMemberHistory.relationship	1
Flag.code	1
Goal.description	1
Group.characteristic.code	1
Group.characteristic.value[x]	1
Immunization.vaccineCode	1
ImmunizationEvaluation.doseStatus	1
ImmunizationEvaluation.targetDisease	1
ImmunizationRecommendation.recommendation.dateCriterion.code	1
ImmunizationRecommendation.recommendation.forecastStatus	1
MedicationAdministration.medication[x]	1
MedicationDispense.medication[x]	1
MedicationRequest.medication[x]	1
MedicationRequest.substitution.allowed[x]	1
MedicationStatement.medication[x]	1
Observation.code	1
Observation.component.code	1
ObservationDefinition.code	1
PaymentReconciliation.detail.type	1
PlanDefinition.goal.description	1
Practitioner.qualification.code	1
SpecimenDefinition.typeTested.container.additive.additive[x]	1
Substance.code	1
Substance.ingredient.substance[x]	1
SupplyRequest.item[x]	1
Task.input.type	1
Task.output.type	1
VisionPrescription.lensSpecification.product	1
extensible	116
0..*	54
ActivityDefinition.jurisdiction	1
AdverseEvent.category	1
Appointment.participant.type	1
AppointmentResponse.participantType	1
AuditEvent.agent.purposeOfUse	1
AuditEvent.purposeOfEvent	1
CapabilityStatement.jurisdiction	1
CapabilityStatement.rest.security.service	1
ChargeItemDefinition.jurisdiction	1
CodeSystem.jurisdiction	1
ConceptMap.jurisdiction	1
Condition.category	1
Device.statusReason	1
DocumentReference.securityLabel	1
EffectEvidenceSynthesis.certainty.certaintySubcomponent.rating	1
EffectEvidenceSynthesis.certainty.rating	1
EffectEvidenceSynthesis.jurisdiction	1
Encounter.participant.type	1
EventDefinition.jurisdiction	1
Evidence.jurisdiction	1
EvidenceVariable.jurisdiction	1
ExampleScenario.jurisdiction	1
GraphDefinition.jurisdiction	1
ImagingStudy.procedureCode	1
ImplementationGuide.jurisdiction	1
Library.jurisdiction	1
Location.type	1
Measure.jurisdiction	1
Measure.supplementalData.usage	1
Measure.type	1
MessageDefinition.jurisdiction	1
MetadataResource.jurisdiction	1
NamingSystem.jurisdiction	1
Observation.component.interpretation	1
Observation.interpretation	1
OperationDefinition.jurisdiction	1
Patient.contact.relationship	1
PlanDefinition.jurisdiction	1
Provenance.reason	1
Questionnaire.jurisdiction	1
ResearchDefinition.jurisdiction	1
ResearchElementDefinition.jurisdiction	1
ResearchStudy.location	1
RiskEvidenceSynthesis.certainty.certaintySubcomponent.rating	1
RiskEvidenceSynthesis.certainty.rating	1
RiskEvidenceSynthesis.jurisdiction	1
SearchParameter.jurisdiction	1
Specimen.condition	1
StructureDefinition.jurisdiction	1
StructureMap.jurisdiction	1
Substance.category	1
TerminologyCapabilities.jurisdiction	1
TestScript.jurisdiction	1
ValueSet.jurisdiction	1
0..1	54
ActivityDefinition.subject[x]	1
AuditEvent.agent.type	1
Claim.accident.type	1
Consent.policyRule	1
Contract.legalState	1
Contract.term.offer.decision	1
Coverage.costToBeneficiary.type	1
Coverage.relationship	1
EffectEvidenceSynthesis.certainty.certaintySubcomponent.type	1
EffectEvidenceSynthesis.effectEstimate.precisionEstimate.type	1
EffectEvidenceSynthesis.effectEstimate.type	1
EffectEvidenceSynthesis.effectEstimate.variantState	1
EffectEvidenceSynthesis.resultsByExposure.variantState	1
EffectEvidenceSynthesis.studyType	1
EffectEvidenceSynthesis.synthesisType	1
EventDefinition.subject[x]	1
ExplanationOfBenefit.accident.type	1
FamilyMemberHistory.sex	1
ImagingStudy.series.performer.function	1
Immunization.performer.function	1
InsurancePlan.contact.purpose	1
Library.subject[x]	1
Measure.compositeScoring	1
Measure.group.population.code	1
Measure.scoring	1
Measure.subject[x]	1
MeasureReport.group.population.code	1
MeasureReport.group.stratifier.stratum.population.code	1
Media.type	1
NamingSystem.type	1
NutritionOrder.enteralFormula.routeofAdministration	1
Observation.component.dataAbsentReason	1
Observation.dataAbsentReason	1
ObservationDefinition.qualifiedInterval.context	1
ObservationDefinition.quantitativeDetails.customaryUnit	1
ObservationDefinition.quantitativeDetails.unit	1
Organization.contact.purpose	1
Patient.maritalStatus	1
PlanDefinition.action.subject[x]	1
PlanDefinition.action.type	1
PlanDefinition.subject[x]	1
PlanDefinition.type	1
Provenance.activity	1
Provenance.agent.type	1
RequestGroup.action.type	1
ResearchDefinition.subject[x]	1
ResearchElementDefinition.subject[x]	1
ResearchStudy.primaryPurposeType	1
RiskEvidenceSynthesis.certainty.certaintySubcomponent.type	1
RiskEvidenceSynthesis.riskEstimate.precisionEstimate.type	1
RiskEvidenceSynthesis.riskEstimate.type	1
RiskEvidenceSynthesis.studyType	1
RiskEvidenceSynthesis.synthesisType	1
Specimen.collection.fastingStatus[x]	1
1..*	1
Consent.category	1
1..1	7
Claim.type	1
ClaimResponse.type	1
Consent.provision.actor.role	1
Consent.scope	1
Coverage.class.type	1
ExplanationOfBenefit.type	1
Library.type	1
preferred	52
0..*	16
Appointment.reasonCode	1
Appointment.specialty	1
Encounter.hospitalization.specialArrangement	1
Encounter.hospitalization.specialCourtesy	1
Encounter.reasonCode	1
HealthcareService.communication	1
HealthcareService.specialty	1
Observation.category	1
OrganizationAffiliation.specialty	1
Practitioner.communication	1
PractitionerRole.specialty	1
RelatedPerson.relationship	1
Schedule.specialty	1
Slot.specialty	1
Task.performerType	1
VerificationResult.primarySource.pushTypeAvailable	1
0..1	30
Appointment.appointmentType	1
ClaimResponse.processNote.language	1
Composition.section.emptyReason	1
Composition.section.orderedBy	1
Condition.severity	1
Coverage.type	1
DetectedIssue.code	1
DeviceMetric.unit	1
DocumentReference.type	1
Encounter.diagnosis.use	1
Encounter.hospitalization.admitSource	1
EpisodeOfCare.diagnosis.role	1
ExplanationOfBenefit.processNote.language	1
Goal.achievementStatus	1
Goal.priority	1
List.emptyReason	1
List.orderedBy	1
MedicationAdministration.category	1
MedicationDispense.category	1
MedicationStatement.category	1
Observation.referenceRange.type	1
PlanDefinition.goal.priority	1
Procedure.focalDevice.action	1
ResearchStudy.objective.type	1
Slot.appointmentType	1
VerificationResult.failureAction	1
VerificationResult.need	1
VerificationResult.primarySource.canPushUpdates	1
VerificationResult.primarySource.validationStatus	1
VerificationResult.validationType	1
1..1	6
Composition.type	1
DetectedIssue.mitigation.action	1
DeviceMetric.type	1
DiagnosticReport.code	1
Patient.communication.language	1
RelatedPerson.communication.language	1
required	14
0..1	14
AdverseEvent.outcome	1
AdverseEvent.severity	1
AllergyIntolerance.clinicalStatus	1
AllergyIntolerance.verificationStatus	1
Condition.clinicalStatus	1
Condition.verificationStatus	1
EffectEvidenceSynthesis.effectEstimate.unitOfMeasure	1
InsurancePlan.plan.specificCost.benefit.cost.applicability	1
Measure.improvementNotation	1
MeasureReport.improvementNotation	1
MolecularSequence.structureVariant.variantType	1
ResearchElementDefinition.characteristic.unitOfMeasure	1
RiskEvidenceSynthesis.riskEstimate.unitOfMeasure	1
SupplyDelivery.type	1
Coding	42
example	14
0..*	4
Contract.term.securityLabel.category	1
Contract.term.securityLabel.control	1
Questionnaire.code	1
Questionnaire.item.code	1
0..1	4
ExplanationOfBenefit.supportingInfo.reason	1
ImagingStudy.series.bodySite	1
ImagingStudy.series.laterality	1
QuestionnaireResponse.item.answer.value[x]	1
1..1	6
Contract.term.securityLabel.classification	1
MessageDefinition.event[x]	1
MessageHeader.event[x]	1
Questionnaire.item.answerOption.value[x]	1
Questionnaire.item.enableWhen.answer[x]	1
Questionnaire.item.initial.value[x]	1
extensible	25
0..*	8
AuditEvent.entity.securityLabel	1
AuditEvent.source.type	1
AuditEvent.subtype	1
Consent.provision.class	1
Consent.provision.purpose	1
Consent.provision.securityLabel	1
ImagingStudy.modality	1
StructureDefinition.keyword	1
0..1	8
AuditEvent.agent.media	1
AuditEvent.entity.lifecycle	1
AuditEvent.entity.role	1
AuditEvent.entity.type	1
CodeSystem.concept.designation.use	1
Contract.term.asset.relationship	1
TestScript.setup.action.operation.type	1
ValueSet.compose.include.concept.designation.use	1
1..1	9
AuditEvent.type	1
CapabilityStatement.messaging.endpoint.protocol	1
Encounter.class	1
Encounter.classHistory.class	1
Endpoint.connectionType	1
ImagingStudy.series.instance.sopClass	1
ImagingStudy.series.modality	1
TestScript.destination.profile	1
TestScript.origin.profile	1
preferred	3
0..1	2
DocumentReference.content.format	1
Location.operationalStatus	1
1..1	1
Contract.signer.type	1
Grand Total	975