smart-on-fhir / fhir-parser Goto Github PK

A Python FHIR specification parser and class generator

License: Other

Python 100.00%

fhir-parser's Introduction

Python FHIR Parser

A Python FHIR specification parser for model class generation. If you've come here because you want Swift or Python classes for FHIR data models, look at our client libraries instead:

Swift-FHIR and Swift-SMART
Python client-py

The main branch is currently capable of parsing R4 and has preliminary support for R5.

This work is licensed under the APACHE license. FHIR® is the registered trademark of HL7 and is used with the permission of HL7.

Tech

The generate.py script downloads FHIR specification files, parses the profiles (using fhirspec.py) and represents them as FHIRClass instances with FHIRClassProperty properties (found in fhirclass.py). Additionally, FHIRUnitTest (in fhirunittest.py) instances get created that can generate unit tests from provided FHIR examples. These representations are then used by Jinja templates to create classes in certain programming languages, mentioned below.

This script does its job for the most part, but it doesn't yet handle all FHIR peculiarities and there's no guarantee the output is correct or complete. This repository does not include the templates and base classes needed for class generation, you must do this yourself in your project. You will typically add this repo as a submodule to your framework project, create a directory that contains the necessary base classes and templates, create settings and mappings files and run the script. Examples on what you would need to do for Python classes can be found in Default/settings.py, Default/mappings.py and Sample/templates*.

Use

Add fhir-parser as a submodule/subdirectory to the project that will use it
Create the file mappings.py in your project, to be copied to fhir-parser root. First, import the default mappings using from Default.mappings import * (unless you will define all variables yourself anyway). Then adjust your mappings.py to your liking by overriding the mappings you wish to change.
Similarly, create the file settings.py in your project. First, import the default settings using from Default.settings import * and override any settings you want to change. Then, import the mappings you have just created with from mappings import *. The default settings import the default mappings, so you may need to overwrite more keys from mappings than you'd first think. You most likely want to change the topmost settings found in the default file, which are determining where the templates can be found and generated classes will be copied to.
Install the generator's requirements by running pip3 (or pip):
```
pip3 install -r requirements.txt
```
Create a script that copies your mappings.py and settings.py file to the root of fhir-parser, _cd_s into fhir-parser and then runs generate.py. The generate script by default wants to use Python 3, issue python generate.py if you don't have Python 3 yet.
- Supply the -f flag to force a re-download of the spec.
- Supply the --cache-only (-c) flag to deny the re-download of the spec and only use cached resources (incompatible with -f).

NOTE that the script currently overwrites existing files without asking and without regret.

Languages

This repo used to contain templates for Python and Swift classes, but these have been moved to the respective framework repositories. A very basic Python sample implementation is included in the Sample directory, complementing the default mapping and settings files in Default.

To get a sense of how to use fhir-parser, take a look at these libraries:

Tech Details

This parser still applies some tricks, stemming from the evolving nature of FHIR's profile definitions. Some tricks may have become obsolete and should be cleaned up.

How are property names determined?

Every “property” of a class, meaning every element in a profile snapshot, is represented as a FHIRStructureDefinitionElement instance. If an element itself defines a class, e.g. Patient.animal, calling the instance's as_properties() method returns a list of FHIRClassProperty instances – usually only one – that indicates a class was found in the profile. The class of this property is derived from element.type, which is expected to only contain one entry, in this matter:

If type is BackboneElement, a class name is constructed from the parent element (in this case Patient) and the property name (in this case animal), camel-cased (in this case PatientAnimal).
Otherwise, the type is taken as-is (e.g. CodeableConcept) and mapped according to mappings' classmap, which is expected to be a valid FHIR class.

TODO: should http://hl7.org/fhir/StructureDefinition/structuredefinition-explicit-type-name be respected?

fhir-parser's People

Contributors

Stargazers

Watchers

fhir-parser's Issues

Unit tests don't pass for R5

After generating models for R5, the unit tests don't pass. You get errors like:

Wrong type <class 'str'> for property "size" on <class 'models.attachment.Attachment'>, expecting <class 'models.integer64.Integer64'>
'Non-optional property "attachment" on <models.documentreference.DocumentReferenceContent object at 0x7ca3eb277e60> is missing'
'Non-optional property "content" on <models.documentreference.DocumentReference object at 0x7ca3eb274710> is missing'

It mostly works! 6 out of 533 fail. So a little more work to go. Once fixed, the tests should be turned on in the github CI.

exception about copying a file

INFO:fhirparser:Copying manual profiles in __init__.py to ../models/__init__.py
Traceback (most recent call last):
  File "generate.py", line 25, in <module>
    spec.write()
  File "/home/tim/fhir-parser/fhirspec.py", line 216, in write
    renderer.copy_files()
  File "/home/tim/fhir-parser/fhirrenderer.py", line 60, in copy_files
    shutil.copyfile(filepath, tgt)
  File "/home/tim/fhir-parser/env/lib/python3.4/shutil.py", line 109, in copyfile
    with open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '../models/__init__.py'

I'm not sure what's going on here... I had put in a hack to get around the issue in #6 and then I hit this exception.

exception when running generate.py

running generate.py gives:

INFO:root:Install "colorlog" to enable colored log messages
INFO:fhirparser:Using cached resources, supply "-f" to re-download
DEBUG:fhirparser:Created class "bool"
DEBUG:fhirparser:Created class "str"
DEBUG:fhirparser:Created class "float"
DEBUG:fhirparser:Created class "int"
DEBUG:fhirparser:Created class "FHIRElement"
DEBUG:fhirparser:Created class "FHIRResource"
DEBUG:fhirparser:Created class "ContainedResource"
DEBUG:fhirparser:Created class "FHIRReference"
DEBUG:fhirparser:Created class "FHIRDate"
DEBUG:fhirparser:Created class "FHIRSearch"
INFO:fhirparser:Parsing profile "integer"
WARNING:fhirparser:Already have profile "integer", discarding
INFO:fhirparser:Parsing profile "dateTime"
WARNING:fhirparser:Already have profile "dateTime", discarding
INFO:fhirparser:Parsing profile "unsignedInt"
WARNING:fhirparser:Already have profile "unsignedInt", discarding
INFO:fhirparser:Parsing profile "code"
WARNING:fhirparser:Already have profile "code", discarding
INFO:fhirparser:Parsing profile "date"
WARNING:fhirparser:Already have profile "date", discarding
INFO:fhirparser:Parsing profile "decimal"
WARNING:fhirparser:Already have profile "decimal", discarding
INFO:fhirparser:Parsing profile "uri"
WARNING:fhirparser:Already have profile "uri", discarding
INFO:fhirparser:Parsing profile "id"
WARNING:fhirparser:Already have profile "id", discarding
INFO:fhirparser:Parsing profile "base64Binary"
WARNING:fhirparser:Already have profile "base64Binary", discarding
INFO:fhirparser:Parsing profile "time"
WARNING:fhirparser:Already have profile "time", discarding
INFO:fhirparser:Parsing profile "oid"
WARNING:fhirparser:Already have profile "oid", discarding
INFO:fhirparser:Parsing profile "positiveInt"
WARNING:fhirparser:Already have profile "positiveInt", discarding
INFO:fhirparser:Parsing profile "string"
WARNING:fhirparser:Already have profile "string", discarding
INFO:fhirparser:Parsing profile "boolean"
WARNING:fhirparser:Already have profile "boolean", discarding
INFO:fhirparser:Parsing profile "uuid"
WARNING:fhirparser:Already have profile "uuid", discarding
INFO:fhirparser:Parsing profile "instant"
WARNING:fhirparser:Already have profile "instant", discarding
INFO:fhirparser:Parsing profile "Period"
DEBUG:fhirparser:Created class "Period"
INFO:fhirparser:Parsing profile "Coding"
DEBUG:fhirparser:Created class "Coding"
INFO:fhirparser:Parsing profile "Range"
DEBUG:fhirparser:Created class "Range"
INFO:fhirparser:Parsing profile "Quantity"
DEBUG:fhirparser:Created class "Quantity"
INFO:fhirparser:Parsing profile "Attachment"
DEBUG:fhirparser:Created class "Attachment"
INFO:fhirparser:Parsing profile "Ratio"
DEBUG:fhirparser:Created class "Ratio"
INFO:fhirparser:Parsing profile "SampledData"
DEBUG:fhirparser:Created class "SampledData"
INFO:fhirparser:Parsing profile "Reference"
DEBUG:fhirparser:Created class "Reference"
INFO:fhirparser:Parsing profile "CodeableConcept"
DEBUG:fhirparser:Created class "CodeableConcept"
INFO:fhirparser:Parsing profile "Identifier"
DEBUG:fhirparser:Created class "Identifier"
Traceback (most recent call last):
  File "generate.py", line 24, in <module>
    spec = fhirspec.FHIRSpec(spec_source, settings)
  File "/home/tim/fhir-parser/fhirspec.py", line 41, in __init__
    self.read_profiles()
  File "/home/tim/fhir-parser/fhirspec.py", line 86, in read_profiles
    profile.process_profile()
  File "/home/tim/fhir-parser/fhirspec.py", line 321, in process_profile
    snap_class, subs = self.main_element.create_class()
  File "/home/tim/fhir-parser/fhirspec.py", line 533, in create_class
    properties = child.as_properties()
  File "/home/tim/fhir-parser/fhirspec.py", line 579, in as_properties
    props.append(fhirclass.FHIRClassProperty(self, type_obj))
  File "/home/tim/fhir-parser/fhirclass.py", line 118, in __init__
    self.reference_to_names = [spec.class_name_for_profile(type_obj.profile)] if type_obj.profile is not None else []
  File "/home/tim/fhir-parser/fhirspec.py", line 176, in class_name_for_profile
    mappedname = self.mapped_name_for_profile(profile_name)
  File "/home/tim/fhir-parser/fhirspec.py", line 170, in mapped_name_for_profile
    type_name = profile_name.split('/')[-1]     # may be the full Profile URI, like http://hl7.org/fhir/Profile/MyProfile
AttributeError: 'list' object has no attribute 'split'

I tried catching the exception there and profile_name has the value ['http://hl7.org/fhir/StructureDefinition/Organization'].

Support for Extensions

Hello,
I've been looking into how to use fhir-parser for generating Swift objects (looked at Swift-FHIR already). I was wondering if fhir-parser currently supports extensions of FHIR models and how that works? And if not, what would be required to support them?

Thanks,
Quyen
[email protected]

No LICENSE file in the repository

Looking to use this parser but I need to know what LICENSE this code is under so I see if it plays well other software packages.

I see the other repositories on the smart-on-fhir are apache 2.0, maybe this one is too but is just missing the LICENSE file?

Valueset elementProperties of the form "Foo.str"

Not quite sure what's going on there. Using the default templates, settings, etc, we get things like the following (taken from generated account.py)

("status", "status", AccountStatus.str, False, None, True),

where I'm pretty sure that AccountStatus.str should actually be a str (or maybe some sort of enum). Either way, I'm reasonably sure it's wrong and not sure where to start debugging. Any thoughts?

Enable Travis repo

@raheelsayeed Now that #43 is merged, can you enable https://travis-ci.com/github/smart-on-fhir/fhir-parser please? (and fix the build badge so it stops pointing to my repo)

Help Need : Step by Step guide for beginner

Please can someone help with step by step guide to set up this code and fetch the request from the FHIR repository http://fhir.hl7fundamentals.org/r4/Patient/3.

fhir-parser rendering some strange ASCII...

Hey Pascal,

I almost have my RealmSwiftFHIR and fhir-parser templates all wrapped up into one nice little package with a bow on top. There's one outstanding issue, and that is about 50% of the time, the fhir-parser generates the code and I get the following compilation errors (roughly 13 of them).

Unprintable ASCII chracter found in source file

I suspect this has to do with the tab characters in the JSON, but... how on earth are you getting around this? I should note I've not modified the portions of the template-unittest.swift file which writes the expectation for the test, so this is stock FHIR-Parser (version: 70f814c).

Thoughts?

Need warning if cache doesn't match source

If the cache is being used and "-c" isn't specified, a warning should be issued if the source of the cache (downloads) directory doesn't match the source in the settings.py.

More unit tests needed

The unit tests needs to verify that the emitted JSON is recognized by a target FHIR server.

Part of this could be accomplished by an ordered comparison to the FHIR examples, but we need to have tests that use the python API to generate FHIR objects and then use a FHIR server/validator to check correctness.

Identifier type doesn't check valid characters

The identifier field allows underscores ('_') and other characters that aren't valid. This is particularly problematic as the underscore causes the HAPI fhir server to claim there is no payload at all...

Need full implementation of FHIR data types

FHIR data types need a robust implementation library

FHIRSearchElement has conflicting property/method names

For example a subject() method is generated for FHIRSearchElement, which interferes with the subject property that the class uses throughout.

Date/time format incorrect

The output format of the date time field isn't in the same format as expected by the FHIR server. In particular, it needs a 'T' where the generated code emits a space

generate.py fails to download FHIR specs file

Using master branch (4607aad), I copied the default settings.py and mappings.py at top-level then ran ./generate.py:

INFO:root:Install "colorlog" to enable colored log messages
INFO:fhirparser:Downloading version.info
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): hl7.org:80
DEBUG:urllib3.connectionpool:http://hl7.org:80 "GET /fhir/2018May//version.info HTTP/1.1" 404 6977
Traceback (most recent call last):
  File "./generate.py", line 25, in <module>
    spec_source = loader.load(force_download=force_download, force_cache=force_cache)
  File "/path/to/fhir-parser/fhirloader.py", line 48, in load
    filename = self.download(remote)
  File "/path/to/fhir-parser/fhirloader.py", line 75, in download
    raise Exception("Failed to download {}".format(url))
Exception: Failed to download http://hl7.org/fhir/2018May//version.info

validation?

Maybe this is more of a question about clarification of function then an actual "issue", but I'm not really sure how to contact you directly...

Do these models perform any sort of validation?

for example:

>>> x = models.device.Device({"manufacturer":"me", "wrong":"so so wrong"})
>>> x.as_json()
{'manufacturer': 'me'}

If you pass things in that don't map properly they seem to just get dropped instead of throwing an exception.

Also, I found it a little weird that the output of Device.as_json didn't automatically add a 'resourceType': 'Device' and explicitly adding one into the constructor seemed to not change that.

Question about unit tests

How do I verify that changes I have made to the parser haven't broken anything? At the moment, I get the following "out of the box".

Steps:

copy settings.py and mappings.py from Defaults to Sample directory
pip install -r requirements.txt
cd sample
../generate.py

...
  DEBUG    | Created class "VerificationResultValidator", module verificationresult
  INFO     | Parsing profile "VisionPrescription"
  DEBUG    | Created class "VisionPrescription", module visionprescription
  DEBUG    | Created class "VisionPrescriptionLensSpecification", module visionprescription
  DEBUG    | Created class "VisionPrescriptionLensSpecificationPrism", module visionprescription
  INFO     | Parsing profile "MetadataResource"
  DEBUG    | Created class "MetadataResource", module metadataresource
Traceback (most recent call last):
  File "../generate.py", line 33, in <module>
    spec.write()
  File "/Users/solbrig/git/smart-on-fhir/fhir-parser/fhirspec.py", line 246, in write
    renderer.render()
  File "/Users/solbrig/git/smart-on-fhir/fhir-parser/fhirrenderer.py", line 91, in render
    imports = profile.needed_external_classes()
  File "/Users/solbrig/git/smart-on-fhir/fhir-parser/fhirspec.py", line 527, in needed_external_classes
    raise Exception('There is no class "{}" for property "{}" on "{}" in {}'.format(prop_cls_name, prop.name, klass.name, self.name))
Exception: There is no class "Http://hl7.org/fhirpath/System.String" for property "id" on "Element" in Element

Python tabs vs. spaces for indenting.

It is common, idiomatic and uncontroversial pep-8 python to use spaces instead of tabs for python code. As indentation is significant in python, it is important to remove the confusion that can exist with tabs, especially as tabs can be set to any value of indent in the user's editor. It is easy for tabs and spaces to be confused, as they are normally undistinguished in editors and on the web. The common idiom is 4 spaces per indent level.

Outdated default python settings?

When using the settings.py, as directed, from the "Python" folder, line 590 in fhirspec.py:

type_code = self.profile.spec.settings.default_base.get(self.profile.structure.kind)

Produces an error as settings.py (from the "Python" folder) has no attribute 'default_base'. Adding:

default_base = { 'contained': 'FHIRElement', 'resource': 'FHIRResource' }

To settings.py allows the parser to run without issue. This mirrors the Swift settings.py, but I'm enough of a FHIR newbie that I cannot fully comprehend whether or not that has any unwelcomed consequences.

Can't locate Sample files

Can't locate the following sample files that were mentioned within README.md file:
Sample/settings.py, and Sample/mappings.py.

Element ordering

The elements in the emitted json objects are alphabetized and, as such, are not valid input to a FHIR server

utf-8 and python files

Python is usually very good with cross platform programming, and I note you consistently use path.join(), so I assume you are too.

One place python falls down is default encoding, which can be platform dependent. Unfortunately, windows does not use utf-8 as it's default and it is a bear to try and force the issue. So, it's become idiomatic to always specify encoding="utf-8" when opening a text file.

Anyhow, without the encoding being specified, generate fails on my windows 8 box, works on my *nix. With, works *nix and windows.

support for python3 type system

Currently the type information is written in docstrings. python3 supports defining type system can this parser to generate python3 variant with supported type declarations?

Unable to run generate.py without error

Similar to #33

"Out of the box"

clone and cd into fresh repo, pip install -r requirements
copy ./Defaults/settings.py & mappings.py to ../
cd into ./Samples
run ./generate.py

Got.. Exception: Unable to create a member name for enum '!=' in http://hl7.org/fhir/questionnaire-enable-operator. You may need to add '!=' to mappings.enum_map

So I went ahead and added '!=': 'ne" to mappings.enum (actually had to change both ./Default/mappings.py and ../mappings.py

Now I run generate and it appears to create all the models but fails during unit test creation:
Traceback (most recent call last):
File "./generate.py", line 31, in
spec.write()
File "/root/fhir-parser/fhirspec.py", line 277, in write
self.parse_unit_tests()
File "/root/fhir-parser/fhirspec.py", line 245, in parse_unit_tests
controller.find_and_parse_tests(self.directory)
File "/root/fhir-parser/fhirunittest.py", line 31, in find_and_parse_tests
test = self.unittest_for_resource(resource)
File "/root/fhir-parser/fhirunittest.py", line 63, in unittest_for_resource
return FHIRUnitTest(self, resource.filepath, resource.content, klass)
File "/root/fhir-parser/fhirunittest.py", line 102, in init
self.expand()
File "/root/fhir-parser/fhirunittest.py", line 137, in expand
tests.extend(item.create_tests(self.controller))
File "/root/fhir-parser/fhirunittest.py", line 174, in create_tests
test = FHIRUnitTest(controller, self.filepath, self.value, self.klass, prefix)
File "/root/fhir-parser/fhirunittest.py", line 102, in init
self.expand()
File "/root/fhir-parser/fhirunittest.py", line 136, in expand
item = FHIRUnitTestItem(self.filepath, path, val, propclass, False, prop.enum)
File "/root/fhir-parser/fhirunittest.py", line 157, in init
self.enum = enum_item['name'] if enum_item is not None else None
TypeError: 'FHIRValueSetEnum' object is not subscriptable

Any ideas what I am doing wrong? My guess is it has something to with me adding '!=': 'ne'

OperationOutcome?

Hello, I’m using the following Construction

v = ValueSet.read(some_id, fhirserver)

In case then all go right and Resource from some_id was founded, then v - object ValueSet.

But in case then request_json() return from server json with resourceType = OperationOutcome - v keep the same object ValueSet.

May be I misuse this function in some way ?

Needs Refactoring

This thing has grown way past what a script is supposed to do. Refactor into internal data representations that can spit out classes and search params.