Giter Club home page Giter Club logo

plan_executor's Introduction

Plan Executor Build Status

Plan Executor runs test suites against a FHIR server.

STU3

Updated to support the FHIR STU3 Candidate branch.

Getting Started

$ bundle install
$ bundle exec rake -T

Listing Test Suites

List all the available Test Suites, excluding supported TestScripts. Pass the version, which can currently either be dstu2 or stu3.

$ bundle exec rake crucible:list_suites[dstu2]

Executing a Test Suite

Crucible tests can be executed by suite from the command-line by calling the crucible-execute rake task with the following parameters:

  • url the FHIR endpoint
  • version the FHIR version (sequence). Currently dstu2 and stu3 are supported.
  • test the name of the test suite (see crucible:list_suites)
  • resource (optional) limit the test (applicable to "ResourceTest" or "SearchTest" suites) to a given resource (e.g. "Patient")

Run a DSTU2 Suite

$ bundle exec rake crucible:execute[http://fhirtest.uhn.ca/baseDstu2,dstu2,TransactionAndBatchTest]

Run a STU3 Suite limited by Resource

$ bundle exec rake crucible:execute[http://fhirtest.uhn.ca/baseDstu3,stu3,ResourceTest,Patient]

Adding a New Test Suite

  1. Fork the repo
  2. Write the test suite in Ruby
  3. Issue a pull request

Add a Test Suite by adding a Ruby file to lib/tests/suites that extends Crucible::Tests::BaseTest -- for example, FooTest:

module Crucible
  module Tests
    class FooTest < BaseSuite

      def id
        'FooTest'
      end

      def description
        'FooTest is an example of adding a new test suite.'
      end

      def initialize(client1, client2=nil)
        super(client1, client2)
        @category = {id: 'connectathon', title: 'Connectathon'}
      end

      def setup
        # create any fixtures you need here
        @patient = ResourceGenerator.generate(FHIR::Patient,3)
        reply = @client.create(@patient)
        @id = reply.id
        @body = reply.body
      end

      def teardown
        # perform any clean up here
        @client.destroy(FHIR::Patient, @id)
      end

      # test 'KEY', 'DESCRIPTION'
      test 'FOO', 'Foo Test checks headers' do
        metadata {
          links "#{REST_SPEC_LINK}#read"
          requires resource: "Patient", methods: ["create", "read"]
          validates resource: "Patient", methods: ["read"]
        }

        assert(@id, 'Setup was unable to create a patient.',@body)
        reply = @client.read(FHIR::Patient, @id)
        assert_response_ok(reply)
        assert_equal @id, reply.id, 'Server returned wrong patient.'
        warning { assert_valid_resource_content_type_present(reply) }
        warning { assert_etag_present(reply) }
        warning { assert_last_modified_present(reply) }
      end
    end
  end
end

Every Test Suite needs to override the following methods:

  • id The unique id of the test, typically matches the class name
  • description The description that is displayed within the Crucible web app
  • initialize Use the example above. Change the @category -- the id and title determine where the test suite is categorized within the Crucible web app
  • setup (optional) Use this method to create fixtures and perform any required assertions prior to execution of individual test blocks.
  • test These blocks are the individual tests within the suites. Each block should start with a metadata section so Crucible knows how to tie the success or failures to portions of the FHIR specification (displayed in the web app with a starburst). See lib/FHIR_structure.json for the values associated with the name keys that you can link to.
  • teardown (optional) Use this method to perform any clean up, so you don't leave a trail of test data behind.

License

Copyright 2014-2016 The MITRE Corporation

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

plan_executor's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

plan_executor's Issues

X030_DeviceRequest intermittently fails

This can be seen on the Crucible server: https://www.projectcrucible.org/servers/593a8f8e63727513d6020000#5a3af9fe04ebd03841000000/resourcetest_devicerequest/X030_DeviceRequest

This test sometimes fails due to generating an invalid request, the error message (from the FHIR test server) is:

DeviceRequest, Element "DeviceRequest.code[x]" (DeviceRequest.code[x]): minimum required = 1, but only found 0

When the test is re-run, sometimes a code will be sent, which passes.

Ref: https://github.com/fhir-crucible/plan_executor/blob/master/lib/tests/suites/resource_test.rb#L225

R4: ReadTest can fail if no Patient resources on client

In the R005 test of the ReadTest, a fatal error can occur:

     ERROR r005_read__summary=text_test: Fatal Error: undefined method `text' for #<FHIR::Bundle:0x000055be105c2168 @id=nil, @meta=nil, @implicitRules=nil, @language=nil, @identifier=nil, @type="searchset", @timestamp=nil, @total=0, @link=[], @entry=[], @signature=nil>
            ----------------------------------------
            /home/p/.rbenv/versions/2.5.8/lib/ruby/gems/2.5.0/gems/fhir_models-4.0.2/lib/fhir_models/bootstrap/model.rb:58:in `method_missing'
            /home/p/plan_executor/lib/tests/suites/read_test.rb:105:in `block in <class:ReadTest>'
            /home/p/plan_executor/lib/tests/suites/base_suite.rb:145:in `instance_eval'

During setup, the code attempts to get a Patient resource from the client under test, otherwise it will create a new one. Unfortunately the created resource is just a local ruby object, it is not created on the client. This causes problems when trying to fetch the resource in test R005 itself.

X010: Subscription: Create New

This test creates a subscription with criteria <criteria value="5OZ3cHdGHZ8yxDPkwqtxqQ=="/>, but this criteria does not make sense and should presumably be rejected by any server that actually plans on honoring the subscription (as HAPI is doing).

Bugs in tests found

AdverseEvent test has element which does not exist in the schema
Imaging Manifest test has element which does not exist in the schema
Claim + Claim Reponse test has element should be
Spinkler Test - SE04P + G searches on patient name which does exist - but your test expecting 0 records
and a few of the other Sprinkler tests similar to that
History Test - HI08 returning a bundle but it is not recognised as a bundle "History should be a Bundle"

CT01G: Expand a specific ValueSet (GET)

This test and a number of other terminology-specific tests assume that the server is pre-populated with certain FHIR-defined CodeSystems, ValueSets, etc
This is an invalid assumption - the tests should ensure test fixtures like these are created first.

The Role FHIR Serve Capability Statement (e.g. fhirVersion) plays in testing and reporting results

In order to actively and accurately test any HL7 FHIR Implementations - an HL7 FHIR Test Platform would be required to successfully interrogate a FHIR Server Capability Statement - fhirVersion element detail.

Such that if the fhirVersion published is found to be invalid (e.g. FHIR VERSION: 1.5.0) according to FHIR Specifications - Crucible would be able to validate/verify the FHIR Server Version/Implementation conformance or interoperability and report accordingly?

Recommendation: FHIR Server Capability Statement be validated prior to any test platform test execution - to ensure the information available to a FHIR Client would not result in non-interoperable exchange of information.

Further information is available if needed - contact the author of this issue

Text field not returned on _summary when it does not exist

I was running tests on Crucible and one of the findings was that our server does not return the text element when _summary is used. I tried to reproduce this, but Vonk does return the text element when it is present in the original resource. However, I see that the created Patient for this test is missing the text element. Therefore, it’s also missing in the response of our server.
Here’s the link to the test result:
https://projectcrucible.org/servers/5aeabdd404ebd02256000000#5b4ef50504ebd07ddf000005/readtest/R005
I think it should be added in the Patient resource created for this test?

Patch Track is Broken with Missing Fixture

C12PATCH_1_(JSON): Get Existing MedicationRequest by JSON

Message
No such file or directory @ rb_sysopen - /usr/local/rvm/gems/ruby-2.1.2/bundler/gems/plan_executor-2b875f49199c/fixtures/patch/medicationorder-simple.xml

FHIR Testing Match Conformance/Interoperability tests between Capability Statement and FHIR Published Specifications

As a FHIR Server I am required by FHIR Specifications to publically publish a Capability (formerly Conformance) Statement - in accordance with the FHIR Specification the Capability Statement (http://hl7.org/fhir/2017Jan/capabilitystatement.html) includes an element fhirVersion (http://hl7.org/fhir/2017Jan/capabilitystatement-definitions.html#CapabilityStatement.fhirVersion) - valid published FHIR Version(s) are viewable here - http://hl7.org/fhir/directory.html

(#1) As a FHIR Server I would like Crucible to identify and ensure the Server fhirVersion is actively part of the test cases chosen and are specific to the published version of FHIR being tested. Note: FHIR Version STU 3 Candidate or Ballot currently has available four valid published version(s) - with active systems running 1.4.0, 1.6.0 and now 1.8.0 coming online daily.

(#2) As a FHIR Server I would like Crucible to identify which version of FHIR test cases (suite) were used to test my server. e.g. Dec 6, 2016 build 1.8.0 FHIR STU3 Candidate + Connectathon 14 (San Antonio)

(#3) Please ensure Crucible does NOT publically publish test results where active public servers running a valid version of FHIR are being tested against none compatible published standards - that would represent erroneous reporting by Crucible (could potentially introduce harm to the Organization being reported against). Given our proximity to HIMSS 2017 - a number of Organizations are planning very high profile FHIR announcements (accurate reporting by public test platforms will be critical to the healthcare marketplace, the industry and HL7).

e.g. AEGIS WildFHIR (URL - http://wildfhir.aegis.net/fhir1-6-0) self identifies as a FHIR 1.6.0 implementation, and should only be tested against FHIR STU3 version 1.6.0 Baltimore Connectathon profile. If the Crucible Team updates the public version of Crucible to support/test 1.8.0 - please ensure testing of 1.8.0 occurs only against AEGIS WildFHIR (URL - http://wildfhir.aegis.net/fhir1-8-0) self identifies as a FHIR 1.8.0.

This notes that there are significant and breaking changes being published between the various STU3 versions. This conversation will be posted to Zulip for more feedback

Fatal Error: undefined method `client=' for nil:NilClass

I pulled the most recent updates on master, when through bundle install and ran the tests and started seeing errors that I wasnt getting before.
Fatal Error: undefined method client=' for nil:NilClass`

Previously, these tests were passing fine on 962a643.

Here's some of the output from bundle exec rake crucible:execute[http://localhost:6006/test_stu3dev,ResourceTest,Patient]

I'd love any suggestions on a dependency or something else I may be missing.

ResourceTest_Patient
      PASS x000_read_type_test: 
     ERROR x010_create_new_test: Fatal Error: undefined method `client=' for nil:NilClass
            ----------------------------------------
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/bundler/gems/fhir_client-79229522dcee/lib/fhir_client/sections/crud.rb:168:in `base_create'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/bundler/gems/fhir_client-79229522dcee/lib/fhir_client/sections/crud.rb:123:in `create'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/bundler/gems/fhir_client-79229522dcee/lib/fhir_client/ext/model.rb:73:in `create'
            /Users/rickysahu/Desktop/git/plan_executor/lib/tests/suites/resource_test.rb:95:in `block (2 levels) in <class:ResourceTest>'
            /Users/rickysahu/Desktop/git/plan_executor/lib/tests/base_test.rb:136:in `ignore_client_exception'
            /Users/rickysahu/Desktop/git/plan_executor/lib/tests/suites/resource_test.rb:95:in `block in <class:ResourceTest>'
            /Users/rickysahu/Desktop/git/plan_executor/lib/tests/suites/base_suite.rb:78:in `instance_eval'
            /Users/rickysahu/Desktop/git/plan_executor/lib/tests/suites/base_suite.rb:78:in `block in test'
            /Users/rickysahu/Desktop/git/plan_executor/lib/tests/base_test.rb:89:in `call'
            /Users/rickysahu/Desktop/git/plan_executor/lib/tests/base_test.rb:89:in `execute_test_method'
            /Users/rickysahu/Desktop/git/plan_executor/lib/tests/base_test.rb:74:in `block in execute_test_methods'
            /Users/rickysahu/Desktop/git/plan_executor/lib/tests/base_test.rb:70:in `each'
            /Users/rickysahu/Desktop/git/plan_executor/lib/tests/base_test.rb:70:in `execute_test_methods'
            /Users/rickysahu/Desktop/git/plan_executor/lib/tests/suites/resource_test.rb:25:in `execute'
            lib/tasks/tasks.rake:136:in `execute_test'
            lib/tasks/tasks.rake:98:in `block (3 levels) in <top (required)>'
            /Users/rickysahu/.rvm/rubies/ruby-2.3.3/lib/ruby/2.3.0/benchmark.rb:293:in `measure'
            lib/tasks/tasks.rake:94:in `block (2 levels) in <top (required)>'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/gems/rake-11.3.0/lib/rake/task.rb:248:in `block in execute'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/gems/rake-11.3.0/lib/rake/task.rb:243:in `each'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/gems/rake-11.3.0/lib/rake/task.rb:243:in `execute'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/gems/rake-11.3.0/lib/rake/task.rb:187:in `block in invoke_with_call_chain'
            /Users/rickysahu/.rvm/rubies/ruby-2.3.3/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/gems/rake-11.3.0/lib/rake/task.rb:180:in `invoke_with_call_chain'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/gems/rake-11.3.0/lib/rake/task.rb:173:in `invoke'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/gems/rake-11.3.0/lib/rake/application.rb:152:in `invoke_task'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/gems/rake-11.3.0/lib/rake/application.rb:108:in `block (2 levels) in top_level'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/gems/rake-11.3.0/lib/rake/application.rb:108:in `each'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/gems/rake-11.3.0/lib/rake/application.rb:108:in `block in top_level'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/gems/rake-11.3.0/lib/rake/application.rb:117:in `run_with_threads'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/gems/rake-11.3.0/lib/rake/application.rb:102:in `top_level'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/gems/rake-11.3.0/lib/rake/application.rb:80:in `block in run'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/gems/rake-11.3.0/lib/rake/application.rb:178:in `standard_exception_handling'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/gems/rake-11.3.0/lib/rake/application.rb:77:in `run'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/gems/rake-11.3.0/exe/rake:27:in `<top (required)>'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/bin/rake:22:in `load'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/bin/rake:22:in `<main>'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/bin/ruby_executable_hooks:15:in `eval'
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/bin/ruby_executable_hooks:15:in `<main>'
            ----------------------------------------
     ERROR x012_conditional_create_(no_matches)_test: Fatal Error: undefined method `client=' for nil:NilClass
            ----------------------------------------
            /Users/rickysahu/.rvm/gems/ruby-2.3.3/bundler/gems/fhir_client-79229522dcee/lib/fhir_client/sections/crud.rb:168:in `base_create'

Potential Allergy intolerance invariant issue

Server: http://fhirtest.uhn.ca/baseDstu3
X010: AllergyIntolerance: Create New

Message
error: invariant: AllergyIntolerance.clinicalStatus SHALL NOT be present if verification Status is entered-in-error [verificationStatus!='entered-in-error' or clinicalStatus.empty()],warning: code-invalid: The value provided ('es-ES') is not in the value set http://hl7.org/fhir/ValueSet/languages (http://hl7.org/fhir/ValueSet/languages, and a code should come from this value set unless it has no suitable code),warning: code-invalid: Display Name for SNO must be one of 'Serial Number',warning: code-invalid: No code provided, and a code should be provided from the value set http://hl7.org/fhir/ValueSet/identifier-type (http://hl7.org/fhir/ValueSet/identifier-type,warning: code-invalid: No code provided, and a code should be provided from the value set http://hl7.org/fhir/ValueSet/identifier-type (http://hl7.org/fhir/ValueSet/identifier-type,warning: code-invalid: No code provided, and a code should be provided from the value set http://hl7.org/fhir/ValueSet/identifier-type (http://hl7.org/fhir/ValueSet/identifier-type

X010_Questionnaire intermittently fails

The resource generated in X010_Questionnaire will intermittently be invalid. The failure stems from the questionnaire being generated with an item with type "question".

Below is an example of the generated incorrect resource:

{
    "meta": {
        "tag": [
            {
                "system": "http://projectcrucible.org",
                "code": "testdata"
            }
        ]
    },
    "language": "de-CH",
    "text": {
        "status": "generated",
        "div": "<div>T6tPj1uc3d4ualMAfivP7g==</div>"
    },
    "url": "http://projectcrucible.org/AUSfhpLj/8lZR0yBd75FWQ==",
    "identifier": [
        {
            "use": "temp",
            "type": {
                "coding": [
                    {
                        "system": "http://hl7.org/fhir/identifier-type",
                        "code": "SNO",
                        "display": "Serial Number",
                        "userSelected": false
                    }
                ],
                "text": "iC4K65SDmRxEHLri5KiqNQ=="
            },
            "system": "http://projectcrucible.org/qqMfks7XhJ4D/Json+1h1w==",
            "value": "3ARJOnXnQIxwzQ6eNluOjg==",
            "period": {
                "start": "2018-02-06T21:51:51.410Z",
                "end": "2018-02-06T21:51:51.410Z"
            },
            "assigner": {
                "identifier": {
                    "use": "temp",
                    "system": "http://projectcrucible.org/hnw1ODASgp/ZcXiQWhG/bQ==",
                    "value": "FQicUYUeaqUJvZEb1Qab/Q=="
                },
                "display": "Organization 7wLywYCNnl0ps7PhTxP3+g=="
            }
        }
    ],
    "name": "RtB5GteDJhbgvYWVCL5TyQ==",
    "title": "6sIoxw633leSxlTDvx8edA==",
    "status": "draft",
    "experimental": false,
    "date": "2018-02-06T21:51:51.410Z",
    "publisher": "wg7FJ3D7CXDpKZSwAeeDbQ==",
    "description": "y1ooCDcbinl1PlIEYv9qPg==",
    "purpose": "w0Jyu9/EgV/9l8REwJztFw==",
    "approvalDate": "2018-02-06",
    "lastReviewDate": "2018-02-06",
    "effectivePeriod": {
        "start": "2018-02-06T21:51:51.410Z",
        "end": "2018-02-06T21:51:51.410Z"
    },
    "useContext": [
        {
            "code": {
                "system": "http://hl7.org/fhir/usage-context-type",
                "code": "task",
                "display": "Workflow Task",
                "userSelected": false
            },
            "valueRange": {
                "low": {
                    "value": 0.7393112961738725,
                    "comparator": "<=",
                    "unit": "lJFJlxK8XeQbmwg9kmgVGQ==",
                    "system": "http://projectcrucible.org/H+PpWB6xCWVGt9w3eirUDg==",
                    "code": "WNGVe8IzIPlxhhLdryY9EA=="
                },
                "high": {
                    "value": 0.7411792367291858,
                    "comparator": "<",
                    "unit": "nVt8136gKCTVJVdGTxmIow==",
                    "system": "http://projectcrucible.org/S8ZuuIhnk47ohU/So0bbnA==",
                    "code": "s9OT/Y4V3gZfbSBWC9oX6A=="
                }
            }
        }
    ],
    "jurisdiction": [
        {
            "coding": [
                {
                    "system": "https://www.usps.com/",
                    "code": "TX",
                    "userSelected": true
                }
            ],
            "text": "JxHb9AFk1UwMZdYmmitjzg=="
        }
    ],
    "contact": [
        {
            "name": "BgH6MSZIKn8nIhpNYlMIpg==",
            "telecom": [
                {
                    "system": "pager",
                    "value": "Kw7H0esOYtXHMKzE1Fqa1A==",
                    "use": "work",
                    "rank": 50,
                    "period": {
                        "start": "2018-02-06T21:51:51.420Z",
                        "end": "2018-02-06T21:51:51.420Z"
                    }
                }
            ]
        }
    ],
    "copyright": "8yyDnk1wu9+Dy2xRh5S6Qg==",
    "code": [
        {
            "system": "http://loinc.org",
            "code": "10389-5",
            "display": "Blood product.other [Type]",
            "userSelected": true
        }
    ],
    "subjectType": [
        "PaymentNotice"
    ],
    "item": [
        {
            "linkId": "4GbGnce5EwG8Oaj1UQYT4w==",
            "definition": "http://projectcrucible.org/QAUcRKYhSTDhWfWj5s5A0w==",
            "code": [
                {
                    "system": "http://loinc.org",
                    "code": "10190-7",
                    "display": "Mental status Narrative",
                    "userSelected": false
                }
            ],
            "prefix": "SeXCeAzEQua+LjnTHqEtPg==",
            "text": "Tg+i+wEyZ5NhLqlDC7T9mA==",
            "type": "question",
            "enableWhen": [
                {
                    "question": "xYj55jr4/HoyVrwCWk+nXQ==",
                    "answerCoding": {
                        "system": "http://snomed.info/sct",
                        "userSelected": false
                    }
                }
            ],
            "required": true,
            "repeats": true,
            "readOnly": false,
            "initialReference": {
                "identifier": {
                    "use": "secondary",
                    "system": "http://projectcrucible.org/RTZYuqmbDKiShE4h4uOIug==",
                    "value": "La2CcHOmXWXwckyPwsE9Gw=="
                },
                "display": "Resource +0vHiUuyNi+iXEeeuiel/Q=="
            }
        }
    ],
    "resourceType": "Questionnaire"
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.