The AWS FHIR Server allows any search parameter name and returns an empty list. It should return an error.

For an example error, see http://hapi.fhir.org/baseR4/Observation?foo=bar

{{HTTP 400 Bad Request}}

"diagnostics": "Unknown search parameter "foo" for resource type "[_id, _language, based-on, category, code, code-value-concept, code-value-date, code-value-quantity, code-value-string, combo-code, combo-code-value-concept, combo-code-value-quantity, combo-data-absent-reason, combo-value-concept, combo-value-quantity, component-code, component-code-value-concept, component-code-value-quantity, component-data-absent-reason, component-value-concept, component-value-quantity, data-absent-reason, date, derived-from, device, encounter, focus, has-member, identifier, method, organization, part-of, patient, performer, specimen, status, subject, value-concept, value-date, value-quantity, value-string]". Valid search parameters for this search are: {2}"

AC:

AWS FHIR Server returns 400 and an appropriate error message for invalid search params instead of an empty list.

Currently our search is completed by using 'keyword'. This leaves much to be desired and required from the FHIR spec. Such as:

• Date range search (#61)
• Greater/less than quantity search (#58)
• Support for parameters for all resources
  • Sorting on any parameter is not supported yet
• Search result parameters (ie sorting results)
  • Sort available by date type field only (#60)

FHIR spec for context:

• https://www.hl7.org/fhir/http.html#search
• https://www.hl7.org/fhir/search.html

A/C:

• Add functionality
• Update integration test

Hi AWS team,
Can you give us a status update on custom search parameters in ES please? I can see that some of this has been done in the feat-search-igs branch but that work looks to have stalled for the moment.
Many thanks,
Allan

The HAPI FHIR Server lists searchable fields in the capability statement. For example, the Test HAPI FHIR Server Capability Statement lists all of these searchable fields in Patient:

"searchParam": [ {"name": "_language","type": "string","documentation": "The language of the resource"}, {"name": "birthdate","type": "date","documentation": "The patient's date of birth"}, {"name": "deceased","type": "token","documentation": "This patient has been marked as deceased, or as a death date entered"}, {"name": "address-state","type": "string","documentation": "A state specified in an address"}, {"name": "gender","type": "token","documentation": "Gender of the patient"}, {"name": "link","type": "reference","documentation": "All patients linked to the given patient"}, {"name": "language","type": "token","documentation": "Language code (irrespective of use value)"}, {"name": "mothersMaidenName","type": "string","documentation": "Search by parent reference."}, {"name": "address-country","type": "string","documentation": "A country specified in an address"}, {"name": "death-date","type": "date","documentation": "The date of death has been provided and satisfies this search value"}, {"name": "phonetic","type": "string","documentation": "A portion of either family or given name using some kind of phonetic matching algorithm"}, {"name": "organizationIdentifier","type": "token","documentation": "Search for the identifier of the managingOrganization III"}, {"name": "SearchByMsgDate","type": "date","documentation": "Search by patient for a MSH.7 agreement, which is stored in Patient Resource"}, {"name": "telecom","type": "token","documentation": "The value in any kind of telecom details of the patient"}, {"name": "address-city","type": "string","documentation": "A city specified in an address"}, {"name": "email","type": "token","documentation": "A value in an email contact"}, {"name": "given","type": "string","documentation": "A portion of the given name of the patient"}, {"name": "identifier","type": "token","documentation": "A patient identifier"}, {"name": "gdaddresstype","type": "token","documentation": "SearchParameter for the freg type of gd-Addresses"}, {"name": "address","type": "string","documentation": "A server defined search that may match any of the string fields in the Address, including line, city, district, state, country, postalCode, and/or text"}, {"name": "race","type": "token","documentation": "Returns patients with a race extension matching the specified code."}, {"name": "birthplace-city","type": "token","documentation": "Search by Birthplace/City"}, {"name": "general-practitioner","type": "reference","documentation": "Patient's nominated general practitioner, not the organization that manages the record"}, {"name": "addresscontenttype","type": "token","documentation": "SearchParameter for the content type of gd-Addresses"}, {"name": "sex","type": "string","documentation": "Search for Patients by their biological sex"}, {"name": "active","type": "token","documentation": "Whether the patient record is active"}, {"name": "middlename","type": "string","documentation": "SearchParameter for the Norwegian middlename extension http://hl7.no/fhir/StructureDefinition/no-basis-middlename"}, {"name": "gender2","type": "token","documentation": "Search by url for a participation agreement, which is stored in a DocumentReference"}, {"name": "SearchByDate","type": "date","documentation": "Search by patient for a MSH.7 agreement, which is stored in a any resource"}, {"name": "address-postalcode","type": "string","documentation": "A postalCode specified in an address"}, {"name": "FinancialTransactionDate","type": "date","documentation": "Search by patient for a MSH.7 agreement, which is stored in a any resource"}, {"name": "verfall","type": "date","documentation": "Bla blablbla."}, {"name": "phone","type": "token","documentation": "A value in a phone contact"}, {"name": "commentExtension","type": "string","documentation": "To search for value of comment-extension"}, {"name": "organization","type": "reference","documentation": "The organization that is the custodian of the patient record"}, {"name": "address-use","type": "token","documentation": "A use code specified in an address"}, {"name": "name","type": "string","documentation": "A server defined search that may match any of the string fields in the HumanName, including family, give, prefix, suffix, suffix, and/or text"}, {"name": "nachname","type": "string","documentation": "Search for \"Nachname\" of patient"}, {"name": "birthtime","type": "date","documentation": "Search by url for a participation agreement, which is stored in a DocumentReference"}, {"name": "_id","type": "token","documentation": "The ID of the resource"}, {"name": "family","type": "string","documentation": "A portion of the family name of the patient"}, {"name": "eyecolour","type": "token"} ]}

The AWS FHIR Server specifies searchParam in each Resource like this:

"searchParam": [{"name": "ALL","type": "composite","documentation": "Support all fields."}]

This cannot be properly consumed by Smart on FHIR Apps.

AC:

List all the search params in the AWS FHIR Server Capability statement.

When generating the search for a String field that uses the ":contains" modifier, because a wildcard query is generated, should these fields should be appended with the ".keyword" suffix? Since a wildcard query is term-level.

Looking at QueryBuilder/typeQueries/stringQuery.ts
It's appended for the ":exact" modifier
const keywordSuffix = modifier === 'exact' ? '.keyword' : '';
But not for the ":contains" modifier
const queries = fields.map((field) => ({ wildcard: { [field]: { value: ${value.toLowerCase()}, }, }, }));

After testing with what is generated for the "address:contains" and running it against some test data (with mappings auto generated), data was only returned when I added in the .keyword field, for example when using the "address:contains" output found in the test file.

I suppose if your mappings had the all of the address fields were of type keyword you would not need it, if I understand this correctly. However I since I don't see the "uses_mappings" check in this portion of the codebase I'm unsure if this is intended.

https://www.elastic.co/guide/en/elasticsearch/reference/8.5/term-level-queries.html

Please let me know what you think.
Thanks!

According to the Aegis Touchstone test suites, searching by an invalid search parameter should return an HTTP 400

Currently we only support ResourceType search. Meaning queries like fhir.api/Patient?name=bob work. This task is to enable searching across the all ResourceType, so queries like fhir.api?name=bob will find Bob the Patient and Bob the Practitioner.

FHIR spec for context: https://www.hl7.org/fhir/search.html

A/C:

• Add functionality
• Update integration test
• Update deployment package

Describe the bug
When a request results in paging, if you run the same request multiple times the results returned in the set are not always the same.

To Reproduce
Steps to reproduce the behavior:

1. Call request something like {{API_URL}}/tenant/7db7dfb5-2f73-4b96-bcf1-131410518850/Observation?patient=Patient/8743177f-ad8b-4dba-b05d-933419da86df&category=vital-signs&_getpagesoffset=0&_count=20
   In our case it was returning 20 of 47.

Expected behavior
Results returned should be the same when making the same call.

Additional context
If we add a _sort=date or _sort=_lastUpdated to our request does seem to resolve the issue.I have also attached the results of both runs
Run1.txt
.
Run2.txt

Describe the bug
Currently the include/revinclude parameters are not strictly validated, they are silently dropped during the query execution if they are invalid

To Reproduce
Steps to reproduce the behavior:
Perform a DocuumentReference search with invalid parameters: /DocumentReference?_include=DocumentReference:sourcePatientInfo

Expected behavior
Validation to be performed on the parameters instead of being silently dropped

Bug Report:

Currently the code that translates a FWoA SearchFilter into an Elasticsearch query filter honor the operator for multivalue filters, but has no way to specify the operator for multi-filter match queries.

For example, when calling getPatientByMid('foo'), the following SearchFilter(s) are passed to ElasticSearchService.buildElasticSearchFilter():

Which produces a query filter that looks like this (notice the absence of the joining operators):

And results in an ES query that looks like this (notice the presence of the OR operators):

{
  "from": 0,
  "size": 20,
  "query": {
    "bool": {
      "filter": [
        {
          "match": {
            "documentStatus": {
              "query": "AVAILABLE",
              "operator": "OR",
              "prefix_length": 0,
              "max_expansions": 50,
              "fuzzy_transpositions": true,
              "lenient": false,
              "zero_terms_query": "NONE",
              "auto_generate_synonyms_phrase_query": true,
              "boost": 1
            }
          }
        },
        {
          "match": {
            "__internalMeta.contexts.identifiers.value": {
              "query": "foo",
              "operator": "OR",
              "prefix_length": 0,
              "max_expansions": 50,
              "fuzzy_transpositions": true,
              "lenient": false,
              "zero_terms_query": "NONE",
              "auto_generate_synonyms_phrase_query": true,
              "boost": 1
            }
          }
        }
      ],
      "adjust_pure_negative": true,
      "boost": 1
    }
  }
}

This occurs because when an operation is omitted in a search query, ES defaults to an OR operator.

See:

The __internalMeta.contexts.identifiers.value field is being analyzed by ES with an analyzer that is splitting up the value by common delimiters (i.e spaces, dashes, underscores). In the case of Cambia’s business identifiers (i.e. foo-bar-baz) this results in search for all documents that match __internalMeta.contexts.identifiers.value = foo OR bar OR baz

Resolution:

The ElasticSearchService.buildElasticSearchFilterPart() method needs to be modified so that it can honor the operator that was passed in via the SearchFilter objects. However we need to confirm with AWS if these operators were intended for multi-value match queries OR multi-match queries filter.

We need change the analyzer used for this field so that the field value is not split along common delimiters.

Is your feature request related to a problem? Please describe.
Here is an example of the query we are trying to execute.
{{API_URL}}/Encounter?status=in-progress&patient:Patient.name=John&_include:iterate=Encounter:patient
Basically we are trying to get all encounters that match the patients name in the Patient resource.

Describe the solution you'd like
Query to return the correct records

Additional context

1. The modifier to be explicit about the intended reference type (the “patient:Patient” part)
   https://www.hl7.org/fhir/search.html#reference
2. The use of chained parameters (the “.name” part)
   https://www.hl7.org/fhir/search.html#chaining

Is your feature request related to a problem? Please describe.
In several cases, client applications can send unsupported or unknown search parameters. Listed below are a couple of possible reasons for this to occur.

1. Different implementations of FHIR: Different servers may support different search parameters - this could either be due to a difference in interpretation of the standard or a custom requirement. Client applications that are designed to be implementation agnostic may hence send search parameters that are not supported expecting the server to ignore any unknown parameters.
2. Additional parameters beyond client's control: Various HTTP stacks and proxies may add parameters that are not under the control of the client application.

While the FHIR implementation guide expects the server to ignore any unknown or unsupported parameters, using either a strict validation or no validation of parameters presents obstacles to onboarding new client applications.

This becomes a more pressing problem needing custom workarounds such as proxies to filter or modify parameters, if the fix for issue reported in #167 always performs strict validation,

Describe the solution you'd like
The recommended method to address this would be support for the "prefer" header (see Unknown and unsupported parameters) on Search- Handling Errors. The server could have a default configuration of prefer: handling = strict while a specific client can override with prefer: handling = lenient.

This would allow control (either by the client application or a proxy) on the validation that is expected for the specific client.

Describe alternatives you've considered

1. Have a proxy modify the search parameters to ensure only valid/supported search parameters are passed to the FHIR server.

Additional context
Add any other context or screenshots about the feature request here.

Updates from previous conversations

Search Resource by multiple identifiers does not work.

• Example: https:///Patient/?identifier.system=AccountNumber&identifier.value=21334&identifier.system=mrn&identifier.value=1234
• HAPI FHIR sample that works with multiple identifier search: http://hapi.fhir.org/baseR4/Patient?identifier=http://hospital.smarthealthit.org|2906be73-5a0b-438b-91adffca6692692c&identifier=http://hl7.org/fhir/sid/us-ssn|999-45-7927&_pretty=true
• FHIR spec also mentions about using pipeline (|) character with code and system search. Fireworks doesn’t seem to be searching with pipeline
  • https://www.hl7.org/fhir/search.html
  • A parameter may refer to a non-existent resource e.g. GET [base]/Observation?patient.identifier=http://example.com/fhir/identifier/mrn|123456
• Range date search is not possible
  • https:///AllergyIntolerance?onsetDateTime=gt2001-10-20

FHIR Works on AWS MVP release on 28th August 2020 included only basic search functionality, with a view to gaining customer feedback to identify what enhancements should be prioritized. Near-term roadmap functionality for search will support _include and _revinclude parameters.

Is your feature request related to a problem? Please describe.
Inferno search by patient tests fail

Describe the solution you'd like
Both forms of the search to return the same results

Describe alternatives you've considered

Additional context
US Core search guidance states servers shall support both forms of the reference search

http://hl7.org/fhir/us/core/general-guidance.html#search-syntax

What's on your mind?

1. For search requests, is it safe to assume the closest match is sorted at the beginning of the result array? Is there a sorting order for searches?
2. When will the search.score feature be available in search?

Versions (please complete the following information):

• 3.4.0

Describe the bug
When using _include with location it is not returning the location resource.

To Reproduce
Steps to reproduce the behavior:
You will need a patient with an encounter that contains a location.
Then Run:
/Encounter?_include:iterate=Encounter:patient&_include:iterate=Encounter:location&patient=Patient/2e3bc8ff-8a25-46ee-9cd7-c87b05cacb5d

Expected behavior
I would expect it to also return the location resource as part of the _include.

Additional context
It does return encounter and patient resource using the _include.

Describe the bug
For API request GET DetectedIssue?patient:Patient.name=aaa . I got all DetectedIssue Resource with patient fields

To Reproduce
Steps to reproduce the behavior:

1. Git clone smart-mainline branch git clone -b smart-mainline [email protected]:awslabs/fhir-works-on-aws-deployment.git
2. Deploy AWS FHIR Works
3. Create Patient and DetectedIssue Resources.

POST /Patient
{
    "resourceType": "Patient",
    "name": [
        {
            "family": "Chalmers"
        }
    ]
}

POST /DetectedIssue
{
    "resourceType": "DetectedIssue",
    "patient": {
        "reference": "Patient/{{Patient ID from the above API request}}"
    }
}

4. Search DetectedIssue with Patient with any name.

GET /DetectedIssue?patient:Patient.name=fdsafds

Search DetectedIssue with Patient with empty name

GET /DetectedIssue?patient:Patient.name=

Expected behavior
Empty list of DetectedIssue is returned

Actual behavior
A DetectedIssue is returned

Describe the bug
I would like to query QuestionnaireResponses based on the value of the "questionnaire" field, where the value of the questionnaire field points to a url like https://my.questionnaires.com/questionnaire:qr_type:1.0.0.

The goal is to get a list of QuestionnaireResponses for a specific Questionnaire.

My request looks like this:

GET {{FHIR_API_URL}}/fhir/{{TENANT_ID}}/QuestionnaireResponse?questionnaire=https://my.questionnaires.com/questionnaire:qr_type:1.0.0

where {{FHIR_API_URL}}/fhir points to my AWS fhir works instance.

This query results in an empty searchset, even when I URL encode the request.

To Reproduce
Steps to reproduce the behavior:

1. Have a running fhir works instance.
2. Upload test QuestionnaireResponse with some specific value for the questionnaire.
3. Make GET request for QuestionnaireResponses referencing specific questionnaire value.

Expected behavior
All QuestionnaireResponses of specific Questionnaire value are returned.

Screenshots
POST QuestionnaireResponse with questionnaire = https://my.questionnaires.com/questionnaire:qr_type:1.0.0. Status 201 Created.

Query of QuestionnaireResponse with questionnaire value https://my.questionnaires.com/questionnaire:qr_type:1.0.0 results in empty searchset. Status 200 OK.

Versions (please complete the following information):

• Release Version installed: v3.12.0

Additional context
Add any other context about the problem here.

Is your feature request related to a problem? Please describe.

• Right now the elastic search schema mapping is created during dynamo db to elastic sync using one index per resource type. If the index for the current resource type isn't found, one is generated using a basic mapping that's not specific to the resource type.
• Dynamic mapping takes care of creating all the other attribute type mappings whenever the first resource is being loaded into the index. This leads to an unpredictable mapping if the imported data isn't all looking exactly the same.
  It also leads to a bloat since a lot of attributes are being mapped that are never used in search.

Describe the solution you'd like

• Use the IG compilation step to generate the schema mapping based on SearchParameters
• Disable dynamic mapping

Describe the bug
When I try to make an extension field searchable, I got an error "Unexpected "(". Instead, I was expecting to see one of the following:" Here is the official example regarding SearchParameter on Extension

To Reproduce
Steps to reproduce the behavior:

1. Create file /implementationGuides/example/SearchParameter.json with official example
2. Create /impementationGuides/example/.index.json with content:

{
  "index-version": 1,
  "files": [
    {
      "filename": "SearchParameter.json",
      "resourceType": "SearchParameter",
      "id": "CustomSearchParameter",
      "url": "http://fhir.example.com/SearchParameter/CustomSearchParameter",
      "version": "1.0.0",
      "type": "reference"
    }
  ]
}

3. run command yarn run compile-igs
4. See error

Expected behavior
Compile igs without any errors.

Versions:

• v2.2.0-smart

Attaching the Q&A from a separate conversation:

1. What is the mechanism for Elastic Search (ES) failover ? Because search operation is tightly coupled with ES.

A: This has not yet been tested; we will put this on the backlog to perform. The expected behavior is that the Lambda function triggered by DynamoDB Streams when the write to DynamoDB occurs will fail during the ES failover; we would like to address this more gracefully.

2. What is the mechanism of Elastic Search sync (Dynamo Stream Sync) after failover.

A: This has not yet been tested; we will put this on the backlog to perform. The expected behavior is that the Lambda function triggered by DynamoDB Streams when the write to DynamoDB occurs will be able to write to the cluster as before.

Describe the bug
When running a revInclude for Provenance on a MedicationRequest Resource it is only returning 10 provenance records when there are more than that tied to it.

To Reproduce
Example:
https://{{API_URL}}/tenant/7db7dfb5-2f73-4b96-bcf1-131410518850/MedicationRequest?patient=Patient/cc8d11df-2df0-4c0a-b478-8169a9228ae3&intent=order&_revinclude=Provenance:target&_count=20

Expected behavior
The expectation is that it would return all provenance records for the MedicationRequest resource.

What's on your mind?
We're performing load testing on some of our custom observation resource types and seeing very unstable behavior from elastic search when performing queries on Observation.code. Specifically, we're seeing our response times balloon into taking between 30-60 seconds and in some cases have caused the data nodes to go offline and 504 responses are returned from the elastic search load balancer for periods of time. Looking more into the issue, we were able to narrow down the unstable behavior to the use of multi_ match in the query when specifying an Observation.code value. Adding a system to the query helped marginally but it was until we went in and removed all of the token types besides CodeableConcept from the fields list did our load tests stabilize and we started getting a response time and p99 values all under 1 second.

There's a comment in the code section about needing the StructureDefinitions here to be able to prune the fields down or even use a match instead of multi_match. Has there been any new developments here? https://github.com/awslabs/fhir-works-on-aws-search-es/blob/mainline/src/QueryBuilder/typeQueries/tokenQuery.ts#L29

I'm seeing where the de-Observation.code StructureDefinition in the FHIR r4 specs and it is defining the field as a CodeableConcept. Am I correct in thinking the solution would be to add another field to the CompiledSearchParam types and regenerating the schema file to get the valid token data types into the tokenQuery context?

Versions (please complete the following information):
v2.5.1-smart prod stage

Is your feature request related to a problem? Please describe.
We have third party apis that retrieve data from our FHIR server and are expecting the "data" section of the JSON to contain the object. Currently a pre-signed URL is returned with in the "presignedGetUrl".
Example:
{
"meta": {
"lastUpdated": "2022-03-01T17:57:13.228583Z",
"versionId": "1"
},
"contentType": "application/xml",
"resourceType": "Binary",
"data": "",
"id": "af227b54-f2d9-4eb5-821c-18614af7ab12",
"presignedGetUrl": "Generated URL"
}
Describe the solution you'd like
We would like to be able to choose which section gets populated. For the object to actually be returned with in the data section.

Describe alternatives you've considered
Currently we are evaluating whether we can adjust these sections using a proxy after returning but before sending it to the third parties.

Additional context

We are attempting to run the fhir-service lambda locally with the environment variable IS_OFFLINE=true, but while still communicating with real DynamoDB and ElasticSearch instances in the us-east-1 AWS region.

When the IS_OFFLINE environment variable is set, the AWS Javascript SDK is reconfigured to use the (hard-coded) us-west-2 region. This region should also be detected from environment variables so that it can be dynamically configured. This would allow a local instance of the fhir-service server to communicate with AWS resource in the region of choice.

Version: v2.0.1

Is your feature request related to a problem? Please describe.
We are hitting situations where the max chained result size is exceeding 100 and providing more precise queries isn't reasonable for the user

Describe the solution you'd like
Allow the max chained params result to be set through an environment variable

Describe alternatives you've considered
We are researching possibly breaking up the search into multiple requests.

Additional context
One example is searching for an encounter by a patient name where the Last Name has more that 100 results:
Encounter?patient:Patient.name=LastName&status=planned,arrived,triaged,in-progress,onleave,finished&_include:iterate=Encounter:patient

Describe the bug
Some chained search parameter searches are erroring with '...points to multiple resource types, please specify'. FHIR specifies that chained params must resolve to exactly one resourceType but the currently the code only looks at the target array when deciding if chained parameters are ambiguous

To Reproduce
Steps to reproduce the behavior:
Perform a DocumentReference search request: /DocumentReference?patient.identifier=2.16.840.1.113883.3.1579|8889154591540

Expected behavior
a search against the DocumentReference patient identifier

Additional context
The search param definition has "target" : ["Patient", "Group"] but its expression is “DocumentReference.subject.where(resolve() is Patient)”. Logically this means that the search parameter can only match Patient

Hi AWS team,
We are seeing lots of Cloudwatch log entries from the search module when doing routine FHIR operations:

2020-10-12T11:04:46.974Z 767c85a9-9a61-4adf-8ce7-eb57dec7d1c2 INFO Iterative inclusion search starts
2020-10-12T11:04:46.975Z 767c85a9-9a61-4adf-8ce7-eb57dec7d1c2 INFO Iteration 0 found zero results. Stopping

Looks like these need disabling or setting debug-only?

Many thanks,
Allan

Searching fields based on prefix or modifiers is not working.

Ex - GET [base]/Patient?birthDate=ne2013-01-14

https://www.hl7.org/fhir/search.html

Prefixes

Modifiers:

We would like to see the contains string search added:

[base]/Patient?given:contains=eve

Any patients with a name with a given part containing "eve" at any position. This would include patients with the given name "Eve", "Evelyn", and also "Severine".

Of course it would be nice to support all 3 string search scenarios mentioned in 3.1.1.4.8:

• [base]/Patient?given=eve
  Any patients with a name containing a given part with "eve" at the start of the name. This would include patients with the given name "Eve", "Evelyn".

• [base]/Patient?given:contains=eve
  Any patients with a name with a given part containing "eve" at any position. This would include patients with the given name "Eve", "Evelyn", and also "Severine".

• [base]/Patient?given:exact=Eve
  Any patients with a name with a given part that is exactly "Eve". Note: This would not include patients with the given name "eve" or "EVE".

Originally posted by @medhost-bfindeisen in #7 (comment)

Describe the bug
When searching by multiple identifiers it is not returning multiple resources

To Reproduce
Steps to reproduce the behavior:
Example:
/Observation?patient=Patient/1a6e4b54-b756-49a6-ae7e-5f7ec27d5896&category=vital-signs&code=8867-4,9279-1,85354-9

Expected behavior
Should return multiple resources based on the codes.

Additional context
http://hl7.org/fhir/R4/observation-vitalsigns.html