overture-stack / maestro Goto Github PK

Read up on:

• Aliasing
• Multi-Index behaviour
• Nesting
• Text search /Analyzers

Update analysis_centric mapping based on this PR: https://github.com/icgc-argo/argo-metadata-schemas/pull/27/files

There is one issue:

Right now Maestro only supports removing ES documents when an analysis is requested to be removed( either by using rest endpoint /index/repository/{repositoryCode}/study/{studyId}/analysis/{analysisId},
or by listening to kafka topics).

Requirement:

1. Extend and refactor DefaultIndexer - removeAnalysis method to include removing either file documents or analysis documents upon removing analysis.

Install kafka on QA env (ansible)

cc. @lepsalex and @Buwujiu

• Design only.
• Include ARGO requirements when designing the index.

• Java JDK
• Maven project with submodules

the current maestro is outdated on argo-qa, update the values in argo-infra
overture-qa
and the jenkins task

Maestro needs to be updated to reflect fields changes made in overture-stack/SONG#573.

Implement a POST endpoint /index/analysis to analysis_centric_index.

Users should have the ability to define indexing rules from SONG.
Rules should be based on any field or combination of fields found in SONG. Eg, this could be by:

Study
Analysis Type
Data
Rules for configuration should be based on an INCLUDE or EXCLUDE basis. Eg, this could be by:

INCLUDE in index only records with status = published
EXCLUDE records with analysis_type = vcf

• Currently maestro is configured to work with the relational song database
• What needs to be done to enable it to work with dynamic song?
• Develop a technical specification to update maestro

What happens when 2 records do not match in 2 repositories?

Expected Behaviour

If the data of the same Analysis-ID does not match in two different song repositories, then:

1. The analysis document should be deleted from the index.
2. An alert for data-conflict-resolution should be sent to a message queue recording the repository and both analysis ids (is there a unique SONG record id for each of these as well?)

Critera to delete a record -
All fields must have the exact same value EXCEPT for date-time values like created_date or updated_date OR the info fields. If they have a single field that does not match, then delete the record.

ability to send failing messages to a retry queue to be looked at later by admins

V1: https://github.com/icgc-argo/argo-models/blob/master/es-mappings/file-centric.mapping.yaml

• Review mappings and assignments

Update file_centric mapping based on this PR: https://github.com/icgc-argo/argo-metadata-schemas/pull/27/files

This is to update the entity ids, which were normalized, for example:
id --> donor_id
id ---> specimen_id
id ---> sample_id

• Update the mapping file to the mapping above
• Once tests fail, update the code and fixtures.

Problem: when indexing multiple indices, endpoint index/repo return only one index result, even both indices have been successfully indexed:

Expected: should return index results for both indices.

• Make the existing POST endpoint /index/repository/ to index an entire repository to both analysis_centric and file_centric.
• Add a generic method in Indexer API to allow repository multi-index.

Rest Endpoints to:

• see current exclusion rules
• update exclusion rules. This endpoint will write back to the config file

List of fields that need to be changed:

• all submitter id fields need to be "submittter_id"
• add gender
• add new specimen fields
• include analysis type
• index repository base url

mapping updates in progress with @rosibaj and @blabadi are here:
icgc-argo/argo-metadata-schemas#18

• only the maestro mapping is relevent
• review the difference for missing fields

Expected Outcome

We can see in ES (whether thru Kibana or another method) Maestro indexing from a song a file_repository index with the new mapping.

Update the analysis_centric mapping to correctly reflect the Song base schema. Some changes include:

• all submitter id fields need to be "submittter_id"
• add gender
• add new specimen fields
• include analysis type
• index repository base url

https://github.com/icgc-argo/argo-metadata-schemas/blob/master/mappings/maestro_analysis_centric.json

Expected Outcome

We can see in ES (whether thru Kibana or another method) Maestro indexing from a song a analysis_repository index with the new mapping.

Maestro right now is target toward indexing a single index, file_centric. This means that many of the functions and classed are not prepared to work within the scope of a second index type - they are only programmed to work with a file_centric view.

Thus, we need to adjust the code in maestro to be able to respond to a multi-index structure.

• refactor DefaultIndexer to remove the duplication of code
• refactor FileCentricElasticSearchAdapter + AnalysisCentricElasticSearchAdapter to be combined into a single adapter

• Make any changes to make very generic
• Create the final mapping based on discussions

• deploy Jenkins to collab from configurations
• Write a Kubernetes deploy scripts

add ability for Maestro to index single analysis

• Please delete EPPIC-CA from the current db, so I can use this for live demo tomorrow

Changes needed for Maestro:

Background:
The dynamic schema portion in SONG is called data, maestro needs to read thedata portion in order to index analyses, studies, and repositories.

experiment is a required field, other fields under data are optional depending on the schema.
Example, the data field could be:

"data" : {
  "experiment" : {  "myField1": "" },
  "myField2" : {.....},
  "myField3" {......},
  }
}

To do:

• update FileCentricAnalysis model to use type : {name, version}
• update FileCentricAnalysis model, by replacing experiment field with data which will include the dynamic data portion
• update FileCentricDocumentConverter.java . getExperiement method, to be getData method

Update Indexer API to provide a generic way of indexing both file and analysis centric indices.

maestro part of #407 is song

• add to a vm for review

This is to expand upon the work that has been done for conflict resolution see : #25
the goal of this ticket is to remove the deletion step of the conflict resolution and refuse the new conflicting document but notify on failure.

that's based on the discussion that happened with @christinayung we want this as MVP to avoid unfolding the work needed in song to handle consistency (decide and check primary file owner, replication process consistency etc). however that to be handled and planned for later on since inconsistency is an issue

• We need to option to delete analysis
• Deletion is based on the Analysis (file) level based on SUPPRESSION in SONG

We need to update these classes to update the logic to handle file_centric and analysis_centric. Currently there one topic for all. The event processing logic needs to updated to process both file_centric and analysis_centric outputs.

• refactor SongAnalysisStreamListener to use indexAnalysis()
• refactor IndexingMessagesStreamListener

Ability to index a whole song repository in one shot

• Set up the QA cluster correctly with volumes
• mimic the prod setup so that we will be ready when the time comes
• not containerized
• Dusan has an ansible playbook ready, 3 vms

• index form multiple songs

When maestro gets song study TEST-PR from song-qa and dev, following exception is throen: springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144.

Cause: TEST-PR exceeds the default spring codec max in memory size which is 256KB.

Slack it .. for conflict issues and other critical errors.

Webcall to slack and refactor some authenicatino schtuff

Users should have the ability to define indexing rules.

• For MVP, go with a config file
• Exclude basis for Study, Donor, File and Sample
• For every Analysis, do a check against the rules
• check to see if it matched any of the excluded ids

Users should be able to index discreet units of data, not the whole database each time, configured by an event system

Indexer should be able to listen to multiple event streams

• Decide between Kafka, RabbitMQ or possible another if there is something better
  -----> We decided on Kafka
• Song is already sending message queues

Currently the indexer fails silently

• Auto - Retry indexing
• In the case that re-try fails, store failed messages for review

• Set up Read the Docs
• Installation
• How to set the configs
• How to hook up with Song & Arranger

Problem: When indexing an analysis fails, the returned result is not showing which index has failed(should be either file_centric or analysis_centric or both). It's currently showing "indexName:": null.

Expected results: indexName should be either file_centric_1.0 or analysis_centric_1.0 depending on which index has failed.

SONG allows the "suppression: of an analysis.

When an analysis is suppressed, the corresponding records needs to be redacted from the index