opensearch-project / asynchronous-search Goto Github PK

Description

JCenter has been turned off and should be removed as a referenced repository.

Please Remove any direct dependency on jcenter() within a repositories block in gradle files.

Related: opensearch-project/opensearch-build#1456

Coming from opensearch-project/.github#21.

The correct copyright for open-source projects in opensearch-project is "Copyright OpenSearch Contributors". Please correct any places that say otherwise, especially where it says copyright Amazon. Make sure NOTICE.txt and README match. See opensearch-project/.github#24 for an example.

Is your feature request related to a problem? Please describe.
OpenSearch repository is going to replace the terminology "master"with "cluster manager".
issue: opensearch-project/OpenSearch#472 (comment), with the plan for its terminology replacement.

Although the existing usages with "master" will be supported in OpenSearch version 2.x to keep the backwards compatibility, please prepare for the nomenclature change in advance, and replace all the usages with "master" terminology in the code base.
All the OpenSearch REST APIs and settings that contain "master" terminology will be deprecated in 2.0, and alternative usages will be added.

Describe the solution you'd like
Replace the terminology "master" with "cluster manager".

When being compatible with OpenSearch 2.0:

• Replace "master" in code comment and internal variable / method name.
• Replace OpenSearh REST API that contains "master" with the provided inclusive REST API:
  replace "master_timeout" with "cluster_manager_timeout"

When being compatible with OpenSearch 3.0:

• Replace "master" in the reference to OpenSearch Java API
  (new APIs are available in version 2.2)
  • DiscoveryNodes.builder().masterNodeId() -> clusterManagerNodeId()
  • DiscoveryNodeRole.MASTER_ROLE -> CLUSTER_MANAGER_ROLE
  • event.localNodeMaster() -> localNodeClusterManager()

Describe alternatives you've considered
None.

Additional context

We are coming from this issue: opensearch-project/opensearch-build#2078 in your 2.0 branch.

Note: This needs to be completed before 05/17 which is feature freeze day for 2.0.0-GA.

1. If you have OpenSearch plugin in your repo, make sure your build.gradle remove rc1 reference, leave as "".

opensearch_version = System.getProperty("opensearch.version", "2.0.0-SNAPSHOT")
buildVersionQualifier = System.getProperty("build.version_qualifier", "")

2. If you have OpenSearch-Dashboards plugin in your repo, make sure your package.json remove rc1 reference, leave as 2.0.0.0.

"version": "2.0.0.0",

3. If you have any .github/workflows contains hardcode rc1 reference, please remove.

Thanks.

Coming from opensearch-plugins#12, enable upgrade to OpenSearch, while supporting backward compatibility.

• Rename Namespaces (e.g. com.amazon.opendistroforelasticsearch to org.opensearch)
• Rename Classes, Methods, Variables
• Rename Remaining Identifiers (e.g. Opendistro to OpenSearch).
• Rename, and Backwards Compatibility for Rest APIs
• Rename, and Backwards Compatibility for Settings
• Rename, and Backwards Compatibility for Indices
• Run in a backwards compatible way on top of OpenSearch 1.0 that has joined an ES 7.10.x cluster
• Run in a backwards compatible way on top of OpenSearch 1.0 that has joined an ODFE 1.13.x cluster
• Drop in replacement for the Opendistro version 1.13 of the plugin

Describe the bug

Unrelated CI failure in #64.

REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.search.asynchronous.integTests.SubmitAsynchronousSearchSingleNodeIT.testSubmitAsynchronousSearchWithRetainedResponse" -Dtests.seed=E63408297C8370E6 -Dtests.security.manager=false -Dtests.locale=fr-LU -Dtests.timezone=America/Coral_Harbour -Druntime.java=14

org.opensearch.search.asynchronous.integTests.SubmitAsynchronousSearchSingleNodeIT > testSubmitAsynchronousSearchWithRetainedResponse FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([E63408297C8370E6]:0)


org.opensearch.search.asynchronous.integTests.SubmitAsynchronousSearchSingleNodeIT > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 1200000 msec).
        at __randomizedtesting.SeedInfo.seed([E63408297C8370E6]:0)
REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.search.asynchronous.integTests.SubmitAsynchronousSearchSingleNodeIT.testSubmitAsynchronousSearchWithRetainedResponse" -Dtests.seed=E63408297C8370E6 -Dtests.security.manager=false -Dtests.locale=fr-LU -Dtests.timezone=America/Coral_Harbour -Druntime.java=14

Is your feature request related to a problem? Please describe.
Creating this issue to gather thoughts on moving this repo to core. Historically some decisions were made before the fork was done and plugins were those limited ways to accomplish building asynchronous search. Reasons why moving to core makes sense

Pros

1. Asynchronous search is a native mechanism to perform search, most search requests could be modelled asynchronously
2. Other plugins kNN/PPL should be easily integrated with asynchronous search
3. Kibana core integration with browser session, triggering a search in asynchronous mode for queries running longer, auto-cancellation on session termination
4. Ability to store response can be extended for synchronous searches as well
5. Out of the box upgrades, simplifies the plugin release management
6. Easier integration with search related enhancements like search resource tracking, back-pressure mechanisms etc

Cons

1. Migration effort for existing users
2. Proxy overhead of maintaining a common endpoint for both sync and async modes
3. Would introduce a tighter coupling of the code/artefacts

Describe the solution you'd like
Move the current repo to a module in OpenSearch core
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Hello,

I am looking for ways to invoke asynchronous search from client application that uses RestHighLevelClient
From looking at RestHighLevelClient I found no way to achieve that.

Java client for Elastic supports that as the following:
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-asyncsearch-asyncsearch-submit.html

Is similar functionality available for OpenSearch or is it scheduled to be supported?

Thank you

Coming from opensearch-project/opensearch-build#1750, replace checked in ZIP(s) with dynamic dependencies.

./asynchronous-search/src/test/resources/org/opensearch/search/asynchronous/bwc/1.13.0.1/opendistro-asynchronous-search-1.13.0.1.zip

Coming from opensearch-project/opensearch-plugins#96, add MacOS (Darwin) support.

• Passing CI on MacOS
• Documentation

Is your feature request related to a problem?
With the existing implementation of integration tests, opensearch plugins use the same test framework as local integration tests which spin up test clusters by itself. This is a hacky way to override the endpoint from local spun up test clusters (via opensearch.testclusters gradle plugin) with the remote cluster endpoint.

The solution doesn't work when we would like to test the opensearch bundle as part of the release. The test frameworks expect the artifact to be available before we could test them. To clean this up, plugins have to create gradle scripts for remote endpoints to run https://github.com/opensearch-project/opensearch-build#integration-tests for the release cycle.

What solution would you like?
Refer to: opensearch-project/anomaly-detection#298 and make similar changes to the plugin.
Also if you have a custom name to invoke these tests, update the integtest.sh script in https://github.com/opensearch-project/opensearch-build/blob/main/scripts/default/integtest.sh

Coming from opensearch-project/opensearch-plugins#92.

Each plugin has a short descriptive text blurb that says what it does: this appears in plugin-descriptor.properties (OpenSearch plugins) or package.json (OpenSearch Dashboards plugins), as well as on the project website's "source" page and in the "About" section of every repo. These were created one-by-one over the years as plugins were created, so looking at them all together now it would be hard for somebody new to the project to use these to understand what the plugins do.

When running the integ tests for this plugin in 1.3.0, seeing failures related to JUnit versioning. Not seeing issues for any other plugins (Jenkins job ohltyler-integ-test-stable #73).

Trace:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileTestJava'.
> Could not resolve all dependencies for configuration ':testCompileClasspath'.
   > Conflict(s) found for the following module(s):
       - junit:junit between versions 4.13.2 and 4.12
     Run with:
         --scan or
         :dependencyInsight --configuration testCompileClasspath --dependency junit:junit
     to get more insight on how to solve the conflict.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 21s

Coming from opensearch-project/opensearch-plugins#95, add Windows support.

• Passing CI on Windows
• Documentation

Coming from opensearch-project/opensearch-build#1632, add support for -Dbuild.version_qualifier=.

The distribution build process now supports passing additional optional qualifiers other than the 3 digit versions, e.g. 2.0.0-alpha1. Add the ability to pass additional qualifier to build scripts, and generate artifacts with a version that includes the qualifier.

Coming from opensearch-project/opensearch-plugins#127.

We should standardize the file location across plugins by implementing following conventions:

• All supporting files for plugins should stay within the plugin's own folder: /plugins/my-plugin
• Within plugin's folder, we recommend following directory structure:
  • ./config: config files
  • ./bin: executables
  • ./lib: for plugins that use native libraries (e.g. libKNNIndexV2_0_11.so)
  • ./install: scripts called to bootstrap the plugin before first use.
  • ./docs: document files
  • ./certs: certificates for the security plugin
  • ./data: test/demo data
• All scripts should come in *nix and .bat flavors, should use OpenSearch's defaults for environment variables like $JAVA_HOME, so users can override these but don't need to specify them by default. Scripts should have 0755 permissions by default.

Description

With existing setup the plugin jars are only published to maven repo, but not the generated plugin zips.

Related META issue

opensearch-project/opensearch-build#1916

Current Behavior

Existing all maven publications target the jar files to publish to maven, but not the generated distribution zips.

Expected Behavior

Along with plugin jars, plugin zips should also be published to maven repo with right maven coordinates, so user can download these plugins using dependancy model.

Steps To Reproduce

1. Run the gradle assemble task that outputs jar and zips.
2. Using build setup the jars are published to local staging maven repo.
3. Zip's are just added to distribution's folder during build runtime.
4. Finally once build is done, just the jar files are added to local maven repo and updates to the manifest file, but not the zips.

Proposed solution

A custom gradle plugin solution that can create the publication task for the zips to get published to local staging maven repo with right maven coordinates.
Add/Configure the custom gradle plugin openseearch.pluginzip to the gradle project and test the generated plugin zip's to upload to local maven repo.
More details on how to use the Plugin:
https://github.com/opensearch-project/opensearch-plugins/blob/main/BUILDING.md#opensearchpluginzip

Level up and cleanup markdown content.

See .github#8 for details and recommendations, and common-utils for an example.

• README.md
• CONTRIBUTING.md
• DEVELOPER_GUIDE.md
• MAINTAINERS.md
• ADMINS.md
• SECURITY.md
• RELEASING.md
• TESTING.md
• other .md files

Coming from opensearch-build#567, release version 1.2. Please follow the following checklist.

Preparation

• Assign this issue to a release owner.
• Finalize scope and feature set and update the Public Roadmap.
• Create, update, triage and label all features and issues targeted for this release with v1.2.0.

CI/CD

• Increment version on main to 1.2.0.0.
• Ensure working and passing CI.
• Re(add) this repo to the manifest.

Pre-Release

• Branch and build from a 1.2 branch.
• Update your branch in the manifest.
• Feature complete, pencils down.
• Fix bugs that target this release.

Release

• Complete documentation.
• Gather, review and publish release notes.
• Verify all issued labeled for this release are closed or labelled for the next release.

Post Release

• Create a release tag.
• Suggest improvements to this template.
• Conduct a postmortem, and publish its results.

Description

Ensure MAJOR_VERSION.x branch exists, the main branch acts as source of truth effectively working on 2 versions at the same time.

Related META issue

opensearch-project/opensearch-plugins#142

Current Behavior

Currently plugins follow a branching strategy where they work on main for the next development iteration, effectively working on 2 versions at the same time. This is not always true for all plugins, the release branch or branch pattern is not consistent, the lack of this standardization would limit multiple automation workflows and alignment with core repo. More details on META ISSUE

Proposed solution

Follow OpenSearch core branching. Create 1.x and 2.x branches, do not create 2.0 as a branch of main, instead create main -> 2.x -> 2.0. Maintain working CI for 3 releases at any given time.

This is a component issue for 2.0.0 (GA).
Coming from opensearch-build 2086. Please follow the following checklist.
Please refer to the DATES in that post.

Preparation

• Assign this issue to a release owner.
• Finalize scope and feature set and update the Public Roadmap.
• All the tasks in this issue have been reviewed by the release owner.
• Create, update, triage and label all features and issues targeted for this release with v2.0.0.

CI/CD

• All code changes for 2.0.0 are complete.
• If this repo is core repo, use 2.0.0 version for your component; if this repo is plugin repo, use 2.0.0.0 version for your component. Qualifier should be empty '' see this issue for details: opensearch-project/opensearch-build#2078.
• Ensure working and passing CI.
• Check that this repo is included in the distribution manifest.

Pre-Release

• Update to the 2.0 release branch in the distribution manifest.
• Increment the version on the parent branch to the next development iteration.
• Gather, review and publish release notes following the rules and back port it to the release branch.git-release-notes may be used to generate release notes from your commit history.
• Confirm that all changes for 2.0.0 have been merged.

Release Testing

• Find/fix bugs using latest tarball and docker image provided in parent release issue and update the release notes if necessary.
• Code Complete: Test within the distribution, ensuring integration, backwards compatibility, and performance tests pass.
• Sanity Testing: Sanity testing and fixing of critical issues found.
• File issues for all intermittent test failures.

Release

• Complete documentation.
• Verify all issued labeled for this release are closed or labelled for the next release.

Post Release

• Prepare for an eventual security fix development iteration by incrementing the version on the release branch to the next eventual patch version.
• Add this repo to the manifest of the next eventual security patch version.
• Suggest improvements to this template.
• Conduct a retrospective, and publish its results.

Test failure link

REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.search.asynchronous.integTests.AsynchronousSearchStatsIT.testStatsAcrossNodes" -Dtests.seed=DC38CE361FE21FA -Dtests.security.manager=false -Dtests.locale=es-AR -Dtests.timezone=Pacific/Johnston -Druntime.java=14
org.opensearch.search.asynchronous.integTests.AsynchronousSearchStatsIT > classMethod FAILED

Coming from meta issue opensearch-project/opensearch-plugins#108:

• Add auto-backport PR creation Github action to make sure all the PRs are backported for a particular release.
• Add auto-delete for merged auto-backport PRs to clean up the branches created during auto-backport.
• Add documentation for the same.

Related documentation: https://github.com/opensearch-project/opensearch-plugins/blob/main/BACKPORT.md

This is a component issue for release 2.0.0 (RC1).

Coming from opensearch-project/opensearch-build#1624, please refer to the dates in that post.

Please also follow the following checklist.

You can find all the date in above issue

Preparation

• Assign this issue to a release owner.
• Finalize scope and feature set and update the Public Roadmap.
• All the tasks in this issue have been reviewed by the release owner.
• Create, update, triage and label all features and issues targeted for this release with v2.0.0.

CI/CD

• All code changes for 2.0.0 are complete.
• Ensure working and passing CI.
• Re(add) this repo to the (if not exist) opensearch input manifest.

Pre-Release

• Update your branch in the opensearch input manifest.
• Confirm that all changes for 2.0.0 have been merged.
• Complete integration tests, and update results in the comment, example.
• Find/fix bugs using latest tarball and docker image provided in meta issue.
• Completed release candidate testing build #TBD.
• All intermittent test failures have issues filed.

Release

• Complete documentation.
• Gather, review and publish release notes.
• Verify all issued labeled for this release are closed or labelled for the next release.

Post Release

• Create a release tag.
• Suggest improvements to this template.
• Conduct a retrospect, and publish its results.

Coming from opensearch-project/opensearch-plugins#64, default CI Java version to 11 (LTS), and run tests on 8, 14 and 17.

• Change the default Java version for CI in GitHub actions from 14/15 to 11 (LTS).
• Add Java 8, 14 (used before), and 17 (next LTS) to the CI matrix.
• Document Java versions in DEVELOPER_GUIDE.md.

Add backwards compatibility tests for the plugin to automate end-to-end upgrade paths for faster releases. Developer documentation for implementing bwc tests and hooking them to CI: https://github.com/opensearch-project/opensearch-plugins/blob/main/TESTING.md#backwards-compatibility-testing

Is your feature request related to a problem?
Humans have to remember to document features, or at least to open issues in documentation-website for new features/changes to be documented. This is a manual process, e.g. opensearch-project/OpenSearch#1711 (comment)

What solution would you like?
Add a workflow for creating issues in the documentation-website repo whenever a label needs-documentation is added on a PR. Refer: opensearch-project/OpenSearch#2929

What alternatives have you considered?
Create issues manually in the documentation-website whenever a feature needs to be documented.

Do you have any additional context?
Add any other context or screenshots about the feature request here.

Please update the version consumed in main and 1.x branch according to OpenSearch main and 1.x respectively.

Describe the bug

Snapshot build is broken.

To Reproduce

./gradlew assemble --no-daemon --refresh-dependencies -DskipTests=true -Dopensearch.version=1.0.0-SNAPSHOT -Dbuild.snapshot=true

Fails as follows.

FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring root project 'asynchronous-search'.
> Could not resolve all artifacts for configuration ':classpath'.
   > Could not find org.opensearch.gradle:build-tools:1.0.0.
     Searched in the following locations:
       - file:/home/ubuntu/.m2/repository/org/opensearch/gradle/build-tools/1.0.0/build-tools-1.0.0.pom
       - https://repo.maven.apache.org/maven2/org/opensearch/gradle/build-tools/1.0.0/build-tools-1.0.0.pom
       - https://plugins.gradle.org/m2/org/opensearch/gradle/build-tools/1.0.0/build-tools-1.0.0.pom
       - https://jcenter.bintray.com/org/opensearch/gradle/build-tools/1.0.0/build-tools-1.0.0.pom
     Required by:
         project :

Expected behavior

Work, produce asynchronous-search-1.0.0.0-SNAPSHOT.zip.

Coming from opensearch-project/opensearch-plugins#132, remove support for JDK 14.

In OpenSearch 2.0 we are building the complete distribution and every component with JDK 11, and testing and bundling JDK 17. Support for JDK 14, 15 and other non-LTS versions can be dropped. Remove building and testing with these JDKs from CI.

See https://opensearch.org/blog/technical-post/2022/03/opensearch-java-runtime/ for more information.

Describe the bug

https://github.com/opensearch-project/asynchronous-search/runs/5514037443?check_suite_focus=true

2022-03-11T16:45:31.7781982Z org.opensearch.search.asynchronous.integTests.AsynchronousSearchStatsIT > testNodewiseStats FAILED
2022-03-11T16:45:31.7793454Z 
2022-03-11T16:45:31.7793751Z     java.lang.AssertionError: expected:<1> but was:<0>
2022-03-11T16:45:31.7794233Z         at __randomizedtesting.SeedInfo.seed([ED17663BDAB0F11A:50D6EECED5549CB7]:0)
2022-03-11T16:45:31.7794697Z         at org.junit.Assert.fail(Assert.java:89)
2022-03-11T16:45:31.7795072Z         at org.junit.Assert.failNotEquals(Assert.java:835)
2022-03-11T16:45:31.7795465Z         at org.junit.Assert.assertEquals(Assert.java:647)
2022-03-11T16:45:31.7795835Z         at org.junit.Assert.assertEquals(Assert.java:633)
2022-03-11T16:45:31.7796528Z         at org.opensearch.search.asynchronous.integTests.AsynchronousSearchStatsIT.lambda$testNodewiseStats$1(AsynchronousSearchStatsIT.java:89)
2022-03-11T16:45:31.7797501Z         at java.util.ArrayList.forEach(ArrayList.java:1259)
2022-03-11T16:45:31.7798225Z         at org.opensearch.search.asynchronous.integTests.AsynchronousSearchStatsIT.testNodewiseStats(AsynchronousSearchStatsIT.java:86)
2022-03-11T16:45:36.9802432Z REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.search.asynchronous.integTests.AsynchronousSearchStatsIT.testNodewiseStats" -Dtests.seed=ED17663BDAB0F11A -Dtests.security.manager=false -Dtests.locale=ar-MA -Dtests.timezone=America/Indiana/Vincennes -Druntime.java=8

Release Version 2.2.0

This is a component issue for 2.2.0.
Coming from opensearch-build#2271. Please follow the following checklist.
Please refer to the DATES in that post.

Preparation

• Assign this issue to a release owner.
• Finalize scope and feature set and update the Public Roadmap.
• All the tasks in this issue have been reviewed by the release owner.
• Create, update, triage and label all features and issues targeted for this release with v2.2.0.
• Cut 2.2 branch

CI/CD

• All code changes for 2.2.0 are complete.
• Ensure working and passing CI.
• Check that this repo is included in the distribution manifest.

Pre-Release

• Update to the 2.2.0 release branch in the distribution manifest.
• Increment the version on the parent branch to the next development iteration.
• Gather, review and publish release notes following the rules and back port it to the release branch.git-release-notes may be used to generate release notes from your commit history.
• Confirm that all changes for 2.2.0 have been merged.
• Add this repo to the manifest for the next developer iteration.

Release Testing

• Find/fix bugs using latest tarball and docker image provided in parent release issue and update the release notes if necessary.
• Code Complete: Test within the distribution, ensuring integration, backwards compatibility, and performance tests pass.
• Sanity Testing: Sanity testing and fixing of critical issues found.
• File issues for all intermittent test failures.

Release

• Complete documentation.
• Verify all issued labeled for this release are closed or labelled for the next release.

Post Release

• Prepare for an eventual security fix development iteration by incrementing the version on the release branch to the next eventual patch version.
• Add this repo to the manifest of the next eventual security patch version.
• Suggest improvements to this template.
• Conduct a retrospective, and publish its results.

Coming from opensearch-project/project-meta#17

A Developer Certificate of Origin is required on commits in the OpenSearch-Project.

See doc.yml for an example workflow. Ensure CONTRIBUTING.md to has a section on the DCO per the project template.

• DCO Check Workflow
• CONTRIBUTING.md DCO Section

This is a component issue for release 1.3.0.

Coming from opensearch-build#889, release version 1.3. Please follow the following checklist.

Preparation

• Assign this issue to a release owner.
• Finalize scope and feature set and update the Public Roadmap.
• Create, update, triage and label all features and issues targeted for this release with v1.3.0.

CI/CD

• Increment version on main to 1.3.0.0.
• Ensure working and passing CI.
• Re(add) this repo to the (if not exist) manifest.

Pre-Release

• Branch and build from a 1.3 branch.
• Update your branch in the manifest.
• Feature complete, pencils down.
• Fix bugs that target this release.

Release

• Complete documentation.
• Gather, review and publish release notes.
• Verify all issued labeled for this release are closed or labelled for the next release.

Post Release

• Create a release tag.
• Suggest improvements to this template.
• Conduct a postmortem, and publish its results.

Because Apache Lucene 9 requires JDK 11 or newer, and coming from opensearch-project/opensearch-plugins#110, remove support for JDK 8.

• Remove JDK 8 from CI/CD, default to 11.
• Set Java source and target compatibility in all build.gradle files to JavaVersion.VERSION_11.
• Update DEVELOPER_GUIDE and remove any other mention of JDK 8.

Coming from opensearch-project/opensearch-plugins#45, use SNAPSHOT Maven dependencies from aws.oss.sonatype.org. See https://github.com/opensearch-project/opensearch-plugins/blob/main/BUILDING.md#building-within-the-opensearch-build-system for details.

Is your feature request related to a problem? Please describe.
The OpenSearch Plugins are still built using Gradle 6.6.x whereas the Gradle release train has moved to 7.x already. The OpenSearch core is about to be switched to Gradle 7.3 (see please opensearch-project/OpenSearch#1609), it would make sense to switch all plugins to Gradle 7.3 as well.

Describe the solution you'd like
Update Gradle to 7.3

Describe alternatives you've considered (Optional)
N/A

Additional context
See please opensearch-project/OpenSearch#1246 and opensearch-project/opensearch-plugins#107

Incremented version to 1.2.5 in 1.2 following from opensearch-project/opensearch-build#1600 to prepare for the next possible release.

Backport the change to higher version branches

Check more details opensearch-project/anomaly-detection#124

Create tracking issues for each plugin to follow the same design.

Is your feature request related to a problem? Please describe.
The typed_keys specify whether aggregation and suggester names should be prefixed by their respective types in the response. Current this is only supported on SUBMIT API. Meaning if the request takes longer, future GET requests do not have an ability to return typed_keys any longer

In opensearch-project/OpenSearch#1276 we have benchmarked various JDKs and concluded that Java Runtime 17 gives a non-trivial performance boost. Therefore, coming from opensearch-project/opensearch-plugins#129, add support for JDK 17.

• Add JDK 17 to CI/CD, do not remove 11 and 14.
• Update DEVELOPER_GUIDE and any other notes on Java runtime compatibility.

Is your feature request related to a problem? Please describe.
Targeted fixes for OS 1.2 release

Campaigns
Incremented asynchronous search version to 1.2 #53

This is a component issue for release 1.1.1.
Coming from release issue 1.1.1, release version 1.1.1. Please follow the following checklist.

You can find all the date in above issue

Preparation

• Assign this issue to a release owner.
• All the tasks in this issue have been reviewed by the release owner.
• Create, update, triage and label all features and issues targeted for this release with v1.1.1.

CI/CD

• All code changes for 1.1.1 are complete.
• Ensure working and passing CI.

Pre-Release

• Confirm that all changes for 1.1.1 have been merged.
• Complete integration and sanity tests, and update results in the comment, example.
• Find/fix bugs using latest tarball and docker image provided in meta issue.
• Completed release candidate testing build #TBD.
• All intermittent test failures have issues filed.

Release

• Complete documentation.
• Gather, review and publish release notes.
• Verify all issued labeled for this release are closed or labelled for the next release.

Post Release

• Create a release tag.
• Suggest improvements to this template.
• Conduct a retrospect, and publish its results.

Describe the bug
While running integration tests with security plugin enabled, the CI is failing with error
https://github.com/opensearch-project/asynchronous-search/pull/21/checks?check_run_id=3008459503

REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.search.asynchronous.restIT.AsynchronousSearchRestIT.testSubmitWithRetainedResponse" -Dtests.seed=32319D298A3961CE -Dtests.security.manager=false -Dtests.locale=pl -Dtests.timezone=SystemV/YST9YDT -Druntime.java=14
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:887)
at org.opensearch.client.RestClient.performRequest(RestClient.java:296)
at org.opensearch.client.RestClient.performRequest(RestClient.java:283)
at org.opensearch.test.rest.OpenSearchRestTestCase.initClient(OpenSearchRestTestCase.java:192)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:972)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:832)

Other plugins installed
Security plugin is installed in docker pull and run step in CI workflow

To Reproduce

1. Go to https://github.com/opensearch-project/asynchronous-search/pull/21/checks?check_run_id=3008459503
2. Re-run github action Test and Build Workflow.

Expected behavior
Integ tests pass

Coming from .github#13, standardize release branching to match what OpenSearch is doing.

1. Create a 1.0 branch for the OpenSearch 1.0 release, 1.x branch for next 1.1 release, and use main for 2.0 development. Make sure CI is enabled on those.
2. Update your release/branching documentation. If you don't have one, add a RELEASING.md that links to, or has content from .github/RELEASING.md.
3. If you are using a develop branch, stop. You should be using the default main brach for furthermost, future development.
4. Delete any merged/stale/develop/feature branches that are no longer in use.
5. Communicate any changes in process to your team.

Refer to release branching for more information.

Description

Gradle project: Staging to add gradle tasks (setVersion and versionIncrement) that support version increment automation for supported versions and add/update gradle.properties file to the project.

Related META issue

Part of: opensearch-project/opensearch-build#1375
From solution: opensearch-project/opensearch-build#1375 (comment)

Current Behavior

Version Increment is a process that needs to be done for all plugin repos during and after a specific release.
This is time consuming and includes lot of manual human effort (that can lead to multiple errors) to go through each repo, check for previous merged PR and then create a version increment PR, allow plugin team to review, follow-up, update the PR with suggestions and then finally get merged for CI to run.

Expected Behavior

After every release expect to raise an [AUTO] PR that increments the version.
Sample automation version increment PR's: job-scheduler, common-utils, anomaly-detection, k-NN

Proposed solution

Automate this version increment process across all plugin components, that increment’s the version, modifies the required files and raise a PR that awaits for plugin team approval, this way an engineer need not spend hours of time to increment the version for all plugin components.

Implementation Changes:

Add gradle versionIncrement and setVersion tasks to the project.

setVersion : This task will update the gradle property opensearch.version and updates the value in gradle.properties file.

gradle.properties : Contains gradle project properties that can be used by project during runtime, for version increment uses opensearch.version.

versionIncrement : This task is configurable and depends on the plugin requirement to modify any files where version is present and that needs to incremented, this task uses ant.replaceregexp to parse the required files that are added in include() section of the task.

Sample Implementation PR's: job-scheduler, common-utils, anomaly-detection, k-NN
More examples: opensearch-project/opensearch-build#1375 (comment)

As part of v2.0 release, mapping types are getting removed from OpenSearch engine. Below are the changes in the opensearch-engine.

• Removal of type from rest end-points
• Removal of include_type_name parameter from API requests.

As part of this issue, please verify if type removal change on OpenSearch engine impacts this repository. If yes, then please remove the type references/usage from this plugin.

Top level changes are captured in gist below:
https://gist.github.com/dreamer-89/d76eaf639171e8ab32fa7f8b9d6c93d3

For more detailed changes, please check meta issue below:
Related: OpenSearch-engine meta issue

This is a component issue for 2.1.0.
Coming from opensearch-build#1818. Please follow the following checklist.
Please refer to the DATES in that post.

Preparation

• Assign this issue to a release owner.
• Finalize scope and feature set and update the Public Roadmap.
• All the tasks in this issue have been reviewed by the release owner.
• Create, update, triage and label all features and issues targeted for this release with v2.1.0.

CI/CD

• All code changes for 2.1.0 are complete.
• Ensure working and passing CI.
• Check that this repo is included in the distribution manifest.

Pre-Release

• Update to the 2.1.0 release branch in the distribution manifest.
• Increment the version on the parent branch to the next development iteration.
• Gather, review and publish release notes following the rules and back port it to the release branch.git-release-notes may be used to generate release notes from your commit history.
• Confirm that all changes for 2.1.0 have been merged.
• Add this repo to the manifest for the next developer iteration.

Release Testing

• Find/fix bugs using latest tarball and docker image provided in parent release issue and update the release notes if necessary.
• Code Complete: Test within the distribution, ensuring integration, backwards compatibility, and performance tests pass.
• Sanity Testing: Sanity testing and fixing of critical issues found.
• File issues for all intermittent test failures.

Release

• Complete documentation.
• Verify all issued labeled for this release are closed or labelled for the next release.

Post Release

• Prepare for an eventual security fix development iteration by incrementing the version on the release branch to the next eventual patch version.
• Add this repo to the manifest of the next eventual security patch version.
• Suggest improvements to this template.
• Conduct a retrospective, and publish its results.