lambdaworks / zio-elasticsearch Goto Github PK

Swap usage of List collection with Chunk in all places, except if there would be significant hit on performance.

Description:

Currently, we have two opened PRs for bumping versions related to our website - #174 and #197. This could be related to our docusaurus plugin, and you can ask @mvelimir for more info.

Description:

The rewrite parameter can be used in queries:

• fuzzy
• prefix
• query_string
• regexp
• wildcard

It determines:

• How Apache Lucene calculates the relevance scores for each matching document
• Whether Apache Lucene changes the original query to a bool query or bit set
• If changed to a bool query, which term query clauses are included

Description

Replace the current implementation of WireMock with STTP. You can refer to the documentation here.

WireMock is used to confirm that HttpElasticExecutor is creating and executing expected HTTP requests.

Implement support for https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query-phrase.html

Description:

Currently this issue is blocked, because lib does not support this yet.
Example app already has an endpoint for search: localhost:9000/api/repositories/search, which should be modified to work with this scenario too.

Description:

The current version of the sttp does not support JVM 8.

Currently we use latest version of sttp for zio2.
"com.softwaremill.sttp.client3" %% "zio" % "3.8.3"

We have following issue with
sttp/client3/SttpBackend has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0, which means it is not compatible with JRE 8.

For compatibility we can use some earlier version of sttp which can be found here: https://github.com/softwaremill/sttp/releases

Queries

• fuzzyQuery
• geoBoundingBoxQuery
• geoDistanceQuery
• geoPolygonQuery
• geoShapeQuery
• hasChildQuery
• hasParentQuery
• idsQuery
• intervalsQuery
• matchBooleanPrefixQuery
• matchPhraseQuery
• matchPhrasePrefixQuery
• multiMatchQuery
• nestedQuery
• prefixQuery
• regexpQuery
• scriptQuery
• scriptScoreQuery
• shapeQuery
• simpleQueryStringQuery
• termsQuery
• termsSetQuery
• queryStringQuery

Tips

• Find comprehensive info about the query you are going to implement here
• Define query in zio.elasticsearch.query.Queries
• Define the public method that represents DSL for the query in the zio.elasticsearch.ElasticQuery, add scaladoc and website documentation for it
• Support additional parameters such as "boost", "case insensitive", etc. in the zio.elasticsearch.query package
• Provide unit tests in the zio.elasticsearch.ElasticQuerySpec to confirm behavior
• Provide integration tests (it module) in the zio.elasticsearch.HttpExectorSpec to confirm the behavior
• 😇 Feel free to extend other tests as well
• 😊 Feel free to address any follow-up issue either for the query you are implementing or implemented one already

Note: You can use #48 as a reference. If you feel that the pull request size is growing out of control, feel free to split it but make sure to link this issue in each of the related PRs.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• fix(deps): update dependency @mdx-js/react to v2

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• chore(deps): update elasticsearch docker tag to v8

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Add support for K Nearest Neighbor in ZIO-elasticsearch.

Updates to the api should allow for programatically defining these fields:

• query_vector
• knn
• similarity

Some aggregations support runtime fields. It's good if aggregation needs to be run on something more complex than the values in a single field.
More informations about this topic can be found here: runtime field.

Description:

size and from parameters are used to page through a larger set of search results. By default, Elasticsearch returns the top 10 matching hits, which is often not enough.
The from parameter defines the number of top hits to skip (default is 0). The size parameter is the maximum number of hits to return.

NOTE: By default, you cannot use from and size to page through more than 10,000 hits.

There are sometimes extra letters for terms and for percentiles aggregations in the elasticsearch response.
Something like this happens:
sterms#aggName

For optimization of search after request it is neccessary to add "track_total_hits": false as well as sort: shard_doc.

Currently we have only sort:shard_doc added in requests.

Description:

Scaladoc is a documentation generation system that reads specially formatted comments in Scala source code and generates compiled documentation.

/** This is a Scaladoc.
  * 
  */

The goal of this task is to add Scaladoc to all public api methods, we can describe what is the purpose of a method, and describe its parameters and return value.

Description:

The boost is already supported in MatchAll, Term, and Wildcard queries, so based on that similar needs to be done in Range and Bool queries.

Description:

• Provide missing features documentation to the website
• Address "TBD" in the current website documentation structure
• Provide queries documentation
• Provide requests documentation
• Provide aggregations documentation
• Provide ElasticSort documentation
• Provide ElasticHighlights documentation
• Provide kNN request and query documentation

Description:

At the moment, we have only must and should elements in bool query. The filter clause must appear in the matching documents. Unlike queries specified under must, queries specified under the filter element have no effect on scoring - scores are returned as 0.

GET _search
{
  "query": {
    "bool": {
      "filter": {
        "term": {
          "status": "active"
        }
      }
    }
  }
}

Currently we are using fuzziness parameter in the fuzzy query as a String, it should be considered that we use enum instead.

More informations about Fuzziness can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/common-options.html#fuzziness

At the moment we have only Int supported, but in Elasticsearch documentation there is also percentage and combination of both.

You can find more informations here: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-minimum-should-match.html

Description:

As an example, you can take a look at the ElasticQuerySpec.

The basic idea is to have a simple structure and description of the tests and combine them with the && operator.

The following specs should be restructured:

• FieldDSLSpec
• HighlightsSpec
• IndexNameSpec
• ScriptSpec
• HttpElasticExecutorSpec
• HttpExecutorSpec

Can you help me figure out how to use index patterns with zio-elasticsearch?

Elastic Search Index Pattern Search

GET categories*/_search
{
  "query": {
    "match_all": {}
  }
}

Intuitively I tried something like this

elasticSearch.execute(getById(IndexName("categories*"),DocumentId("documentId")))

This is blocked because IndexName does not allow "*".

I have indexes: categories_1, categories_2, categories_i+1

Script and InnerHits case class can not be created on Scala 2.12. because apply methods are not visible.

Description:

The sort parameter allows us to add one or more sorts on specific fields. It is defined on a per field level, with special field name for _score to sort by score

The order option can have the following values:

• asc (default for all sortings except the _score)
• desc (default for sorting on the _score)

The mode option is used for sorting by array or multi-valued fields, and it can have the following values:

• min (default for asc)
• max (default for desc)
• sum
• avg
• median

POST /_search
{
   "query" : {
      "term" : { "product" : "chocolate" }
   },
   "sort" : [
      {"price" : {"order" : "asc", "mode" : "avg"}}
   ]
}

Elasticsearch also supports: sorting by fields that are inside one or more nested objects, handling docs which are missing the sort field, and more.. https://www.elastic.co/guide/en/elasticsearch/reference/7.17/sort-search-results.html

Description:

Enables you to script document updates. The script can update, delete, or skip modifying the document. The update API also supports passing a partial document, which is merged into the existing document.

More about Update API can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/docs-update.html#docs-update-api-request

Current ElasticConfig implentation does not allow to use https scheme (it uses sttp.model.Uri constructor that assumes http).

Aggregations

• Support adjacencyMatrixAgg
• Support avgAgg
• Support bucketSelectorPipelineAgg
• Support bucketSortPipelineAgg
• Support cardinalityAgg
• Support childrenAgg
• Support dateHistogramAgg
• Support autoDateHistogramAgg
• Support dateRangeAgg
• Support extendedStatsAgg
• Support filterAgg
• Support filtersAgg
• Support geoBoundsAgg
• Support geoDistanceAgg
• Support geoHashGridAgg
• Support geoTileGridAgg
• Support geoCentroidAgg
• Support globalAgg
• Support histogramAgg
• Support ipRangeAgg
• Support maxAgg
• Support minAgg
• Support missingAgg
• Support nestedAgg
• Support percentilesAgg
• Support percentileRanksAgg
• Support rangeAgg
• Support reverseNestedAgg
• Support samplerAgg
• Support scriptedMetricAgg
• Support sigTermsAgg
• Support statsAgg
• Support sumAgg
• Support termsAgg
• Support topHitsAgg
• Support topMetricsAgg
• Support valueCountAgg
• Support variableWidthHistogramAgg
• Support weightedAvgAgg

Tips

• Find comprehensive info about the aggregation you are going to implement here
• Define aggregation in zio.elasticsearch.aggregation.Aggregations
• Define the public method that represents aggregation in the zio.elasticsearch.ElasticAggregation object, add scaladoc and website documentation for it
• Support "withAgg", "withSubAgg" in the zio.elasticsearch.aggregation package, if possible
• Provide unit tests in the zio.elasticsearch.ElasticAggregationSpec to confirm behavior
• Provide integration tests (it module) in the zio.elasticsearch.HttpExectorSpec to confirm the behavior
• 😇 Feel free to extend other tests as well
• 😊 Feel free to address any follow-up issue either for the aggregation you are implementing or implemented one already

Note: You can use #95 as a reference. If you feel that the pull request size is growing out of control, feel free to split it but make sure to link this issue in each of the related PRs.

Add a way to explicitly refresh an index using the refresh API: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-refresh.html

Description:

At the moment we have only bool compound query. We would like to support other queries too:

• - boosting query
• - constant_score query
• - dis_max query
• - function_score query

More about each query can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/compound-queries.html

Only case object Kilometers is plural, everything else is in singular. We should unify under one style.

Class

Currently StreamConfig has parameter that represents page size but this parameter is never used when creating a request.

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: .github/renovate.json
Error type: The renovate configuration file contains some invalid settings
Message: Invalid schedule: 'Invalid schedule: Failed to parse "after 6pm every workday"'

Maybe I oversee something here but I could not find a possibility to use the average aggregation. If they want to be added they should appear here probably, right?

Description:

Updates documents that match the specified query. If no query is specified, performs an update on every document in the data stream or index without modifying the source, which is useful for picking up mapping changes.

POST my-index-000001/_update_by_query?conflicts=proceed

More about Update By Query API can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html

Are there any plans to target Scala 3? Is there anything standing in the way currently? I would be willing to help with the refactor if there is interest.

Description:

Add a search option bar to our website. As you can see, our docs don't have the search option at the moment.

You can explore the possibilities here.

Description:

At the moment, we have only must and should elements. The must_not clause must not appear in the matching documents. Because scoring is ignored, a score of 0 for all documents is returned.

POST _search
{
  "query": {
    "bool" : {
      "must_not" : {
        "range" : {
          "age" : { "gte" : 10, "lte" : 20 }
        }
      }
    }
  }
}

Description:

Get familiar with ZStreams and ElasticSearch's support for streaming query result.

Description:

Update request modify a document using the specified script. The script can update, delete, or skip modifying the document. If the document does not already exists, the contents of the script are inserted as a new document.

POST /<index>/_update/<_id>

The Update API can be used for passing a partial document, which is merged into the existing document.

POST test/_update/1
{
  "doc": {
    "name": "new_name"
  }
}

Also, update request can be used inside the bulk request.

POST /_bulk
{ "update": {"_id": "5", "_index": "index1"} }

Description:

Gets the number of matches for a search query.

GET /my-index-000001/_count?q=user:kimchy

GET /my-index-000001/_count
{
  "query" : {
    "term" : { "user.id" : "kimchy" }
  }
}

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/search-count.html#search-count-api-path-params