{"error":{"root_cause":[{"type":"illegal_argument_exception",
"reason":"Custom normalizer [lowercase] may not use filter [trim]"}],"type":"illegal_argument_exception","reason":"Custom normalizer [lowercase] may not use filter [trim]"},"status":400}"

This is with the latest version of elastic search.

The number of shards is set to 5 which follows what ES used to use as defaults. As of the most recent version they have reduced this to 1 as most people never touched it and were left with ballooning indexes with no real value. As of now, they recommend setting this to 1 unless you know you need otherwise. Also, if you have less than X nodes (we have 2) this doesn't really buy much durability either. Would like to set this to 1 if you have no issues with that.

Consider adding support for ElasticSearch v7

• an issue was reported by a user: https://app.intercom.io/a/apps/l6e7tr01/inbox/inbox/conversation/20726038791
• the reason why this is happening is ES index definition parameter for text field values: ignore_above: 64 -

  rp-indexer/indexer.go

  Line 495 in f57fbc3

  "ignore_above": 64,

  • this parameter prevents indexing of contact field values that are longer than 64 characters
  • https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-above.html
• at the moment we define the max size of a text field value in the project settings: VALUE_FIELD_SIZE = 640
  • https://github.com/nyaruka/rapidpro/blob/41277f92c0e6f6914324ef5826ee63b30d4067c2/temba/settings_common.py#L1067

The options are to redefine the index for text fields and reindex all contacts:

• set "ignore_above": 640
• remove the param

Would allow for much more sophisticated searching of text fields, IE, we could do contains instead of exact matches. May also reduce index size though we would need to experiment to know for sure. Does have some implications as to exact searches however and that would also need to be experimented with.

It has been noticed that while using harmon758/postgresql-action@v1 your postgresql password temba is present in plaintext. Please ensure that secrets are encrypted or not passed as plain text in github workflows.

example:

"number": 700000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

These are currently hardcoded in https://github.com/nyaruka/rp-indexer/blob/main/contacts/index_settings.json

Maybe we split that file into 2 files that have the analysis and mapping sections, and then the settings > index is constructed from config params instead.

We're currently loading up to 500,000 contacts to index which is far from ideal, but was the only way to deal with hundreds of thousands of contacts having the same modified_on. We've gotten better at only ever updating contacts in small batches, so this could be lower.

If we load up to N contacts, and we get N contacts with the same modified_on, then we won't advance properly, so that's a sentry error and something that needs to be addressed ASAP.

Hi!
I'm getting this error:

level=error msg="error indexing contacts" error="pq: SSL is not enabled on the server"

any thoughts? I'm trying to get a full docker-compose setup for rapidpro running here:
https://github.com/cromulus/rapidpro-docker-compose