Search for Umlauts about bleve HOT 2 CLOSED

Thank you for your quick reply. I used the wonderful service https://bleveanalysis.couchbase.com/analysis with export custom mapping to stepwise reduce the source which causes in my opinion the non-optimal search results and I came up with

err = index.AddCustomAnalyzer("thesaurus_german_wortlautmapping_2",
	map[string]interface{}{
		"type":      custom.Name,
		"tokenizer": unicode.Name,
		"token_filters": []interface{}{
			// de.NormalizeName,
			de.StopName,
			de.LightStemmerName,
			lowercase.Name,
		},
	})

So de.NormalizeName is in my opinion causing unwanted Normalizations.

Thank you for your quick reply.

from bleve.

@the42 Your code looks good. My german unfortunately is not :)

Per the rules set for de, the token stream generated for :

• Nägel -> ["nagel"]
• Fußpflege -> ["fusspfleg"]
• Fuß -> ["fuss"]

You can test out analyzer behavior here - https://bleveanalysis.couchbase.com/analysis

So it seems - if just Fuß does not exist in your documents, your search for Fuß will not find anything.

As you can see here, these are the components used by the de analyzer -

• Tokenizer: unicodeTokenizer
• Tokenfilters:
  • toLowerFilter
  • stopDeFilter
  • normalizeDeFilter
  • lightStemmerDeFilter

If these rules do not suffice, I'd recommend building a custom analyzer with these rules and any additional ones that you see fit for your use case. Also, we're happy to accept any contributions in making the stock de analyzer more accurate.

from bleve.