Ability to override stemming in queries about fatcat-scholar HOT 1 OPEN

This came up again via a conversation on mastodon: https://mastodon.archive.org/web/@[email protected]/109426257900886328

Which was a good reminder that we do have exact variants for some fields, like biblio.title.exact (https://scholar.archive.org/search?q=biblio.title.exact%3A%22periodicals%22). And others, like author, are always indexed as "exact" (which does still have some normalization, just not stemming).

It should not be too "expensive" in index size to enable "exact" over the core biblio fields. Enabling it for the "everything" field (including body text) would nearly double the entire index size, if I understand correctly. We are already pretty tight on backing disk (SSD) at the moment, and would need to trade off this feature against things like citation graph, indexing additional search-only fulltext works, etc.

Another option might be to somehow execute "exact" match semantics at query time. Eg, taking a large-ish result set (based on the index with stemming enabled), and narrow the results by re-tokenizing the documents in "exact" mode and checking if they still match. If this is an existing elasticsearch/lucene feature I haven't heard of it. I suspect it wouldn't work well if not built-in, because request sets are often quite large and not returned in full (eg, one million or more results would need to be post-processed).

To summarize:

• should enable exact variants of more biblio fields in next full reindexing
• consider enabling exact variants in fatcat release search index (which is smaller)
• include exact index size increases for everything field as part of future hardware budgeting
• help wanted: feasibility of query-time exact filtering without requiring additional index size

from fatcat-scholar.