Substitutions for special characters and numerals about periodo-reconciler HOT 8 CLOSED

Can we make a basic character substitution map so that special or accented characters match plaintext, or is this too complicated with Unicode?

https://www.npmjs.com/package/normalize-strings

from periodo-reconciler.

Well, there we go.

from periodo-reconciler.

Another way to do this is to normalize input to Unicode NFD (decomposed) and then use a regex to strip all accent class characters (in Ruby this is \p{M}).

from periodo-reconciler.

Yep, that's probably a better approach than relying on a whitelist of characters.

from periodo-reconciler.

Fixed by 48d5efd

from periodo-reconciler.

One issue with replacing Roman numerals with their Arabic equivalents is that the same processing is applied to both period labels and queries. Doing that replacement on a period label is no problem, since it's very unlikely that there would be a stray "I" "V" or "X" token appearing in a period label that was not a Roman numeral. However, there might be a situation where that would be the case in a query-- like if someone wanted all the periods that started with a "V". That's not a real case, but it's not far from one.

from periodo-reconciler.

But this comment is more about free-text searching than reconciliation, right? I would assume that one wouldn't have a dataset for reconciliation that just had letters like 'V' that weren't numerals.

from periodo-reconciler.

Yes, you're right. Implemented in #7 if @rybesh is fine with it!

from periodo-reconciler.