mplatt / fold-to-ascii Goto Github PK

A JavaScript port of the Apache Lucene ASCII Folding Filter that converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the ‘Basic Latin’ Unicode block) into a ASCII equivalents, if they exists.

License: Apache License 2.0

JavaScript 100.00%

fold-to-ascii's Introduction

fold-to-ascii-js

A JavaScript port of the Apache Lucene ASCII Folding Filter that converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the "Basic Latin" Unicode block) into their ASCII equivalents.

Documentation

Installation

npm install fold-to-ascii

Usage

There are two different modes of operation:

Replace all known non-ASCII characters with appropriate replacements, replace the unknown ones with a fallback (foldReplacing).
Replace all known non-ASCII characters with appropriate replacements, maintain the unknown ones (foldMaintaining).

The difference in output only manifests if the inputs contain characters without known replacements:

var ASCIIFolder = require("./lib/ascii-folder");

// Some Characters have no defined replacement.
// Specify a fixed replacement character (defaults to the empty string).
ASCIIFolder.foldReplacing("Lörem 🤧 ëripuît") === "Lorem  eripuit";
ASCIIFolder.foldReplacing("Lörem 🤧 ëripuît", "X") === "Lorem XX eripuit";

ASCIIFolder.foldMaintaining("Lörem 🤧 ëripuît") === "Lorem 🤧 eripuit";

Tests

npm test

Sources

This is a straightforward port of the extensive switch/case statement found in http://svn.apache.org/repos/asf/lucene/java/tags/lucene_solr_4_5_1/lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/ASCIIFoldingFilter.java

fold-to-ascii's People

Contributors

Stargazers

Watchers

Forkers

tlemburg sovanyio avvs yokuze youknowriad monteshaffer lucaswdeveloper jdrew1303 eugef sorunome bemoje pexxi sakari-io zazapeta ntedgi drieschel

fold-to-ascii's Issues

Doesn't handle eggplant

I tried the following in Chrome and Firefox.

> ASCIIFolder.fold("🍆", null)
"null"
> ASCIIFolder.fold("こんにちは", null)
"nullnullnullnullnull"
> ASCIIFolder.fold("お早うございます", null)
"nullnullnullnullnullnullnullnull"

According to the docs, all cases should not have performed any character replacements.

Rename main file from index.js to fold-to-ascii.js

When I'm using main-bower-files, main file from your library is moved to vendor directory of my project. Index.js doesn't say too much. Please, consider renaming it. I hope it doesn't make big difference for you.

ES5 version?

Would it be possible to offer an ES5 version alongside the ES6 version by running it through something like Babel?

Request for a major version release and restore v4 without breaking changes

Fold is not a function

Last changes introduced in this library, specifically the ones introduced in this commit in particular: 4cf57c7#diff-e9ef0426c130f92ae3cda5a4b3d0356fL32. Must be considered as a breaking change and probably should be managed as a major release of the package. v5.0.0

If I do this:

{ fold } = require('fold-to-ascii'),

I end up with an error that says fold is not a function.

The README has a call to foldMaintaining with a 2nd parameter

Looking at the source, foldMaintaining only takes 1

Broken in Typescript 4.4

Hi,
fold-to-ascii has been broken in the new version of Typescript 4.4.

Cannot read property '_fold' of undefined

import { foldReplacing } from "fold-to-ascii";
foldReplacing("test string");

Reproduction: https://codesandbox.io/s/fold-to-ascii-typescript-4-4-broken-xxkc4