unfoldingword / string-punctuation-tokenizer Goto Github PK

View Code? Open in Web Editor NEW

8.0 9.0 1.0 2.2 MB

Small library that provides functions to tokenize a string into an array of words with or without punctuation

Home Page: https://string-punctuation-tokenizer.netlify.app/#/Tokenize

License: MIT License

JavaScript 100.00%

javascript nlp scripture-open-components nlp-library segmentation tokenizers

string-punctuation-tokenizer's Introduction

string-punctuation-tokenizer

Small library that provides functions to tokenize a string into an array of words with or without punctuation

Setup

npm install string-punctuation-tokenizer

Usage

var stringTokenizer = require('string-punctuation-tokenizer');

or ES6

import {tokenize} from 'string-punctuation-tokenizer';

Tokenize with punctuation

import {tokenize} from './src/tokenizers'; // use the import from above instead of this
let words = tokenize({text: 'Hello world, my name is Manny!', includePunctuation: true});
// words = ["Hello", "world", ",", "my", "name", "is", "Manny", "!"]

Tokenize without punctuation

import {tokenize} from './src/tokenizers'; // use the import from above instead of this
let words = tokenize({text: 'Hello world, my name is Manny!'});
// words = ["Hello", "world", "my", "name", "is", "Manny"]

Documentation

See detailed documentation and live WYSIWYG playground here: https://string-punctuation-tokenizer.netlify.app/#/Tokenize

string-punctuation-tokenizer's People

Contributors

Stargazers

Watchers

Forkers

cdolek

string-punctuation-tokenizer's Issues

unexpected word split when encountering "3-inch"

The tokenizer correctly keeps words separated by a single dash (like “goat-hair” Exodus 26:7 NLT 2007), however we discovered that it splits up words when the dash is surrounded by a number and a word (like “3-inch” Exodus 25:25 NLT 2007). Is there a way for us to keep that as a single word?

[6]SPIKE: Normalize Hebrew strings for canonical equivalence according to Unicode standard

Based on https://unicode.org/reports/tr15/#Canonical_Equivalence and following. Note that: The outputs are not required to be identical, only canonically equivalent.

DoD: Opening a canonically equivalent, but different Hebrew text highlights the same as another Hebrew text. Text UHB version 2.1.9

Needs a definition of what characters can be re-ordered. Consult @jag3773

Normalize Hebrew strings for canonical equivalence according to Unicode standard

see SPIKE: #46
Based on https://unicode.org/reports/tr15/#Canonical_Equivalence and following. Note that: The outputs are not required to be identical, only canonically equivalent.

DoD: Opening resources with Hebrew words that have been identified as edge cases and should be highlighted

@ancientTexts-net please add list of edge cases.

Add more info to verbose mode

in verbose mode could you include the token position and character position in the sentence as well?
e.g. given "Hello World", "Hello" is at token position 1 (assuming starting at 0), and character position 6. This shouldn't impact performance and would help to eliminate some processing in the wordmap lexer.

Normalize Greek strings for canonical equivalence according to Unicode standard

see SPIKE: #46
Based on https://unicode.org/reports/tr15/#Canonical_Equivalence and following. Note that: The outputs are not required to be identical, only canonically equivalent.

DoD: Opening resources with Greek words that have been identified as edge cases and should be highlighted

@ancientTexts-net please add list of Greek edge cases.

"½" character removed from word split

The tokenizer doesn’t count fraction characters (i’m guessing it considers it punctuation because I choose not to include punctuation). For example, “7½ feet wide” is split up into 3 words: “7", “feet”, “wide”. Is there a way for us to keep the “½” character together with the “7" like this "7½"? See Exodus 27:1 NLT 2007.

Just curious. Thanks.

Add a license

Would it be possible to add a LICENSE file to the project so we could decide if we can use the package? Thanks!