Two situations where jargon should use the HTML tokenizer (instead of plain text):

• Fetching via the -u flag, where Content-Type starts with text/html
• Reading a file via the -f flag, and the file extension is .html or .htm

Add ability to canonicalize numbers with are expressed with words. Keep it simple for common cases.

"three hundred million" => "300000000"

Building upon #6

On some platforms it's a pain to redirect output correctly (for example, .NET's Process isn't super friendly for that), and without significant overhead. The alternative is having a shell run the actual command, which can also be irritating to do.

Presumably something like -o <outputfile>, to go along with -f, would solve this.

As it stands now, the lexer loads the full string into memory. This is fine and simple enough for small strings.

If someone were to throw a large file at it, we'd want it to come through buffered, presumably using bufio.

Such a change might fit into the current lexer in a few strategic places, replacing the byte-reading bits with ReadByte. Or, deprecate the existing one and rewrite using bufio.Scanner.

(On the other side, out of scope for this issue, we might emit tokens in a buffered way, using channels perhaps.)

Implement the first dictionary of tags & synonyms using the Stack Exchange API.

• Fetch & deserialize JSON
• Write file
• Figure out how to do this in a separate package without a circular dependency

Currently, the jargon command line takes its input by specifying the source via flags.

  -f string
    	A file path to lemmatize
  -s string
    	A (quoted) string to lemmatize
  -u string
    	A URL to fetch and lemmatize

It occurs to me that jargon would play better simply by accepting Stdin.

There are already fine tools for reading files (cat) and fetching URLs (curl). jargon should just accept bytes piped from other tools.

Files

cat file.txt | jargon

replaces

jargon -f file.txt

URLs

curl https://example.com | jargon

replaces

jargon -u https://example.com

Strings

echo "I luv Rails" | jargon

replaces

jargon -s "I luv Rails"

Currently jargon uses a Dictionary based on Stack Exchange tags & synonyms: https://github.com/clipperhouse/jargon/tree/master/stackexchange

What other types of data would you like to see a Dictionary for? Recall, a Dictionary serves two purposes:

• To ‘correct’ (lemmatize, canonicalize) concepts that might expressed different ways
• To identify named entities, even if they don’t need to be ‘corrected’

One idea that comes to mind is cities, or perhaps geographies in general. There is a data source here: https://www.geonames.org/export/

type tokenizer struct {
        sc *bufio.Scanner
}

Needs to be updated to

import (	"github.com/clipperhouse/uax29/iterators" )

type tokenizer struct {
        sc *iterators.Scanner
}

to work with more recent releases of uax29

A suggestion from @kevin-montrose: a Dictionary to canonicalize contractions, e.g.

shouldn’t → should not

The list of English contractions appears short. This would suggest simply encoding that list in a map[string]string, keyed by the contraction.

contractions[“shouldn't"] = "should not"
contractions["he’s"] = "he is”
// etc

(With tolerance for smart apostrophes, and perhaps missing apostrophes.)

Alternatively, we could try something rule-based, e.g.:

s/(.+)n’t/$1 not/

My instinct is that this is likely overkill and prone to undesirable edge cases, compared to the first idea.