ulf1 / kshingle Goto Github PK

TODO:

• Update documentation

J(A, B) = | INTERSECTION(A,B) | / | UNION(A,B) |
C(A,B) = | INTERSECTION(A,B) | / | A | 

Assume A=Q is a shingleset that we want to query against a database full of shinglesets x_i (i.e. B=x_i for specific database example i).

Literature:

• page 2, https://arxiv.org/pdf/1809.00458.pdf
• docs at http://ekzhu.com/datasketch/lshensemble.html

TODO

• Rename functions
• Add shingling_* functions as wrapper with deprecation notice (for version 0.8.0)
• Change README.md examples

• https://codebeat.co/
• https://lgtm.com/

• example: abcdef
• output n=1: abc*ef
• output n=2: ab*ef
• output n=3: a*ef

setuptools>=56.*

Why?
The variable name VOCAB is misleading since MWEs (Phrases) are used (e.g. Mikolov seminal paper), and subwords are used in modern NNs (e.g. BERT).

TODO

• rename functions
• rename variable name
• check docstr
• check README

https://github.com/ulf1/kshingle#select-the-shingles-cews-create-pattern-list-and-encode-data

Examples

• Alphabet, 2-shingle, ..., k-shingle
• alle Worttoken
• Sonderzeichen
• Abkürzungen
• NE-Listen

Usage

import kshingle as ks
memo = {k: db[k] for k in ["i.e.", "e.g."]}
memo = ks.cews(db, threshold=0.9, min_count_split=10, max_wildcards=2)

introduce a wildcard character that stand for 1 or 2 chars.

a*b
axb
axxb

why

H*user
Häuser
Haeuser

k=1

• any unicode utf-8 character
• A-Za-z0-9
• chinese symbols
• punctations
• ...

k>=2

• syllables
• morphemes
• words (e.g. dictionaries)
• multi-words (e.g. NE lists)
• end of word A and start of word B
• word n-grams

or better fix at the source

test

s = "a"
k = 2
is the dim=k

• CEWS is way too slow

TODO

• find something faster, e.g., a) select the rarest shingles, b) select the shortest shingles then the most frequent shingle.
• remove from docs https://github.com/ulf1/kshingle#collectively-exhaustive-wildcard-shingling-cews
• remove file https://github.com/ulf1/kshingle/blob/main/kshingle/cews.py
• remove unit test https://github.com/ulf1/kshingle/blob/main/test/test_cews.py