The ekphrasis from cbaziotis

MUISTI!

ne-am chinuit ca ultimii prosti ca nu vad cue pe voi capu
🤬

tokenizing '20th' to '2','0','th'

How to make this as one token and not separate it. Where is this tokenizing happening?

Updation of url : https://www.dropbox.com/s/a84otqrg6u1c5je/stats.zip?dl=1 required

Unable to download word statistics file due to a broken link. Please update.

Failed during generate_stats.py

computing statistics for file: text8.txt
100%|███████████████████████████████████████████████████████████████████| 1/1 [12:41<00:00, 761.71s/it]

Writing 1-grams...
entries:250,982 - tokens:16,996,178
writing stats to file /home/jj301440/ekphrasis-master/ekphrasis/tools/../stats/text8/counts_1grams.txt
Writing 2-grams...
entries:4,136,483 - tokens:16,996,178
writing stats to file /home/jj301440/ekphrasis-master/ekphrasis/tools/../stats/text8/counts_2grams.txt
Writing 3-grams...
entries:10,327,876 - tokens:16,996,178
writing stats to file /home/jj301440/ekphrasis-master/ekphrasis/tools/../stats/text8/counts_3grams.txt
Traceback (most recent call last):
File "generate_stats.py", line 191, in
write_stats(stats)
File "generate_stats.py", line 147, in write_stats
write_stats_to_file(filename, counter, args.mincount[int(k) - 1])
IndexError: list index out of range

Can you please tell me where the problem is?

what's wrong with "Word statistics files not found!"

teach me， thx

urllib.error.HTTPError: HTTP Error 429: Too Many Requests

I am implementing Segmenter and SpellCorrector using corpus generated by me, and when ever I instantiate corpus for Segmenter or SpellCorrector it return me with the error as

Corpus was generated as mention using script provided at site-package/ekphrasis/tools/

I am running these in docker container and instantiate corpus is implement at init.py
any resolution to it
I also thought that something is wrong with my corpus so i also ran it with corpus="english" and still same problem

Please add a LICENSE to this repo

I really like this tool. Can you add a license to this repo so that it is easy to use it in other projects as a dependency?

This is a good resource around this: https://help.github.com/en/github/creating-cloning-and-archiving-repositories/licensing-a-repository

extracting url

how could i extract the url in the tag it gets replaced

Add tests for regexes

We should add tests for regular expressions.

Remove one character entities on slang dictionary

The slang dictionary automatically removes one-letter long words such as 'n->and' and 'u->you'. I think this should be optional.

she's --> she ' s

Running the text_processor example in the README with the input "she's" returns ["she", "'", "s"].
Is this expected behavior? I'd have taught "she" "is".

spelling correction mostly is not working

Came to this project for spelling in twitter text, but it doesn't quite work most of the time.

spell correction seems to only work when annotate is set as in the example. Now
take the same example and set annotate={} and spell correction is gone:

i saw the new john doe movie and it suuuuucks ! ! ! waisted <money> . . . bad movies <annoyed>

if I restore annotate={"hashtag", "...}, then it corrects suuuuucks to sucks
I'm not sure what is the connection between annotations and spell correction.

spelling-correction doesn't work in general. Again, going back to your pipeline example, change the first input sentence to inject some spelling errors: CANT WAIT for the neww seaason of #TwinPeaks , run it, you get:
cant wait for the neww seaason of twin peaks - i.e. no spell correction.
The spell_correct_elong doesn't seem to make a difference.

Yet, if I run:

from ekphrasis.classes.spellcorrect import SpellCorrector
sp = SpellCorrector(corpus="english") 
print([sp.correct(x) for x in "neww seaason".split()])

It corrects: ['new', 'season']

Memory usage

As i noticed there is no trivial way to free memory allocated in RAM after creating a TextPreProcessor object. I tried remove object reference with: preprocessor = None or del preprocessor but it does not seem to work. Is there something i am missing?

How can the text_processor be parelize?

Hi, i am loving the framework,
anyways, is just taking so long.
I was wondering is this could be parallel ?

The TextPreProcessor class only supports segmenting text with hastags. Required support for normal text segmenter.

The TextPreProcessor class only supports word segmenting if hashtag symbol is there otherwise it fails.

Example:-

# With hashtag it works
s = " question kind infidelity passed sweety not feel sweet #savingyourmarriagebeforeitstarts"
print(" ".join(text_processor.pre_process_doc(s)))
'question kind infidelity passed sweety not feel sweet <hashtag> saving your marriage before it starts </hashtag>'

#without hashtag it fails
s = " question kind infidelity passed sweety not feel sweet savingyourmarriagebeforeitstarts"
print(" ".join(text_processor.pre_process_doc(s)))
" question kind infidelity passed sweety not feel sweet savingyourmarriagebeforeitstarts"

The TextPreProcessor class configuration is similar to what is defined in README.md file.

Kindly review it and if you find that correct, I can send a pull request.

Can Ekphrasis be used in other languages?

My dataset is in Italian. I am wondering if Ekphrasis can also be used in Italian. Or it's only for English?

Log messages print to stdout

First just wanted to say thanks for making this available, this has been very useful- I've been working with reddit and Twitter text and this has been instrumental.

This is altogether a pretty minor issue, but there are a few places where messages are printed. I use this in a script along with GNU parallel to process line delimited json, and merge the results back into one file. So the output is mixed with the messages about loading the models, etc.

I forked this and replaced the print statements with logging calls, so that they'll go to stderr and the user can control the verbosity by setting the logging level. Would you be interested in a pull request for that? I would just need to clean up a few changes I made to keep subreddit names together (just added a regex in the pipeline). I'd be happy to share that too.

Thanks!

Warning regarding using TextPreProcessor as a preprocessing for torchtext.data.Field()

As it can be seen in the code sample below, we get different results if

we pre-process a text with TextPreProcessor text_processor and then create an Example with a torchtext.data.Field() without a preprocessing Pipeline vs
creating an Example with the text as it is, with a torchtext.data.Field(preprocessing=data.Pipeline(lambda x : text_processor(x))

Using the Field preprocessing pipeline, the text_processor will be called on a token level, instead of on a sentence-level, and expressions like "October 10th" that will be converted to <date>, will not be correctly converted, as the text_processor will be called on two separate tokens "October" and "10th", and the last one will be break into "1 0 th" after that call.

from torchtext import data, vocab
import torch.optim as optim
import torch.nn.functional as F
import torch.nn as nn
from ekphrasis.classes.preprocessor import TextPreProcessor
from ekphrasis.classes.tokenizer import SocialTokenizer
from ekphrasis.dicts.emoticons import emoticons

text_processor = TextPreProcessor(
            # terms that will be normalized
            normalize=['url', 'email', 'percent', 'money', 'phone', 'user',
                'time', 'url', 'date', 'number'],
            # terms that will be annotated
            annotate={"hashtag", "allcaps", "elongated", "repeated",
                'emphasis', 'censored'},
            fix_html=True,  # fix HTML tokens

            # corpus from which the word statistics are going to be used
            # for word segmentation
            segmenter="twitter",

            # corpus from which the word statistics are going to be used
            # for spell correction
            corrector="twitter",

            unpack_hashtags=True,  # perform word segmentation on hashtags
            unpack_contractions=True,  # Unpack contractions (can't -> can not)
            spell_correct_elong=False,  # spell correction for elongated words

            # select a tokenizer. You can use SocialTokenizer, or pass your own
            # the tokenizer, should take as input a string and return a list of tokens
            tokenizer=SocialTokenizer(lowercase=True).tokenize,

            # list of dictionaries, for replacing tokens extracted from the text,
            # with other expressions. You can pass more than one dictionaries.
            dicts=[emoticons]
        )

Reading twitter - 1grams ...
Reading twitter - 2grams ...
Reading twitter - 1grams ...

>>> def custom_processing(x, text_processor):
...    text = " ".join(text_processor.pre_process_doc(x))
...    return text

>>> text = "That Mexico vs USA commercial with trump gets your blood boiling. Race war October 10th. Imagine that parking lot. Gaddamnnnnnn VIOLENCE!!!"
	
>>> text_to_process = data.Field(preprocessing=data.Pipeline(lambda x : custom_processing(x, text_processor) ) )
>>> Dataset_input = [data.Example.fromlist(data=[text],fields=[('text', text_to_process)])]
>>> Dataset_input[0]


<torchtext.data.example.Example object at 0x000001ACFA71A748>

>>> Dataset_input[0].text

['that', 'mexico', 'vs', '<allcaps> usa </allcaps>', 'commercial', 'with', 'trump', 'gets', 'your', 'blood', 'boiling .', 'race', 'war', 'october', '1 0 th .', 'imagine', 'that', 'parking', 'lot .', 'gaddamn <elongated>', '<allcaps> violence </allcaps> ! <repeated>']


>>> processed_text = " ".join(text_processor.pre_process_doc(text))
>>> processed_text


'that mexico vs <allcaps> usa </allcaps> commercial with trump gets your blood boiling . race war <date> . imagine that parking lot . gaddamn <elongated> <allcaps> violence </allcaps> ! <repeated>'


>>> print( " ".join(text_processor.pre_process_doc("10th")))

1 0 th

Word Statistics File not Found. | Receiving 404 error while dowloading the file.

While using the library, the word statistics file is again missing from its original source:

Please fix this as soon as possible and please make some more robust solutions for hosting this file. It looks like people faced this problem in the past as well.

Due to this issue, the production service is stopped. Please fix this asap!

Thanks in advance.!

"maximum recursion depth exceeded" Error

Hi,
I am running a segmenter on a big file and I am facing "maximum recursion depth exceeded" error. I am running on a big file. Is there any way to resolve this?

RecursionError Traceback (most recent call last)
in ()
1 message_p = []
2 for i in range(0,len(message), 1000):
----> 3 message_p.append(tweet2lang(message[i:i+1000]))

482 frames
in tweet2lang(message)
18 for w in msg_tokens:
19 if len(w) >3:
---> 20 temp_seg = seg_tw.segment(w)
21 # print(w,temp_seg)
22 message_seg.append(temp_seg)

/usr/local/lib/python3.6/dist-packages/ekphrasis/classes/segmenter.py in segment(self, word)
133 def segment(self, word):
134 if word.islower():
--> 135 return " ".join(self.find_segment(word)[1])
136 else:
137 return self.case_split.sub(r' \1', word).lower()

/usr/local/lib/python3.6/dist-packages/ekphrasis/classes/segmenter.py in find_segment(self, text, prev)
126 return 0.0, []
127 candidates = [self.combine((log10(self.condProbWord(first, prev)), first), self.find_segment(rem, first))
--> 128 for first, rem in self.splits(text)]
129 return max(candidates)
130