Do documents or large texts need special characters to tokenize before translation? about easynmt HOT 2 CLOSED

Yes, sentence delimiters (. ! ?) are needed. Otherwise you can also chunk it your self and provide a list of sentences.

Also I don't know if the models will work well, they have not been trained on such strange input text.

from easynmt.

I removed stopwords and removed special characters as well, I am working in a sentiment analysis report, I think some techniques for detect the sentiment score does not need contextual polarity, that is the reason I am calculating the score based in the average of scores of all words present by document.

I think this could work:

for translation in model.translate_stream(text, show_progress_bar=False, chunk_size=16, source_lang='es', target_lang='en'):
      translated_text.append(translation)

from easynmt.