cjhead / suntranslator Goto Github PK

Stop words need to be removed from the input text. These are words that don't usually add any significant value to a sentence, i.e. 'and', 'is', 'it', etc.

However, some of these common stop words are in the Sun symbol dictionary so the list of stop words should cross reference that first before being removed from the input text.

Input text needs to be cleaned. This includes, but is not limited to, the following:

• Punctuation
• Contractions
• Multiple line feeds
  • i.e. '\n\n'

One of the requirements for the Sun language is for sentences to have seven or less symbols. Many English sentences have many more words so the approach to reducing the overall word count per sentence should be approached with extreme caution.

This should be the result of cleaning the input text.

Parts of speech are going to need to be tagged i.e. nouns, verbs, adjectives, etc. This should help us in many ways, one of which being to achieve the seven word limit for sentences by eliminating words of little value to the concept of the sentence.

Some words that aren't in the Sun symbol dictionary have a more generalized word that is. For example, 'apple' is not in the Sun dictionary but 'fruit' is. Therefore, while processing the input text, we need to find these hypernyms, if available, and reference them to the Sun dictionary.

The input text needs to be lemmatized, i.e. 'running' -> 'run'. Careful consideration needs to be applied here because some words still won't be converted successfully. During my initial testing, words of length four or less seemed to have the greatest inaccuracy. For example, using the NLTK package, 'us' gets lemmatized to 'u'.