incorrect sentiment due to "!" about vadersentiment HOT 4 CLOSED

The problem is that they don't separate bad from "!!!!" within the program, because it is not included in their PUNC_LIST, so that the token gets read as "bad!!!!", which is not contained in the lexicon... so bad is not included when evaluating the valence.

from vadersentiment.

I just ran into this, and this seems to be from not one but two bugs on top of each other. First, the code's tokenization routine for trailing punctuation only considers sequences of one, two, or three exclamation points, but it ignores anything with 4 or more. Second, it also shouldn't be considering your second sentence as neutral since "bad" is a sentiment-laden word in the lexicon (valence is -2.5). So even if it was ignoring the punctuation marks, the scores should at least come out to be the same as your first sentence.

Unfortunately the maintainers don't appear to be responding to pull requests or issue reports. I see several things outstanding, some of which are quite simple to fix. My own work is in PHP though, so I'm just porting the Python. So I don't have any one-off fixes to offer you. My apologies.

from vadersentiment.

I guess I personally don't infer any meaning from "!!!!" (anyone using that much or more punctuation might not be thinking clearly) so I'd expect the same score in either test case.

from vadersentiment.

Got this corrected in the master of this repo. Thanks for pointing it out!

In my local version, this is what I'm seeing:

from vaderSentiment import SentimentIntensityAnalyzer
vader = SentimentIntensityAnalyzer()
sentences = ["This is so bad",
             "This is so bad!",
             "This is so bad!!",
             "This is so bad!!!",
             "This is so bad!!!!",
             "This is so bad!!!!!",
             "This is so bad!!!!!!"
            ]
for sentence in sentences:
    vs = vader.polarity_scores(sentence)
    print("{:-<25} {}".format(sentence, str(vs)))

outputs:

This is so bad----------- {'neg': 0.6, 'neu': 0.4, 'pos': 0.0, 'compound': -0.6696}
This is so bad!---------- {'neg': 0.615, 'neu': 0.385, 'pos': 0.0, 'compound': -0.6988}
This is so bad!!--------- {'neg': 0.628, 'neu': 0.372, 'pos': 0.0, 'compound': -0.7249}
This is so bad!!!-------- {'neg': 0.641, 'neu': 0.359, 'pos': 0.0, 'compound': -0.7482}
This is so bad!!!!------- {'neg': 0.654, 'neu': 0.346, 'pos': 0.0, 'compound': -0.769}
This is so bad!!!!!------ {'neg': 0.654, 'neu': 0.346, 'pos': 0.0, 'compound': -0.769}
This is so bad!!!!!!----- {'neg': 0.654, 'neu': 0.346, 'pos': 0.0, 'compound': -0.769}

from vadersentiment.