text = "我们明天一起去看《摔跤吧！爸爸》好吗？好！"
expected: ["我们明天一起去看《摔跤吧！爸爸》好吗？", "好！"]
actual: ["我们明天一起去看《摔跤吧！", "爸爸》好吗？", "好！"]

it's working well when using double quote marks, for example:

text = "我们明天一起去看“摔跤吧！爸爸”好吗？好！"
expected: ["我们明天一起去看“摔跤吧！爸爸”好吗？", "好！"]
actual: ["我们明天一起去看“摔跤吧！爸爸”好吗？", "好！"]

FYI, "摔跤吧！爸爸" is the Chinese name of movie "Dangal"

single_character = 'a'
result = PragmaticSegmenter::Segmenter.new(text: single_character).segment

result = []

I've been testing the ellipsis rules with . . . replaced with U+2026 (…) and find that pragmatic segmenter fails when given the actual ellipsis character. I'm probably missing something but shouldn't ellipsis.rb contain rules for the actual ellipsis character?

This brings up a bigger question of how all the variants of symbols are covered. I notice that certain end punctuation characters are explicitly defined, e.g., U+FF1F (？) in punctuation_replacer.rb. However, there are many Unicode characters that could stand in for their ASCII equivalents, e.g., U+FE56 (﹖), U+FE16 (︖), etc. for question marks or U+2047 (⁇), U+2048 (⁈), etc. for double end punctuation and so on for all symbols that are used in segmenting decisions, e.g., (), [], -, ., ...
Chasing all these down seems like a nightmare!

Couldn't it make sense to convert everything to ASCII, i.e, unidecode, segment, and then replace the decoded characters with their original characters? This assumes that all 'equivalent' characters have the same meaning but I believe they do, e.g., ፧ is the Ethiopic question mark which carries the same linguistic meaning as in English. If not, those could be the exceptions rather than the rule.

I would love to hear your thoughts.

Thanks for the great library...from my testing it performs better than spacy, segtok, CoreNLP, and Punkt on English wikipedia data.

My sentence ‘What’s the matter, Jerry?’ called Mr Lorry.

is segmented into two parts.

• ‘What’s the matter, Jerry?
• ’ called Mr Lorry.

I need to parse a text to sentences.
Afterwords I need to find the most frequent word.

I saw how to use the parser in the README.
In order to count the words in a sentence I will need to parse it down to words.
Does 'pragmatic_segmenter' include a REGEX to words (abbreviations etc..)?
Under the hood it must be using some kind of word detector,
can you please share it thus I can also parse a sentence to words by the Same Logic.

Currently pragmatic_segmenter returns an instance of PragmaticSegmenter::Text, which is a subclass of String. As pragmatic_tokenizer checks if text.class == String and also returning segmented objects of a different class than initially passed, I suggest to return strings instead of instances of the only internally used subclass.

I wonder if there is a smarter idea than using #to_s when returning the result, as it would unnecessarily duplicate the strings in memory. Maybe instead of using a subclass of String rather extend the String class with a module providing that single method used? (and use a method name which won't have a chance to mess with anyones code surprisingly if they also decide to extend the String class)

If there is abbreviation at the start of the text, it breaks the sentence correctly and sometimes it doesn't.

My sample sentences were:

TAB. ECOSPRIN 75MG ONE TAB 10PM

Output:

TAB.
ECOSPRIN 75MG ONE TAB 10PM

whereas in the case of Cal. is traditional abbreviation for California it worked fine.

Is this because lib/pragmatic_segmenter/languages/common.rb has cal abbreviation in it?

HI,
In french we have a ... at the end of sentence but here it doesn't segment right I think it's because etc is also an abreviation that is written etc.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<wrapper>
<s>J&#39;aime le sport etc..</s>
<s>. Cependant est ce vrai ?</s>
</wrapper>

Should look like this :

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<wrapper>
<s>J&#39;aime le sport etc...</s>
<s>Cependant est ce vrai ?</s>
</wrapper>

Thanks for your time

I'd like to use the segmenter with a large Java application and would prefer not to use JRuby, etc. I'd prefer to run the segmenter as a light-weight JSON service. Apologies, I don't know Ruby or I would read the code, but can you run this project as an always-on service?

replace_parens_in_numbered_list() calls scan_lists() twice with same paramters. I checked the commit which introduced the duplication and it looks like a mistake.

Reference

• c5edc45#diff-8d1d0f65d5ba9ff24847dd238f3db5e7R103

See example below, when 'clean' parameter is 'false', the asterisk after cat is still removed

pry(main)> s = "I am a dog. Cat.*"
=> "I am a dog. Cat.*"

pry(main)> ps = PragmaticSegmenter::Segmenter.new(text: s, language: 'en', clean: false)
=> #<PragmaticSegmenter::Segmenter:0x00007fdf5d6890e0
 @doc_type=nil,
 @language="en",
 @language_module=PragmaticSegmenter::Languages::English,
 @text="I am a dog. Cat.*">

pry(main)> segments = ps.segment
=> ["I am a dog.", "Cat."]

The method PragmaticSegmenter::AbbreviationReplacer#replace_abbreviation_as_sentence_boundary chains eleven .gsub calls for each of the 23 SENTENCE_STARTERS, which each create a copy of the string and gc the old string. That's 253 GC'd strings for each call.

If replaced by gsub!this would avoid redundant copy/assign operations and speedup the method.

Hi, first of all, thanks for a great gem! I get the following error when trying to segment text using this gem in Ruby 3:

> PragmaticSegmenter::Segmenter.new(text: 'This is apple. This is pen. Ah. Apple pen.').segment
NoMethodError: undefined method `apply' for "This is apple. This is pen. Ah. Apple pen.":String
from vendor/bundle/ruby/3.0.0/gems/pragmatic_segmenter-0.3.22/lib/pragmatic_segmenter/processor.rb:37:in `block in split_into_segments'

It seems that this is a feature(?) of Ruby 3 where calling String methods like String#split will return a String type rather than the subclass (https://github.com/ruby/ruby/blob/v3_0_0/NEWS.md) So, calling split on PragmaticSegmenter::Text will turn it into String and break the chaining that happens internally.

I think if we use delegation instead of inheritance for PragmaticSegmenter::Text, and make any necessary changes in calling code, we can have a version of pragmatic_segmenter that is compatible with Ruby 3 - will try to open a PR over the weekend. I ended up just refactoring the code to use vanilla String instead of a String subclass: #68

Some other tools to consider for comparison:

• Elephant: http://www.aclweb.org/anthology/D13-1146
• English, regex-based: http://moin.delph-in.net/WeSearch/DocumentParsing
• Alpino is listed among the tools not yet tested. It is designed for Dutch and in my evaluation it came out as the best tokenizers/segmenter.

I have a corpus of text that often uses explicit non-breaking spaces (NBSP, U+00A0). They are mainly used to keep together words in the same sentence. They often appear after sentence-medial terminal punctuation (.!?) and before short sentence-final words (as in G. F. Handel composed Water Music for George I.). They were used in order to improve both text document layout and parsing.

Consider the following four cases:

1.  'Peter Pan is a J. M. Barrie play.'   # no NBSP
2.  'Peter Pan is a J. M. Barrie play.'   # NBSP after J.
3.  'Peter Pan is a J. M. Barrie play.'   # NBSP after M.
4.  'Peter Pan is a J. M. Barrie play.'   # NBSP after J. and M.

I was surprised that only the first case was segmented correctly out of the box:

1.  ['Peter Pan is a J. M. Barrie play.']
2. *['Peter Pan is a J.', ' M.', 'Barrie play.']
3. *['Peter Pan is a J. M.', ' Barrie play.']
4. *['Peter Pan is a J.', ' M.', ' Barrie play.']

Of course, it would be easy to just translate all of them to normal spaces and call it a day.

But non-breaking spaces can serve as useful disambiguation to a pragmatic sentence segmenter.

Not every sentence can be accurately segmented using a limited number of rules, so taking advantage of non-breaking spaces can improve the results in some trickier cases without making the checks much more complicated.

(Disclaimer: I have not looked at the rules and do not claim to understand the internals of the program.)

I will provide a few examples to demonstrate how this may be useful. You are welcome to incorporate them as new test cases, even if you ultimately decide not to bother with non-breaking spaces.

Cases 5–7 are extremely similar, but 7 surprisingly produces a different result. In case 8, adding an NBSP seems to fix the problem, presumably without making This behave like He and They internally.

5.  'Sri Lanka was conquered by the Cholas and Raja Raja I. They moved the capital.'   # no NBSP
6.  'Sri Lanka was conquered by Raja Raja I. He moved the capital to Polonnaruwa.'     # no NBSP
7.  'Sri Lanka was conquered by Raja Raja I. This moved the capital to Polonnaruwa.'   # no NBSP
8.  'Sri Lanka was conquered by Raja Raja I. This moved the capital to Polonnaruwa.'   # NBSP before I.

5.  ['Sri Lanka was conquered by the Cholas and Raja Raja I.', 'They moved the capital.']
6.  ['Sri Lanka was conquered by Raja Raja I.', 'He moved the capital to Polonnaruwa.']
7. *['Sri Lanka was conquered by Raja Raja I. This moved the capital to Polonnaruwa.']
8.  ['Sri Lanka was conquered by Raja Raja I.', 'This moved the capital to Polonnaruwa.']

Cases 9–10 show the same sort of “fix”:

9.   'Lu Xun wrote The True Story of Ah Q. He was a Chinese author.'   # no NBSP
10.  'Lu Xun wrote The True Story of Ah Q. He was a Chinese author.'   # NBSP before Q.

9.  *['Lu Xun wrote The True Story of Ah Q. He was a Chinese author.']
10.  ['Lu Xun wrote The True Story of Ah Q.', 'He was a Chinese author.']

Cases 11–14 show that it’s much more difficult than it looks as Feng S. He is a plausible Chinese name:

11.  'Feng S. He was a Chinese diplomat who secretly saved 3,000 Austrian Jews.'  # no NBSP
12.  'He said the story of Feng S. He was a secret. He was a Chinese diplomat.'   # no NBSP
13.  'He said the story of Feng S. He was a secret. He was a Chinese diplomat.'   # NBSP after S.
14.  'I learned the son of Feng S. He was a Chinese-American microbiologist.'     # no NBSP

11.  ['Feng S. He was a Chinese diplomat who secretly saved 3,000 Austrian Jews.']
12.  ['He said the story of Feng S. He was a secret.', 'He was a Chinese diplomat.']
13. *['He said the story of Feng S.', ' He was a secret.', 'He was a Chinese diplomat.']
14.  ['I learned the son of Feng S. He was a Chinese-American microbiologist.']

The non-breaking space currently has no effect in cases 15 and 16. However, they could be easily segmented correctly without adding more rules or rare words like .NET to an explicit list.

15.  'I want to learn Microsoft’s .NET framework.'   # no NBSP
16.  'I want to learn Microsoft’s .NET framework.'   # NBSP before .NET

15. *['I want to learn Microsoft’s .', 'NET framework.']
16. *['I want to learn Microsoft’s .', 'NET framework.']

Resolving this issue could help with the following:

1. Improve accuracy for corpora that (partially) use an existing convention for keeping words together using non-breaking spaces.
2. Improve the existing rules by scrutinizing issues with the cases provided.

I favor this program for my use case because of its stance on embedded quotations.

Pragmatic_Segmenter should be able to return segments of sentences of a maximum size.
E.g.
https://github.com/akalsey/textchunk
https://github.com/algolia/chunk-text

The following code example is donated by https://auditus.cc courtesy of @havenwood.
It is used to ensure that the conversion requests stay within the limits of AWS Polly (1500 char limit)

      optimized_sentences = sentences.each_with_object([]) do |sentence, accumulator| # like reduce except better
        if (accumulator.last&.size&. < 1500) && ((accumulator.last&.size&. + sentence.size& + 2) < 1500)
          accumulator.last << sentence
          accumulator.last << ' '
        else
          accumulator << sentence
          accumulator.last << ' '
        end
      end

Given an array of sentences, the above code counts the number of characters in each sentence and concatenates a sentence and the following sentence together if their sum is less than an arbitray number of characters
(in this case 1500, amazon polly char limit)

https://gist.github.com/havenwood/e9c286c524f2de5649586e7d28fec7af

The above code however do not handle cases where the length of the concatenated string exceeds 1500, here is one possible method (also donated by https://auditus.cc):

      stripped_sentences = optimized_sentences.flat_map { |sentence|
        # sentence_too_long? sentence
        if sentence.size > 1500
          # sentence.scan(/.{1, 1500}/)
          sentence.scan(/(\S{1500,}|.{1,1500})(?:\s|$)/)
        else
          sentence
        end
      }

the output for this text is wrong. In total there are 3 sentences in this
"Joe did not calculatingly set out to steal or defraud," his attorneys wrote in their submission to the judge. "Rather, he -- illegally -- shifted risk to unwitting investors by making false statements. This was not the Ponzi scheme it was made out to be in the media reports of his arrest and it was not a ’fictitious business’ where no tickets or ticket deals ever existed and money was simply stolen for selfish reasons."

The following sentence is broken into three segments when it is supposed to be one sentence?

"Some part of the sentence that says refer to paragraphs (a) and (b) above is usually in correct."

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<wrapper>
<s>Some part of the sentence that says refer to paragraphs</s>
<s>(a) and</s>
<s>(b) above but not always that.</s>
</wrapper>

gem install pragmatic_segmenter
Successfully installed pragmatic_segmenter-0.3.22
Parsing documentation for pragmatic_segmenter-0.3.22
Done installing documentation for pragmatic_segmenter after 0 seconds
1 gem installed

But it does not install:
sevilay@sevilay-linux:~$ apt-cache policy pragmatic_segmenter
N: Unable to locate package pragmatic_segmenter

Please anyone could tell me what is the problem?

Hi again,

I came across this surprising behavior. The first three of these are as expected (for comparison), but the last seems like a bug?

PragmaticSegmenter::Segmenter.new(text: "He has high level training", clean: false).segment 
    # => ["He has high level training"]

PragmaticSegmenter::Segmenter.new(text: "He has high-level training", clean: false).segment 
    # => ["He has high-level training"]

PragmaticSegmenter::Segmenter.new(text: "He has Ph.D. level training", clean: false).segment 
    # => ["He has Ph.D. level training"]

PragmaticSegmenter::Segmenter.new(text: "He has Ph.D.-level training", clean: false).segment 
    # => ["He has Ph.D.", "-level training"]

Thanks!

Hi,

When I use this great tool for preprocessing wikipedia dumps, I encountered the infinite loop and failed with NoMemoryError.

Example:

When we input

'' (a '\0 !\0')

with "en" option to pragmatic segmenter,
sub_4 = sub_characters(sub_3, '!', '&ᓴ&') at https://github.com/diasks2/pragmatic_segmenter/blob/master/lib/pragmatic_segmenter/punctuation_replacer.rb#L55
causes the infinite loop.

I'm wondering if we can solve this problem by escaping '\0' in sub_characters function.

def sub_characters(string, char_a, char_b)
      sub = string.gsub(char_a, char_b).gsub('\\0', '\\\\\0')
      @text.gsub!(/#{Regexp.escape(string)}/, sub)
      sub
end

Thanks!

Hi Kevin - first of all, thanks for your work on this gem.

I'd like to report the following unexpected behavior:

Example 1: Unexpected Result:

Note the period in Inc.

PragmaticSegmenter::Segmenter.new(text: 'The parties to this Agreement are PragmaticSegmenterExampleCompanyA Inc. (“Company A”), and PragmaticSegmenterExampleCompanyB Inc. (“Company B”).', clean: false).segment
["The parties to this Agreement are PragmaticSegmenterExampleCompanyA Inc.", "(“Company A”), and PragmaticSegmenterExampleCompanyB Inc.", "(“Company B”)."]

Example 2: Expected Result:

No period in Inc

PragmaticSegmenter::Segmenter.new(text: 'The parties to this Agreement are PragmaticSegmenterExampleCompanyA Inc (“Company A”), and PragmaticSegmenterExampleCompanyB Inc (“Company B”).', clean: false).segment
["The parties to this Agreement are PragmaticSegmenterExampleCompanyA Inc (“Company A”), and PragmaticSegmenterExampleCompanyB Inc (“Company B”)."]

Example 3: Expected Result:

Note period in Inc. but now there's text between Inc. and the parens

PragmaticSegmenter::Segmenter.new(text: 'The parties to this Agreement are PragmaticSegmenterExampleCompanyA Inc., a fake corporation (“Company A”), and PragmaticSegmenterExampleCompanyB Inc., a fake corporation (“Company B”).', clean: false).segment
["The parties to this Agreement are PragmaticSegmenterExampleCompanyA Inc., a fake corporation (“Company A”), and PragmaticSegmenterExampleCompanyB Inc., a fake corporation (“Company B”)."]

Example 4: Expected Result:

Same as Example 3 but without the comma after Inc.

PragmaticSegmenter::Segmenter.new(text: 'The parties to this Agreement are PragmaticSegmenterExampleCompanyA Inc. a fake corporation (“Company A”), and PragmaticSegmenterExampleCompanyB Inc. a fake corporation (“Company B”).', clean: false).segment
["The parties to this Agreement are PragmaticSegmenterExampleCompanyA Inc. a fake corporation (“Company A”), and PragmaticSegmenterExampleCompanyB Inc. a fake corporation (“Company B”)."]

Would it be possible to get instructions for how to use this on the command line in a pipe? e.g.

$ cat ~/corpora/languages/tatar/wikipedia/wiki.txt |  ruby pragmatic_segmenter.rb 

This gives no output...

Hello,

I've read this thread and followed what's written on it :

#18

The problem is that I got this error message :

irb(main):004:0> require 'pragmatic_segmenter'
NameError: uninitialized constant PragmaticSegmenter::Languages::Common::Abbreviation::Set
from /var/lib/gems/2.3.0/gems/pragmatic_segmenter-0.3.9/lib/pragmatic_segmenter/languages/common.rb:12:in module:Abbreviation' from /var/lib/gems/2.3.0/gems/pragmatic_segmenter-0.3.9/lib/pragmatic_segmenter/languages/common.rb:11:inmodule:Common'
from /var/lib/gems/2.3.0/gems/pragmatic_segmenter-0.3.9/lib/pragmatic_segmenter/languages/common.rb:6:in module:Languages' from /var/lib/gems/2.3.0/gems/pragmatic_segmenter-0.3.9/lib/pragmatic_segmenter/languages/common.rb:5:inmodule:PragmaticSegmenter'
from /var/lib/gems/2.3.0/gems/pragmatic_segmenter-0.3.9/lib/pragmatic_segmenter/languages/common.rb:4:in <top (required)>' from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:inrequire'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in require' from /var/lib/gems/2.3.0/gems/pragmatic_segmenter-0.3.9/lib/pragmatic_segmenter/languages.rb:5:in'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in require' from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:inrequire'
from /var/lib/gems/2.3.0/gems/pragmatic_segmenter-0.3.9/lib/pragmatic_segmenter/segmenter.rb:2:in <top (required)>' from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:inrequire'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in require' from /var/lib/gems/2.3.0/gems/pragmatic_segmenter-0.3.9/lib/pragmatic_segmenter.rb:2:in'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:127:in require' from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:127:inrescue in require'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:40:in require' from (irb):4 from /usr/bin/irb:11:in'

Any help would be greatly appreciated.

Thanks

The text in question:

On April 11, our friends at the Financial Times' "Alphachat" podcast invited THE INDICATOR to host a panel at a bar in Washington, D.C. The joint event was called A Night Of Jargon-Free Economics, and there was even a jargon bell that people in the audience could ring each time someone on the panel used jargon.

Based on rule 14: Multi-period abbreviations at the end of a sentence the text should be segmented at the end of Washington, D.C. and before The joint.

The actual result:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<wrapper>
<s>On April 11, our friends at the Financial Times&#39; &quot;Alphachat&quot; podcast invited THE INDICATOR to host a panel at a bar in Washington, D.C. The joint event was called A Night Of Jargon-Free Economics, and there was even a jargon bell that people in the audience could ring each time someone on the panel used jargon.</s>
</wrapper>

Hi, thanks for writing this and making it available, very cool!

I came across an issue, however, involving a word with leading apostrophes. As far as I can tell, a word with a leading apostrophe (represented as a single-quote) gets interpreted as the start of a single-quoted quotation, slurping up all subsequent sentences until a sentence containing another apostrophe gets encountered.

Probably easiest to explain by example:

$ \pry -r pragmatic_segmenter
[1] pry(main)> def seg(s) ; PragmaticSegmenter::Segmenter.new(text: s).segment ; end
:seg
[2] pry(main)> # This works as expected:
[3] pry(main)> seg("I wrote this last year.  It has four sentences.  This is the third, isn't it?  And this is the last")
[
    [0] "I wrote this last year.",
    [1] "It has four sentences.",
    [2] "This is the third, isn't it?",
    [3] "And this is the last"
]
[4] pry(main)> # This gets confused by the leading apostrophe:
[5] pry(main)> seg("I wrote this in the 'nineties.  It has four sentences.  This is the third, isn't it?  And this is the last")
[
    [0] "I wrote this in the 'nineties.  It has four sentences.  This is the third, isn't it?",
    [1] "And this is the last"
]
[6] pry(main)>

(Of course if the text was using a proper unicode right-quote/apostrophe symbol that wouldn't be an issue.)

Brothers, this text not work:

Zur Pflanzengeographie Chinas China oder Sina sagen die einen, andere nennen es Eine einzige Ordnung ist fur das kosmische Leben . Diese letztgenannte Bezeichnung massgebend, und zwar ist es jene Ordnung, die ihm ist eine Ubersetzung des offiziellen chinesischen . 1'' a die Kultur aufpriigt. " mcns, zhong kuo>. Der zweite-ebenfalls offizielle Riesig gross ist China, das Blutenland. 'on der . 1''amc des chinesischen Reichs lautet, Ostgrenze bis zur \Vestgrenze mnfasst es 6o Langen was so'iel heisst wie, ssfumen>-oder, ssfutenlamf, ein grade, was einem Sechstel des Erdumfangs ent Name, der fur dieses Land der Nordhemisphare spricht; Yom Norden zum Suden sind es so Breiten durchaus berechtigt ist, denn hier ist die l.iclfiiltig grade. Das Klima ist kuhl-gemassigt im . 1''orden mit stc, artenreichste Flora der Alten \'elt 1.orhandcn. ariden, wustenhaften Gebieten, im Suden hingegen Zudem sind Blumen, Baume und Straucher nicht gibt es tropische Regenwalder mit entsprechend ho nur Schmuck und Zierde, sondern sie sind seit ur her Luftfeuchtigkeit. Der Chomolongma im Eln alter Zeit mit Bedeutungstiefe 1erankert. Jahreszeit estmassi' ist mit seinen 884H m der hochste Berg der liches Denken und Fuhlen ist in China lebendig Welt. Der tiefste Punkt in China liegt unter dem geblieben. Auch heute noch 11 eiss jedermann, wel Meeresni'eau und misst -'54 m; er befindet sich in che Pflanzen unter den ''ier Edlen> zu 1erstehen der Nahe des Aydingkolsees. Riesenhafte Berge, sind oder 11er d)ie Drei Freunde des Winters> sind."

It seems that pragmatic segmenter does not correctly split Spanish text when there is no space after a period, e.g. "Hola señorita.Espero que muy bien."
This works with language: 'en' but not with language: 'es'

Dear Kevin,

Thank you for your tool and for comparison with other tools. I was actually looking for test cases for WBD and how different approaches perform on them. Though I have found the results of testing, I have not found "a set of distinct edge cases" that you have created.
I would be glad to use your data for testing since Penn Treebank corpora is indeed too expensive for me.

Best,
Artem

I just realised that for an array with 1000 strings with each 50-300 chars length (url titles and description generated by gottfrois/link_thumbnailer), the following causes a much higher memory load…
array.map {|string| PragmaticSegmenter::Segmenter.new(text: string, language: 'de').segment }
…than this here:
PragmaticSegmenter::Segmenter.new(text: array.join('\r'), language: 'de').segment

In my tests it's a 30-50MB difference, I assume objects inside a #map will not get garbage collected sequentially but all at once, when the entire array has been mapped.

@diasks2 would you consider updating the API to also support:
ps = PragmaticSegmenter::Segmenter.new(language: 'de'); array.map {|string| ps.segment(string) }
…which would allow to reuse the Segmenter object and will most likely reduce memory load? It would be possible to support the old API as well, by additionally allowing initialisation without a passed text and adding an optional argument to #segment.

As a side note, I've noticed lots of #gsub which probably can be replaced with #gsub! to reduce the strain on the garbage collector. I'll submit a PR whenever I ever get to it, unfortunately my current work load only allows me to report the issue and not much more.

Thanks!

there is a test for chinese but language definition is missing.

You can see below the last segment was changed. Spaces were removed from inside the [ ]

s =  "A representative office of the Bank in Paris was first established in 2047. It later became licensed as a banking branch (with unlimited duration) on 29 November 1983, commencing business in 1986. Its legal name is Bank of China Limited (“BoC Ltd”). The registered address of this branch in France is 23-25 avenue de la Grande Armée, Paris 75016, France. The branch is located in the centre of Paris, close to the Arc de Triomphe.  Its registration number is 322 284 696 R.C.S. Paris, and its telephone number is [     ].">

[4] pry(main)> ps = PragmaticSegmenter::Segmenter.new(text: s, language: 'en').segment
=> ["A representative office of the Bank in Paris was first established in 2047.",
 "It later became licensed as a banking branch (with unlimited duration) on 29 November 1983, commencing business in 1986.",
 "Its legal name is Bank of China Limited (“BoC Ltd”).",
 "The registered address of this branch in France is 23-25 avenue de la Grande Armée, Paris 75016, France.",
 "The branch is located in the centre of Paris, close to the Arc de Triomphe.",
 "Its registration number is 322 284 696 R.C.S. Paris, and its telephone number is [ ]."]

I'm trying to test this library against some larger english corpora but I'm running into trouble aligning the results back to the original text. Even with "clean" turned off, the resulting sentences have modified whitespace in a seemingly unpredictable way.

Unfortunately I can't supply the data to demonstrate the problem due to license restrictions. Looking at the code it doesn't appear that there is any easy to to ensure that the returned sentence text is fully unmodified, is that correct?

Hi! So basic stuff. Trying to segment text but not able to. I get this error:

/usr/local/lib/ruby/gems/3.0.0/gems/pragmatic_segmenter-0.3.22/lib/pragmatic_segmenter/processor.rb:37:in block in split_into_segments': undefined method apply' for "":String (NoMethodError) from /usr/local/lib/ruby/gems/3.0.0/gems/pragmatic_segmenter-0.3.22/lib/pragmatic_segmenter/processor.rb:37:in map!'
from /usr/local/lib/ruby/gems/3.0.0/gems/pragmatic_segmenter-0.3.22/lib/pragmatic_segmenter/processor.rb:37:in split_into_segments' from /usr/local/lib/ruby/gems/3.0.0/gems/pragmatic_segmenter-0.3.22/lib/pragmatic_segmenter/processor.rb:30:in process'
from /usr/local/lib/ruby/gems/3.0.0/gems/pragmatic_segmenter-0.3.22/lib/pragmatic_segmenter/segmenter.rb:26:in segment'

I am on ruby ruby 3.0.0p0 and a Mac M1 machine.
Can anybody please help with this?

Can you list all the supported languages? It would be helpful to know if I were to use this in a project.

Hi, I am a new one for Ruby. I have installed the pragmatic_segmenter with the command "sudo gem install pragmatic_segmenter". And could you give my a example rb file to run the tool for text? Because I met the following error when I use the command as your usage in the irb environment.

NameError: uninitialized constant PragmaticSegmenter
from (irb):2
from /usr/bin/irb:12:in `

I tried looking in the docs and even in the code and this doesn't seem to be a feature. So this is either a question or a potential feature request:

Is it possible to turn on a setting to preserve the spaces between sentences? It would be ideal to be able to split sentences such that I can later re-create the entire document from the split parts. Like if I had the string "What is a test? This is a test!", I would get the array ['What is a test? ', 'This is a test!']

0.3.2/lib/pragmatic_segmenter/cleaner.rb:65:in search_for_connected_sentences

PragmaticSegmenter::AbbreviationReplacer::SENTENCE_STARTERS is a constant that does not care about the defined language:

SENTENCE_STARTERS = %w(A Being Did For He How However I In It Millions More She That The There They We What When Where Who Why)

I suggest it to be defined in PragmaticSegmenter::Languages::SomeLanguage , with an empty array as default.

If given the explanation/regex on how to analyze a dataset for appropriate sentence starters), I could provide the array for German.

I stumbled upon the following case where (the otherwise wonderful) PragmaticSegmenter trips up:

It will split a sentence containing a telephone number with letter characters 800.ACME.NOW is split after 800.:

    it "Telephone number with letters" do
      sentence = "If you have questions, call ACME Enterprises at 800.ACME.NOW (800.123.4567) or visit our website at: ACME-Enterprises.com."
      ps = PragmaticSegmenter::Segmenter.new(text: sentence, language: "en")
      expect(ps.segment).to eq([sentence])
    end

Hi, I've ported this library to C# since I've been doing some work which required sentence boundary detection in C#.

Link

I wanted to check what you would like in terms of attribution and naming of the project.

Currently it is called PragmaticSegmenterNet but if you wanted it to use a name that was less directly linked then I'd be happy to do this.

Also I have no understanding of how licenses work so I wanted to check you were happy with the license format below:

The MIT License (MIT)

Original work Copyright (c) 2015 Kevin S. Dias
Modified work Copyright (c) 2017 UglyToad

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

But I have no idea if this is the correct way to manage this situation so let me know if you wanted some other form of license text. I've included attribution in the README but if it would also be polite to provide attribution in some kind of Notices.txt file or similar please let me know.

I hope to feed back any changes or bug-fixes to the Ruby code as I make any further modifications. Please let me know if this is useful or you'd rather avoid this.

Feel free to close this issue straight away. Thanks, Eliot

@diasks2 : It's working very nice on few example I have tried on live demo. Could you provide any resource where I can get python equivalent of this repository? or else can you please guide or give general idea of How should I proceed if I wish to make python one?

What kind of doc_types are supported? I have tried html, but it is not working.

hi,
we have used your segmenter to deal with very big corpus(wiki dump) with size 320MB, it is written in Kazakh but the segmenter going to segment a very very long sentence. Because of a very long sentence, I have a problem to use these sentences as input to my application. Please, could you guide us with a solution?

Thanks

I noticed the following is treated as one sentence:

"This is a sentence. And this is another one. And this is the third."

Is that by design? When not surrounded by double quotes, it is split into 3 sentences.

I was segmenting sentences in a novel, and entire paragraphs were surrounded in double quotes and got treated as one huge sentence.

Hi,

I came across this sentence:

"Mix it, put it in the oven, and -- voila! -- you have cake."

And the exclamation mark is being treated as a sentence break:

[
   "Mix it, put it in the oven, and -- voila!",
  "-- you have cake."
]

More generally it seems that a sentence should be able to contain an em dash clause that ends with an exclamation or question mark:

"There are many -- oh so very many! -- similar cases."
"Some can be -- if I may say so? -- a bit questionable."

You're the expert and I'm not, but I couldn't resist pondering a bit how to handle this :) Here's what occurred to me...

A simple-minded rule would be to avoid a break if the subsequent sentence would start with a dash. But I searched a bit and found that there are counterexamples, such as this line from Moby Dick:

What do you see?—Posted like silent sentinels all around the town, stand thousands upon thousands of mortal men fixed in ocean reveries."

So that would imply that a slightly less simple-minded rule would be to suppress a break due to a question or exclamation mark, if that mark is immediately followed by a dash and if there is a dash somewhere earlier in the sentence.

In any case, thanks as always for the great gem

Is there a way to retain/return the white-spacing that was removed between sentences?

I am looking for a way to detect paragraphs breaks (essentially two newlines).

Rs. (or rs.) is standard abbreviation for currency Rupees (INR).

Hello,

I have been using pragmatic segmenter by following the steps below:
sudo apt-get install ruby-full
gem install pragmatic_segmenter

And after install the pragmatic_segmenter I got this:
Successfully installed pragmatic_segmenter-0.3.22
Parsing documentation for pragmatic_segmenter-0.3.22
Done installing documentation for pragmatic_segmenter after 0 seconds
1 gem installed

And using it to sengemet the sentences in a whole file by applying the code block below:
require 'pragmatic_segmenter'
if (ARGV.length < 3)
puts "\nUsage : ruby2.5 sentenceTokenizer.rb 639-1ISOlangCode textFilePath sentencesFilePath"
exit
end
File.open(ARGV[1]).each do |line1|
line1.delete! ('()[]{}<>|$/'"')
ps = PragmaticSegmenter::Segmenter.new(text: line1, language: ARGV[0], doc_type: 'txt')
sentences = ps.segment
File.open(ARGV[2], "a") do |line2|
sentences.each { |sentence| line2.puts sentence }
end
end

But the problem now I got the error below:
Traceback (most recent call last):
2: from sentenceTokenizer.rb:1:in

' 1: from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in require'
/usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require': cannot load such file -- pragmatic_segmenter (LoadError)
Could you point me please? the same code and same steps have been used before and worked? I am curious, why I got this error now. Help