knu / ruby-unf Goto Github PK

View Code? Open in Web Editor NEW

75.0 6.0 10.0 137 KB

A wrapper library to bring Unicode Normalization Form support to Ruby/JRuby

License: BSD 2-Clause "Simplified" License

Ruby 100.00%

ruby-unf's Introduction

ruby-unf

Synopsis

A wrapper library to bring Unicode Normalization Form support to Ruby/JRuby

Description

Uses unf_ext on CRuby and java.text.Normalizer on JRuby.

Normalizes UTF-8 strings into and from NFC, NFD, NFKC or NFKD

  # For bulk conversion
  normalizer = UNF::Normalizer.instance
  a_bunch_of_strings.map! { |string|
    normalizer.normalize(string, :nfc) #=> string in NFC
  }

  # Class method
  UNF::Normalizer.normalize(string, :nfc)

  # Instance methods of String
  string.to_nfc

Installation

gem install unf

License

Licensed under the 2-clause BSD license. See LICENSE for details.

ruby-unf's People

Contributors

Stargazers

Watchers

Forkers

banzaiman nirvdrum iang125 jkingdon michaelabon jrafanie voxik benlangfeld chendagouzi orien

ruby-unf's Issues

0.1.3 won't install on JRuby

It looks like installing 0.1.3 on JRuby now tries to install unf_ext, which naturally fails.

How about upstreaming this to CRuby core?

This gem is a good candidate to be upstreamed to Ruby core. Currently, unf-ext breaks every year when there is a new Ruby release. Any thoughts?

Segmentation fault in normalize on Windown (Ruby 1.9.2p180)

I got a segmentation fault with the following two-line program:

require 'unf'
UNF::Normalizer.normalize "\u00FC", :nfd

This is on Windows 7, Japanese version, with
ruby 1.9.2p180 (2011-02-18) [i386-mingw32]
using RubyInstaller for Windows. The error messages are as follows:

C:/Ruby192/lib/ruby/gems/1.9.1/gems/unf-0.1.3/lib/unf/normalizer.rb:25: [BUG] Segmentation
fault
ruby 1.9.2p180 (2011-02-18) [i386-mingw32]

-- control frame ----------
c:0005 p:---- s:0016 b:0016 l:000015 d:000015 CFUNC :normalize
c:0004 p:0021 s:0011 b:0011 l:000010 d:000010 METHOD C:/Ruby192/lib/ruby/gems/1.9.1/gems/u
nf-0.1.3/lib/unf/normalizer.rb:25
c:0003 p:0033 s:0006 b:0006 l:001a94 d:001984 EVAL test_unf.rb:3
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH

c:0001 p:0000 s:0002 b:0002 l:001a94 d:001a94 TOP

-- Ruby level backtrace information ----------------------------------------
test_unf.rb:3:in <main>' C:/Ruby192/lib/ruby/gems/1.9.1/gems/unf-0.1.3/lib/unf/normalizer.rb:25:innormalize'
C:/Ruby192/lib/ruby/gems/1.9.1/gems/unf-0.1.3/lib/unf/normalizer.rb:25:in `normalize'

[NOTE]
You may have encountered a bug in the Ruby interpreter or extension libraries.
Bug reports are welcome.
For details: http://www.ruby-lang.org/bugreport.html

This may be a bug in the Ruby interpreter itself, but it only appears when using unf, so my current guess is that it's some problem in unf itself.

Make String monkey-patching optional

I'm looking to use unf as a replacement for the "unicode" gem in fog, but the String monkey-patching is a non-starter. As a transitive dependency, this isn't something we could push on all our users. If you're not opposed, I'd like to see the String monkey-patching pulled out of the default require path.

Issue with `unf_ext` version 0.0.9 with multiple linux platforms

Installation of unf_ext version 0.0.9 which was released recently i.e. November 11, 2023 https://rubygems.org/gems/unf_ext/versions/0.0.9 causes the below issue:

D | 2023-11-13T16:18:28+00:00 | Installing unf_ext 0.0.9 with native extensions

D | 2023-11-13T16:18:29+00:00 | Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

D | 2023-11-13T16:18:29+00:00 | 

D | 2023-11-13T16:18:29+00:00 | current directory:

D | 2023-11-13T16:18:29+00:00 | /****/lib/ruby/gems/3.1.0/gems/unf_ext-0.0.9/ext/unf_ext

D | 2023-11-13T16:18:29+00:00 | /****/bin/ruby -I /****/lib/ruby/3.1.0 -r

D | 2023-11-13T16:18:29+00:00 | ./siteconf20231113-28048-seados.rb extconf.rb

D | 2023-11-13T16:18:29+00:00 | checking for -lstdc++... yes

D | 2023-11-13T16:18:29+00:00 | creating Makefile

D | 2023-11-13T16:18:29+00:00 | 

D | 2023-11-13T16:18:29+00:00 | current directory:

D | 2023-11-13T16:18:29+00:00 | /****/lib/ruby/gems/3.1.0/gems/unf_ext-0.0.9/ext/unf_ext

D | 2023-11-13T16:18:29+00:00 | make DESTDIR\= clean

D | 2023-11-13T16:18:29+00:00 | 

D | 2023-11-13T16:18:29+00:00 | current directory:

D | 2023-11-13T16:18:29+00:00 | /****/lib/ruby/gems/3.1.0/gems/unf_ext-0.0.9/ext/unf_ext

D | 2023-11-13T16:18:29+00:00 | make DESTDIR\=

D | 2023-11-13T16:18:29+00:00 | compiling unf.cc

D | 2023-11-13T16:18:29+00:00 | unf.cc:39:3: sorry, unimplemented: non-trivial designated initializers not

D | 2023-11-13T16:18:29+00:00 | supported

D | 2023-11-13T16:18:29+00:00 |    };

D | 2023-11-13T16:18:29+00:00 |    ^

D | 2023-11-13T16:18:29+00:00 | unf.cc:39:3: sorry, unimplemented: non-trivial designated initializers not

D | 2023-11-13T16:18:29+00:00 | supported

D | 2023-11-13T16:18:29+00:00 | make: *** [Makefile:214: unf.o] Error 1

D | 2023-11-13T16:18:29+00:00 | 

D | 2023-11-13T16:18:29+00:00 | make failed, exit code 2

D | 2023-11-13T16:18:29+00:00 | 

D | 2023-11-13T16:18:29+00:00 | Gem files will remain installed in

D | 2023-11-13T16:18:29+00:00 | /****/lib/ruby/gems/3.1.0/gems/unf_ext-0.0.9 for inspection.

Issue is raised on unf_ext knu/ruby-unf_ext#74

Could we pin the version for unf_ext below 0.0.9 til the fix is made on unf_ext's end?

unf 0.1.4 not found by heroku

Hi, just got this - is there any knows issue with https://rubygems.org/ - which I use ?

remote: Your bundle is locked to unf (0.1.4), but that version could not be found in any remote: of the sources listed in your Gemfile. If you haven't changed sources, that remote: means the author of unf (0.1.4) has removed it. You'll need to update your remote: bundle to a different version of unf (0.1.4) that hasn't been removed in order remote: to install.

Thanks :)

UNF::Normalizer.instance isn't thread-safe

The UNF::Normalizer.instance class method isn't thread-safe. The consequences of having multiple instances out there might not be huge, but could lead to confusing debug issues.

Release 0.2.0

It looks like the 0.2.0 release won't require building the extension at all if String.method_defined?(:unicode_normalize) is true. This is a really nice benefit on modern Ruby versions. Is there anything holding up promoting the beta to a proper release?

native extensions do not build on Mac OS X 10.10.3

I was supporting @syzygy333 when he discovered a blocking error in installing unf

➜  ~  gem install unf
Building native extensions.  This could take a while...
ERROR:  Error installing unf:
    ERROR: Failed to build gem native extension.

    /opt/rubies/ruby-2.2.1/bin/ruby -r ./siteconf20150408-2004-b0vn44.rb extconf.rb
checking for main() in -lstdc++... yes
checking for ruby/encoding.h... yes
creating Makefile

make "DESTDIR=" clean

make "DESTDIR="
compiling unf.cc
In file included from unf.cc:1:
In file included from ./unf/normalizer.hh:4:
In file included from /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/vector:265:
In file included from /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/__bit_reference:15:
In file included from /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/algorithm:628:
In file included from /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:604:
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/iterator:341:10: fatal error: '__debug' file not found
#include <__debug>
         ^
1 error generated.
make: *** [unf.o] Error 1

make failed, exit code 2

Gem files will remain installed in /Users/dpickett/.gem/ruby/2.2.1/gems/unf_ext-0.0.6 for inspection.
Results logged to /Users/dpickett/.gem/ruby/2.2.1/extensions/x86_64-darwin-14/2.2.0-static/unf_ext-0.0.6/gem_make.out

I've confirmed this error for ruby versions 2.2.1 and 2.0.0-p576 - for kicks I attempted a reinstall of ruby version 2.2.1 but still had no luck.