While there are specs for the REGEX* constants of the gem, we should also document/specify what the other constants are for (e.g. EXTENDED_PICTOGRAPHIC_NO_EMOJI)

Hi,

I find your gem and it seems awesome to spot emojis but I've got some difficulties with it.

When I use Unicode::Emoji::REGEX_ANY, it matches numbers too (as described in the ReadMe).
To me, a number is not an emoji (but I'm probably wrong as I'm the first one to put this as an issue)

What would be the right regexp to match a maximum of emojis without numbers or "regular" characters ?

Scanning family emojis (e.g.) using the "recommended" REGEX results in a smaller family and a separate kid rather than matching the whole emoji in 15 cases:

Unicode::Emoji.list("People & Body", "family").each do |emoji|
  if emoji.length > emoji.scan(Unicode::Emoji::REGEX)[0].length
    puts "\"#{emoji}\".scan(Unicode::Emoji::REGEX) = #{emoji.scan(Unicode::Emoji::REGEX)}"
  end
end

# "👨‍👩‍👧‍👦".scan(Unicode::Emoji::REGEX) = ["👨‍👩‍👧", "👦"]
# "👨‍👩‍👦‍👦".scan(Unicode::Emoji::REGEX) = ["👨‍👩‍👦", "👦"]                                                          
# "👨‍👩‍👧‍👧".scan(Unicode::Emoji::REGEX) = ["👨‍👩‍👧", "👧"]                                                          
# "👨‍👨‍👧‍👦".scan(Unicode::Emoji::REGEX) = ["👨‍👨‍👧", "👦"]                                                          
# "👨‍👨‍👦‍👦".scan(Unicode::Emoji::REGEX) = ["👨‍👨‍👦", "👦"]                                                          
# "👨‍👨‍👧‍👧".scan(Unicode::Emoji::REGEX) = ["👨‍👨‍👧", "👧"]                                                          
# "👩‍👩‍👧‍👦".scan(Unicode::Emoji::REGEX) = ["👩‍👩‍👧", "👦"]                                                          
# "👩‍👩‍👦‍👦".scan(Unicode::Emoji::REGEX) = ["👩‍👩‍👦", "👦"]                                                          
# "👩‍👩‍👧‍👧".scan(Unicode::Emoji::REGEX) = ["👩‍👩‍👧", "👧"]                                                          
# "👨‍👦‍👦".scan(Unicode::Emoji::REGEX) = ["👨‍👦", "👦"]                                                                
# "👨‍👧‍👦".scan(Unicode::Emoji::REGEX) = ["👨‍👧", "👦"]                                                                
# "👨‍👧‍👧".scan(Unicode::Emoji::REGEX) = ["👨‍👧", "👧"]                                                                
# "👩‍👦‍👦".scan(Unicode::Emoji::REGEX) = ["👩‍👦", "👦"]                                                                
# "👩‍👧‍👦".scan(Unicode::Emoji::REGEX) = ["👩‍👧", "👦"]
# "👩‍👧‍👧".scan(Unicode::Emoji::REGEX) = ["👩‍👧", "👧"]

I think this is because of the order of the generated REGEX. I suspect it finds the smaller match first and stops looking... After modifying the generated REGEX in lib/unicode/emoji/generated/regex.rb to REGEX = /(?:(?:👨‍❤️‍👨|👨‍❤️‍💋‍👨|👨‍👩‍👧‍👧|👨‍👦|👨‍👦‍👦|... (moving the 👨‍👩‍👧‍👧 earlier) that particular emoji started matching correctly.

I have no idea how to update the generation logic to move those emojis to the front of the queue though?

First of all, thanks so much for this gem! I try to minimise dependencies so I spent hours trying to figure out a sane way to check for valid emoji code points and sequences before finding this small, focused, well-maintained gem. Thank you so much!

The issue I'm encountering is that a non-RGI sequence is being allowed through by Unicode::Emoji::REGEX. For me it's matching the "GB AGB" flag that the README says is valid but not recommended and also the Northern Ireland flag which is valid but not recommended.

I had a look at the spec and discovered this is by design. The base flag is matched and the other characters are ignored.

Is there a way to more strictly compare? I'm trying to use this REGEX to validate (not extract) recommended emoji in an ActiveRecord model... For me I'd like it to not match at all if the whole sequence is valid but not recommended.

# Simplified model
class Person < ApplicationRecord
  validates :emoji, allow_blank: true, format: {with: Unicode::Emoji::REGEX, message: "can only contain recommended emoji"}
  validate :emoji_length

  private
  def emoji_length
    if emoji.present? && emoji.grapheme_clusters.length > 1
      errors.add(:emoji, "can only contain 1 emoji")
    end
  end
end

# Simplified test (ignoring setup)
class PersonTest < ActiveSupport::TestCase
  test "Allows no emoji (set implicitly)" do
    assert @person.valid?, "P should be valid with no emoji but isn't\n\n\t#{@person.errors.full_messages}"
  end

  test "Does not allow non-recommended (RGI) emoji" do
    @person.emoji = "🏴󠁧󠁢󠁮󠁩󠁲󠁿"

    assert_not @person.valid?, "P should be not be valid with Northern Ireland flag sequence but is"
  end
end

I suppose one way around it could be to extract the matched portion of the emoji and just save that? 🤔

Hello. I’m trying to use the gem to remove emojis from strings, but I’m getting an error when comparing the result with the expected string.

[33] pry(#<RSpec::Matchers::DSL::Matcher>)> regex_result = '🛤🎯📮📘↕⭕🇬🇶🇼🇸📪🛎👨‍🌾🏺🐚🤯'.gsub(Unicode::Emoji::REGEX_ANY, '')
=> "‍"
[34] pry(#<RSpec::Matchers::DSL::Matcher>)> regex_result == ''
=> false

[36] pry(#<RSpec::Matchers::DSL::Matcher>)> Marshal.dump(regex_result)
=> "\x04\bI\"\b\xE2\x80\x8D\x06:\x06ET"
[37] pry(#<RSpec::Matchers::DSL::Matcher>)> Marshal.dump('')
=> "\x04\bI\"\x00\x06:\x06ET"

What I’m doing wrong here? 😅

Hey there!

First off I absolutely love this library, it's so well organised and designed, it has been a total pleasure to use - thank you for your incredible work.

I'm just wondering, is there a straightforward way for me to update the marshal.gz file with newer emoji? I'm trying to build something that will match newer emoji and want to add them to one of the regexes.

I really love the project! Thank you for your hard work!

Hello.

I noticed that unicode-emoji gem takes more memory than I expected from such a library. Just requiring the gem takes 7-8MB. I know that for today's standards it isn't a huge amount, but if many such gems were used then it could add up to unnecessary memory usage.

Here is a method I used to measure memory usage (I use get_process_mem gem):

require 'get_process_mem'

def mem(&block)
  raise ArgumentError, 'missing block' unless block

  mem = GetProcessMem.new
  before = mem.mb
  block.call
  after = mem.mb
  return after - before
end

puts mem { require "unicode/emoji" }

Running multiple times this script, gives me numbers between 7-9MB (most of the time something around 7.6MB).

I also used memory_profiler:

require 'memory_profiler'
report = MemoryProfiler.report do
  require 'unicode/emoji'
end

report.pretty_print

It gives the information where memory is allocated but also how much of it is retained (which in most cases means it will never be freed).

I'm not sure how exactly people use this gem but looking at the content I suspect that most probably they use one of provided regex constants. And this is the case in the application I work on. We literally use single regex from this library (Unicode::Emoji::REGEX).

What can be done to lower memory usage? Here is the idea:

• instead of generating all the Regex constants, they could be generated offline and included directly in a file
• every constant could go to different file and autoload :FOO, File.expand_path('emoji/foo', __dir__) could be used to lazy load it when it is used
• INDEX could be lazy loaded too. If all regexes were generated offline then it would be only used by methods like properties. No method calls? No constant loaded.
• some regexes are quite big (require "object_space"; ObjectSpace.memsize_of(...)), for example REGEX_VALID_INCLUDE_TEXT is almost 0.5MB. I didn't look closely but I think that some big unions like "||..." could be replaced by range "[char1-charN]" (if it is sequence of subsequent characters of course).

If done properly then for usage scenario like mine (single constant), memory usage would be reduce from 7-8MB to a size of that constant (in our case it is 120kB).

Do you think it is worth looking into it?

Feature description

The REGEX and REGEX_VALID regexes will not match textual emoji like 😴︎ (because there not considered emoji by the Consortium)

However, some software might want to detect these, too (see #4 for example)

We seemed to have an issue with the above emoji not being caught by the gem regex? I don't really know how to debug further, but I can say that this doesn't seem to be working correctly?

Happy to try and debug further in any way I can help? Let me know,

Many thanks.