The current phone regex does not match international format like "+14129996294".
Maybe this regex could be added as a separate option:

/^\+(?:[0-9] ?){6,14}[0-9]$/

Hi @ankane, url encoded emails are no being filtered. Example:

email%40example.com

I you want I can make a PR and change the regex:

# Current regex
EMAIL_REGEX = /\b[\w+\-.]+@[a-z\d\-]+(\.[a-z\d\-]+)*\.[a-z]+\b/i

# Regex with url encoded emails support
EMAIL_REGEX = /\b[\w+\-.]+(@|%40)[a-z\d\-]+(\.[a-z\d\-]+)*\.[a-z]+\b/i

Hi,

The URL_PASSWORD_REGEX appears to be overly greedy and is matching on values that fall outside of the URL, causing user values to be unnecessarily filtered. See the below test string and the match that is causing the log to return a filtered user as:
{\"foo\":\"app_name\",\"bar\":\"./file.rb\",\"level\":\"error\",\"error\":\"HTTP POST request to: http://localhost:3000//url/50000091/call\",\"request-id\":\"e6ce7cb8-054d-415c-a194-45d5df583648\",\"user\":\"[FILTERED]@oreilly.test\",\"time\":\"2023-06-30T14:25:43Z\"}

Test String:
{\"foo\":\"app_name\",\"bar\":\"./file.rb\",\"level\":\"error\",\"error\":\"HTTP POST request to: http://localhost:3000//url/50000091/call\",\"request-id\":\"e6ce7cb8-054d-415c-a194-45d5df583648\",\"user\":\"[email protected]\",\"time\":\"2023-06-30T14:25:43Z\"}

Match:

//localhost:3000//url/50000091/call\",\"request-id\":\"e6ce7cb8-054d-415c-a194-45d5df583648\",\"user\":\"joey.grady@

Would it be possible to update the Regex to be a bit less greedy for url_passwords?

This looks like an awesome project and interesting approach to data obfuscation in logs. Thank you for sharing it publically!

I tried it out in a non-rails environment (sinatra) and was unable to use it. I've ran into following errors:

NoMethodError: undefined method `info' for #<Logger::Formatter:0x000000013b3cec18 @datetime_format=nil>
	/Users/skatkov/.rvm/gems/ruby-2.7.6/gems/logstop-0.2.8/lib/logstop/formatter.rb:31:in `method_missing'
	/Users/skatkov/.rvm/gems/ruby-2.7.6/gems/http-5.1.0/lib/http/features/logging.rb:33:in `wrap_request'
	/Users/skatkov/.rvm/gems/ruby-2.7.6/gems/http-5.1.0/lib/http/client.rb:107:in `block in wrap_request'
	/Users/skatkov/.rvm/gems/ruby-2.7.6/gems/http-5.1.0/lib/http/client.rb:106:in `each'
	/Users/skatkov/.rvm/gems/ruby-2.7.6/gems/http-5.1.0/lib/http/client.rb:106:in `inject'
	/Users/skatkov/.rvm/gems/ruby-2.7.6/gems/http-5.1.0/lib/http/client.rb:106:in `wrap_request'
	/Users/skatkov/.rvm/gems/ruby-2.7.6/gems/http-5.1.0/lib/http/client.rb:55:in `build_request'
	/Users/skatkov/.rvm/gems/ruby-2.7.6/gems/http-5.1.0/lib/http/client.rb:30:in `request'
	/Users/skatkov/.rvm/gems/ruby-2.7.6/gems/http-5.1.0/lib/http/chainable.rb:20:in `get'
	/Users/skatkov/Code/middleware/lib/studio/client.rb:14:in `block in get'

I used it with standard ruby Logger class, that doesn't have a formatter class.

logger = Logger.new STDOUT
logger.formatter # => nil

But also tried Ougai logger, that has formatter class. But doesn't define info and error classes in formatter, yeat, it's defined in a Ougai::Logger class instead.

logger = Ougai::Logger.new STDOUT
logger.formatter.class # => Ougai::Formatters::Bunyan
l.formatter.methods.include?(:info) # => false
l.methods.include?(:info) # => true

Would you be interested in improving compatibility with non-rails loggers? I'm considering forking this and trying to get it to work with Ougai, but not sure if you want me to bring these changes to original repo.

I am looking through the code and noticed the regex only seems to match IPv4 addresses.

Should IPv6 also be matched?

I'm hoping to do this so that I can add a custom scrubber which used IpAnonymizer and lean on the gem-maintained Regex.

Thoughts?

Would it be possible to filter out any password reset tokens from logged emails?

Hello,

I'm having some trouble with my Logstop configuration. Here's the configuration I'm trying to use, in development.rb:

if ENV['RAILS_LOG_TO_STDOUT'].present? logger = ActiveSupport::Logger.new(STDOUT) logger.formatter = config.log_formatter config.logger = ActiveSupport::TaggedLogging.new(logger) Logstop.guard(config.logger) end

However, I'm encountering issues with this setup, the PII in Rails console is not being obfuscated. Could you provide some guidance on how to properly install and configure Logstop in a Rails 7 application? Any help or pointers to relevant documentation would be greatly appreciated.

Thank you!

I've been having some timeout issues, and noticed one error report mentioned logstop.rb in scrub at line 21:

18. msg = msg.to_s.dup
19. 
20. # order filters are applied is important
21. msg.gsub!(URL_PASSWORD_REGEX, FILTERED_URL_STR)
22. msg.gsub!(EMAIL_REGEX, FILTERED_STR)
23. msg.gsub!(CREDIT_CARD_REGEX, FILTERED_STR)
24. msg.gsub!(CREDIT_CARD_REGEX_DELIMITERS, FILTERED_STR)

logstop.rb in scrub at line 21
logstop/formatter.rb in call at line 12
logger.rb in format_message at line 582
logger.rb in add at line 472
active_record/session_store/extension/logger_silencer.rb in add_with_threadsafety at line 38
active_support/logger_thread_safe_level.rb in add at line 53
logger.rb in info at line 525

Have you ran into any issues like this before?

Feature request

Add an option of Active Record scrubbing. This would allow for filtering Active Record query logs by passing attribute names.

Describe the solution

For the time being logstop scrubs Personal Identifiable Information (PII) based on default and custom regular expressions. This is cool! 🥇

However, when using this gem in Rails, I would like to have an option of providing a list of attributes (via config initializer) to be scrubbed from Active Record query logs.