The Railtie provides an additional way to config via Rails.application.config.elastic_apm but this isn't included when starting eg. Sidekiq workers. Maybe streamline the two?

We want to not show the root.span "app" as a span in the Timeline view in the Elastic APM UI. Here's a current example;

Uncommon queries like this example:

select months.month, count(created_at) from (select DATE '2017-06-09'+(interval '1' month * generate_series(0,11)) as month, DATE '2017-06-10'+(interval '1' month * generate_series(0,11)) as next) months left outer join subscriptions on created_at < month and (soft_destroyed_at IS NULL or soft_destroyed_at >= next) and (suspended_at IS NULL OR suspended_at >= next)group by month order by month desc

result in the abbreviated SQL SELECT FROM (select.

Work is underway to add support for Metrics to the API/server: elastic/apm-server#964
Support needs to be added to this agent too.
The default minimum set of metrics should be supported. TBD which additional metrics will be supported for this agent.

Instead of always waiting for the next transaction to trigger sending recorded transactions to apm-server we should follow the example of the Python agent and have a timer thread that guarantees to at least send every x seconds.

env: ELASTIC_APM_VERIFY_SERVER_CERT

Like python's https://www.elastic.co/guide/en/apm/agent/python/2.x/configuration.html#config-span-frames-min-duration-ms

Vomiting anything to STDOUT should be suppressed by default, and enablable by configuration.

Currently it is the other way around, and it pollutes STDOUT, it's really annoying and cumbersome to configure each of the apps using the agent.

I would suggest instead of:

Doing something like:

if config.enable_environment_warning?

Every Instrumenter has a Subscriber now. The Subscriber is only used when tracing ActiveSupport instrumentation. To remove AS we need to make the Subscriber optional.

Config consists of some things that are actual config options and some things that are more in the vein of application context info. Like, view_paths for example isn't really a config option but rather information about the application context and so we should probably distinguish between the two kinds.

I'm getting the following error from the Elastic-Apm-Server, both with the released gem and master:

** [ElasticAPM] Flushing transactions
** [ElasticAPM] POST returned an unsuccessful status code (400)
** [ElasticAPM] {"error":"Problem validating JSON document against schema: I[#] S[#] doesn't validate with "transaction#"\n I[#] S[#/required] missing properties: "app""}

My app is a simple Hello World to play around with Elastic-Apm using Rails 5.2 and Ruby 2.5.0. The Elastic stack is at version 6.1.2 on a Ubuntu box.

Work is underway to add support for distributed tracing to the API/server: elastic/apm-server#913 / elastic/apm-server#968.
Support needs to be added to this agent too.

I am using mongoid 6.4 in Rails 5.2.
When accessing mongo in the rails console while considering using elastic apm
The following error occurred.
(elastic-apm master a6d6cdd)

[1] pry(main)> Company.first
NoMethodError: undefined method `done' for nil:NilClass
from /usr/local/lib/ruby/gems/2.5.0/bundler/gems/apm-agent-ruby-a6d6cddedc72/lib/elastic_apm/spies/mongo.rb:50:in `pop_event'
[2] pry(main)> Mongo::Monitoring::Global.subscribers
=> {"Command"=>
  [#<ElasticAPM::Spies::MongoSpy::Subscriber:0x000055f77b40fbc8
    @events={1=>nil}>]}

Because transactions do not start in the Rails console, ElasticAPM.span returns nil, so you can not execute pop_event.

The work on the new TimedWorker (#92) brought up a few issues and edge cases that shouldn't be a problem for most uses but for environments with very high load and high error rates (bad deploy, broken external service).

The problem being that we handle both the transmitting of data and checking for flush intervals in the same thread forcing us to sleep for a split second in the loop but then also maybe making us wait before sending the next error or bundle of transactions.

I've got this work in Production ,however would like to support also in test/development ?

rails s -e test / development

Hi,
I'm having some trouble using Elastic APM ruby agent on a Rails app hosted using puma (3.11.2 (ruby 2.4.1-p111)) in production environment (not really a production app, but hosted like so ;) ).
I get errors like (agent side):

[ElasticAPM] {"error":"Problem validating JSON document against schema: I[#] S[#] doesn't validate with \"transaction#\"\n  I[#/process/pid] S[#/properties/process/properties/pid/type] expected number, but got null"}

Occuring only in production mode, when using the rails s in DEV env mode it works like a charm (also using puma as dev server when running rails s also`).

Here is the full apm rails agent output:

D, [2018-02-07T01:11:58.377672 #7856] DEBUG -- : ** [ElasticAPM] Flushing transactions
E, [2018-02-07T01:11:59.144111 #7856] ERROR -- : ** [ElasticAPM] POST returned an unsuccessful status code (400)
D, [2018-02-07T01:11:59.144372 #7856] DEBUG -- : ** [ElasticAPM] {"error":"Problem validating JSON document against schema: I[#] S[#] doesn't validate with \"transaction#\"\n  I[#/process/pid] S[#/properties/process/properties/pid/type] expected number, but got null"}

And here are the apm-server logs:

2018-02-07T01:11:15.732+0100    INFO    [request]       beater/handlers.go:150  handled request {"request_id": "c426b616-b81c-404e-8431-2e0ee8768008", "response_code": 400, "method": "POST", "URL": "http://livia.cypressxt.net:8200/v1/trans
actions", "content_length": 94047, "remote_address": "192.168.1.106", "user_agent": "elastic-apm/ruby 0.3.0"}
2018-02-07T01:11:44.152+0100    INFO    [monitoring]    log/log.go:124  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"apm-server":{"processor":{"transaction":{"validation":{"count":1,"errors":1}}},"server":{"requests
":{"counter":1},"response":{"errors":1}}},"beat":{"cpu":{"system":{"ticks":210,"time":216},"total":{"ticks":1250,"time":1260,"value":1250},"user":{"ticks":1040,"time":1044}},"info":{"ephemeral_id":"feabd8ce-1786-47a7-b306-1d430a4c8dcb","up
time":{"ms":3240010}},"memstats":{"gc_next":4194304,"memory_alloc":2640816,"memory_total":146758632}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"system":{"load":{"1":0.03,"15":0.09,"5":0.0
8,"norm":{"1":0.015,"15":0.045,"5":0.04}}}}}}
2018-02-07T01:11:59.113+0100    ERROR   [request]       beater/handlers.go:384  error handling request  {"request_id": "2bec79d8-1bc8-4aa8-8fc8-a5ac2430059f", "error": "Problem validating JSON document against schema: I[#] S[#] doesn't val
idate with \"transaction#\"\n  I[#/process/pid] S[#/properties/process/properties/pid/type] expected number, but got null"}

And last, here is my puma configuration file used:

#!/usr/bin/env puma

directory '/var/www/myapp/current'
rackup "/var/www/myapp/current/config.ru"
environment 'preproduction'
daemonize true
pidfile "/var/www/myapp/shared/tmp/pids/puma.pid"
state_path "/var/www/myapp/shared/tmp/pids/puma.state"
stdout_redirect '/var/www/myapp/shared/log/puma_access.log', '/var/www/myapp/shared/log/puma_error.log'
threads 5,5
bind 'unix:///var/www/myapp/shared/tmp/sockets/puma.sock'
workers 0
prune_bundler

Don't know exactly why but the agent seems to parse the pid type as null in this configuration.

2018-05-07T15:44:25.647+0200    ERROR    [request]    beater/handlers.go:384    error handling request    {"request_id": "f0472126-608a-41b8-81fb-1741d457e632", "error": "Problem validating JSON document against schema: I[#] S[#] doesn't validate with \"transaction#\"\n  I[#/transactions/0/spans/1/context/db/statement] S[#/properties/transactions/items/properties/spans/items/properties/context/properties/db/properties/statement/type] expected string or null, but got object"}

How I should proceed with debugging to find out what the issue is?
What information should I attach?

The most common use of filters by now is ignoring specific endpoints. We could probably make things a bit easier by including a built-in version of that with an option like ignored_endpoint_patterns or something similar – perhaps matching an option name from one of the other agents.

It seems that commit dfc64b7 (pull request #42 ) introduced a bug that prevents, at least Rails apps, from starting.

 2: from /lib/ruby/gems/2.5.0/bundler/gems/apm-agent-ruby-38fd266dc376/lib/elastic_apm/railtie.rb:15:in `block in <class:Railtie>'
 1: from /lib/ruby/gems/2.5.0/bundler/gems/apm-agent-ruby-38fd266dc376/lib/elastic_apm/railtie.rb:15:in `each'

/lib/ruby/gems/2.5.0/bundler/gems/apm-agent-ruby-38fd266dc376/lib/elastic_apm/railtie.rb:16:in `block (2 levels) in class:Railtie': can't modify frozen ElasticAPM::Config (FrozenError)

I removed the offending line ( elastic_apm/railtie.rb:16 ) and could start again, only getting the "Problem validating JSON document" error again.

I've been using elastic-apm for several weeks now, and it's great at giving me high-level metrics, but it's not very useful for troubleshooting pathological cases.

I'd like elastic-apm to have a configurable runtime threshold, and "sample" any request which runs for longer than that threshold. I realize that this is not how the project is currently designed, architecturally, but samples are way more important for the slow, infrequent cases than the common fast case. When I have slow actions, I want to be able to figure out why they were slow.

Is there anything like that on the horizon?

The sinatra specs sometimes fail on JRuby because of what looks like a race condition eg. here.

With a config/elastic_apm.yml

---
server_url: 'http://monitoring.apm:8200'
log_path: 'log/elastic_apm.log'

I don't see any logs.

Expected behaviour:

There should be a file named elastic_apm.log under RAILS_ROOT/log containing the log information for the apm agent.

Used gem is version 0.7.1

Hello, our team has recently setup apm and we are really impressed. @mikker thanks for making the ruby agent a reality!

Wondering if there has been any discussion on setting default context and tags that would be added onto every transaction. In our typical usage of other agent beats, we end up utilizing that "fields" configuration option in order to add global metadata. For example, one common use case for us is to always add the "team/department" shipping the log because we shard our ELK indices by that field for retention / security.

Our first attempt to solve this was to just set those things inside rack middleware, but then once we added in sidekiq, we of course realized that wasn't sufficient for other spies.

Is this something you would be open to adding to the feature set? If so and you are short on time, I would be happy to submit a PR for review.

env: ELASTIC_APM_TRANSACTION_MAX_SPANS

APM supports capping span count. When a max span count is reached we should stop collecting more and instead bump transaction.span_count.dropped.total.

Before going 1.0 we should spend some more time investigating the performance of the agent. There might be a lot to win with relatively small changes as we've spend very little time specifically on performance so far.

We have some legacy code which is using Redis for session management, which among other things uses Marshal.dump to store Ruby objects in Redis. This results in non-UTF-8 byte sequences in the command, which breaks Elastic APM when it tries to dump the payload to UTF-8 JSON.

Here's a quick repro case (I added this to the Redis injector, but it doesn't really belong there. It demonstrates the break though.)

    it "breaks on binary query payloads" do
      redis = ::Redis.new
      ElasticAPM.start

      transaction = ElasticAPM.transaction 'T' do
        redis.set("session", Marshal.dump(0xdeadbee))
      end.submit 200

      ElasticAPM.agent.instance_variable_get("@serializers").transactions.build_all([transaction]).to_json
      ElasticAPM.stop
    end

  2) Injectors::RedisInjector breaks on binary query payloads
     Failure/Error:
                 alias :"__without_apm_#{method}" :"#{method}"

                 def #{method}(*args, &block)
                   unless ElasticAPM.current_transaction
                     return __without_apm_#{method}(*args, &block)
                   end

                   ElasticAPM.span "#{name}", "#{type}" do
                     __without_apm_#{method}(*args, &block)

     Encoding::UndefinedConversionError:
       "\xEE" from ASCII-8BIT to UTF-8
     # /home/chris/.rvm/gems/ruby-2.4.3/gems/activesupport-4.2.10/lib/active_support/core_ext/object/json.rb:34:in `encode'
     # /home/chris/.rvm/gems/ruby-2.4.3/gems/activesupport-4.2.10/lib/active_support/core_ext/object/json.rb:34:in `to_json'
     # /home/chris/.rvm/gems/ruby-2.4.3/gems/activesupport-4.2.10/lib/active_support/core_ext/object/json.rb:34:in `to_json_with_active_support_encoder'
     # /home/chris/.rvm/gems/ruby-2.4.3/gems/activesupport-4.2.10/lib/active_support/json/encoding.rb:57:in `to_json'
     # ./lib/elastic_apm/span_helpers.rb:20:in `generate'
     # /home/chris/.rvm/gems/ruby-2.4.3/gems/activesupport-4.2.10/lib/active_support/json/encoding.rb:101:in `stringify'
     # /home/chris/.rvm/gems/ruby-2.4.3/gems/activesupport-4.2.10/lib/active_support/json/encoding.rb:35:in `encode'
     # /home/chris/.rvm/gems/ruby-2.4.3/gems/activesupport-4.2.10/lib/active_support/json/encoding.rb:22:in `encode'
     # /home/chris/.rvm/gems/ruby-2.4.3/gems/activesupport-4.2.10/lib/active_support/core_ext/object/json.rb:37:in `to_json_with_active_support_encoder'
     # ./spec/elastic_apm/injectors/redis_spec.rb:35:in `block (2 levels) in <module:ElasticAPM>'

Tangentially, I'd like the ability to exclude the arguments to the Redis command from the APM, since I don't really want large Redis payloads with sensitive being captured and stored in the APM itself.

elastic/apm-server#554

Seems the community has settled on http://rubybunny.info as the go-to lib for this.

Now that we support worker libs that have their own threads and processes we can't count on the user to either use Rails (and load config file in Railtie) or load the file themselves.

Instead let's always load config/elastic_apm.yml in ElasticAPM.start if it exists.

As Resque (by default) forks the process around every job this was a bit tricky. We want this eventually but for now it's put off until after 1.0.

We both used elastic-apm (0.5.1) and newrelic_agent newrelic in our rails app ( rails 5.1.5, ruby 2.4.1)
and we got bad performance down here, newrelic shows use elastic-apm made 5~10 times slower than before.

I know the agent status is in BETA , but we still hope someone can help find out what makes the performance down and offer a solution. thanks

apm-server accepts compressed requests and there's a lot of bytes to be saved with our kind of data.

Prob. using https://ruby-doc.org/stdlib-2.5.0/libdoc/zlib/rdoc/Zlib.html

Makes it obsolete as it caches one (or more?) levels above.

• Update ruby instructions and remove the beta tag (elastic/kibana#20482)
• Docs FAQ section
• Troubleshooting guide (#182)
• Debug information - Log levels, where the logs are located, etc.

Instead users having to discover available integrations by themselves we should come set up with all bells and whistles and instead let them disabled the stuff they don't want.

Changes enabled_injectors to disabled_injectors.

My thinking is they'll do it by returning nil.