newrelic / elixir_agent Goto Github PK

Hi there,

we don't seem to be able to run the agent from our Elixir service running from AWS Fargate.

The AWS metadata endpoint:
https://github.com/newrelic/elixir_agent/blob/master/lib/new_relic/util.ex#L68

is not available from an Fargate/ECS task as per this forum post:
https://forums.aws.amazon.com/thread.jspa?messageID=830807&tstart=0

and we are getting this error upon start of the Elixir service:

Elixir.ArgumentError argument error 
    gen_tcp.erl:153 :gen_tcp.connect/4
    APP_PATH/deps/hackney/src/hackney_connect.erl:247 :hackney_connect.do_connect/5
    APP_PATH/deps/hackney/src/hackney_connect.erl:37 :hackney_connect.connect/5
    APP_PATH/deps/hackney/src/hackney.erl:316 :hackney.request/5
    lib/httpoison/base.ex:630 HTTPoison.Base.request/9
    lib/new_relic/util.ex:80 NewRelic.Util.aws_vendor_hash/1
    lib/new_relic/util.ex:71 NewRelic.Util.maybe_add_vendors/2
    lib/new_relic/harvest/collector/agent_run.ex:62 NewRelic.Harvest.Collector.AgentRun.connect_payload/0

Is Fargate not supported yet or are we missing anything?

Thanks!
Angel

Environment

Erlang/OTP 21 [erts-10.0.5] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [hipe]

Elixir 1.7.2 (compiled with Erlang/OTP 21)

• Agent version 1.3.1

When instrumenting for Distributed Tracing, we know the URL of the External call, so we can use it in favor of the standard {module, function, arity} to identify the external

https://github.com/newrelic/elixir_agent/blob/master/lib/new_relic.ex#L72-L80

https://github.com/newrelic/elixir_agent/blob/master/lib/new_relic/tracer/report.ex#L100

https://github.com/newrelic/elixir_agent/blob/master/lib/new_relic/tracer/report.ex#L121-L124

External/host/all
External/host/client/method

Is your feature request related to a problem? Please describe.
Error reports that contain session information can be considered sensitive and often not desired to be stored in any third party.

Describe the solution you'd like
Either: disable the error reporter from starting or be able to filter out certain error types from the error.

2 This could potentially be difficult depending on the type of error. For instance, an unmatched clause in a phoenix controller will raise an error with the entire plug contents (including session)

Describe alternatives you've considered
The sasl logger could be manually removed at application boot time. This may actually be an acceptable answer, but could end up re-initializing the error handler if the new relic application crashes and restarts.

I would really like to see a separate library (e.g new_relic_phoenix) with Phoenix instrumentation support. That library should probably come with Ecto support out of the box too.

Describe the bug

When using NewRelic.set_transaction_name/1, I expect that the output transaction name is visually correct in New Relic. However, it seems to not match with the recommended way of setting a name. For example:

NewRelic.set_transaction_name("GET//OkrViewController/show")

should produce a GET transaction called OkrViewController/show. In practice, the leading // is kept, even though the GET/POST/PUT is picked up correctly:

Environment

• Elixir & Erlang version (elixir -v):

Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Elixir 1.7.3 (compiled with Erlang/OTP 20)

• Agent version (mix deps | grep new_relic_agent): new_relic_agent 1.0.2 (Hex package) (mix)

Describe the bug
We seem to be getting 413 status with an empty body from New Relic.

Environment

• Elixir & Erlang version (elixir -v):
  Erlang/OTP 21 [erts-10.0.5] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]
  Elixir 1.7.3 (compiled with Erlang/OTP 21)

• Agent version (mix deps | grep new_relic_agent):
  new_relic_agent 1.2.0 (Hex package) (mix)
  locked at 1.2.0 (new_relic_agent) ce96cb01

• Relevant log messages

[mana-production-5dcb56c996-k6x4l] [INFO] - 2018-11-16 07:08:36 - Completed Error Event harvest - size: 0
[mana-production-5dcb56c996-k6x4l] [INFO] - 2018-11-16 07:08:36 - Completed Error Trace harvest - size: 0
[mana-production-5dcb56c996-k6x4l] [INFO] - 2018-11-16 07:08:36 - Completed Transaction Trace harvest - size: 7
[mana-production-5dcb56c996-k6x4l] [INFO] - 2018-11-16 07:08:36 - Completed Metric harvest - size: 24
[mana-production-5dcb56c996-k6x4l] [INFO] - 2018-11-16 07:08:36 - Completed Custom Event harvest - size: 75
[mana-production-5dcb56c996-k6x4l] [INFO] - 2018-11-16 07:08:37 - Completed Transaction Event harvest - size: 58
[mana-production-5dcb56c996-k6x4l] [ERROR] - 2018-11-16 07:08:37 - (413) {}
[mana-production-5dcb56c996-k6x4l] [ERROR] - 2018-11-16 07:08:38 - (413) {}
[mana-production-5dcb56c996-k6x4l] [INFO] - 2018-11-16 07:08:38 - Completed Span Event harvest - size: 6168

is there a way to show traces in breakdown table and App server breakdown Graphs?

Describe the bug
Every time we do hot upgrades (mix edeliver deploy upgrade to production) the app stops reporting data to newrelic. It is solved by doing mix edeliver restart production but obviusly that breaks the purpose of doing hot upgrades.

Environment

• Elixir & Erlang version: Elixir 1.9.1, Erlang/OTP 22
• Agent version : 1.8.0

Is your feature request related to a problem? Please describe.
We have paths for health-checks and swagger documentation which we don't want to report to New Relic.

Describe the solution you'd like
It would be nice to include an env variable like NEW_RELIC_RULES_IGNORE_URL_REGEXES, which allows to have a list of regexes in a comma-separated string to be ignored.

Another option could be a route option like newrelic_ignore.

Describe the bug
When I added @trace adnotations in few places in the code and run all tests I experience a lot of errors from new_relic_agent

Environment

• Elixir & Erlang version (elixir -v): 20 and 1.6.5
• Agent version (mix deps | grep new_relic_agent): 1.0.2

Samples:

17:08:44.865 [error] ** Task <0.6007.0> terminating
** Started from 'Elixir.NewRelic.Harvest.Collector.ErrorTrace.HarvestCycle'
** When function  == #Fun<Elixir.NewRelic.Harvest.Collector.ErrorTrace.Harvester.0.43805112>
**      arguments == []
** Reason for termination ==
** {{killed,{'Elixir.GenServer',call,[<0.1559.0>,send_harvest,5000]}},[{'Elixir.GenServer',call,3,[{file,"lib/gen_server.ex"},{line,836}]},{'Elixir.NewRelic.Harvest.Collector.ErrorTrace.Harvester','-complete/1-fun-0-',1,[{file,"lib/new_relic/harvest/collector/error_trace/harvester.ex"},{line,42}]},{'Elixir.Task.Supervised',do_apply,2,[{file,"lib/task/supervised.ex"},{line,88}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
17:08:44.865 [error] CRASH REPORT Process <0.6007.0> with 0 neighbours exited with reason: {killed,{'Elixir.GenServer',call,[<0.1559.0>,send_harvest,5000]}} in 'Elixir.Task.Supervised':exit/4 line 121
17:08:44.866 [error] ** Task <0.6005.0> terminating
** Started from 'Elixir.NewRelic.Harvest.Collector.TransactionTrace.HarvestCycle'
** When function  == #Fun<Elixir.NewRelic.Harvest.Collector.TransactionTrace.Harvester.0.71467580>
**      arguments == []
** Reason for termination ==
** {{killed,{'Elixir.GenServer',call,[<0.1558.0>,send_harvest,5000]}},[{'Elixir.GenServer',call,3,[{file,"lib/gen_server.ex"},{line,836}]},{'Elixir.NewRelic.Harvest.Collector.TransactionTrace.Harvester','-complete/1-fun-0-',1,[{file,"lib/new_relic/harvest/collector/transaction_trace/harvester.ex"},{line,47}]},{'Elixir.Task.Supervised',do_apply,2,[{file,"lib/task/supervised.ex"},{line,88}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
17:08:44.867 [error] CRASH REPORT Process <0.6005.0> with 0 neighbours exited with reason: {killed,{'Elixir.GenServer',call,[<0.1558.0>,send_harvest,5000]}} in 'Elixir.Task.Supervised':exit/4 line 121
17:08:44.868 [error] ** Task <0.6009.0> terminating
** Started from 'Elixir.NewRelic.Harvest.Collector.TransactionErrorEvent.HarvestCycle'
** When function  == #Fun<Elixir.NewRelic.Harvest.Collector.TransactionErrorEvent.Harvester.0.66859507>
**      arguments == []
** Reason for termination ==

17:18:18.518 [error] Supervisor 'Elixir.NewRelic.Harvest.Collector.TransactionEvent.HarvesterSupervisor' had child 'Elixir.NewRelic.Harvest.Collector.TransactionEvent.Harvester' started with 'Elixir.NewRelic.Harvest.Collector.TransactionEvent.Harvester':start_link() at <0.1369.0> exit with reason killed in context child_terminated
17:18:21.601 [error] Task #PID<0.5404.0> started from NewRelic.Harvest.Collector.TransactionEvent.HarvestCycle terminating
** (stop) exited in: GenServer.call(#PID<0.1369.0>, :send_harvest, 5000)
    ** (EXIT) killed
    (elixir) lib/gen_server.ex:836: GenServer.call/3
    (new_relic_agent) lib/new_relic/harvest/collector/transaction_event/harvester.ex:46: anonymous fn/1 in NewRelic.Harvest.Collector.TransactionEvent.Harvester.complete/1
    (elixir) lib/task/supervised.ex:88: Task.Supervised.do_apply/2
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Function: #Function<0.60780878/0 in NewRelic.Harvest.Collector.TransactionEvent.Harvester.complete/1>
    Args: []
17:18:21.603 [error] Task #PID<0.5402.0> started from NewRelic.Harvest.Collector.CustomEvent.HarvestCycle terminating
** (stop) exited in: GenServer.call(#PID<0.1371.0>, :send_harvest, 5000)
    ** (EXIT) killed
    (elixir) lib/gen_server.ex:836: GenServer.call/3
    (new_relic_agent) lib/new_relic/harvest/collector/custom_event/harvester.ex:54: anonymous fn/1 in NewRelic.Harvest.Collector.CustomEvent.Harvester.complete/1
    (elixir) lib/task/supervised.ex:88: Task.Supervised.do_apply/2
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Function: #Function<0.93471100/0 in NewRelic.Harvest.Collector.CustomEvent.Harvester.complete/1>
    Args: []

but crashes like this as well:

17:09:47.419 [error] GenServer #PID<0.6013.0> terminating
** (UndefinedFunctionError) function :hackney_meck_original.request/5 is undefined (module :hackney_meck_original is not available)

Describe the bug
I am receiving a no function clause matches error on https://github.com/newrelic/elixir_agent/blob/v1.9.0/lib/new_relic/transaction/complete.ex#L27

I am not sure what transform_name_attrs/1 is actually receiving

The stacktrace from honeybadger is just:

lib/new_relic/transaction/complete.ex:27:in `transform_name_attrs/1'
lib/new_relic/transaction/complete.ex:12:in `run/2'
lib/task/supervised.ex:89:in `do_apply/2'
proc_lib.erl:249:in `init_p_do_apply/3'

Environment

• Elixir & Erlang version (elixir -v): 1.7.4 Erlang/OTP 21 [erts-10.2.3]
• Agent version (mix deps | grep new_relic_agent): * new_relic_agent (Hex package) (mix) locked at 1.9.0 (new_relic_agent) 6d8d4332

Is your feature request related to a problem? Please describe.
Elixir applications in New Relic are not linked to Kubernetes. As a result, if the application is running in Kubernetes, CPU% does not show correctly because of the broken link.

Describe the solution you'd like
A link between the New Relic Elixir APM agent and Kubernetes. Documentation from New Relic here: https://docs.newrelic.com/docs/integrations/kubernetes-integration/link-your-applications/link-your-applications-kubernetes

i added include plug use NewRelic.Transaction to my project but i did not see database details. Should i config something else?

Problem

A) How the @trace works for :datastore events

We're attempting to adopt New Relic as Elixir monitoring tool and have stumbled upon some limitations of new_relic_agent. We've picked that NewRelic.Tracer provides the @trace {:op, {category: :datastore}} decorator form as the only available means for tracing database operations but NR reports generated by this API are lacking in places that are taken for granted e.g. in Ruby library. This includes:

• Databases >> [pick operation] >> Time consumption by caller - we should be able to see a chart with caller transactions and an ability to click any of them to inspect the situation from its end

• Databases >> [pick operation] >> Slow queries - we should be able to see slowest SQL statements and click on them to see the full SQL and stack trace of that specific call

• Transactions >> [pick transaction] >> Breakdown table - database operations should be tagged with category "Database" and the name in "Segment" column should be a link to relevant operation in Databases section

• Transactions >> [pick transaction] >> Transaction traces >> [pick trace] >> Database queries - we should be able to see a list of all database operations involved in specific transactions grouped by SQL with a total duration and number of calls (useful for picking N+1 issues)

• Transactions >> [pick transaction] >> Transaction traces >> [pick trace] >> Trace details - database operations should have the database icon which expands upon clicking into specific SQL query details and stack trace like in Databases section

This basically means that we're missing on ability to cross-reference transactions and database operations back and forth, an ability that's SUPER useful for database-centric applications such as ours, but it's the case of most Phoenix projects.

B) The lack of "post-factum" API alternative to @trace

In addition, with the NewRelic.Tracer as the only API for reporting spans with "datastore" category, we're missing on ability to report spans post-factum e.g. when reports come through telemetry only after the operation ends - such as the Ecto case, also important for most Phoenix users.

I can only guess that this is why the effort started in binaryseed/new_relic_ecto#4 was not finished, but also wasn't taken over by the community - because the base library seems to lack required API. As established in #53, such API should of course be database- and library-agnostic and is not expected to be automatic - it should merely lay out the foundation for new_relic_ecto and similar.

I believe we're talking here about one of biggest blockers for pushing forward adoption of New Relic in Elixir/Phoenix projects for many companies such as ours which successfully use & love NR with many stacks & for years but got stuck on Elixir app database monitoring.

Solution proposal

It'd be great to see API similar to following:

NewRelic.report_datastore_event(
  datastore, # module or arbitrary string
  resource, # it may be table name, schema name or misc DB resource name string
  operation, # insert / update / delete / select... - arbitrary string 
  query, # sql statement or equivalent for any (e.g nosql) data store
  duration # in nanoseconds as returned from System.monotonic_time() that the agent uses
)

With sample call as follows:

NewRelic.report_datastore_event(
  MyApp.Repo,
  "users",
  "insert",
  "SELECT count(id) FROM users",
  5_000
)

This function would report all kinds of transaction spans, distributed traces and aggregates such as NewRelic.Tracer.Report.call/5 does, but fill in the aforementioned gaps of that function, i.e.

• being callable post-factum
• making database events appear as such in transactions
• reversely - linking transactions to database operations
• including stacktrace and statement string in database operations
• naming database operation with actual datastore name instead of "Database"

What may be helpful for resolving this issue and useful for those looking for instant fill-in for Ecto instrumentation is our current solution available in https://github.com/surgeventures/new_relic_addons that tries to get the most out of current agent API in order to bring in Ecto instrumentation and some handy function decorators similar to @trace.

I'm not familiar with NR APIs and can't work around this issue myself, but please let me know if I can be of any assistance here. Thanks!

It would be nice to get Heroku Dyno names reported to New Relic (web.1, web.2 etc).

Here's how we solved it in New Relixir:
TheRealReal/new-relixir@4452728

Store the number of processes spawned during the request as an attribute on the Transaction event

Describe the bug
We occasionally use :recon_trace.calls({Module, function, :return_trace}, n) in production to debug issue (if the logs are not sufficient to track down the issue). With new_relic_agent enabled, this leads to following error, although the trace appears to work

** can only have one tracer per process

1. is there any way to fix this?
2. if not possible, is it possible to disable/enable new_relic for the duration via remote_console? I tried to stop new_relic via Application.stop(x), but it didn't go well, as all the calls to newrelic modules were failing.

Environment

• Elixir & Erlang version (elixir -v):

Erlang/OTP 21 [erts-10.2.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]

Elixir 1.7.4 (compiled with Erlang/OTP 21)

• Agent version (mix deps | grep new_relic_agent):

new_relic_agent 1.9.3

Is it possible to wrap HTTPoison with New Relic instrumentation in modules where HTTPoison is wrapped with a macro already?

defmodule Foo do
  use HTTPoison.Base
end

Describe the bug
I believe I've some undocumented behaviour of the New Relic agent, which may be a bug. When you have multiple functions with the same name + arity but different guards etc and apply a @trace to only one of them, the others seem to be ignored.

For example:

defmodule Example do
  use NewRelic.Tracer

  def random(value) when value == [], do: nil

  @trace :random
  def random(value), do: <something you really don't want to do if value is []>
end

In the above, all calls to random/1 would go to the second definition, regardless of value. I can't find any documentation of this behaviour and haven't yet dug into why it happens exactly (I assume NR is replacing the function in the module attributes with it's own macro-ed function and in doing so removes the existing ones). Adding a trace to both functions prevents this.

Environment

• Elixir & Erlang version (elixir -v): Elixir 1.7.4 (compiled with Erlang/OTP 21)
• Agent version (mix deps | grep new_relic_agent): 1.9.7

Hi,

I’ve deployed an Elixir application (based on Phoenix) recently and the NewRelic reporting doesn’t seem to make sense to me. Maybe I’m missing something but the Elixir metric is always twice as high as the Response time.

Am I missing something? Thanks for your help, the agent is working pretty well otherwise 😄

Environment

• Elixir & Erlang version: 1.8.1
• Agent version: new_relic_agent 1.10.0

Describe the bug

When running in a test environment we have disabled NewRelic agent through the configuration, however, sampling still occurs. This presents a problem if the process being sampled has already exited, as Process.info/2 called in sampler/process.ex#L75 returns nil. This results in an argument error as seen below:

2019-08-14 13:16:08.501 [error] GenServer NewRelic.Sampler.Process terminating
** (ArithmeticError) bad argument in arithmetic expression
    (new_relic_agent) lib/new_relic/sampler/process.ex:79: NewRelic.Sampler.Process.take_sample/1
    (new_relic_agent) lib/new_relic/sampler/process.ex:62: NewRelic.Sampler.Process.store_pid/3
    (new_relic_agent) lib/new_relic/sampler/process.ex:24: NewRelic.Sampler.Process.handle_cast/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:"$gen_cast", {:sample_process, #PID<0.6739.0>}}
State: %{pids: %{#PID<0.1314.0> => true, #PID<0.1315.0> => true, ..., reductions: 46}}}

Environment

• Elixir & Erlang version (elixir -v):

Erlang/OTP 21 [erts-10.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Elixir 1.9.1 (compiled with Erlang/OTP 21)

• Agent version (mix deps | grep new_relic_agent):

new_relic_agent 1.9.11

Note: Although we are a few versions behind, the code path for this still exists in master.

Is your feature request related to a problem? Please describe.

Our app uses proxies to get through to the Phoenix server. We want to track time spent in proxy as request queueing.

Describe the solution you'd like

The same request queueing feature that Ruby has, based on X-Request-Start header

Describe alternatives you've considered

I have not

Is your feature request related to a problem? Please describe.
I would like to have Non-web transactions represented in NewRelic transactions list. As an example, I would like to see ConsumerSupervisor children processes as transactions.

Describe the solution you'd like
It would be great to have module NewRelic.NonWebTransaction which I could use in my module and then annotate function, which I would like to create transaction for (similar to NewRelic.Trace, but created transaction instead of trace span).

I would be eager to help implementing it.

Describe alternatives you've considered
I'm wrapping my functions with something like:

def new_relic_transaction(name, fun) do
  NewRelic.Transaction.Reporter.start()
  NewRelic.set_transaction_name(name)
  fun.()
after
  conn = %Plug.Conn{method: "NONWEB"} # Fake connection
  NewRelic..Transaction.Reporter.stop(conn)
end

But it does report transactions as Web.

Instead of seeing the URL of external services I'm tracing with the New Relic instrumentation, I'm seeing the function name, e.g NewRelic.Instrumented.HTTPoison.get

Am I doing something wrong or is it a bug?

Describe the bug

When setting a transaction name using set_transaction_name/1, the result comes into New Relic UI as "Root Path". We've tried the names "Elixir.Api.Method.FrontendInstallations.Index" and "/Elixir.Api.Method.FrontendInstallations.Index". The data appears to make it to the Transactions event correctly in New Relic Insights.

before and after the /

          "event": {
            "framework_name": "/Phoenix/PortalWeb.FrontController/execute",
            "custom_name": "/Elixir.Api.Method.FrontendInstallations.Index",
            "name": "WebTransaction/Elixir.Api.Method.FrontendInstallations.Index",
            "timestamp": 1571171661879
          }

Environment

• Elixir & Erlang version (elixir -v):

Erlang/OTP 21 [erts-10.1.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]

Elixir 1.7.3 (compiled with Erlang/OTP 19)

That compiled with looks weird, but I don't think it's related

• Agent version (mix deps | grep new_relic_agent):
  1.12.0

Hello team.

Any chance we can get a new release published? We are after a fix that prevents the agent to crash when the AWS metadata endpoint is not available (when running within an AWS Fargate task).

Thanks a lot!
Angel

Describe the bug
Not able to customize NewRelic.Logger formatting

Currently we have unified custom log format(json) across the org. We use custom formatting and as nearly all the libraries use the standard Logger, this works without any issue.

Any specific reason for rolling custom logger instead of the standard Logger. It would be great if the library can use the standard Logger

Environment

• Elixir & Erlang version (elixir -v):

Erlang/OTP 21 [erts-10.1.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]

Elixir 1.7.3 (compiled with Erlang/OTP 21)

• Agent version (mix deps | grep new_relic_agent):

new_relic_agent 1.2.0

In the Readme it's said that there should be "some general BEAM VM stats", but I'm not seeing it anywhere. Am I missing something?

Environment

• Elixir 1.6.6 (running inside a docker container)
• docker 17.03.1-ce
• :new_relic_agent, "1.0.2"
• :phoenix, "1.3.2"
• :cowboy, "1.1.2"

When starting an application and we're unable to connect to collector.newrelic.com (like on an airplane), the new_relic_agent crashes on startup.

** (Mix) Could not start application new_relic_agent: NewRelic.Application.start(:normal, []) returned an error: shutdown: failed to start child: NewRelic.Harvest.Supervisor
    ** (EXIT) shutdown: failed to start child: NewRelic.Harvest.Collector.Supervisor
        ** (EXIT) shutdown: failed to start child: NewRelic.Harvest.Collector.AgentRun
            ** (EXIT) an exception was raised:
                ** (CaseClauseError) no case clause matching: {:failed_connect, [{:to_address, {'collector.newrelic.com', 443}}, {:inet, [:inet], :nxdomain}]}
                    (new_relic_agent) lib/new_relic/harvest/collector/agent_run.ex:20: NewRelic.Harvest.Collector.AgentRun.init/1
                    (stdlib) gen_server.erl:374: :gen_server.init_it/2
                    (stdlib) gen_server.erl:342: :gen_server.init_it/6
                    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

This becomes a significant problem if our production systems temporarily can't resolve collector.newrelic.com, because it means we can't start our production apps. Ideally the application would still start up and periodically try to reconnect.

Environment

• Elixir version: 1.9.1
• Erlang version: 22.0.7
• Agent version: 1.10.0

Describe the bug

We're seeing a few error reports daily with the following error message:

Erlang error: {:key_exists, {2688, 1554362690713334865}}

coming from lib/new_relic/sampler/top_process.ex:51:in measure_and_insert/2.

Environment

• Elixir & Erlang version (elixir -v): 1.8.1
• Agent version (mix deps | grep new_relic_agent): 1.8.0

Is your feature request related to a problem? Please describe.
Traced functions are so fast to use and work great, but they include the args of the function, and that contains a private API key in my case.

Describe the solution you'd like
I'd like all of the same tracing, but the args marked as [] or [:redacted]

Describe alternatives you've considered
I am thinking about finding a way to structure the function without accepting the input. Unfortunately I don't see a way yet. I am also considering replicating what @trace does, but that ends up being fairly complex.

Is your feature request related to a problem? Please describe.
As resolved in PR #173, the metadata doesn't capture the results of a query that returns an error tuple. Previously this would raise and detach. Now it will simply ignore the result.

Describe the solution you'd like
Instead of ignoring this potentially useful information, we could try to capture it.

Here is an example of an error (including params):

%{
  params: [9000000000, %{account_id: 1, …}, 18508, \"1\", ~N[2020-02-04 21:42:30], ~N[2020-02-04 21:42:30]],
  query: \"INSERT INTO \\\"table\\\" (\\\"column1\\\",\\\"column2\\\",\\\"column3\\\",\\\"column4\\\",\\\"inserted_at\\\",\\\"updated_at\\\") VALUES ($1,$2,$3,$4,$5,$6) RETURNING \\\"id\\\"\",
  repo: MyApp.Repo,
  result: {:error, %Postgrex.Error{
    connection_id: 21417,
    message: nil,
    postgres: %{
      code: :foreign_key_violation,
      constraint: \"table_foreign_id_fkey\",
      detail: \"Key (id)=(9000000000) is not present in table \\\"table\\\".\",
      file: \"ri_triggers.c\",
      line: \"2474\",
      message: \"insert or update on table \\\"table\\\" violates foreign key constraint \\\"table_foreign_id_fkey\\\"\",
      pg_code: \"23503\",
      routine: \"ri_ReportViolation\",
      schema: \"public\",
      severity: \"ERROR\",
      table: \"table\",
      unknown: \"ERROR\"
    },
    query: nil}
  },
  source: nil,
  type: :ecto_sql_query
}

We currently use the Logger option for new_relict_agent.logger. We set the application log level to :info, but info level doesn't seem to be ideal level for new_relic_agent

new_relic_agent - Completed Error Event harvest - size: 0
new_relic_agent - Completed Span Event harvest - size: 86
new_relic_agent - Completed Metric harvest - size: 20
new_relic_agent - Completed Error Trace harvest - size: 0
new_relic_agent - Completed Custom Event harvest - size: 68
new_relic_agent - Completed Transaction Trace harvest - size: 0
new_relic_agent - Completed Transaction Event harvest - size: 12
new_relic_agent - Completed Metric harvest - size: 20
new_relic_agent - Completed Transaction Trace harvest - size: 0

We are not interested in the above information and would probably be only interested in warnings/errors. is there any way to control the log level of the agent? I initially thought I could use compile time purging, but as new_relic_agent uses Logger.log directly, I am not sure whether anything could be done at compile time.

Describe the bug
The moment we include new_relic_agent, even if we don't configure it or reference it anywhere, local 500s lose their stack traces making it impossible to diagnose problems.

This is obviously frustrating as our ops team loves the integration in production but we are forced to remove it from our mix files (as well as comment out all the places that configure it) to get any work done.

Is it reasonable to expect the plugin to have no affect on our app if it's not configured? This is how Sentry and Appsignal work; behavior we've come to expect.

Example:
Simply by including new_relic_agent in our application, w/ no references to it anywhere:

14:38:52.634 request_id=956da5e18863478bf4cc6a0f20365dcc [info] GET /consumer/programs/8wWw
14:38:52.657 request_id=956da5e18863478bf4cc6a0f20365dcc [debug] Processing with ConsumerWeb.ProgramController.show/2
14:38:52.693 request_id=956da5e18863478bf4cc6a0f20365dcc [info] Sent 500 in 58ms

when we comment out that line from our deps, the same request:

14:41:31.063 request_id=2df4a040ffa54883af01c902ba5c7e62 [info] GET /consumer/programs/8wWw
14:41:31.114 request_id=2df4a040ffa54883af01c902ba5c7e62 [debug] Processing with ConsumerWeb.ProgramController.show/2
14:41:31.167 request_id=2df4a040ffa54883af01c902ba5c7e62 [info] Sent 500 in 103ms
14:41:31.179 [error] #PID<0.3230.0> running API.Endpoint (connection #PID<0.3229.0>, stream id 1) terminated
Server: noreaga:80 (http)
Request: GET /consumer/programs/8wWw
** (exit) an exception was raised:
    ** (FunctionClauseError) no function clause matching in ConsumerWeb.ProgramResolverFixture.program_by_id/2
        stack trace continues...

Thanks for any guidance here!

Environment

• Elixir & Erlang version: Elixir 1.8.1 (compiled with Erlang/OTP 21)
• Agent version: 1.9.5

Describe the bug
I was wondering if the error I'm getting is due to old deps. Upgraded Elixir from 1.3.4 to 1.7.3 to be able to use new relic's agent

== Compilation error in file lib/employer_gateway.ex ==
** (RuntimeError) no routes defined in module MyApp using Plug.Router
    (plug) expanding macro: Plug.Router.__before_compile__/1
    lib/my_app.ex:1: MyApp (module)
    (elixir) lib/kernel/parallel_compiler.ex:206: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/6

Deps

defp deps do
    [
      {:cors_plug, "~> 1.2"},
      {:cowboy, "~> 2.0"},
      {:credo, "~> 0.3", only: [:dev, :test]},
      {:excoveralls, "~> 0.5", only: :test},
      {:gettext, "~> 0.9"},
      {:httpoison, "~> 1.0"},
      {:jiffy, "~> 0.14.7", override: true},
      {:joken, "~> 1.5.0"},
      {:jose, "~> 1.3"},
      {:maru, "~> 0.13"},
      {:new_relic_agent, "~> 1.2.0"},
      {:phoenix, "~> 1.4"},
      {:poison, "~> 2.2.0"}, # here be dragons
      {:plug, "~> 1.6"},
      {:pubsub_client, "~> 0.3.1", git: "[email protected]:itriage/pubsub_elixir_client.git", tag: "test-elixir-1.7-rc.0" },
      {:uuid, "~> 1.1"}
   ]
  end

Environment
Development

• Elixir & Erlang version (elixir -v): Elixir 1.7.3
• Agent version (mix deps | grep new_relic_agent):

Environment :

• Elixir (elixir -v):

Erlang/OTP 20 [erts-9.3.3.11] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:10] [hipe] [kernel-poll:false]

Elixir 1.8.2 (compiled with Erlang/OTP 20)

Describe the bug

Error while upgrading from v1.10.0 to 1.10.1 with a minimal configuration (just the minimal on the readme) without changing anything except the following :

Changes :

1. Before update :

  defp deps do
    [{...,
     {:new_relic_agent, "~> 1.10"},
     {:new_relic_phoenix, "~> 0.2"},
   ]
  end

• Agent version (mix deps | grep new_relic_agent):

* new_relic_agent (Hex package) (mix)
  locked at 1.10.0 (new_relic_agent) 008a074a

2. After update :

  defp deps do
    [{...,
     {:new_relic_agent, "== 1.10.1"},
     {:new_relic_phoenix, "~> 0.2"},
   ]
  end

• Agent version (mix deps | grep new_relic_agent):

* new_relic_agent (Hex package) (mix)
  locked at 1.10.1 (new_relic_agent) e200b5d2

Generate the following error :

[error] GenServer NewRelic.Harvest.Collector.Metric.HarvestCycle terminating
** (ArgumentError) argument error
    (stdlib) :ets.lookup(NewRelic.Harvest.Collector.AgentRun, :data_report_period)
    (new_relic_agent) lib/new_relic/harvest/collector/agent_run.ex:108: NewRelic.Harvest.Collector.AgentRun.lookup/1
    (new_relic_agent) lib/new_relic/harvest/collector/harvest_cycle.ex:124: NewRelic.Harvest.Collector.HarvestCycle.trigger_
harvest_cycle/1
    (new_relic_agent) lib/new_relic/harvest/collector/harvest_cycle.ex:63: NewRelic.Harvest.Collector.HarvestCycle.handle_in
fo/2
    (stdlib) gen_server.erl:616: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:686: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: :harvest_cycle
State: %{harvest_cycle_key: :data_report_period, harvester: nil, name: NewRelic.Harvest.Collector.Metric.HarvestCycle, super
visor: NewRelic.Harvest.Collector.Metric.HarvesterSupervisor, timer: nil}
[warn] Logger dropped 180 OTP/SASL messages as it had 555 messages in its inbox, exceeding the amount of :discard_threshold
500 messages. The number of messages was reduced to 375 (75% of the threshold)

Any ideas ?
Can it be related to #138 ?

Describe the bug
I've installed the NewRelic agent and used the NewRelic.Tracer module to capture spans. However when I look at the trace details in the NewRelic dashboard, instead of ModuleName.function_name as described in the docs, I see PIDs:

Here is what our instrumentation looks like:

# In our router
defmodule MyApp.Router do
  pipeline :pipeline do
    plug(NewRelic.Transaction)
    ...
  end

  ...
end

# In the modules we are trying to instrument.
defmodule MyModule do
  ...
  use NewRelic.Tracer
  
  @trace :function
  def function(arg) do
    ...
  end

  @trace :other_function
  def other_function do
    ...
  end
end

Is this a bug with the agent or a configuration issue on our end?

Environment

• Elixir & Erlang version (elixir -v): Elixir 1.8.2 (compiled with Erlang/OTP 21)
• Agent version (mix deps | grep new_relic_agent): 1.9.11

Since upgrading to new_relic_agent 1.15.0, database transactions are not uploaded to New Relic.

The issue seems to be that Telemetry.Ecto.Handler expects config for hostname:, port: and database:, which are optional Ecto configuration fields.

Here's the top of the error log.

[error] Handler {:new_relic_ecto, :axdb} has failed and has been detached. Class=:error
Reason={:badmatch, %{database: "development", pool_size: 10}}
Stacktrace=[{NewRelic.Telemetry.Ecto.Handler, :handle_event, 4, [file: 'lib/new_relic/telemetry/ecto/handler.ex', line: 18]},

As far I see in the code, there is no support for nested segments for Transactions. Example:

defmodule MyMod do
  use NewRelic.Tracer

@trace :outer
def outer() do
  :timer.sleep(1000)
  inner()
end

@trace :inner
def inner() do
  :timer.sleep(1000)
  :ok
  end
end

The segments will be sent to NewRelic in flat namespace, i.e there will be no indication that the inner function is nested in outer part. It may lead to confusing data displayed in the Transaction traces in the Trace details tab.

Most likely I will develop this feature in the elixir-agent plugin for our purposes (and maybe create PR here later on). Hence my question is:
Where can I find the documentation of REST API of the NewRelic?
I mean the API that the plugin uses for pushing transactions, segments and spans.
I believe there is a support for nested segments in NewRelic in general, so how would look like http request ?

Describe the bug
We're seeing new relic elixir processes getting stuck and chewing up our available CPU. After enough traffic these processes never die and continue to process while memory slowly grows and eventually our pods are killed and restarted.

Using the observer_cli (https://github.com/zhongwencool/observer_cli) to monitor what is using the most memory and allocated the most CPU work (reductions) we have seen that consistently Elixir.Newrelic.Harvest.Collector.Tran, Elixir.NewRelic.Transaction.Monitor, and especially Elixir.NewRelic.Transaction.Reporter topping our lists. After digging around this is our best guess:

Elixir.NewRelic.Transaction.Reporter seems to be the most harmful stuck process. The "Current Function" section always reports it is in Elixir.Enum:-reduce/3-lists^foldl and these are the processes reporting the largest reductions.

Looking through the elixir docs and this code base our best guess is that it is stuck in NewRelic.Sampler.Process.record_samples

None of us here are well versed in Elixir and are trying our best to explain this. If I can provide any additional information please let me know what would be useful. I'll include a few screenshots as well. Most likely we will be removing new relic until we can get this issue resolved.

Environment

Erlang/OTP 21 [erts-10.3.4] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]
Elixir 1.8.1 (compiled with Erlang/OTP 21)

New Relic agents:
1.9.4 (https://github.com/newrelic/elixir_agent/releases/tag/v1.9.4)
1.10.0 (https://github.com/newrelic/elixir_agent/releases/tag/v1.10.0)

Describe the bug
When using the elixir_agent we are having issue with making dialyzer happy. The code works fine (we can see the output in new relic website) but when we run dialyzer it comes up with an error. I believe it's caused by the elixir_agent because when we remove :new_relic_agent dependency and its setup code, dialyzer passes.

Here is the dialyzer error:

[:asn1, :certifi, :compiler, :cowboy, :cowlib, :crypto, :elixir, :exsync, :hackney, :httpoison, :idna, :kernel, :logger, :metrics, :mime, :mimerl, :plug, :plug_cowboy, :plug_crypto, :public_key, :ranch, :reverse_proxy_plug, :ssl, :ssl_verify_fun, :stdlib]
Looking up modules in dialyxir_erlang-22.0.4_elixir-1.9.0_deps-dev.plt
Finding applications for dialyxir_erlang-22.0.4_elixir-1.9.0_deps-dev.plt
Finding modules for dialyxir_erlang-22.0.4_elixir-1.9.0_deps-dev.plt
Removing 10 modules from dialyxir_erlang-22.0.4_elixir-1.9.0_deps-dev.plt
:dialyzer.run error: Analysis failed with error:
Could not scan the following file(s):
  Could not get Core Erlang code for: anubis/_build/dev/lib/certifi/ebin/certifi.beam

Last messages in the log cache:
  Reading files and computing callgraph...
Checking 733 modules in dialyxir_erlang-22.0.4_elixir-1.9.0_deps-dev.plt
** (CaseClauseError) no case clause matching: {{:nocatch, {:dialyzer_error, [78, 111, 116, 32, 97, 32, 114, 101, 103, 117, 108, 97, 114, 32, 102, 105, 108, 101, 58, 32, 'hireup/anubis/_build/dev/lib/hackney/ebin/hackney_metrics.beam', 10]}}, [{:dialyzer_plt, :compute_md5_from_file, 1, [file: 'dialyzer_plt.erl', line: 543]}, {:dialyzer_plt, :compute_new_md5_1, 3, [file: 'dialyzer_plt.erl', line: 509]}, {:dialyzer_plt, :check_plt1, 3, [file: 'dialyzer_plt.erl', line: 479]}, {:dialyzer_plt, :"-subproc/1-fun-0-", 1, [file: 'dialyzer_plt.erl', line: 603]}]}
    dialyzer_cl.erl:250: :dialyzer_cl.check_plt/3
    dialyzer_cl.erl:183: :dialyzer_cl.plt_common/3
    dialyzer.erl:160: :dialyzer.run/1
    lib/dialyxir/plt.ex:209: Dialyxir.Plt.plt_run/1
    lib/dialyxir/plt.ex:202: Dialyxir.Plt.plt_check/2
    lib/dialyxir/plt.ex:146: Dialyxir.Plt.check_beams/3
    lib/dialyxir/plt.ex:43: Dialyxir.Plt.check_plt/4
    (elixir) lib/enum.ex:1948: Enum."-reduce/3-lists^foldl/2-0-"/3

Setup code:
in mix.exs

    [
      extra_applications: [:logger, :httpoison, :new_relic_agent],
      mod: {Interfaces.Application, []}
    ]
  end

in our entry point code

use Plug.Router
if Mix.env() == :prod, do: use(NewRelic.Transaction)

Would you guys know what the reason is?

Environment

• Elixir & Erlang version (elixir -v): elixir 1.9.0 and Erlang/OTP 22
• Agent version (mix deps | grep new_relic_agent): 1.9.12

Describe the bug
When I add the @trace directive before a function I get an unused variable warning for args on the following code.

  @trace {:get_user_details, category: :external}
  def get_user_details(args = %{open_id: _}, headers) do
    "/api/public/v2/user/details"
    |> get(add_open_id_header(headers, args))
    |> handle_response(%{"data" => %User{}})
  end

the full warning is:

warning: variable "args" is unused

Note variables defined inside case, cond, fn, if and similar do not leak. If you want to conditionally override an existing variable "args", you will have to explicitly return the variable. For example:

    if some_condition? do
      atom = :one
    else
      atom = :two
    end

should be written as

    atom =
      if some_condition? do
        :one
      else
        :two
      end

Unused variable found at:
  lib/mana_web/services/user_service.ex:9

Environment

• Elixir & Erlang version (elixir -v):
  Erlang/OTP 21 [erts-10.0.5] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]
  Elixir 1.7.3 (compiled with Erlang/OTP 21)
• Agent version (mix deps | grep new_relic_agent):
  new_relic_agent 1.6.2 (Hex package) (mix)
  locked at 1.6.2 (new_relic_agent) a20b514d

When using the Pre-Instrumented Modules - HTTPoison and doing pattern matching in HttpPoison response, eg.:

{:error, %HTTPoison.Error{reason: reason}} = response

it raises an error at compilation time:

NewRelic.Instrumented.HTTPoison.Error.__struct__/0 is undefined, cannot expand struct NewRelic.Instrumented.HTTPoison.Error

I don't know if there is a trick to do in NewRelic.Instrumented.HTTPoison to solve this error, I solved the problem using a different alias:

alias NewRelic.Instrumented.HTTPoison, as: NRHTTPoison

and then: NRHTTPoison.post(...)

but it looks ugly 😢

Environment

• Elixir & Erlang version (elixir -v):
  Erlang/OTP 20 [erts-9.0] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]
  Elixir 1.7.3 (compiled with Erlang/OTP 20)

• Agent version (mix deps | grep new_relic_agent):
  new_relic_agent 1.0.2 (Hex package) (mix)
  locked at 1.0.2 (new_relic_agent) 4528c139

• OTP 21

Any plans on supporting Cowboy 1.0? I think that would increase adoption of Phoenix based apps.

The Instrumented implementation of HTTPoison does not provide the request/5 function:
https://hexdocs.pm/httpoison/HTTPoison.html#request/5

Environment

• Elixir & Erlang version (elixir -v):
  Erlang/OTP 21 [erts-10.0.7] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [dtrace]

Elixir 1.7.3 (compiled with Erlang/OTP 21)

• Agent version (mix deps | grep new_relic_agent):

• new_relic_agent (Hex package) (mix)
  locked at 1.0.2 (new_relic_agent) 4528c139

Describe the bug
We are deployed on Heroku and a few days ago our app name started to be incorrect. We use the environment variables for configuration:

jason@Jasons-MacBook-Pro-2 ~/n/messages> heroku config|grep NEW_RELIC
 ›   Warning: heroku update available from 7.21.0 to 7.22.3
NEW_RELIC_APP_NAME:                                My App Production
NEW_RELIC_LICENSE_KEY:                             <key>
NEW_RELIC_LOG:                                     stdout

However, the app name in New Relic is "My App Production (Elixir)", the old "My App Production" still exists but no data is being recorded to it. Do you have any idea what might be causing this?

Environment

• Elixir & Erlang version (elixir -v): 1.7.4 Erlang/OTP 21 [erts-10.2.3]
• Agent version (mix deps | grep new_relic_agent): * new_relic_agent (Hex package) (mix) locked at 1.9.0 (new_relic_agent) 6d8d4332

Describe the bug
No Apdex sample is reported by Agent so we can't use thresholds and alert conditions based on this metric.

Environment

• Elixir 1.7.4 & Erlang 21
• Agent version (1.6.2)