Using ActiveRecord .eager_load creates a span for each object it instantiates and also a span for each child object.

Example:

  describe 'eager loading' do
    it 'traces' do
      users = 3.times.map { User.create! }
      users.each do |user|
        user.articles.create!(title: 'test article1')
        user.articles.create!(title: 'test article2')
      end
      exporter.reset

      User
        .eager_load(:articles)
        .map { |u| [u.id, u.articles.length] }

      puts spans.map(&:name).inspect
      # => [
      #  "User.instantiate",
      #  "Article.instantiate",
      #  "Article.instantiate",
      #  "User.instantiate",
      #  "Article.instantiate",
      #  "Article.instantiate",
      #  "User.instantiate",
      #  "Article.instantiate",
      #  "Article.instantiate"
      # ]
    end
  end

In this example I have 3 users in the database, each having 2 articles. Fetching all users with eager loading their articles creates a total of 9 spans even though there's only one SQL query. This gets worse when the number of records grows.

Compare this to fetching all objects without eager_load:

    it 'traces' do
      users = 3.times.map { User.create! }
      exporter.reset
      User.all.to_a
      puts spans.map(&:name).inspect
      # => ["User.find_by_sql"]

Here it only creates one span.

One idea I had was to just remove tracing from .instantiate method. But in that case, there wouldn't be any spans created when using eager_load.

I would be fine submitting a PR, but I'm not really sure how to improve this.

spawning off some discussion here open-telemetry/opentelemetry-ruby#947, there are two areas that ought to be addressed in instrumentation

1. Be more defensive with version compatibility
2. Have a way to safely test against latest versions

Currently our instrumentation has the concept of a compatible block, which is just an arbitrary proc an instrumentation author can use to define logic for whether the instrumentation is compatible with the specific Gem version that has been installed. It could also be used to check arbitrary things like, for example, whether it's a jruby environment, or whether specific permissions are available, etc.

In practice, afaik, this is only used to do a MIN_VERSION comparison, like in action_pack, for example.

Over time, as new major versions come out, it becomes harder for us to say with confidence that our instrumentation is "compatible" with them, and so it might be worthwhile to make a pass through the instrumentation and add a MAX_VERSION comparison as well. In between the outcomes of "This instrumentation unexpectedly doesn't install and logs a warning to the user" and "This instrumentation causes an application crash during runtime", the former seems like a more reasonable posture for us to take.

With that posture we would also need to have some sort of mechanism for reliably testing latest major version releases. One solution here could be to include a latest or canary nightly build in CI, that pulls in the latest major+minor versions of everything , runs the test suite, and informs us of what fails. In this way we could establish some confidence when we want to bump the MAX_VERSION compatibility, or have some context to point to if contributors/users open issues asking for a major version bump.

lastly, as @arielvalentin suggests in this issue comment, open-telemetry/opentelemetry-ruby#947 (comment) , a DSL for declaring min/max version may make it easier to maintain this approach long term. Perhaps some semantic conventions around what constants to use for MIN_VERSION and MAX_VERSION combined with a modification to the base compatible() method could accomplish this (and also make it easier for a canary build to inform us when the latest gem version pulled down has a delta with the instrumentation's MAX version).

The current instrumentation for mysql2 is tracing the full query with all the column values which may contain sensitive information and act as an hindrance for adoption in production environment.

https://github.com/open-telemetry/opentelemetry-ruby/blob/master/instrumentation/mysql2/lib/opentelemetry/instrumentation/mysql2/patches/client.rb#L37

def query(sql, options = {})
  tracer.in_span(
    database_span_name(sql),
    attributes: client_attributes.merge(
      'db.statement' => sql
    ),
    kind: :client
  ) do
    super(sql, options)
  end
end

Also Opentelemetry specification mentions that sensitive information can be excluded from the db.statement span attribute.

It will help us in faster adoption if we can follow a similar approach like Newrelic which has an Obfuscation util to mask values from query statements. https://github.com/newrelic/newrelic-ruby-agent/blob/006dd1bb8174e6f49c495c7e1a8ca543de9ceb93/lib/new_relic/agent/database/obfuscator.rb#L57

Convert Datadog instrumentation for aws-sdk to OpenTelemetry.

See

• Datadog porting guide
• Semantic Conventions
• Datadog implementation

(I see that this repo doesn't have any real content yet, but I guess that it doesn't hurt to add the link to the OTel website now :))

Under the About section of this repo (which you can access from the main repo page -- click the gear icon next to the About title):

• Under Website, add https://opentelemetry.io

We're looking to do this uniformly for all OTel org repos.

For example:

This seems to be in v1.11.0 of the spec, although it didn't make it into the release notes. The relevant section of the resource spec states:

Resource detectors that populate resource attributes according to OpenTelemetry semantic conventions MUST ensure that the resource has a Schema URL set to a value that matches the semantic conventions. Empty Schema URL SHOULD be used if the detector does not populate the resource with any known attributes that have a semantic convention or if the detector does not know what attributes it will populate (e.g. the detector that reads the attributes from environment values will not know what Schema URL to use).

This is not a huge deal, since we have a grand total of one resource detector, of course. And, it's implicitly blocked by open-telemetry/opentelemetry-ruby#1237.

Description of the bug

This is not actually a bug. I investigate heavy traces with many many active record spans and found the following:

1. active record instrument the public api functions, few of those are: create, save, transaction.
2. When create is invoked, active record checks if the argument is an array, and if so, it invokes create for each element in the array:

      def create(attributes = nil, &block)
        if attributes.is_a?(Array)
          attributes.collect { |attr| create(attr, &block) }
        else
          object = new(attributes, &block)
          object.save
          object
        end
      end

3. Each create invokes the save function on each object, which is also instrumented
4. There is also I think an invocation of the transaction method which is also instrumented. Could not reproduce it myself but I can see it generated in a some other environment.

For example, the following function invocation:

User.create([{:username => "foo", :password => "1234"}, {:username => "bar", :password => "5678"}])

Creates the following output in Jaeger:

I am thinking about whether it gives good value to instrument the public methods when they are invoked from code inside the active record module itself. This follows the actual code being executed in reality, but create large traces with many spans:

1. More costs to users
2. Visual load in UI
3. More performance impact on user's application
4. Confusing traces: create span with n children of same operation create with child of save each. This, I suspect, is not intuitive to the user which was only invoking create once. I was confues myself until I looked at the rails source code to make sense of what I am seeing.

I post here this issue to hear opinions and consult if you are open to changing this behavior to only include functions invoked directly by users. This can also be implemented as instrumentation config option I guess.

Here is an example:

https://github.com/open-telemetry/opentelemetry-ruby/runs/6307139168?check_suite_focus=true#step:6:2024

opentelemetry-instrumentation-que: test
>> BUNDLE_GEMFILE=/home/runner/work/opentelemetry-ruby/opentelemetry-ruby/instrumentation/que/gemfiles/que_1.0.gemfile bundle exec rake test
Run options: --seed 32798

# Running:

/opt/hostedtoolcache/Ruby/3.1.2/x64/lib/ruby/gems/3.1.0/gems/activesupport-6.1.5.1/lib/active_support/core_ext/class/subclasses.rb:30: warning: method redefined; discarding old subclasses
/opt/hostedtoolcache/Ruby/3.1.2/x64/lib/ruby/gems/3.1.0/gems/opentelemetry-instrumentation-pg-0.20.0/lib/opentelemetry/instrumentation/pg/patches/connection.rb:20: warning: forwarding async_exec to async_exec_params and send_query to send_query_params is deprecated
F.................

Finished in 1.795210s, 10.0267 runs/s, 32.3082 assertions/s.

  1) Failure:
OpenTelemetry::Instrumentation::Que::que poller::when trace_poller is disabled#test_0001_does not create any spans [/home/runner/work/opentelemetry-ruby/opentelemetry-ruby/instrumentation/que/test/opentelemetry/instrumentation/que_test.rb:261]:
--- expected
+++ actual
@@ -1 +1 @@
-[]
+[#<struct OpenTelemetry::SDK::Trace::SpanData name="DELETE postgres", kind=:client, status=#<OpenTelemetry::Trace::Status:0xXXXXXX @code=1, @description="">, parent_span_id="\x00\x00\x00\x00\x00\x00\x00\x00", total_recorded_attributes=9, total_recorded_events=0, total_recorded_links=0, start_timestamp=1651758287512252886, end_timestamp=1651758287515648612, attributes={"db.system"=>"postgresql", "db.user"=>"postgres", "db.name"=>"postgres", "net.peer.name"=>"localhost", "net.transport"=>"IP.TCP", "net.peer.ip"=>"::1", "net.peer.port"=>"5432", "db.operation"=>"DELETE", "db.statement"=>"DELETE FROM public.que_lockers WHERE pid = pg_backend_pid() OR NOT EXISTS (SELECT 1 FROM pg_stat_activity WHERE pid = public.que_lockers.pid)"}, links=nil, events=nil, resource=#<OpenTelemetry::SDK::Resources::Resource:0xXXXXXX @attributes={"service.name"=>"unknown_service", "process.pid"=>11561, "process.command"=>"/opt/hostedtoolcache/Ruby/3.1.2/x64/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/rake_test_loader.rb", "process.runtime.name"=>"ruby", "process.runtime.version"=>"3.1.2", "process.runtime.description"=>"ruby 3.1.2p20 ([2022](https://github.com/open-telemetry/opentelemetry-ruby/runs/6307139168?check_suite_focus=true#step:6:2022)-04-12 revision 4491bb740a) [x86_64-linux]", "telemetry.sdk.name"=>"opentelemetry", "telemetry.sdk.language"=>"ruby", "telemetry.sdk.version"=>"1.0.3"}>, instrumentation_library=#<struct OpenTelemetry::SDK::InstrumentationLibrary name="OpenTelemetry::Instrumentation::PG", version="0.20.0">, span_id="\x8F~\xEAI\x9C\x8A\b\x82", trace_id="\xEB\x0F\xC8+\x0E\"xb\xB8\x02.\xE4\xED\xB9\xB2H", trace_flags=#<OpenTelemetry::Trace::TraceFlags:0xXXXXXX @flags=1>, tracestate=#<OpenTelemetry::Trace::Tracestate:0xXXXXXX @hash={}>>, #<struct OpenTelemetry::SDK::Trace::SpanData name="INSERT postgres", kind=:client, status=#<OpenTelemetry::Trace::Status:0xXXXXXX @code=1, @description="">, parent_span_id="\x00\x00\x00\x00\x00\x00\x00\x00", total_recorded_attributes=9, total_recorded_events=0, total_recorded_links=0, start_timestamp=1651758287516412718, end_timestamp=1651758287517783529, attributes={"db.system"=>"postgresql", "db.user"=>"postgres", "db.name"=>"postgres", "net.peer.name"=>"localhost", "net.transport"=>"IP.TCP", "net.peer.ip"=>"::1", "net.peer.port"=>"5432", "db.operation"=>"INSERT", "db.statement"=>"INSERT INTO public.que_lockers (pid, worker_count, worker_priorities, ruby_pid, ruby_hostname, listening, queues) VALUES (pg_backend_pid(), $1::integer, $2::integer[], $3::integer, $4::text, $5::boolean, $6::text[])"}, links=nil, events=nil, resource=#<OpenTelemetry::SDK::Resources::Resource:0xXXXXXX @attributes={"service.name"=>"unknown_service", "process.pid"=>11561, "process.command"=>"/opt/hostedtoolcache/Ruby/3.1.2/x64/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/rake_test_loader.rb", "process.runtime.name"=>"ruby", "process.runtime.version"=>"3.1.2", "process.runtime.description"=>"ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]", "telemetry.sdk.name"=>"opentelemetry", "telemetry.sdk.language"=>"ruby", "telemetry.sdk.version"=>"1.0.3"}>, instrumentation_library=#<struct OpenTelemetry::SDK::InstrumentationLibrary name="OpenTelemetry::Instrumentation::PG", version="0.20.0">, span_id="x\v\r;\xB2)\xBCo", trace_id="\x80[\xB7\x8A\x9A\x81\x90l \xFBx\xC90 nW", trace_flags=#<OpenTelemetry::Trace::TraceFlags:0xXXXXXX @flags=1>, tracestate=#<OpenTelemetry::Trace::Tracestate:0xXXXXXX @hash={}>>]


18 runs, 58 assertions, 1 failures, 0 errors, 0 skips
rake aborted!
Command failed with status (1)
/opt/hostedtoolcache/Ruby/3.1.2/x64/bin/bundle:25:in `load'
/opt/hostedtoolcache/Ruby/3.1.2/x64/bin/bundle:25:in `<main>'
Tasks: TOP => test
(See full trace by running task with --trace)
FAILURE

We support TruffleRuby on a best effort basis (see open-telemetry/opentelemetry-ruby#1295). Our rdkafka instrumentation does not work properly with TruffleRuby, however.

This ticket will be complete when our CI build successfully runs specs instrumentation-rdkafka.

To run these specs in CI, delete the gem from the TruffleRuby filter stanzas (1, 2) in .github/workflows/ci.yml.

Description of the bug

Using Sidekiq with Sidekiq's instrumentation propagation_style set to :child, when delaying the execution of a worker using perform_in(sec, params) or perform_at(timestamp, params), a span is produced for the enqueueing (send) but no span is ever produced for the execution itself (process).

If propagation_style is not set (it defaults to link), the span for process will be produced.

Executing the worker asynchronously, with perform_async(params), works correctly regardless of the value of propagation_style.

Share details about your runtime

Operating system details: Linux, Debian 10.10 (in docker)
RUBY_ENGINE: "ruby"
RUBY_VERSION: "3.0.1"
RUBY_DESCRIPTION: "ruby 3.0.1p64 (2021-04-05 revision 0fb782ee38) [x86_64-linux]"

Share a simplified reproduction if possible

docker run -ti --rm -w /perform_in_bug_report ruby:3.0.1-buster bash
gem install rails
rails new --skip-git --skip-keeps --skip-action-mailer --skip-action-mailbox --skip-action-text --skip-active-record --skip-active-job --skip-active-storage --skip-action-cable --skip-asset-pipeline --skip-javascript --skip-hotwire --skip-jbuilder --skip-test --skip-system-test --skip-bootsnap --skip-bundle .
bundle add opentelemetry-api opentelemetry-sdk opentelemetry-instrumentation-sidekiq sidekiq
apt-get update && apt-get install -y vim redis
mkdir app/workers

# config/initializers/opentelemetry.rb
span_processor = OpenTelemetry::SDK::Trace::Export::SimpleSpanProcessor.new(
  OpenTelemetry::SDK::Trace::Export::ConsoleSpanExporter.new
)

OpenTelemetry::SDK.configure do |c|
  c.add_span_processor(span_processor)
  c.use 'OpenTelemetry::Instrumentation::Sidekiq', {
    propagation_style: :child,
    span_naming: :job_class
  }
end

# app/workers/example_worker.rb
class ExampleWorker
  include Sidekiq::Worker

  sidekiq_options retry: 0

  def perform(id)
    OpenTelemetry::Trace.current_span.set_attribute('worker.parameters.id', id)
  end
end

redis-server &
bundle exec sidekiq &

bin/rails c
> ExampleWorker.perform_async(1)
> ExampleWorker.perform_in(1.second, 1)

The perform_async call will show a span for ExampleWorker send and another for ExampleWorker process, while the perform_in call will only show a span for ExampleWorker send.

We support TruffleRuby on a best effort basis (see open-telemetry/opentelemetry-ruby#1295). Our sidekiq instrumentation does not work properly with TruffleRuby, however.

This ticket will be complete when our CI build successfully runs specs opentelemetry-instrumentation-sidekiq.

To run these specs in CI, delete the gem from the TruffleRuby filter stanzas (1, 2) in .github/workflows/ci.yml.

We support TruffleRuby on a best effort basis (see open-telemetry/opentelemetry-ruby#1295). Our actionpack instrumentation does not work properly with TruffleRuby, however.

This ticket will be complete when our CI build successfully runs specs opentelemetry-instrumentation-action_pack.

To run these specs in CI, delete the gem from the TruffleRuby filter stanzas (1, 2) in .github/workflows/ci.yml.

GraphQL instrumentation currently provides poor visibility into failed GraphQL requests as the typical configuration is done in the form of a post which returns a 200 even if the query is malformed or fails for other reasons.

Some exploration was while creating the instrumentation gem, however it was removed to be addressed in a separate PR.

Our various gems are on a pretty old version of Rubocop (see @dazuma's comment open-telemetry/opentelemetry-ruby#1096 (comment) for context). ActiveModel Serializers gemspec is at 0.73.0.

We should update each instrumentation to the latest Rubocop.

Placeholder for @ericmustin @arielvalentin to determine a strategy for migrating the relevant portions of https://github.com/open-telemetry/opentelemetry-ruby into this repository

Description of the bug
When instrumenting a sidekiq "perform_async" call, two spans are generated, one for sidekiq, and another for the underlying operation to redis (assuming that redis instrumentation is also installed). I expected the redis span to be child of sidekiq span, as it is executed by it inernally and in sidekiq context (line 3 in image is sibling to line 2 instead of child):

Share a simplified reproduction if possible

# config/initializers/opentelemetry.rb

require 'opentelemetry/sdk'
require 'opentelemetry/exporter/otlp'
require 'opentelemetry/instrumentation/all'

OpenTelemetry::SDK.configure do |c |
    c.service_name = 'ruby-monolith'
    c.use_all 'OpenTelemetry::Instrumentation::ActionPack' => { enable_recognize_route: true }

    c.add_span_processor(
    OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
        OpenTelemetry::Exporter::OTLP::Exporter.new(endpoint: '***', headers: {
        'Authorization' => '***'
        })
      )
    )
end

# app/controllers/api/v1/sidekiq_controller.rb

class Api::V1::SidekiqController < ApplicationController

    # GET /sidekiq
    def index
        HardWorker.perform_async('bob', 5)
        render status: 200
    end

end

Convert Datadog instrumentation for Grape to OpenTelemetry.

See

• Datadog porting guide
• Semantic Conventions
• Datadog implementation

Convert Datadog instrumentation for Rake to OpenTelemetry.

See

• Datadog porting guide
• Semantic Conventions
• Datadog implementation

Following up on open-telemetry/opentelemetry-ruby#978, we noticed we're missing some detail for our multiple kafka instrumentations

The specification states that peer.service SHOULD be set for kafka producers.

https://github.com/open-telemetry/opentelemetry-specification/blob/a1a8676a43dce6a4e447f65518aef8e98784306c/specification/trace/semantic_conventions/messaging.md#apache-kafka

For Apache Kafka producers, peer.service SHOULD be set to the name of the broker or service the message will be sent to. The service.name of a Consumer's Resource SHOULD match the peer.service of the Producer, when the message is directly passed to another service. If an intermediary broker is present, service.name and peer.service will not be the same.

This should get added across instrumentation's in 1 go

We support TruffleRuby on a best effort basis (see open-telemetry/opentelemetry-ruby#1295). Our kafka instrumentation does not work properly with TruffleRuby, however.

This ticket will be complete when our CI build successfully runs specs opentelemetry-instrumentation-ruby_kafka.

To run these specs in CI, delete the gem from the TruffleRuby filter stanzas (1, 2) in .github/workflows/ci.yml.

Convert Datadog instrumentation for gRPC to OpenTelemetry.

See

• Datadog porting guide
• Semantic Conventions
• Datadog implementation

Description of the bug

After installing the latest release of opentelemetry-ruby (v0.2.2) via opentelemetry-instrumentation-all and configuring it using use_all, I see the following message in the logs when starting up my app:

WARN -- : Instrumentation: OpenTelemetry::Instrumentation::ActiveRecord failed to install

After doing some digging I found this PR which appears to fix the issue: open-telemetry/opentelemetry-ruby#1043

However, when I inspect the source that was installed the changes from that PR are not present. It looks like maybe a release with those changes failed, or I'm just misunderstanding the release cadence for this lib.

I am able to work around this with the following monkey-patch placed before my instrumentation setup:

module OpenTelemetry
  module Instrumentation
    module ActiveRecord
      class Instrumentation < OpenTelemetry::Instrumentation::Base
        compatible do
          true
        end
      end
    end
  end
end

That silences the warning, but I'm a bit concerned about running on this version w/out the additional changes included in that PR.

Share details about your runtime

Operating system details: Linux, Ubuntu 20.04 LTS
RUBY_ENGINE: "ruby"
RUBY_VERSION: "3.1.0"
RUBY_DESCRIPTION: "ruby 3.1.0p0 (2021-12-25 revision fb4df44d16) [x86_64-darwin20]"

Share a simplified reproduction if possible

# In Gemfile
  
gem 'opentelemetry-sdk'
gem 'opentelemetry-exporter-oltp'
gem 'opentelemetry-instrumentation-all'


# config/initializers/opentelemetry.rb
OpenTelemetry::SDK.configure do |c|
  c.use_all()
end

Spec release 1.12.0 is available:

• Release notes
• Diff v1.11.0...v1.12.0

Most of this spec release concerns semantic conventions, and metrics changes. The most interesting changes for the -contrib repo will likely be the new bits about required semconv attributes.

TODO:

• Update opentelemetry-semantic_conventions
• Audit changelog for anything that may concern this repo and create follow-up issues

not blocking bc it's a huge change and out of scope for this PR but, along with some broad http helpers that oughta be abstracted into a gem, to be DRY we should also probably move the mysql obfuscation regex into a db helper gem of some sort eventually/ideally as an early todo in 2022.

Originally posted by @ericmustin in open-telemetry/opentelemetry-ruby#1063 (comment)

There is structural duplication in SQL instrumentation gems, e.g. obfuscation and semconv, as well as knowledge duplication between the mysql2 and triology instrumentations. To minimize maintenance burden of these libraries it will likely be best to extract reusable code and provides more focused test coverage.

In addition to that there is not any test coverage for UDS connections and should update semantic conventions to reflect these values.

Our Rack instrumentation allows configuration of a URL quantization function https://github.com/open-telemetry/opentelemetry-ruby/blob/1c37b00dc82d490dace1f92739977aa4674d7039/instrumentation/rack/lib/opentelemetry/instrumentation/rack/middlewares/tracer_middleware.rb#L122-L137

We should add the same functionality to the Sinatra instrumentation. @genebean reported this requirement in Gitter:

I am using the auto instrumentation of the Sinatra but need to account for url paramters so that /api/v2/status/queue/info/1600289926679 is instead traced as /api/v2/status/queue/info/{job_id} - any tips on how to do this?

I've added Rack, Rails and Mysql2 instrumentation to an unremarkable Rails 6.0 project.

I get the Rack and Rails instrumentation details in one trace for the get request and then each sql statement has its own trace so they can't put on the same tree as they should be. It looks like this:

Where on the left I'm printing trace ids and every trace there except http get has one only span with the sql statement.
I do not see any obvious reason why this should end up like this, the mysql2 instrumentation uses the same tracer method than the rack instrumentation, both inherit from the same base that gets the tracer from the same place.

Any pointers?

Description

NOTE: We should block on #33 before addressing this issue because the Resource Detectors should go in the same opentelemetry-sdk-extension-aws gem as the ID Generator.

We should add Resource Detectors to automatically populate the resource.* attributes of all spans in certain attributes. For example, if an app is running on EC2, users should be able to use the OpenTelemetry EC2 Resource Detector to automatically populate information about the EC2 instance they are running on.

Resource Detectors already exist in several other OTel Languages:

OTel Java:

• SDK Extension Package (IDs Generator + AWS Resource Detectors)

OTel Python

• SDK Extension Package (IDs Generator + AWS Resource Detectors)

These languages have resource detectors, but put them in separate packages:

OTel Go:

• AWS Resource Detectors Package

OTel JS:

• IDs Generator Package
• AWS Resource Detectors Package

.NET has all X-Ray necessary components in one package, but should split the propagator out eventually

OTel .NET

• SDK Extension (Resource Detectors + ID Generator + Propagator

The X-Ray team would be happy to help contribute these in the near future! 🙂

Currently we only patch the request method, but it would be good to add visibility around connect.

It would also be nice to be able to allow / deny list functionality for context propagation. This would likely be something we'd like to apply across the board to other instrumentation if we go this route.

We support Truffleruby on a best effort basis (see open-telemetry/opentelemetry-ruby#1295). Our redis instrumentation does not work properly with truffleruby, however.

This ticket will be complete when our CI build successfully runs specs for opentelemetry-instrumentation-redis.

To run these specs in CI, delete the gem from the TruffleRuby filter stanzas (1, 2) in .github/workflows/ci.yml.

Most test suites rely on users executing tests using the docker-compose managed services in local development and Action managed services in CI.

The sidekiq test suite attempts to start a redis server during tests with specific configurations and if you are running redis via docker-compose or as part of Actions CI it results in a backtrace but the test suite passes:

@arielvalentin ➜ /workspaces/opentelemetry-ruby/instrumentation/sidekiq (use-gem-versions) $ bundle exec appraisal sidekiq-6.3 rake test
>> BUNDLE_GEMFILE=/workspaces/opentelemetry-ruby/instrumentation/sidekiq/gemfiles/sidekiq_6.3.gemfile bundle exec rake test
Traceback (most recent call last):
        43: from /home/codespace/.ruby/current/bin/bundle:23:in `<main>'
        42: from /home/codespace/.ruby/current/bin/bundle:23:in `load'
        41: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.5/exe/bundle:36:in `<top (required)>'
        40: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.5/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors'
        39: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.5/exe/bundle:48:in `block in <top (required)>'
        38: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.5/lib/bundler/cli.rb:25:in `start'
        37: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.5/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
        36: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.5/lib/bundler/cli.rb:31:in `dispatch'
        35: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.5/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
        34: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.5/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
        33: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.5/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
        32: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.5/lib/bundler/cli.rb:484:in `exec'
        31: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.5/lib/bundler/cli/exec.rb:23:in `run'
        30: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.5/lib/bundler/cli/exec.rb:58:in `kernel_load'
        29: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.5/lib/bundler/cli/exec.rb:58:in `load'
        28: from /home/codespace/.ruby/current/bin/rake:23:in `<top (required)>'
        27: from /home/codespace/.ruby/current/bin/rake:23:in `load'
        26: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
        25: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
        24: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
        23: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
        22: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
        21: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
        20: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
        19: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
        18: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
        17: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
        16: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
        15: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
        14: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
        13: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
        12: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `invoke_prerequisites'
        11: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `each'
        10: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:243:in `block in invoke_prerequisites'
         9: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
         8: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
         7: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
         6: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
         5: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
         4: from /opt/ruby/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
         3: from /workspaces/opentelemetry-ruby/instrumentation/sidekiq/Rakefile:15:in `block in <top (required)>'
         2: from /workspaces/opentelemetry-ruby/instrumentation/sidekiq/Rakefile:15:in `fork'
         1: from /workspaces/opentelemetry-ruby/instrumentation/sidekiq/Rakefile:15:in `block (2 levels) in <top (required)>'
/workspaces/opentelemetry-ruby/instrumentation/sidekiq/Rakefile:15:in `exec': No such file or directory - redis-server (Errno::ENOENT)
2022-03-28T14:28:31.938Z pid=12109 tid=7mp INFO: Booting Sidekiq 6.4.1 with redis options {:password=>"REDACTED", :url=>"redis://127.0.0.1:16379/0"}
Run options: --seed 45071

# Running:

.......................................

Finished in 6.324330s, 6.1667 runs/s, 18.1837 assertions/s.

39 runs, 115 assertions, 0 failures, 0 errors, 0 skips

We should change the Sidekiq test suite to behave the same way as the rest of the tests.

Rails 6 / unicorn application with c.use_all

This happens with and without prometheus instrumentation enabled. So may or may not be related to open-telemetry/opentelemetry-ruby#352

/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-0.12.1/lib/opentelemetry/context.rb:110:in `value': stack level too deep (SystemStackError)
	from /app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-0.12.1/lib/opentelemetry/trace.rb:62:in `current_span'
	from /app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-sdk-0.12.1/lib/opentelemetry/sdk/trace/samplers/constant_sampler.rb:30:in `should_sample?'
	from /app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-sdk-0.12.1/lib/opentelemetry/sdk/trace/samplers/parent_based.rb:56:in `should_sample?'
	from /app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-sdk-0.12.1/lib/opentelemetry/sdk/trace/tracer.rb:47:in `start_span'
	from /app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-0.12.1/lib/opentelemetry/trace/tracer.rb:28:in `in_span'
	from /app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-instrumentation-redis-0.12.0/lib/opentelemetry/instrumentation/redis/patches/client.rb:18:in `call'
	from (eval):9:in `call'
	from /app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-instrumentation-redis-0.12.0/lib/opentelemetry/instrumentation/redis/patches/client.rb:23:in `block in call'
	 ... 9346 levels...
	from /app/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `load'
	from /app/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `<main>'
	from /app/vendor/bundle/ruby/2.6.0/bin/ruby_executable_hooks:24:in `eval'
	from /app/vendor/bundle/ruby/2.6.0/bin/ruby_executable_hooks:24:in `<main>'

Per a discussion in the SIG on April 12, we'd like to have a list of attributes published on spans in various instrumentation gems. So, for instance, you could look at the mysql2 instrumentation gem readme and see the attributes that users should expect to see on spans generated by that instrumentation. I'd personally like to see the config flag that toggles the inclusion of an attributes, when relevant, but implementing might be a bit annoying/complex.

@ericmustin pointed us to how dd-trace does it, and that's a great step for us.

• extract attribute names to constants file in each instrumentation
• write toys or make command to update READMEs with attribute names

Convert Datadog instrumentation for SuckerPunch to OpenTelemetry.

See

• Datadog porting guide
• Semantic Conventions
• Datadog implementation

We support JRuby on a best effort basis. Unfortunately, some of our instrumentation is not compatible with JRuby. Specifically, the following gems are not compatible:

• instrumentation-action_pack
• instrumentation-action_view
• instrumentation-active_model_serializers
• instrumentation-active_record
• instrumentation-active_support
• instrumentation-aws_sdk
• instrumentation-delayed_job
• instrumentation-graphql
• instrumentation-http
• instrumentation-http_client
• instrumentation-koala
• instrumentation-lmdb
• instrumentation-rack
• instrumentation-rails

This ticket will be complete when our CI build successfully runs JRuby specs against every gem in the list above.

To run the specs for a given gem with JRuby in CI, delete the gem from the JRuby filter stanzas (1, 2) in .github/workflows/ci.yml.

Incremental progress against this issue (i.e. fixing one gem at a time!) is very welcome. We will not worry about fixing the gems in one pass!

Note: this ticket came out of #4

Convert Datadog instrumentation for Elasticsearch to OpenTelemetry.

See

• Datadog porting guide
• Semantic Conventions
• Datadog implementation

Convert Datadog instrumentation for Racecar to OpenTelemetry.

See

• Datadog porting guide
• Semantic Conventions
• Datadog implementation

Add instrumentation adapters for Rails components via ActiveSupport notifications. @martinisoft can fill in the details.

The work done to add event notifications was removed from the PR to add the ruby kafka instrumentation.

Some of these notifications could be better served as events, and some as spans. This was removed from the original PR so we could keep it more focused and to give these parts of instrumentation more focused attention in their own follow up PR against the ruby kafka instrumentation.

Description

Today, the X-Ray ID Generator lives inside the opentelemetry-propagator-xray gem instead of being in its own package.

This is a problem for projects that want to install either the Propagator or the ID Generator, but not the other. For example, the https://github.com/open-telemetry/opentelemetry-lambda repository creates Lambda Layers that can be used to export to any backend. The specifications for instrumenting an AWS Lambda function say it should use the AWS X-Ray Propagator and so Lambda Instrumentation packages normally take it as a dependency.

However, Lambda Instrumentations should not be taking the AWS X-Ray ID Generator a dependency, since it is not required. Because of the way the OTel Ruby opentelemetry-propagator-xray gem is set up, it will have no choice but to take the ID Generator.

Instead, we should split the ID Generator into its own gem called opentelemetry-sdk-extension-xray, because it is meant to "extend" on the OTel SDK's notion of "Id Generators". (I know in OTel Ruby the concept of an "ID Generator" is directly in the opentelemetry-api package, but the specification describes Id Generator as only an SDK concept, not an API concept).

This would match what other OpenTelemetry SIGs are doing:

OTel JS:

• Propagators Package
• IDs Generator Package
• AWS Resource Detectors Package

OTel Java:

• Propagator Package
• SDK Extension Package (IDs Generator + AWS Resource Detectors)

OTel Python

• Propagator Package
  • The PR where we split it out into its own package
• SDK Extension Package (IDs Generator + AWS Resource Detectors)

Related to: open-telemetry/opentelemetry-ruby#769

We have a number of instrumentation helper methods and options, (specifically related to http in the above pr but also potentially db/cache services) which all http instrumentation rely on for shared/utility methods and behavior.

We should add an instrumentation-helpers gem to support these shared utils, and which open-telemetry/opentelemetry-ruby#769 can then rebase off of.

Example:

_(
  easy.instance_eval { @otel_original_headers['traceparent'] }
).must_equal "00-#{span.trace_id.unpack1('H*')}-#{span.span_id.unpack1('H*')}-01"

This is a very common pattern in the adapter tests. It’s worth thinking about extracting this to a helper in the API or SDK. It can be accomplished with something like:

OpenTelemetry::Trace::Propagation::TraceContext::TraceParent.from_context(span).to_s

but that’s pretty wordy, may not work exactly like this, and doesn’t really reveal the intent terribly well. Something like the following would be better:

OpenTelemetry::Trace.to_traceparent(trace_id: span.trace_id, span_id: span.span_id, sampled: true)

I guess that’s even longer, but I think it is clearer.

Originally posted by @fbogsany in open-telemetry/opentelemetry-ruby#280

open-telemetry/opentelemetry-ruby#631
googleapis/google-api-ruby-client#3185

Convert Datadog instrumentation for presto-client to OpenTelemetry.

See

• Datadog porting guide
• Semantic Conventions
• Datadog implementation

The Redis telemetry will mark the spans as containing errors if there is a CommandError
https://github.com/open-telemetry/opentelemetry-ruby/blob/main/instrumentation/redis/lib/opentelemetry/instrumentation/redis/patches/client.rb#L45-L51

But the redis library has logic to retry and recover from this kind of error, depending on the situation.
https://github.com/redis/redis-rb/blob/6542934f01b9c390ee450bd372209a04bc3a239b/lib/redis/cluster.rb#L217-L237

The result is that we see many spans with status Unhandled exception of type: Redis::CommandError but it is due things being moved around, the redis-rb recovers from this internally and the application itself never sees an error. Which is misleading.

Given that we have Rack instrumentation as well as Sinatra instrumentation there is the possibility of having nested spans with SpanKind.SERVER. With Faraday and Net::HTTP instrumentation, it's also possible to have nested spans with SpanKind.CLIENT. Ideally, each request will have a single server and client span.

The actual SpanKind will vary depending on the installed instrumentation. We should discuss how to handle this, and update our instrumentation accordingly.

Convert Datadog instrumentation for Shoryuken to OpenTelemetry.

See

• Datadog porting guide
• Semantic Conventions
• Datadog implementation

We use the appraisal gem to run instrumentation specs against different versions of the gems being instrumented. As an example, we run action_pack's tests against Rails 5.2, 6.0, 6.1, and 7.0:

https://github.com/open-telemetry/opentelemetry-ruby/blob/4ab6fdf6a4840bfde704a9e8e1f068df102a2291/instrumentation/action_pack/Appraisals#L8-L26

These Appraisal files get out of date in multiple ways. As such, we should have a GitHub action (or actions) that updates the Appraisal file with a new appraise block if there's a new release of the instrumented gem. If opening up a PR ends up being more work than we want to do, we can just open an issue in this repo notifying us to add an appraisal for a new version of the gem.

Alternate solutions are very welcome! This seemed to make sense since we're already using GH actions 😄

Follow on from open-telemetry/opentelemetry-ruby#230 and open-telemetry/opentelemetry-ruby#263.

Convert Datadog instrumentation for Sequel to OpenTelemetry.

See

• Datadog porting guide
• Semantic Conventions
• Datadog implementation