cmd/trace-agent: resource replace rules get applied to "sql.query" tag too about datadog-trace-agent HOT 15 CLOSED

Thanks!

from datadog-trace-agent.

Thanks a lot for taking the time to move the issue here.

from datadog-trace-agent.

@mbyio would you be able to use our set of replace rules to do this? You could target the resource and use a regexp to make replacements. An example is available here.

from datadog-trace-agent.

I'll see if we can try it out this week, thanks.

from datadog-trace-agent.

I finally got a chance to try this out. It seems to work okay in the sense that it does mitigate the problem. But a major downside is now we can't see the original query in a trace, which makes it harder to diagnose performance issues.

from datadog-trace-agent.

In that case I guess we can consider this issue closed, right? To add back any extra information, you could use tags - wouldn't that work?

from datadog-trace-agent.

Sure, we get the tags from the Go tracing integration. Should we make a separate feature request for adding that feature?

from datadog-trace-agent.

I think it should be possible to accomplish what you want without making changes to the code. But I'll need your help to better understand what you're trying to do.

Can you please let me know what integration you are using, what tags and resource name you are getting and what you are expecting to see but aren't?

from datadog-trace-agent.

Sure! We are using https://github.com/DataDog/dd-trace-go. To get SQL database tracing, we are using the sqlx integration. We are using the replace rules feature that you described here to cleanup the SQL queries. Since the sqlx integration uses the SQL queries as resource names, we are using the replace rules feature to cleanup the resource names.

Using the replace rules feature somewhat solves the problem of SQL queries not being grouped together, so now we can get some useful data by query. But because the resource name is being modified, we can no longer see the original query associated with a span, and we need that information to help diagnose performance problems.

from datadog-trace-agent.

Even though the resource name is modified, are you not getting the sql.query tag set to the original query? Isn't that what you are after? Sorry if I'm not completely understanding the issue.

from datadog-trace-agent.

Hm, we don't seem to be getting a sql.query tag. Actually, we aren't getting any tags at all on the SQL spans. I'll try moving to a more recent version of the Go tracing package, hopefully sometime this week.

from datadog-trace-agent.

You should definitely be getting that tag. If not, let's look into it. And yes, upgrading will most likely help (at least to a >1.0 version if you haven't already). We haven't encountered any problems with missing tags for the jmoiron/sqlx integration yet, and the unit tests confirm it. The only potential problem I can think of is perhaps the replace rules which might be removing them? You could maybe paste those here so we can take a look, if you also think that there is a possibility for it to be so. Otherwise, I'll wait for your update.

from datadog-trace-agent.

We were already > 1.0; we upgraded today, and now we don't seem to have the sql.query tag, and the replace rules stopped working too. 😞 I will reach out via datadog support tomorrow so you, or whoever works on it, can see the actual data coming in.

from datadog-trace-agent.

Sorry to hear that. We and other customers are also using these features and integrations, so I'm sure it's just a matter of settings this up correctly. I'm happy to help further via the support channel.

Let's close this issue for now, as it fixes your cardinality issue. We can keep track of this via support then. Thanks for being patient.

from datadog-trace-agent.

This is a bug, we are addressing it in #477. Hoping to release it as part of 6.6.0 mid-October. Thanks for working with us to pin the problem down. This is indeed an unfortunate oversight.

from datadog-trace-agent.