intergral / deep Goto Github PK

Review and clean up module:

• remove references to temp
• remove references to traces (except for monitoring purposes)
• enable any disabled tests
• review module for redundant functionality
• review module for missing functionality

Is your feature request related to a problem? Please describe.
There is a module for usage reports, that sends data to grafana (remnant of tempo). This is still there and sends data to grafana.com, which doesn't work. So this module should be removed or a new usage report end point created.

Describe the solution you'd like
A endpoint we can run that collects the data for reporting.

Describe alternatives you've considered
Remove the module, as it currently wont / cant work.

Review and clean up pkg directory:

• remove references to temp
• remove references to traces (except for monitoring purposes)
• enable any disabled tests
• review module for redundant functionality
• review module for missing functionality

Is your feature request related to a problem? Please describe.
As a user that is bound by security policies (SOC, ISO27001, etc). Any change to production needs to be approved and reviewed. It should be possible to have this ability with DEEP tracepoints.

As a tracepoint can collect any data, and expose it to others. It should be subject to the same rules as any MR/PR. It should therefore be possible to allow a tracepoint to be created, then put into review. Where another user would need to approve it before it is activated by the service.

In a similar vain this would also require audit logs of the tracepoints being created/modified and approved. See ...

Describe the solution you'd like
Once a tracepoint is created it is stored as 'pending' then a review can be performed to ensure these tracepoints are ok. Once approved the tracepoint can be activated and sent to the clients.

It could also be possible to preapprove certain tracepoints, if they are deemed safe.

Describe alternatives you've considered
Possible integration into Github/GitLab to provide a repo that contains the configs. This would allow the existing MR/PR approaches to be used.

Additional context
Add any other context or screenshots about the feature request here.

Tempo supports forwarders that allow received traces to be forwarded to other services.

Deep has this functionality but it has not been updated to use snapshots.

• distributor: needs update to create snapshot forwarders
• config: forward config refers to otl, this should change to snapshots
• example/test: should create a simple receiver that can be configured as a forwarder to allow testing/example
• Change metric forwarder to be the same/similar to the dynamic types?

Check the licenses of all the projects that would be releases with the beta to ensure that they are correct.

In deep there is a limit on the way native search works. It will only search attribute/resource tags that are strings. This is the same as tempo. We can (I believe) fix this, but I assume this was done in tempo for a reason (maybe performance).

The open question is, do we need to support the complex types that tempo supports. Or do we just want these tags to be strings? at the moment a tab can be int, boolean, string, double, array, bytes, map. This makes it quite complex to render and search.

I cannot see a reason why these tags would be anything but strings.

Tempo/Otel has the concept of receivers that can accept spans/traces in a variety of formats and sources. They then convert the data to a common format to be processed by tempo.

We currently have only a single source for deep, but this could easy change with there being a few possibilities:

• FusionReactor snapshots
• NerdVision Snapshots
• Datadog snapshots
• Kafka streams

Each would need to connect to the data source or accept incoming data and convert to the common deep format.

Is your feature request related to a problem? Please describe.
Connect Deep into alerting in some fashion.

Describe the solution you'd like
When a snapshot/tracepoint is triggered generate an alert. This would possible need to support a variety of options, for now look at integration into alert manager/prometheus/grafana.

Additional context
Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe.
Make it possible to add fire count a global item. Essentially allow a fire count of 1 mean that 1 trace point will fire even if there are more than 1 client connected.

Describe the solution you'd like
This would need Deep to track fire counts of the tracepoints as the snapshots arrive, then disable the tracepoints in the service once they have exceeded the count.

Considerations:

• Scale of the control - how would a deep cluster of 10-20-1000 nodes handle this?
• How could we track the fire count (as a metric)? - maybe look at the global rate limit control for ingest.
• How would we re arm these disabled tracepoints?
• How would a local deep-agent/collector handle these?

Additional context
Add any other context or screenshots about the feature request here.

Describe the bug
When reviewing the meta.json for the blocks there are some data points that seem to be unused or inconsistent.

• totalObjects vs totalRecords - objects are the number of snapshots in the block, records always seems to be 1. They are then referred to inconsistently in the code in compactor.go#60 for example.
• indexPageSize - seems to be passed around during requests but never set or read
• encoding and dataEncoding - should be reviewed to ensure these are correct

Expected behavior
Naming and usage should be consistent and clear.

Describe the bug
Not sure if this is considered a bug but I expected the example to be runnable to test the project without creating a sample app.

When trying to run the docker-compose.yml in the local example, I get an unauthorized error from the docker daemon.

docker compose up -d
[+] Running 4/4
 ✘ prometheus Error                                                                     0.4s
 ✘ test_app Error                                                                       0.4s
 ✘ grafana Error                                                                        0.4s
 ✘ deep Error                                                                           0.4s
Error response from daemon: Head "https://ghcr.io/v2/intergral/deep-python-client/manifests/simple-app": unauthorized

To Reproduce
Try to pull the images without logging into/authorizing docker against any docker registries.

Expected behavior
I expected to be able to run the example locally by running the docker compose file.

Desktop (please complete the following information):

• OS: Pop!_OS 22.04 LTS
• Kernel: 6.4.6-76060406-generic
• Docker version: 24.0.5, build ced0996

Review and clean up module:

• remove references to temp
• remove references to traces (except for monitoring purposes)
• enable any disabled tests
• review module for redundant functionality
• review module for missing functionality

Review and clean up module:

• remove references to temp
• remove references to traces (except for monitoring purposes)
• enable any disabled tests
• review module for redundant functionality
• review module for missing functionality

Describe the bug
When calling the api for tag values the tag service.name doesn't return any values.

To Reproduce
Steps to reproduce the behavior:

1. connect a client (all clients must send service.name tag)
2. use api to get values /api/search/tag/service.name/values

Expected behavior
Should get a list of all service.name seen

Screenshots

Create a query language based on promql/traceql to allow users to query for snapshot data.

Features:

• find by id e.g. "112dd12d112d"
• find by resource e.g. "{ resource.language="java" }" - would return all snapshots that have triggered on a java application
• find by attribute e.g. "{ attribute.path ~= ".*/some_file.py" }" - would return all snapshots that triggered in a file called 'some_file.py'

Review and clean up pkg/deepql:

remove references to temp
remove references to traces (except for monitoring purposes)
enable any disabled tests
review module for redundant functionality
review module for missing functionality

Review and clean up module:

• remove references to temp
• remove references to traces (except for monitoring purposes)
• enable any disabled tests
• review module for redundant functionality
• review module for missing functionality

Check the readmes on all the projects for the beta to ensure they are correct.

Is your feature request related to a problem? Please describe.
As a user that is bound by security policies (SOC, ISO27001, etc). Any change to production needs to be recorded in an audit log for historic review.

Deep should be able to provide such a log that can be exported to external services.

Describe the solution you'd like
A way to export an audit log from DEEP that contains information about tracepoints that have been created, modified, deleted etc.

This log should contains at minimum:

• The user making the change
• The time/date the change was made
• The details of the change (before/after or delta)

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Review and clean up module:

• remove references to temp
• remove references to traces (except for monitoring purposes)
• enable any disabled tests
• review module for redundant functionality
• review module for missing functionality

Review and clean up pkg/deepdb:

remove references to temp
remove references to traces (except for monitoring purposes)
enable any disabled tests
review module for redundant functionality
review module for missing functionality

Review and clean up module:

• remove references to temp
• remove references to traces (except for monitoring purposes)
• enable any disabled tests
• review module for redundant functionality
• review module for missing functionality