liatrio / liatrio-otel-collector Goto Github PK

View Code? Open in Web Editor NEW

34.0 9.0 2.0 1.66 MB

The Liatrio distribution of the OpenTelemetry Collector

Home Page: https://openo11y.dev

License: Apache License 2.0

Go 97.54% Makefile 2.13% Dockerfile 0.33%

o11y go metrics monitoring observability opentelemetry opentelemetry-collector

liatrio-otel-collector's Introduction

liatrio-otel-collector

The Liatrio OTEL Collector is an upstream distribution of the Open Telemetry collector, rebuilt with custom packages hosted within this repository. These custom packages are by default targeted for downstream contribution to Open Telemetry; pursuant acceptance by the community.

Quick Start Guide

Before diving into the codebase, you'll need to set up a few things. This guide is designed to help you get up and running in no time!

Here are the steps to quickly get started with this project:

Install Go: Use Homebrew to install Go with the following command:
```
brew install go
```
Install pre-commit: With the help of Homebrew, install the pre-commit as follows:
```
brew install pre-commit
```
Initialize pre-commit: Once installed, initiate pre-commit by running:
```
pre-commit install
```
Set up GitHub and/or GitLab scraper(s).
Start the collector: Finally, initiate the program:
```
make run
```

Configure GitHub Scraper

To configure the GitHub Scraper you will need to make the following changes to config/config.yaml:

Uncomment/add extensions.basicauth/github section

basicauth/github:
    client_auth:
        username: ${env:GITHUB_USER}
        password: ${env:GITHUB_PAT}

Uncomment/add receivers.gitprovider.scrapers.github.auth section
```
auth:
    authenticator: basicauth/github
```

Add basicauth/github to the service.extensions list

extensions: [health_check, pprof, zpages, basicauth/github]

Set environment variables: GITHUB_ORG, GITHUB_USER, and GITHUB_PAT

Configure GitLab Scraper

To configure the GitLab Scraper you will need to make the following changes to config/config.yaml

Uncomment/add extensions.bearertokenauth/gitlab section

bearertokenauth/gitlab:
    token: ${env:GITLAB_PAT}

Uncomment/add receivers.gitprovider.scrapers.gitlab.auth section
```
auth:
    authenticator: bearertokenauth/gitlab
```

Add bearertokenauth/gitlab to the service.extensions list

extensions: [health_check, pprof, zpages, bearertokenauth/gitlab]

Set environment variables: GL_ORG and GITLAB_PAT

Exporting to Grafana Cloud

If you want to export your data to Grafana Cloud through their OTLP endpoint, there's a couple of extra things you'll need to do.

Run export GRAF_USER and export GRAF_PAT with your instance id and cloud api key
Update the config/config.yaml file with the following:

extensions:
  ...

  basicauth/grafana:
    client_auth:
      username: ${env:GRAF_USER}
      password: ${env:GRAF_PAT}
...

exporters:
  ...
  otlphttp:
    auth:
      authenticator: basicauth/grafana
    endpoint: https://otlp-gateway-prod-us-central-0.grafana.net/otlp

...

service:
  extensions: [..., basicauth/grafana]
  pipelines:
    metrics:
      receivers: [otlp, gitprovider]
      processors: []
      exporters: [..., otlphttp]

Debugging

To debug through vscode:

Create a .debug.env file in the root of the repo, make a copy of the example
run make build-debug
run cd build && code .
With your cursor focused in main.go within the build directory. Launch the Launch Otel Collector in debug" debug configuration.

OTEL Intro

OTEL is a protocol used for distributed logging, tracing, and metrics. To collect metrics from various services, we need to configure receivers. OTEL provides many built-in receivers, but in certain cases, we may need to create custom receivers to meet our specific requirements.

A custom receiver in OTEL is a program that listens to a specific endpoint and receives incoming log, metrics, or trace data. This receiver then pipes the content to a process, for it to then be exported to a data backend.

Creating a custom receiver in OTEL involves implementing the receiver interface and defining the endpoint where the receiver will listen for incoming trace data. Once the receiver is implemented, it can be deployed to a specific location and configured to start receiving trace data.

Prereqs

There is currently a guide to build a custom trace receiver. It is a long read, requires a fairly deep understanding, and is slightly out of date due to non-backwards compatible internal API breaking changes. This document and receiver example attempts to simplify that to an extent.

There are a few main concepts that should help you get started:

Get familiar with the ocb tool. It is used to build custom collectors using a build-config.yaml file.
Get familiar with Go & the Factory design pattern.
Clearly define what outcome you want before building a customization.
Get familiar with Go interfaces.
Get familiar with delv the go debugger.

References & Useful Resources

liatrio-otel-collector's People

Contributors

Stargazers

Watchers

Forkers

caseyw laurentmarchelli

liatrio-otel-collector's Issues

Add the ability to filter down a search in GitLab to focus on team or topic

          ok. it definitely would be ideal to be able to look at a group by default, but be able to add a search query (like by topic) in the event that you want to narrow down.

For example, if we had a liatrio group on gitlab, but we had 500 repo,s we'd only want to look at ones for a specific team(topic).

Originally posted by @adrielp in #97 (comment)

GitHub and GitLab OnPremise

Hi guys,

Your work looks promising !!!
However, is it reserved for official GitHub and Gitlab instances or "On Premise" instances will be supported as well with the appropriate configuration ?
For instance :

config.yaml

---
extensions:
  bearertokenauth/github:
    token: "ghp_thisismytoken"
receivers:
  gitprovider:
    # https://github.com/open-telemetry/opentelemetry-collector/tree/main/config/confighttp
    # https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md
    endpoint: "https://ghetest.mycompany.grp/api/graphql"
    tls:
      insecure: false
      insecure_skip_verify: true
    scrapers:
      github:
        github_org: "target_organization"
        auth:
          authenticator: "bearertokenauth/github"
        metrics:
          git.repository.branch.count:
            enabled: true
          git.repository.branch.time:
            enabled: true
          git.repository.contributor.count:
            enabled: true
          git.repository.count:
            enabled: true
          git.repository.pull_request.time:
            enabled: true
        resource_attributes:
          git.vendor.name:
            enabled: true
          organization.name:
            enabled: true
exporters:
  logging:
    verbosity: detailed
service:
  extensions: [bearertokenauth/github]
  pipelines:
    metrics:
      receivers: [gitprovider]
      exporters: [logging]

Validate or improve go routine definition in gitlabscraper

          I'm thinking we don't need this and just need to have a case and send `done` to the channel in the even that we're done? i might be wrong

Originally posted by @adrielp in #97 (comment)

chore: improve error catching of login query

The login_query in the github_scraper.go file is used to check if the provided org in the config.yml file is an org or a user. Currently if the provided config is a user, an error is caught from the query and an info message logged.

It would be better to check after getting an error on the login query against an org, if a query against a user produces an error. If an error returns on user query, then we should log an error to indicate that the config provided cannot be found at all.

move this logic to its own function, with the return result being a boolean if the org provided in the config.yml is an organization or user
convert login_query to org_login_query
add another login query but for users

	var user_login_query struct {
		Organization struct {
			Login githubv4.String
		} `graphql:"user(login: $login)"`
	}

add additional if else statements to catch the error when an org its found, then check if user can be found, and if that errors, log the error. If user can be found, set provided_org_is_org to false

that section might look something like this

	// do we get an error when looking up the provided org config as an org?
	err_org := graphqlClient.Query(context.Background(), &org_login_query, variables)
	if err_org != nil {
		ghs.logger.Sugar().Info("Org provided not found; it may be a user")
		// do we get an error when looking up the provided org config as a user?
		err_user := graphqlClient.Query(context.Background(), &user_login_query, variables)
		if err_user != nil {
			// if we get here, we got an error trying to look up the provided org config as an org and user
			ghs.logger.Sugar().Errorf("Error finding Org config provided, not found as org or user", zap.Error(err_user))
		} else {
			// no error found when looking up provided org config as a user
			provided_org_is_org = false
		}
	} else {
		// no error found when looking up provided org config as an org
		provided_org_is_org = true
	}
	// now that we have determined if login is org or user, we can define the query

Contribute second PR for git provider receiver

Overview

We need to contribute the git provider receiver to the OTEL contribution repository. This process has already been started with this issue on the OTEL project.

The second PR needs to have core functionality bringing it to a working state.

It may require more than one PR as well to keep them small.

Tasks

Beta Give feedback

add functionality to receiver
add tests and ensure proper test coverage
create pr
accept feedback and iterate
Options

Improve endpoint handling for github and gitlab scrapers

#156 (comment)

Address docker multi-platform builds & performance of CI in liatrio otel collector

Add `factory_test.go` for gitlab scraper's `factory.go`

This file has two functions we need to write tests for.

func (f *Factory) CreateDefaultConfig() internal.Config { // stuff }

func (f *Factory) CreateMetricsScraper(
	ctx context.Context,
	params receiver.CreateSettings,
	cfg internal.Config,
) (scraperhelper.Scraper, error) { //stuff }

SPIKE on OTEL implementation of SDC

Utilizing work started by Adriel, build out a GitHub receiver to demonstrate how the OTEL Collector pattern can address future SDC needs and act as a POC for the future of Team Effectiveness metrics

Repo where new GitHub receiver is being implemented

In these go routines, we should add buffered channels to reduce the need for extreme resources caused by multiple threads.

          In these go routines, we should add buffered channels to reduce the need for extreme resources caused by multiple threads.

Originally posted by @adrielp in #106 (comment)

Additionally we should cleanup these go routines & ensure they follow go routine best practices

Refactor the github scraper to be more performant and functional

In the gitHub_scraper.go file, we should create the githubv4 once and pass that githubv4 to each function instead of creating a new client in each function.

update all functions (except for scrape()) that have graphqlClient := githubv4.NewClient(ghs.client) to accept graphqlClient as an argument and delete their graphqlClient creation line
in the scrape function, pass the graphqlClient created to all the function now accepting that as an arguement
and sneak this cleanup recommendation of variables where applicable

Requests are sent to the server even they are disabled in the configuration

Version : 0.30.0-386
Image : ghcr.io/liatrio/liatrio-otel-collector:0.30.0-386
Tests is realized with an authentication token with limited capacity (read only).
All Metrics except git.repository.count are disabled.

config.yaml

...
        scrapers:
          github:
            # https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/README.md
            # https://github.com/open-telemetry/opentelemetry-collector/tree/main/config/confighttp
            # https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md
            github_org: "target_organization"
            endpoint: "https://ghetest.mycompany.grp:443/api"
            tls:
              insecure: false
              insecure_skip_verify: true
            auth:
              authenticator: "bearertokenauth/github"
            metrics:
              git.repository.branch.commit.aheadby.count:
                enabled: false
              git.repository.branch.commit.behindby.count:
                enabled: false
              git.repository.branch.count:
                enabled: false
              git.repository.branch.line.addition.count:
                enabled: false
              git.repository.branch.line.deletion.count:
                enabled: false
              git.repository.branch.time:
                enabled: false
              git.repository.count:
                enabled: true
              git.repository.pull_request.approval.time:
                enabled: false
              git.repository.pull_request.deployment.time:
                enabled: false
              git.repository.pull_request.merge.time:
                enabled: false
              git.repository.pull_request.merged.count:
                enabled: false
              git.repository.pull_request.open.count:
                enabled: false
              git.repository.pull_request.time:
                enabled: false
              # Optional Metrics
              git.repository.contributor.count:
                enabled: false
...

Log output :

2023-10-19T14:21:36.740Z error githubscraper/github_scraper.go:261 error getting branch data%!(EXTRA zapcore.Field={error 26 0  input:9: Your token has not been granted the required scopes to execute this query. The 'aheadBy' field requires one of the following scopes: ['repo'], but your token has only been granted the: ['public_repo', 'read:audit_log', 'read:org', 'read:project', 'read:public_key', 'read:repo_hook', 'read:user', 'repo:status', 'repo_deployment', 'user:email'] scopes. Please modify your token's scopes at: https://ghetest.mycompany.grp/settings/tokens.
input:10: Your token has not been granted the required scopes to execute this query. The 'behindBy' field requires one of the following scopes: ['repo'], but your token has only been granted the: ['public_repo', 'read:audit_log', 'read:org', 'read:project', 'read:public_key', 'read:repo_hook', 'read:user', 'repo:status', 'repo_deployment', 'user:email'] scopes. Please modify your token's scopes at: https://ghetest.mycompany.grp/settings/tokens.
}) {"kind": "receiver", "name": "gitprovider/build-tools", "data_type": "metrics"}
github.com/liatrio/liatrio-otel-collector/pkg/receiver/gitproviderreceiver/internal/scraper/githubscraper.(*githubScraper).getBranches
 /home/runner/work/liatrio-otel-collector/liatrio-otel-collector/pkg/receiver/gitproviderreceiver/internal/scraper/githubscraper/github_scraper.go:261
2023-10-19T14:21:36.790Z error githubscraper/github_scraper.go:261 error getting branch data%!(EXTRA zapcore.Field={error 26 0  input:9: Your token has not been granted the required scopes to execute this query. The 'aheadBy' field requires one of the following scopes: ['repo'], but your token has only been granted the: ['public_repo', 'read:audit_log', 'read:org', 'read:project', 'read:public_key', 'read:repo_hook', 'read:user', 'repo:status', 'repo_deployment', 'user:email'] scopes. Please modify your token's scopes at: https://ghetest.mycompany.grp/settings/tokens.
input:10: Your token has not been granted the required scopes to execute this query. The 'behindBy' field requires one of the following scopes: ['repo'], but your token has only been granted the: ['public_repo', 'read:audit_log', 'read:org', 'read:project', 'read:public_key', 'read:repo_hook', 'read:user', 'repo:status', 'repo_deployment', 'user:email'] scopes. Please modify your token's scopes at: https://ghetest.mycompany.grp/settings/tokens.
}) {"kind": "receiver", "name": "gitprovider/build-tools", "data_type": "metrics"}
github.com/liatrio/liatrio-otel-collector/pkg/receiver/gitproviderreceiver/internal/scraper/githubscraper.(*githubScraper).getBranches
 /home/runner/work/liatrio-otel-collector/liatrio-otel-collector/pkg/receiver/gitproviderreceiver/internal/scraper/githubscraper/github_scraper.go:261

Add tests for the getBranches function in the Github scraper

Add ability to search within the organization to prune down information

Overview

With large orgs there's the potential for not being able to actually grab all the data. For example, we've seen some orgs with 15,000+ repositories. Thus it'd be performant and ideal to be able to additionally prune down the search in those cases, such that you can search for the things you want.

Acceptance Criteria

tested
documentation updated
configuration optional
allows for searching by topic, team, or repo name

Complete metrics for GitHub reciever

The current implementation of the GitHub receiver is able to collect information needed for calculating repo & branch metrics. There is some remaining work to complete the calculation and create new data points

Add ability to get time delta between PR approval, merge, then publish to production in GitHub

Overview

For metrics on deployments, mirroring the basic DORA metrics, we want to be able to calculate the time delta from pr approval, to merge, to deployment to production.

Optimize the build & release process for the otel collector

Overview

We need to optimize the build & release process as it currently takes some time.

Acceptance criteria

caching is done through workflows through and through
makefiles are updated to be faster and in parallel
builds for goreleaser faster
workflows are blocking on main when there are many merges

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

dockerfile

Dockerfile

alpine 3.20.0

github-actions

.github/workflows/auto-label-issues.yml

liatrio/poc-label-issue-workflow v0.1.0

.github/workflows/build.yml

actions/checkout v4

actions/setup-go v5

actions/checkout v4

actions/setup-go v5

docker/setup-qemu-action v3

docker/setup-buildx-action v3

actions/create-github-app-token v1

goreleaser/goreleaser-action v6

actions/checkout v4

actions/setup-go v5

codecov/codecov-action v4

actions/create-github-app-token v1

go-semantic-release/action v1

.github/workflows/pr_conventional.yml

actions/create-github-app-token v1

amannn/action-semantic-pull-request v5

.github/workflows/pr_labeler.yml

actions/labeler v5

.github/workflows/release.yml

actions/checkout v4

docker/login-action v3

actions/setup-go v5

docker/setup-qemu-action v3

docker/setup-buildx-action v3

goreleaser/goreleaser-action v6

.github/workflows/scorecard.yml

actions/checkout v4

ossf/scorecard-action v2.3.3@dc50aa9510b46c811795eb24b2f1ba02a914e534

actions/upload-artifact v4.3.3@65462800fd760344b1a7b4382951275a0abb4808

github/codeql-action v3.25.8@2e230e8fe0ad3a14a340ad0815ddb96d599d2aff

gomod

cmd/compgen/go.mod

go 1.22

github.com/spf13/cobra v1.8.0

github.com/stretchr/testify v1.9.0

extension/githubappauthextension/go.mod

go 1.22

github.com/bradleyfalzon/ghinstallation/v2 v2.11.0

github.com/stretchr/testify v1.9.0

go.opentelemetry.io/collector/component v0.101.0

go.opentelemetry.io/collector/confmap v0.101.0

go.opentelemetry.io/collector/extension v0.101.0

go.opentelemetry.io/collector/extension/auth v0.101.0

go.opentelemetry.io/otel/metric v1.26.0

go.opentelemetry.io/otel/trace v1.26.0

go.uber.org/goleak v1.3.0

go.uber.org/zap v1.27.0

go.mod

go 1.22

go 1.22.2

internal/tools/go.mod

go 1.22

github.com/Khan/genqlient v0.7.0

github.com/golangci/golangci-lint v1.59.0

github.com/google/osv-scanner v1.7.4

github.com/goreleaser/goreleaser/v2 v2.0.0

github.com/securego/gosec/v2 v2.20.1-0.20240525090044-5f0084eb01a9@5f0084eb01a9

go.opentelemetry.io/build-tools/crosslink v0.13.0

go.opentelemetry.io/build-tools/multimod v0.13.0

go.opentelemetry.io/collector/cmd/builder v0.102.1

go.opentelemetry.io/collector/cmd/mdatagen v0.101.0

golang.org/x/tools v0.21.1-0.20240514024235-59d9797072e7@59d9797072e7

golang.org/x/vuln v1.1.1

honnef.co/go/tools v0.4.7

receiver/gitproviderreceiver/go.mod

go 1.22

go 1.22.2

github.com/Khan/genqlient v0.7.0

github.com/google/go-cmp v0.6.0

github.com/google/go-github/v62 v62.0.0

github.com/open-telemetry/opentelemetry-collector-contrib/pkg/golden v0.101.0

github.com/open-telemetry/opentelemetry-collector-contrib/pkg/pdatatest v0.101.0

github.com/stretchr/testify v1.9.0

go.opentelemetry.io/collector/component v0.102.1

go.opentelemetry.io/collector/config/confighttp v0.102.1

go.opentelemetry.io/collector/confmap v0.102.1

go.opentelemetry.io/collector/consumer v0.102.1

go.opentelemetry.io/collector/filter v0.101.0

go.opentelemetry.io/collector/otelcol v0.101.0

go.opentelemetry.io/collector/pdata v1.9.0

go.opentelemetry.io/collector/receiver v0.101.0

go.opentelemetry.io/collector/semconv v0.101.0

go.uber.org/goleak v1.3.0

go.uber.org/zap v1.27.0

github.com/xanzy/go-gitlab v0.105.0

go.opentelemetry.io/otel/metric v1.27.0

go.opentelemetry.io/otel/trace v1.27.0

ocb

config/manifest.yaml

go.opentelemetry.io/collector 0.102.1

github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/connector/countconnector v0.102.0

go.opentelemetry.io/collector/exporter/debugexporter v0.102.1

go.opentelemetry.io/collector/exporter/otlpexporter v0.102.1

go.opentelemetry.io/collector/exporter/otlphttpexporter v0.102.1

github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsxrayexporter v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awscloudwatchlogsexporter v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/exporter/influxdbexporter v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/exporter/zipkinexporter v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/exporter/lokiexporter v0.102.0

go.opentelemetry.io/collector/extension/zpagesextension v0.102.1

go.opentelemetry.io/collector/extension/ballastextension v0.102.1

github.com/open-telemetry/opentelemetry-collector-contrib/extension/basicauthextension v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/extension/pprofextension v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/extension/bearertokenauthextension v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/extension/sigv4authextension v0.102.0

go.opentelemetry.io/collector/processor/batchprocessor v0.102.1

go.opentelemetry.io/collector/processor/memorylimiterprocessor v0.102.1

github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/processor/k8sattributesprocessor v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/processor/attributesprocessor v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/processor/groupbyattrsprocessor v0.102.0

go.opentelemetry.io/collector/receiver/otlpreceiver v0.102.1

github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kubeletstatsreceiver v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/receiver/jaegerreceiver v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/receiver/zipkinreceiver v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/receiver/webhookeventreceiver v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscloudwatchmetricsreceiver v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscloudwatchreceiver v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8seventsreceiver v0.102.0

github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sobjectsreceiver v0.102.0

Check this box to trigger a request for Renovate to run again on this repository

Use the same states variable in the pr count and pr data queries

Since we're passing states into the count query, does it make to sense to use the same states variable in this query to ensure the same states are passed into the count & the getPullRequestData queries.

Originally posted by @adrielp in #77 (comment)

POC out a collector for the "people you've met" spread sheet

Overview

Liatrio maintains a google sheet internally called the "Meeting and Greet Tracker". This sheet lets the company know how many people across the organization have met each other in person.

We could create a quick prototype that pulls that data from the sheet & generates metrics from it (using our practices & backends). We could use this metrics as part of the pulse and combine it with NPS metrics on how the company is doing/feeling.

Add `config_test.go` for gitlab scraper's `config.go`

I am not sure how we should approach this. The only consists of a single struct. We may not need test coverage for it. @adrielp @4lch4

Add codecov as a workflow & badge to the liatrio-otel-collector

Overview

The otel collector repo needs to have a codecov & codeql badges & workflows, helping us set a target of test coverage & displaying this coverage to the users.

Acceptance Criteria

a workflow is added for codecov that runs after unit tests
a badge is updated in the readme showing the codecov status
a codeql badge is added and formatted the same as the other badges

Examples:

Below are some great examples from the opentelemetry contrib repository.
https://github.com/search?q=repo%3Aopen-telemetry%2Fopentelemetry-collector-contrib%20codecov&type=code

I think this could be done better. I think we just can use `restClient, err := graphql.NewClient("https://gitlab.com/api/v4", gls.client)` instead. Let's fix that up in a subsequent PR

          I think this could be done better. I think we just can use `restClient, err := graphql.NewClient("https://gitlab.com/api/v4", gls.client)` instead. Let's fix that up in a subsequent PR

Originally posted by @adrielp in #115 (comment)

Figure out something to do about the copious amounts of things that GitLab can return

          I'm wondering if we should make a comment + code that says if number of repo's > x number, then emit metric, but don't go and make x number of requests for all the different repos with specific branch information, etc.

Originally posted by @adrielp in #97 (comment)

Organization attributes in metrics

I would expect to be able to filter metrics via an organization or individual attribute for the use case where more than one org or individual is being collected from Git Hub or other git providers.

Add more test cases when unmarshalling config in the git provider receiver

          We should add some additional test cases here. Let's merge this in and iterate on another  tasks to add more test cases.

Originally posted by @adrielp in #144 (comment)

Add pull request approval time metric to the gitlab scraper

Overview

Add the pull request approval time metric to the gitlab scraper.

AC

Docs updated.
Scraper is tested.
Any added functions are tested.

bug: github performance issues on graphql query at 100

Bug

I am seeing the following error when trying the following query in postman and in the o11y-demo against the aa client repos.

error

"message": "Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `4A9B:716A:296B9:53EA6:64DCF39F` when reporting this issue."

query

{
  repository(name: "bgs-ramplink", owner: "aainternal") {
    id
    refs(refPrefix: "refs/heads/", first: 100) {
      totalCount
      nodes {
        name
        target {
          ... on Commit {
            id
            history {
              totalCount
              edges {
                node {
                  committedDate
                }
              }
            }
          }
        }
      }
    }
  }
}
```

I tested out 60 and 70 for the the `first` field and 60 did not produce the error, while 70 did.

# Suggestion

Lets adjust the query for now (until Github deals with its graphql performance issues?

Add pull request time metric to the gitlab scraper

Overview

Add the pull request time metric to the gitlab scraper.

AC

Docs updated.
Scraper is tested.
Any added functions are tested.

Add contributor count metric to the gitlab scraper

Overview

Add the contributor count metric to the gitlab scraper.

AC

Docs updated.
Scraper is tested.
Any added functions are tested.

Add pull request deployment time metric to the gitlab scraper

Overview

Add the pull request deployment time metric to the gitlab scraper.

AC

Docs updated.
Scraper is tested.
Any added functions are tested.

bug: collector errors on repo scraping

Locally the collector failed on a local make run against the Liatrio org when scraping for repo information on the gitops-example repo (likely when it attempted to get the PR info).

Definition of Done

fix the scraping function for repos so it can handle either an archived repo or repo with PRs.

Add missing unit tests for the github scraper

There is no unit testing in 6 functions (at the time of writing this issue) in the github_scraper.go file and 0 unit tests in the github_scraper_test.go

write unit tests for each

Add branch time metric to the gitlab scraper

Overview

Add the branch time metric to the gitlab scraper.

AC

Docs updated.
Scraper is tested.
Any added functions are tested.

add addition function with test to replace inline addition

I'm thinking that this should be a simple add function that we can unit test to ensure the output of the function is always going to be what we expect.

Originally posted by @adrielp in #77 (comment)

Add search as the only default option to clean up code

Right now there's some insane if conditionals going on when determining login and which code paths to take, resulting in significant code duplication. The search option should be the default and only option for gather repo's.

Move waitgroup and channels outside of functions for Github scraper

Add `generated_test.go` for gitlab scraper's `generated.go`

This file is generated.

Update liatrio-otel-collector distribution with functioning skeleton

Overview

The liatrio/liatrio-otel-collector repository will be viewed as the repo hosting the liatrio distribution of the otel collector. This repo will serve as the landing place for our development and distribution, prior to publishing upstream. We need to update this repo with core functionality to be able to build and publish the collector.

Acceptance Criteria

the repo has some basic CI to publish the collector via OCB
the repo has the basic file structure needed to add packages and build them into the OTEL collector
documentation in that repo and in o11y docs is updated accordingly including how to contribute, etc.

Examples

Below are a couple distributions that exist and are maintained by other companies in the industry to serve as examples on how to maintain your own distro.

Sumo logic: https://github.com/SumoLogic/sumologic-otel-collector
Splunk: https://github.com/signalfx/splunk-otel-collector

Tasks

Beta Give feedback

No tasks being tracked yet.

Options

Panic in git provider

I'm seeing a panic after about 20-25 minutes when running the collector against about 100 repos.

Add branch count metric to the gitlab scraper

Overview

Add the branch count metric to the gitlab scraper

AC

docs updated
scraper is tested
added functions are tested
pageanation works

Tasks

Beta Give feedback

Update docs
Add tests for scraper
Add tests for new functions
Ensure pagination works
Options

test: provider receiver `config.go`

Adding tests for /liatrio-otel-collector/pkg/receiver/gitproviderreceiver/config.go

There are a few tests already but there aren't any for the Unmarshal functions.

Add metric for size of diff between open branch and main

This metric interweaves with branch age and count.

Add tests to github_scraper.go

Add documentation in the collector to run the calculator for requests and provide options

Overview

Add the documentation to calculate the rate limit and how it effects the resolution of the collector. Also document that you can increase it with an org wide pat.

Contribute a developer experience improvement to OTEL via the OTEL issue

Overview

There are a few things in the OTEL contrib local developer experience that could be improved when developing a new component. An issue has been opened on the project here

Acceptance Criteria

the issue is assigned
the pr is approved and merge
docs are updated accordingly to reflect the improvement
a blog post is made

Duplicate metrics when visualizing metrics in grafana

I'm seeing duplicate entries when viewing the metrics in Grafana, I'm unsure if this is related to the collector implementation or my configuration.

Add pull request merge time metric to the gitlab scraper

Overview

Add the pull request merge time metric to the gitlab scraper.

AC

Docs updated.
Scraper is tested.
Any added functions are tested.

Add `gitlab_scraper_test.go` for gitlab scraper's `gitlab_scraper.go`

Tasks

Beta Give feedback

getBranches
getMergeRequest
scrape
newGitLabScraper
start
Options

We should use buffered chans.

          We should use buffered chans.

Originally posted by @adrielp in #123 (comment)

liatrio / liatrio-otel-collector Goto Github PK

liatrio-otel-collector's Introduction

liatrio-otel-collector

Quick Start Guide

Configure GitHub Scraper

Configure GitLab Scraper

Exporting to Grafana Cloud

Debugging

OTEL Intro

Prereqs

References & Useful Resources

liatrio-otel-collector's People

Contributors

Stargazers

Watchers

Forkers

liatrio-otel-collector's Issues

Overview

Tasks

Overview

Acceptance Criteria

Overview

Overview

Acceptance criteria

Rate-Limited

Open

Detected dependencies

Overview

Overview

Acceptance Criteria

Examples:

Overview

AC

Bug

Overview

AC

Overview

AC

Overview

AC

Overview

AC

Overview

Acceptance Criteria

Examples

Tasks

Overview

AC

Tasks

Overview

Overview

Acceptance Criteria

Overview

AC

Tasks

Recommend Projects

Recommend Topics

Recommend Org