Related asyncapi/cli#1300

Related asyncapi/cli#1300

Related to #1

Problem

Collect CLI metrics that we need to complete #1

Solution

Once asyncapi/cli#841 is merged we need to find how we collect:

• Time to first API Design
• System errors
• Validation errors
• Time to fix a validation error
• Number of created AsyncAPI files
• AsyncAPI 3.0x Adoption

Having a Correlation ID or a way to identify a user session is a must. see asyncapi/cli#841 (comment)

Scope

• asyncapi/cli#841
• asyncapi/cli#1161

Out of bounds

n/a

Problem

We aim to provide a dashboard for monitoring our Developer Experience (DX) metrics, which should be collected in asyncapi/community#879

Solution

The dashboard must have the capability to showcase Developer Experience Working group metrics and enable other working groups to add their respective metrics.

• Time to first API Design
• System errors
• Validation errors
• Time to fix a validation error
• Number of created AsyncAPI files
• AsyncAPI 3.0x Adoption

Look & Feel

Link to Figma

Where do we get metrics:

This initiative focuses on the CLI metrics

Definition of metrics

Time to first API Design

It's the time between the creation of a new AsyncAPI file and generation (doc, code...) for a specific user.

 asyncapi generate - asyncapi new file

System errors

Refer to any issue or malfunction of our tools that are not generated by the user (eg. validation errors).
Example: asyncapi/cli#1137

Validation errors

Errors from asyncapi validate

Time to fix a validation error

The delta between a failing validation and successful validation.

 asyncapi validate (success) - asyncapi validate (error)

Number of created AsyncAPI File

From asyncapi new file

AsyncAPI 3.0x Adoption

We can parse the version when a user triggers asyncapi validate

Tech stack

• React/NextJS
• Library: https://www.tremor.so/ or NewRelic Widgets

Note: This is a suggestion! anything else relevant with a valid argument can work as well.

Rabbit holes

1. Assurance in the Metrics Collection:

   • Due to the pending merge of asyncapi/community#879, uncertainty surrounds whether we'll have the required metrics in time. To proactively address this risk, let's proceed without being blocked and develop using mocked data.

2. Rate Limit Consideration for New Relic API:

   • Should we opt to construct our own component rather than relying on NewRelic widgets, it's imperative that the backend incorporates a caching mechanism. Refer to [this discussion]((asyncapi/community#879 (comment)) for more details.

Scope

• #2
• #3
• #4

Out of bounds

• We show only Developer Experience (DX) metrics

Success criteria

• The dashboard should be accessible to the public either through the website or a designated URL such as metrics.asyncapi.com.
• Maintainers need to have a clear understanding of the metrics presented on the dashboard.

Related to #1

Related to #1

• Time to first API Design
• System errors
• Validation errors
• Time to fix a validation error
• Number of created AsyncAPI files
• AsyncAPI 3.0x Adoption

Problem

New Relic's 1-hour raw aggregation limit on timestamps prevents us from accurately tracking metrics like "Time To First API Design" and "Time To Fix a Validation Error" over a 3-month period.

1)The query will round older data, leading to inaccurate results for these metrics.
2) We can't use the TIMESERIES feature to request historical data.

Possible Solutions

1. Extend the query time: Using the RAW query modifier to access unaggregated data for up to 2 days.
2. Implement Local Caching: Implement a local caching system to store data for these specific metrics for 3 months.