support metric format from dbt about metriql HOT 6 CLOSED

It's implemented in 0b133b7! 🎉

from metriql.

Great question @danielefrigo! Already started the implementation but there is only one issue and I would love to get your opinion about it. dbt's metrics implementation requires dimensions to be set for each metric whereas Metriql lets you define metrics under models and automatically makes them available for all the metrics. It's unfortunately impossible to implement this behavior for most of the downstream applications so we did not design it that way.

We can explicitly make the dimensions hidden/visible for the metrics potentially but I don't really see any viable use-case since it's not possible to pass this information to downstream applications so I think that I will just ignore that property.

WDYT?

from metriql.

Hi @buremba, sorry for the late answer.
This detail about dimensions definition let me really doubtful when I read it in dbt docs.
I agree with you, it's much more intuitive defining them at model level.

In order to leverage dbt metadata, an idea could be to expose at model level the intersection of the dimensions defined in the related metrics. It's more redundant to define, but could at least guarantee compatibility with the dbt metadata format (hoping they'll change their mind on this point in the future).

Makes sense?

from metriql.

@danielefrigo Actually, I had more time to dive into it for the last few days and found a way to integrate dbt's metric definitions to the downstream tools: We may expose these metrics as database tables rather than the measures. The dimensions definition picks the relevant columns from the target model and exposes them to the downstream tools in that case and the dataset can have a single metric definition.

We should still support our metric definitions under models.meta though and merge the dbt's metric definitions with our existing metric definitions approach for so that the dimension references can point to the dimension references under models.meta or models.columns for reusability and backward compatibility.

That's similar to your idea and we will be able to support metrics as a first-class citizens.

from metriql.

Maybe I misunderstood your explanation, but does this mean that using dbt metric definitions metriql will expose one separate dataset for each measure?
I understand where this comes from the dbt metric - dimension link, but on the other hand this could create great performance degredation and modeling complexities in BI tools since you often want to query multiple measures from the same fact table.

from metriql.

@danielefrigo , you're right and that's why we will continue supporting our own metric definitions. All the BI tools including Looker follow our approach but the metric stores such as Transform and Supergrain follow a similar approach to dbt's metric definitions. I believe that it's the feature because non-technical people know metrics, not database tables. We will probably have a feature to categorize the metrics and the downstream tools will have to adapt their user interfaces over time.

Until that day, we need to find a bridge with Metriql as it's our responsibility. Don't worry about the performance degradation because the underlying architecture will be the same, the metrics will actually point to the measures under models at Metriql so they will re-use the same Aggregates. For the modeling complexities, I think that it's a mental shift we will need to adapt over time for the sake of reducing the complexity for non-technical people.

from metriql.