The computed branch is deleted and recreated each run. If the users enter a wrong name (like "main") they will lose all the data.
Computed branch should not be a parameter, but a default value based on the metric name

Problem

installation of datagit requires to setup a pipeline that runs the datagit package script.
For DBT users we want to leverage metric definitions, and pipeline runs.

Features

• As a DBT user I can configure a git repository, with authentication details
• As a DBT user I can configure a table to be historized in datagit
• As a DBT user when I run a dbt command (probably dbt run), the datagit package is called with github token and the dataframe containing the table

Technical strategy

From current knowledge, dbt offer either dbt packages or dbt adapters.
dbt adapters are made for SQL engines, and datagit is no sql engine. But a datagit repository looks like a DWH since you can read and write.
An implementation could be a SQL facade that push the data to github. An other implementation could be an Adapter run method and implement only the write.

For dbt packages it seems not possible to invoke python packages, or to make API calls. It should be used for python utils operations.

Pray the lord so it won't be too long

TODO:

• Add a dual table that scrolls
• Design tables to represent red and green
• Deploy storybook
• Display the commit diff
  • display ellipsis between hunks
  • make table header sticky
  • header complet
• Month filter
  • I see the month in a dropdown
  • When I select a month on the dropdown, the diff is filtered on that
• deploy in production (example)
• Display diff on metric

• Add datadrift drilldown link to issue
• Add comment to changelog (in notion report)

• Format number with comma for thousands
• Metabase style for hover

We should use drift_handler and new_data_handler instead of a class having both handlers.

Why we need this

When looking at the summary, users then go to find the detailed reports for the KPI that drifted the most

Solution Design

Simplified nav to detailed reports by directly adding below the summary chart the filtered database

Tasks

tbd

Why we need this

Datagit document needs to be improved for faster & smoother installation

Task

• Add PyGithub import
• Add auto_merge drift in the documentation. Auto_merge should be by default in the store_metric function
• Add step to authorize your datadrift app in your notion page
• Remove date_column in the notion config json
• Add to round numbers as a best practice when setting up your query
• Add lineage to the documentation

Why we need this

When opening a changelog event page on Notion, users do not have much context. They need to click on the commit link to have any.

Solution Design

Directly add context in the body of the event page. Context should include:

• total nb of diff introduced by the commit
• impact day by day

Task

tbd

When metric starts at 0, all the modification that introduce a 0 drift are considered a "Initial Value" event.

In the changelog database, the Initial value has 0 impact, while it should have the initial value

Implement a driftdb-connector in the python library. It should look like the local-connector, but uses the API of the datadrift backend.

Why we need this

Users want to compare two metrics in time and see by how much/why it has drifted. Use cases: compare with the previous investor update or board pack.

je veux comparer le MRR EoP d’aujourd’hui à celui qui a été communiqué au board en juillet 2023 (et pas par rapport au dernier main ou depuis la 1e historisation)

Solution Design

tbd

Task

tbd

• Rename all metric to table
• Remove Prints, use python logging
• Use isort within the formatter
• Use 120 lines width
• Use a class for connectors and drift evaluator

In order to detect new data, from historical data update (drift). The dataframes needs a date column which was not provided by the query builder

• handle default case when user has not defined a guard
• Add a verbose parameter
• Make the cli work without a date, with an option "latest snapshot of today"
• Configure alert connector in the datadrift file
• More colorful logs
• Documentation

Why we need this

The graph is not easily understandable and does not provide waw effect. Users do not understand:

• what the x and y axis represent
• which data series is which
• The charts url expires

Solution Design

Implement a new chart lib and new features for the chart to allow users to "align timeline" as in Github Star History

Tasks

• Fixed color code for Y and M
• On hover, display the metric name (metric + date)
• Display unit on X axis
• Decide between timeline or cohort display mode

Apply same pattern as for the database connection

Issue

datagit.github_connector.store_metric() function has a strange behavior regarding opening or merging PRs:

• If assignees is set, then a PR is opened
• If not, no PR is open, but:
  • if branch happens to be set to main, then changes are pushed to main branch and can be used in future drift computation
  • if branch is not set / set to something else, drift will be simply stored in a branch

We have 3 different behaviors (related to "how to handle changes?") that are controlled by 2 params that seem not really related.

Also having changes pushed in main branch is nice, but what's be nicer would be having a PR opened and then automatically merged --> better for keeping track of who changed what

Proposal

• New param: merge_policy = [ no_pr, open_pr, auto_merge_pr ]
  • no_pr -> Simply push changes to a branch, and do nothing more
  • open_pr -> Same as no_pr but also open a PR
  • auto_merge_pr -> Same as open_pr but immediately auto-merge it
• We keep param assignee, but if it's None/empty list then -> PR is opened with no assignee
• We keep param branch, it simply controls where things are pushed

We could use an Enum for that

class MergePolicy(Enum):
    NoPR = "NoPR"
    OpenPR = "OpenPR"
    AutoMergePR = "AutoMergePR"


github_connector.store_metric(
    ghClient=Github(github_api_token),
    dataframe=dataframe,
    merge_policy=MergePolicy.AutoMergePR,
    assignees=[],
    branch=branch,
)

As of now, the report page is found with a reportId generated from the metric name, timegrain, dimension and period. But the content of the report is deleted and recreated at each refresh.
There is a check on the reportHash to avoid recreation.

The feature:

What:

• When there is a report update we only update what is mandatory (new graph, updated summary) and we append data when mandatory (new line in the changelog). Nothing is deleted

Why:

• Notion page behaves as expected instead of being deleted. User can add comments & content.
• Avoid too many calls to Notion API. Rate limiting during a report refresh ends with the report being partially deleted, then partially recreated.

How:

• Store the Ids of the block that need to be updated, and the id of the block where changelog will be append.
• Store those Ids in a database. Making the project stateful, and harder to self host