[BUG] expect_row_values_to_have_recent_data with dbt utils 1.0.0 about dbt-expectations HOT 16 CLOSED

Ok, merged the PR and released as part of 0.8.1, which should hopefully be on dbt package hub with the next hour or so. Thanks for everyone's help in tracking this down! 👍

from dbt-expectations.

Hi @clausherther and @olivierdet, looking into it now.
Will report my findings shortly.

from dbt-expectations.

Ok, I released both updates that address the dbt.* issues in dbt-date in 0.7.2, which should be on dbt package hub in the next hour or so. The current release of dbt-expectations (0.8.2) should just pull that in once available. Thanks everyone!
(Leaving this open for now.)

from dbt-expectations.

dbt-date 0.7.2 seems so have resolved this for me 👌 thanks!

from dbt-expectations.

Hi @olivierdet, as of dbt-expectations release 0.8.0, we no longer have a dependency on dbt-utils. Make sure you run dbt clean && dbt deps to remove old versions and download the 0.8.0 release.
Have you checked with elementary_data if this is caused by something on their end?
It would also help if you provided more of the error log, and also the contents of your packages.yml file.

from dbt-expectations.

Hello,
Yes, I've run the clean and deps.
Here you'll find my package

packages:
  - package: dbt-labs/dbt_utils
    version: 1.0.0
  - package: tnightengale/dbt_meta_testing
    version: 0.3.6
  - package: Snowflake-Labs/dbt_constraints
    version: [">=0.5.0", "<0.6.0"]
  - package: calogica/dbt_expectations
    version: 0.8.0
  - package: elementary-data/elementary
    version: 0.6.6

If we remove the following, the error disapear.

  - package: elementary-data/elementary
    version: 0.6.6

Here the log message :

�[0m16:45:18.883710 [debug] [MainThread]: Flushing usage events
�[0m16:45:19.443596 [error] [MainThread]: Encountered an error:
Compilation Error in test dbt_expectations_expect_row_values_to_have_recent_data_fact_admin_site_event_tracking_event_at__day__1___event_at_modules_datetime_date_today_modules_datetime_timedelta_14_ (models/lakehouse/core/fact_admin_site_event_tracking.yml)
  'dbt_utils' is undefined
  
  > in macro default__test_expect_row_values_to_have_recent_data (macros/schema_tests/table_shape/expect_row_values_to_have_recent_data.sql)
  > called by macro test_expect_row_values_to_have_recent_data (macros/schema_tests/table_shape/expect_row_values_to_have_recent_data.sql)
  > called by macro type_timestamp (macros/utils/data_types/data_type.sql)
  > called by test dbt_expectations_expect_row_values_to_have_recent_data_fact_admin_site_event_tracking_event_at__day__1___event_at_modules_datetime_date_today_modules_datetime_timedelta_14_ (models/lakehouse/core/fact_admin_site_event_tracking.yml)

Here the extraction of my yml file

        # Expect the model to have rows every day
        tests:
          - dbt_expectations.expect_row_values_to_have_recent_data:
              datepart: day
              interval: 1
              row_condition: " event_at >= '{{ modules.datetime.date.today() - modules.datetime.timedelta(14) }}' "

from dbt-expectations.

Looks like this error comes from the inclusion of the elementary package then. We no longer use dbt_utils at all. It's possible they've overridden the expect_row_values_to_have_recent_data test and are still using dbt_utils in their version.

from dbt-expectations.

Hi,
We were able to recreate and understand why it happens.
To support both dbt 1.0-1.2 and 1.3 and above, we created our own macros that call dbt if version>1.3 and dbt_utils if version<1.3.
One of these macros is named type_timestamp, which uses dbt_utils.

We import dbt_utils, so I don't fully understand why it is undefined, but it sure happens although everything is installed properly:

Anyway, we are thinking of a temp workaround for you @olivierdet, and will figure out the best solution for this. Also trying to understand if this is a bug in how dbt searches macros in packages, so we could report to them if needed.

@clausherther I see you removed the dbt_utils dependency. Could you share why?

from dbt-expectations.

@Maayan-s thanks for digging into this! I removed dependencies on dbt-utils b/c managing the chained relationship between dbt-date, dbt-utils and dbt-expectations got too much. Once dbt-core contained all the necessary date macros, I was able to remove dbt-utils from dbt-date and downstream. For type_timestamp we actually have a local version in dbt-expectations b/c the dbt-core version returns an imprecise value, I filed an issue on dbt-core for that.

I don't super understand how the dbt elementary-data package works. How come this issue surfaces in a dbt-expectations macro?

from dbt-expectations.

Hi @olivierdet @clausherther @Maayan-s, I'll try to elaborate on why we believe that happens.
The way dbt works is that upon compiling, it iterates over all the macros in the project.
That includes macros of packages as well.

dbt will work out the dependencies of each package and compile the dependencies first until it compiled all the macros throughout the project. For instance, for a user that uses elementary, it'll first compile dbt-utils which is a dependency package, and only then elementary itself.

For a user that installs both dbt-expectations and elementary as @olivierdet,
and given that dbt-expectations's macros don't currently use package-scoping for the macro lookups (using type_timestamp() rather than dbt.type_timestamp().

It could happen that type_timestamp() will resolve to elementary.type_timestamp() which in return will do the following lookup {% set macro = dbt.type_timestamp or dbt_utils.type_timestamp %} as @Maayan-s explained.

The problem here is that dbt_utils's macros were not yet loaded (compiled) into the context object and therefore caused a crash when trying to access it.

We were able to fix it by immediately returning the macros that we override and are likely to appear on other packages, we're releasing a hotfix now.

Hopefully that's clearer.

from dbt-expectations.

@clausherther
Changing locally the code of dbt-expectations to explicit macro refs (type_timestamp() -> dbt_expectations.type_timestamp()) will prevent this from happening with other packages in the future.
Happy to open a PR if thats ok with you.

from dbt-expectations.

Ah, good to know, thanks both @Maayan-s and @elongl!
I'll change our references to be explicit, should probably do that for the dbt-core macros then as well? i.e. dbt.type_int() to be safe?

from dbt-expectations.

Yes, that's is generally the recommended practice 👌🏽
Thanks for your help! Much appreciated.

from dbt-expectations.

Hello, I try to install the dbt_expectation 0.8.2, but I’ve got exactly the same issue :-(

Compilation Error in test dbt_expectations_expect_row_values_to_have_recent_data_fact_admin_site_event_tracking_event_at__day__1___event_at_modules_datetime_date_today_modules_datetime_timedelta_14_ (models/lakehouse/core/fact_admin_site_event_tracking.yml)
  'dbt_utils' is undefined

  > in macro now (macros/calendar_date/now.sql)
  > called by macro default__test_expect_row_values_to_have_recent_data (macros/schema_tests/table_shape/expect_row_values_to_have_recent_data.sql)
  > called by macro test_expect_row_values_to_have_recent_data (macros/schema_tests/table_shape/expect_row_values_to_have_recent_data.sql)
  > called by macro current_timestamp (macros/utils/cross_db_utils/current_timestamp.sql)
  > called by test dbt_expectations_expect_row_values_to_have_recent_data_fact_admin_site_event_tracking_event_at__day__1___event_at_modules_datetime_date_today_modules_datetime_timedelta_14_ (models/lakehouse/core/fact_admin_site_event_tracking.yml)

from dbt-expectations.

Same issue here, looks as though the current_timestamp() here is getting resolved as an expected macro from dbt-utils or something? As it was pre-1.0.0, which seems consistent with the dbt-core ticket opened in the above chain

from dbt-expectations.

@gary-beautypie ugh, I missed that in dbt-date yesterday, thanks for catching that. To be clear, this is only an issue if used in conjunction with dbt-utils. I'll merge your PR later today and release a patch on dbt-date.

from dbt-expectations.