Comments (16)
nicor88
commented on May 28, 2024
1
@jurgispods After some thoughts, I don't think that the issue will be fix in dbt-core. There are cases where we want to drop a relation and no config context is available, e.g. for some table dropping on testing.
As agreed with @Jrmyy we will try to propose a solution that ignore totally the config, and check only the table type in order to do the right drop.
from dbt-athena.
jurgispods
commented on May 28, 2024
1
This seems to work for me as well, thank you @nicor88!
The nice thing about it is that it addresses another issue that I was having from time to time during development: When you switch a table type between "hive" and "iceberg", the drop_table macro would not delete the existing S3 paths on the next run, and the Athena DROP TABLE statement would time out. This can't happen anymore with #119 as it is always checked with Glue what table type we are dealing with instead of relying on user config.
Btw, I had other cases with DROP TABLE statements on Iceberg tables timing out and filed issues with AWS support. In one case, they found a bug related to trailing slashes in the S3 location property which should be fixed soon.
from dbt-athena.
nicor88
commented on May 28, 2024
the issue seems to be due to the query comments attached before your command.
if we add drop in here https://github.com/dbt-athena/dbt-athena/blob/main/dbt/adapters/athena/query_headers.py#L19-L20 I'm pretty sure that it will work.
from dbt-athena.
jurgispods
commented on May 28, 2024
I'm not sure what query comments have to do with this. However, I can now reliably reproduce the error. The issue seems to be that the macro drop_relation defined in this repo is not getting picked up properly. In fact, in a clean repository after poetry install (empty target folder) and running dbt --debug test I get the following log output:
============================== 2022-12-14 16:46:28.472566 | 98722af2-3808-4de6-b36c-ca07314d38cb ==============================
�[0m16:46:28.472569 [info ] [MainThread]: Running with dbt=1.3.1
�[0m16:46:28.472794 [debug] [MainThread]: running dbt with arguments {'debug': True, 'write_json': True, 'use_colors': True, 'printer_width': 80, 'version_check': True, 'partial_parse': True, 'static_parser': True, 'profiles_dir': '/Users/jurgispods/projects/data-importer', 'send_anonymous_usage_stats': True, 'fail_fast': True, 'event_buffer_size': 100000, 'quiet': False, 'no_print': False, 'target': 'prelive', 'indirect_selection': 'eager', 'which': 'test', 'rpc_method': 'test'}
�[0m16:46:28.472940 [debug] [MainThread]: Tracking: tracking
�[0m16:46:28.485131 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10686b610>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10686b730>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10686b7f0>]}
�[0m16:46:28.503194 [info ] [MainThread]: Partial parse save file not found. Starting full parse.
�[0m16:46:28.503334 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'partial_parser', 'label': '98722af2-3808-4de6-b36c-ca07314d38cb', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1068b8e20>]}
�[0m16:46:28.530774 [debug] [MainThread]: Parsing macros/salted_hash.sql
�[0m16:46:28.531271 [debug] [MainThread]: Parsing macros/get_next_n_partitions.sql
�[0m16:46:28.534501 [debug] [MainThread]: Parsing macros/prepend_zeros.sql
�[0m16:46:28.534831 [debug] [MainThread]: Parsing macros/generate_schema_name.sql
�[0m16:46:28.535693 [debug] [MainThread]: Parsing macros/get_partition_count.sql
�[0m16:46:28.536112 [debug] [MainThread]: Parsing macros/materializations/snapshots/snapshot.sql
�[0m16:46:28.536504 [debug] [MainThread]: Parsing macros/materializations/models/incremental/column_helpers.sql
�[0m16:46:28.539752 [debug] [MainThread]: Parsing macros/materializations/models/incremental/merge.sql
�[0m16:46:28.542454 [debug] [MainThread]: Parsing macros/materializations/models/incremental/helpers.sql
�[0m16:46:28.549435 [debug] [MainThread]: Parsing macros/materializations/models/incremental/incremental.sql
�[0m16:46:28.556955 [debug] [MainThread]: Parsing macros/materializations/models/incremental/on_schema_change.sql
�[0m16:46:28.559877 [debug] [MainThread]: Parsing macros/materializations/models/table/table.sql
�[0m16:46:28.561545 [debug] [MainThread]: Parsing macros/materializations/models/table/create_table_as.sql
�[0m16:46:28.567031 [debug] [MainThread]: Parsing macros/materializations/models/view/view.sql
�[0m16:46:28.567651 [debug] [MainThread]: Parsing macros/materializations/models/view/create_or_replace_view.sql
�[0m16:46:28.569592 [debug] [MainThread]: Parsing macros/materializations/models/view/create_view_as.sql
�[0m16:46:28.569926 [debug] [MainThread]: Parsing macros/materializations/seeds/helpers.sql
�[0m16:46:28.572418 [debug] [MainThread]: Parsing macros/utils/ddl_data_type.sql
�[0m16:46:28.573782 [debug] [MainThread]: Parsing macros/utils/datatypes.sql
�[0m16:46:28.574272 [debug] [MainThread]: Parsing macros/utils/dateadd.sql
�[0m16:46:28.574763 [debug] [MainThread]: Parsing macros/utils/timestamps.sql
�[0m16:46:28.575155 [debug] [MainThread]: Parsing macros/utils/right.sql
�[0m16:46:28.575543 [debug] [MainThread]: Parsing macros/utils/datediff.sql
�[0m16:46:28.578727 [debug] [MainThread]: Parsing macros/utils/safe_cast.sql
�[0m16:46:28.579028 [debug] [MainThread]: Parsing macros/utils/hash.sql
�[0m16:46:28.579307 [debug] [MainThread]: Parsing macros/utils/any_value.sql
�[0m16:46:28.579564 [debug] [MainThread]: Parsing macros/utils/array_concat.sql
�[0m16:46:28.579850 [debug] [MainThread]: Parsing macros/utils/bool_or.sql
�[0m16:46:28.580101 [debug] [MainThread]: Parsing macros/utils/date_trunc.sql
�[0m16:46:28.580392 [debug] [MainThread]: Parsing macros/utils/array_construct.sql
�[0m16:46:28.580925 [debug] [MainThread]: Parsing macros/utils/array_append.sql
�[0m16:46:28.581209 [debug] [MainThread]: Parsing macros/adapters/relation.sql
�[0m16:46:28.582291 [debug] [MainThread]: Parsing macros/adapters/metadata.sql
�[0m16:46:28.587041 [debug] [MainThread]: Parsing macros/adapters/columns.sql
�[0m16:46:28.587890 [debug] [MainThread]: Parsing macros/materializations/hooks.sql
�[0m16:46:28.590045 [debug] [MainThread]: Parsing macros/materializations/configs.sql
�[0m16:46:28.591293 [debug] [MainThread]: Parsing macros/materializations/snapshots/snapshot_merge.sql
�[0m16:46:28.592280 [debug] [MainThread]: Parsing macros/materializations/snapshots/strategies.sql
�[0m16:46:28.600972 [debug] [MainThread]: Parsing macros/materializations/snapshots/helpers.sql
�[0m16:46:28.608297 [debug] [MainThread]: Parsing macros/materializations/snapshots/snapshot.sql
�[0m16:46:28.615326 [debug] [MainThread]: Parsing macros/materializations/tests/test.sql
�[0m16:46:28.617860 [debug] [MainThread]: Parsing macros/materializations/tests/helpers.sql
�[0m16:46:28.618889 [debug] [MainThread]: Parsing macros/materializations/tests/where_subquery.sql
�[0m16:46:28.619882 [debug] [MainThread]: Parsing macros/materializations/models/incremental/column_helpers.sql
�[0m16:46:28.624260 [debug] [MainThread]: Parsing macros/materializations/models/incremental/merge.sql
�[0m16:46:28.633425 [debug] [MainThread]: Parsing macros/materializations/models/incremental/is_incremental.sql
�[0m16:46:28.634222 [debug] [MainThread]: Parsing macros/materializations/models/incremental/strategies.sql
�[0m16:46:28.637587 [debug] [MainThread]: Parsing macros/materializations/models/incremental/incremental.sql
�[0m16:46:28.643228 [debug] [MainThread]: Parsing macros/materializations/models/incremental/on_schema_change.sql
�[0m16:46:28.652813 [debug] [MainThread]: Parsing macros/materializations/models/table/table.sql
�[0m16:46:28.655691 [debug] [MainThread]: Parsing macros/materializations/models/table/create_table_as.sql
�[0m16:46:28.657569 [debug] [MainThread]: Parsing macros/materializations/models/view/view.sql
�[0m16:46:28.660566 [debug] [MainThread]: Parsing macros/materializations/models/view/helpers.sql
�[0m16:46:28.661278 [debug] [MainThread]: Parsing macros/materializations/models/view/create_or_replace_view.sql
�[0m16:46:28.662975 [debug] [MainThread]: Parsing macros/materializations/models/view/create_view_as.sql
�[0m16:46:28.664108 [debug] [MainThread]: Parsing macros/materializations/seeds/seed.sql
�[0m16:46:28.667739 [debug] [MainThread]: Parsing macros/materializations/seeds/helpers.sql
�[0m16:46:28.678699 [debug] [MainThread]: Parsing macros/get_custom_name/get_custom_alias.sql
�[0m16:46:28.679531 [debug] [MainThread]: Parsing macros/get_custom_name/get_custom_schema.sql
�[0m16:46:28.680823 [debug] [MainThread]: Parsing macros/get_custom_name/get_custom_database.sql
�[0m16:46:28.681634 [debug] [MainThread]: Parsing macros/generic_test_sql/relationships.sql
�[0m16:46:28.682118 [debug] [MainThread]: Parsing macros/generic_test_sql/not_null.sql
�[0m16:46:28.682592 [debug] [MainThread]: Parsing macros/generic_test_sql/unique.sql
�[0m16:46:28.682982 [debug] [MainThread]: Parsing macros/generic_test_sql/accepted_values.sql
�[0m16:46:28.683697 [debug] [MainThread]: Parsing macros/etc/statement.sql
�[0m16:46:28.686530 [debug] [MainThread]: Parsing macros/etc/datetime.sql
�[0m16:46:28.691021 [debug] [MainThread]: Parsing macros/utils/except.sql
�[0m16:46:28.691498 [debug] [MainThread]: Parsing macros/utils/replace.sql
�[0m16:46:28.692179 [debug] [MainThread]: Parsing macros/utils/concat.sql
�[0m16:46:28.692711 [debug] [MainThread]: Parsing macros/utils/length.sql
�[0m16:46:28.693296 [debug] [MainThread]: Parsing macros/utils/dateadd.sql
�[0m16:46:28.693994 [debug] [MainThread]: Parsing macros/utils/intersect.sql
�[0m16:46:28.694442 [debug] [MainThread]: Parsing macros/utils/escape_single_quotes.sql
�[0m16:46:28.694984 [debug] [MainThread]: Parsing macros/utils/right.sql
�[0m16:46:28.695683 [debug] [MainThread]: Parsing macros/utils/listagg.sql
�[0m16:46:28.696941 [debug] [MainThread]: Parsing macros/utils/datediff.sql
�[0m16:46:28.697578 [debug] [MainThread]: Parsing macros/utils/safe_cast.sql
�[0m16:46:28.698130 [debug] [MainThread]: Parsing macros/utils/hash.sql
�[0m16:46:28.698689 [debug] [MainThread]: Parsing macros/utils/cast_bool_to_text.sql
�[0m16:46:28.699217 [debug] [MainThread]: Parsing macros/utils/any_value.sql
�[0m16:46:28.699694 [debug] [MainThread]: Parsing macros/utils/position.sql
�[0m16:46:28.700243 [debug] [MainThread]: Parsing macros/utils/literal.sql
�[0m16:46:28.700717 [debug] [MainThread]: Parsing macros/utils/data_types.sql
�[0m16:46:28.704005 [debug] [MainThread]: Parsing macros/utils/array_concat.sql
�[0m16:46:28.704544 [debug] [MainThread]: Parsing macros/utils/bool_or.sql
�[0m16:46:28.705022 [debug] [MainThread]: Parsing macros/utils/last_day.sql
�[0m16:46:28.705923 [debug] [MainThread]: Parsing macros/utils/split_part.sql
�[0m16:46:28.707062 [debug] [MainThread]: Parsing macros/utils/date_trunc.sql
�[0m16:46:28.707636 [debug] [MainThread]: Parsing macros/utils/array_construct.sql
�[0m16:46:28.708464 [debug] [MainThread]: Parsing macros/utils/array_append.sql
�[0m16:46:28.709102 [debug] [MainThread]: Parsing macros/adapters/schema.sql
�[0m16:46:28.710260 [debug] [MainThread]: Parsing macros/adapters/timestamps.sql
�[0m16:46:28.711987 [debug] [MainThread]: Parsing macros/adapters/indexes.sql
�[0m16:46:28.713462 [debug] [MainThread]: Parsing macros/adapters/relation.sql
�[0m16:46:28.721523 [debug] [MainThread]: Parsing macros/adapters/freshness.sql
�[0m16:46:28.722639 [debug] [MainThread]: Parsing macros/adapters/apply_grants.sql
�[0m16:46:28.729437 [debug] [MainThread]: Parsing macros/adapters/persist_docs.sql
�[0m16:46:28.731609 [debug] [MainThread]: Parsing macros/adapters/metadata.sql
�[0m16:46:28.735253 [debug] [MainThread]: Parsing macros/adapters/columns.sql
�[0m16:46:28.740264 [debug] [MainThread]: Parsing macros/python_model/python.sql
�[0m16:46:28.743537 [debug] [MainThread]: Parsing tests/generic/builtin.sql
�[0m16:46:28.745264 [debug] [MainThread]: Parsing macros/web/get_url_host.sql
�[0m16:46:28.746421 [debug] [MainThread]: Parsing macros/web/get_url_path.sql
�[0m16:46:28.747769 [debug] [MainThread]: Parsing macros/web/get_url_parameter.sql
�[0m16:46:28.748551 [debug] [MainThread]: Parsing macros/generic_tests/fewer_rows_than.sql
�[0m16:46:28.750950 [debug] [MainThread]: Parsing macros/generic_tests/equal_rowcount.sql
�[0m16:46:28.753120 [debug] [MainThread]: Parsing macros/generic_tests/relationships_where.sql
�[0m16:46:28.754581 [debug] [MainThread]: Parsing macros/generic_tests/recency.sql
�[0m16:46:28.756446 [debug] [MainThread]: Parsing macros/generic_tests/not_constant.sql
�[0m16:46:28.757591 [debug] [MainThread]: Parsing macros/generic_tests/accepted_range.sql
�[0m16:46:28.758898 [debug] [MainThread]: Parsing macros/generic_tests/not_accepted_values.sql
�[0m16:46:28.760401 [debug] [MainThread]: Parsing macros/generic_tests/at_least_one.sql
�[0m16:46:28.761557 [debug] [MainThread]: Parsing macros/generic_tests/unique_combination_of_columns.sql
�[0m16:46:28.763778 [debug] [MainThread]: Parsing macros/generic_tests/cardinality_equality.sql
�[0m16:46:28.764835 [debug] [MainThread]: Parsing macros/generic_tests/expression_is_true.sql
�[0m16:46:28.765843 [debug] [MainThread]: Parsing macros/generic_tests/not_null_proportion.sql
�[0m16:46:28.767514 [debug] [MainThread]: Parsing macros/generic_tests/sequential_values.sql
�[0m16:46:28.769560 [debug] [MainThread]: Parsing macros/generic_tests/equality.sql
�[0m16:46:28.771512 [debug] [MainThread]: Parsing macros/generic_tests/not_empty_string.sql
�[0m16:46:28.772531 [debug] [MainThread]: Parsing macros/generic_tests/mutually_exclusive_ranges.sql
�[0m16:46:28.778325 [debug] [MainThread]: Parsing macros/jinja_helpers/pretty_log_format.sql
�[0m16:46:28.778930 [debug] [MainThread]: Parsing macros/jinja_helpers/_is_relation.sql
�[0m16:46:28.779524 [debug] [MainThread]: Parsing macros/jinja_helpers/pretty_time.sql
�[0m16:46:28.780162 [debug] [MainThread]: Parsing macros/jinja_helpers/log_info.sql
�[0m16:46:28.780743 [debug] [MainThread]: Parsing macros/jinja_helpers/slugify.sql
�[0m16:46:28.781539 [debug] [MainThread]: Parsing macros/jinja_helpers/_is_ephemeral.sql
�[0m16:46:28.782667 [debug] [MainThread]: Parsing macros/sql/date_spine.sql
�[0m16:46:28.785013 [debug] [MainThread]: Parsing macros/sql/nullcheck_table.sql
�[0m16:46:28.785875 [debug] [MainThread]: Parsing macros/sql/get_relations_by_pattern.sql
�[0m16:46:28.787832 [debug] [MainThread]: Parsing macros/sql/generate_series.sql
�[0m16:46:28.790155 [debug] [MainThread]: Parsing macros/sql/get_relations_by_prefix.sql
�[0m16:46:28.792022 [debug] [MainThread]: Parsing macros/sql/get_tables_by_prefix_sql.sql
�[0m16:46:28.792905 [debug] [MainThread]: Parsing macros/sql/star.sql
�[0m16:46:28.795995 [debug] [MainThread]: Parsing macros/sql/unpivot.sql
�[0m16:46:28.799310 [debug] [MainThread]: Parsing macros/sql/safe_divide.sql
�[0m16:46:28.799919 [debug] [MainThread]: Parsing macros/sql/union.sql
�[0m16:46:28.807230 [debug] [MainThread]: Parsing macros/sql/groupby.sql
�[0m16:46:28.807959 [debug] [MainThread]: Parsing macros/sql/deduplicate.sql
�[0m16:46:28.809992 [debug] [MainThread]: Parsing macros/sql/surrogate_key.sql
�[0m16:46:28.810812 [debug] [MainThread]: Parsing macros/sql/safe_add.sql
�[0m16:46:28.811915 [debug] [MainThread]: Parsing macros/sql/nullcheck.sql
�[0m16:46:28.812757 [debug] [MainThread]: Parsing macros/sql/get_tables_by_pattern_sql.sql
�[0m16:46:28.816500 [debug] [MainThread]: Parsing macros/sql/get_column_values.sql
�[0m16:46:28.819888 [debug] [MainThread]: Parsing macros/sql/pivot.sql
�[0m16:46:28.822108 [debug] [MainThread]: Parsing macros/sql/get_filtered_columns_in_relation.sql
�[0m16:46:28.823589 [debug] [MainThread]: Parsing macros/sql/width_bucket.sql
�[0m16:46:28.826561 [debug] [MainThread]: Parsing macros/sql/get_query_results_as_dict.sql
�[0m16:46:28.827831 [debug] [MainThread]: Parsing macros/sql/generate_surrogate_key.sql
�[0m16:46:28.829213 [debug] [MainThread]: Parsing macros/sql/get_table_types_sql.sql
�[0m16:46:28.830055 [debug] [MainThread]: Parsing macros/sql/get_single_value.sql
�[0m16:46:28.831593 [debug] [MainThread]: Parsing macros/sql/haversine_distance.sql
�[0m16:46:28.834619 [debug] [MainThread]: Parsing macros/get_date_dimension.sql
�[0m16:46:28.848404 [debug] [MainThread]: Parsing macros/get_base_dates.sql
�[0m16:46:28.851060 [debug] [MainThread]: Parsing macros/_utils/date_spine.sql
�[0m16:46:28.853148 [debug] [MainThread]: Parsing macros/_utils/generate_series.sql
�[0m16:46:28.855405 [debug] [MainThread]: Parsing macros/fiscal_date/get_fiscal_year_dates.sql
�[0m16:46:28.857996 [debug] [MainThread]: Parsing macros/fiscal_date/get_fiscal_periods.sql
�[0m16:46:28.859090 [debug] [MainThread]: Parsing macros/calendar_date/tomorrow.sql
�[0m16:46:28.859467 [debug] [MainThread]: Parsing macros/calendar_date/next_week.sql
�[0m16:46:28.859793 [debug] [MainThread]: Parsing macros/calendar_date/next_month_name.sql
�[0m16:46:28.860171 [debug] [MainThread]: Parsing macros/calendar_date/next_month.sql
�[0m16:46:28.860477 [debug] [MainThread]: Parsing macros/calendar_date/day_name.sql
�[0m16:46:28.862219 [debug] [MainThread]: Parsing macros/calendar_date/to_unixtimestamp.sql
�[0m16:46:28.863093 [debug] [MainThread]: Parsing macros/calendar_date/n_days_away.sql
�[0m16:46:28.863468 [debug] [MainThread]: Parsing macros/calendar_date/week_start.sql
�[0m16:46:28.864823 [debug] [MainThread]: Parsing macros/calendar_date/iso_week_start.sql
�[0m16:46:28.866135 [debug] [MainThread]: Parsing macros/calendar_date/n_days_ago.sql
�[0m16:46:28.866762 [debug] [MainThread]: Parsing macros/calendar_date/last_week.sql
�[0m16:46:28.867094 [debug] [MainThread]: Parsing macros/calendar_date/now.sql
�[0m16:46:28.867425 [debug] [MainThread]: Parsing macros/calendar_date/periods_since.sql
�[0m16:46:28.867886 [debug] [MainThread]: Parsing macros/calendar_date/today.sql
�[0m16:46:28.868190 [debug] [MainThread]: Parsing macros/calendar_date/last_month.sql
�[0m16:46:28.868493 [debug] [MainThread]: Parsing macros/calendar_date/day_of_year.sql
�[0m16:46:28.869477 [debug] [MainThread]: Parsing macros/calendar_date/round_timestamp.sql
�[0m16:46:28.869838 [debug] [MainThread]: Parsing macros/calendar_date/from_unixtimestamp.sql
�[0m16:46:28.872717 [debug] [MainThread]: Parsing macros/calendar_date/n_months_ago.sql
�[0m16:46:28.873237 [debug] [MainThread]: Parsing macros/calendar_date/date_part.sql
�[0m16:46:28.873983 [debug] [MainThread]: Parsing macros/calendar_date/n_weeks_away.sql
�[0m16:46:28.874520 [debug] [MainThread]: Parsing macros/calendar_date/day_of_month.sql
�[0m16:46:28.875224 [debug] [MainThread]: Parsing macros/calendar_date/yesterday.sql
�[0m16:46:28.875596 [debug] [MainThread]: Parsing macros/calendar_date/day_of_week.sql
�[0m16:46:28.879805 [debug] [MainThread]: Parsing macros/calendar_date/iso_week_end.sql
�[0m16:46:28.880985 [debug] [MainThread]: Parsing macros/calendar_date/n_weeks_ago.sql
�[0m16:46:28.881507 [debug] [MainThread]: Parsing macros/calendar_date/month_name.sql
�[0m16:46:28.883013 [debug] [MainThread]: Parsing macros/calendar_date/last_month_name.sql
�[0m16:46:28.883417 [debug] [MainThread]: Parsing macros/calendar_date/week_of_year.sql
�[0m16:46:28.884393 [debug] [MainThread]: Parsing macros/calendar_date/convert_timezone.sql
�[0m16:46:28.886454 [debug] [MainThread]: Parsing macros/calendar_date/n_months_away.sql
�[0m16:46:28.886928 [debug] [MainThread]: Parsing macros/calendar_date/iso_week_of_year.sql
�[0m16:46:28.888224 [debug] [MainThread]: Parsing macros/calendar_date/week_end.sql
�[0m16:46:28.889368 [debug] [MainThread]: Parsing macros/calendar_date/next_month_number.sql
�[0m16:46:28.889723 [debug] [MainThread]: Parsing macros/calendar_date/last_month_number.sql
�[0m16:46:28.890077 [debug] [MainThread]: Parsing macros/utils/datatypes.sql
�[0m16:46:28.891495 [debug] [MainThread]: Parsing macros/utils/groupby.sql
�[0m16:46:28.892192 [debug] [MainThread]: Parsing macros/regex/regexp_instr.sql
�[0m16:46:28.894251 [debug] [MainThread]: Parsing macros/math/log_natural.sql
�[0m16:46:28.894976 [debug] [MainThread]: Parsing macros/math/rand.sql
�[0m16:46:28.895856 [debug] [MainThread]: Parsing macros/math/median.sql
�[0m16:46:28.896173 [debug] [MainThread]: Parsing macros/math/percentile_cont.sql
�[0m16:46:28.897147 [debug] [MainThread]: Parsing macros/schema_tests/_generalized/_ignore_row_if_expression.sql
�[0m16:46:28.898364 [debug] [MainThread]: Parsing macros/schema_tests/_generalized/expression_between.sql
�[0m16:46:28.900520 [debug] [MainThread]: Parsing macros/schema_tests/_generalized/expression_is_true.sql
�[0m16:46:28.902584 [debug] [MainThread]: Parsing macros/schema_tests/_generalized/equal_expression.sql
�[0m16:46:28.911146 [debug] [MainThread]: Parsing macros/schema_tests/_generalized/_truth_expression.sql
�[0m16:46:28.911639 [debug] [MainThread]: Parsing macros/schema_tests/string_matching/expect_column_values_to_match_like_pattern.sql
�[0m16:46:28.912330 [debug] [MainThread]: Parsing macros/schema_tests/string_matching/expect_column_values_to_match_like_pattern_list.sql
�[0m16:46:28.913524 [debug] [MainThread]: Parsing macros/schema_tests/string_matching/expect_column_values_to_match_regex.sql
�[0m16:46:28.914388 [debug] [MainThread]: Parsing macros/schema_tests/string_matching/expect_column_value_lengths_to_equal.sql
�[0m16:46:28.915089 [debug] [MainThread]: Parsing macros/schema_tests/string_matching/expect_column_value_lengths_to_be_between.sql
�[0m16:46:28.916050 [debug] [MainThread]: Parsing macros/schema_tests/string_matching/expect_column_values_to_not_match_regex.sql
�[0m16:46:28.917027 [debug] [MainThread]: Parsing macros/schema_tests/string_matching/expect_column_values_to_not_match_regex_list.sql
�[0m16:46:28.918319 [debug] [MainThread]: Parsing macros/schema_tests/string_matching/expect_column_values_to_match_regex_list.sql
�[0m16:46:28.919599 [debug] [MainThread]: Parsing macros/schema_tests/string_matching/expect_column_values_to_not_match_like_pattern_list.sql
�[0m16:46:28.920863 [debug] [MainThread]: Parsing macros/schema_tests/string_matching/_get_like_pattern_expression.sql
�[0m16:46:28.921285 [debug] [MainThread]: Parsing macros/schema_tests/string_matching/expect_column_values_to_not_match_like_pattern.sql
�[0m16:46:28.922007 [debug] [MainThread]: Parsing macros/schema_tests/table_shape/expect_row_values_to_have_recent_data.sql
�[0m16:46:28.924101 [debug] [MainThread]: Parsing macros/schema_tests/table_shape/expect_table_columns_to_contain_set.sql
�[0m16:46:28.925568 [debug] [MainThread]: Parsing macros/schema_tests/table_shape/expect_table_row_count_to_equal_other_table.sql
�[0m16:46:28.926106 [debug] [MainThread]: Parsing macros/schema_tests/table_shape/expect_table_columns_to_not_contain_set.sql
�[0m16:46:28.927414 [debug] [MainThread]: Parsing macros/schema_tests/table_shape/expect_grouped_row_values_to_have_recent_data.sql
�[0m16:46:28.931396 [debug] [MainThread]: Parsing macros/schema_tests/table_shape/expect_column_to_exist.sql
�[0m16:46:28.932771 [debug] [MainThread]: Parsing macros/schema_tests/table_shape/expect_table_row_count_to_equal.sql
�[0m16:46:28.934399 [debug] [MainThread]: Parsing macros/schema_tests/table_shape/expect_table_row_count_to_be_between.sql
�[0m16:46:28.935269 [debug] [MainThread]: Parsing macros/schema_tests/table_shape/expect_table_row_count_to_equal_other_table_times_factor.sql
�[0m16:46:28.935723 [debug] [MainThread]: Parsing macros/schema_tests/table_shape/expect_table_columns_to_match_set.sql
�[0m16:46:28.937109 [debug] [MainThread]: Parsing macros/schema_tests/table_shape/_get_column_list.sql
�[0m16:46:28.937625 [debug] [MainThread]: Parsing macros/schema_tests/table_shape/expect_table_columns_to_match_ordered_list.sql
�[0m16:46:28.939269 [debug] [MainThread]: Parsing macros/schema_tests/table_shape/_list_intersect.sql
�[0m16:46:28.939845 [debug] [MainThread]: Parsing macros/schema_tests/table_shape/expect_table_column_count_to_equal_other_table.sql
�[0m16:46:28.940496 [debug] [MainThread]: Parsing macros/schema_tests/table_shape/expect_table_column_count_to_equal.sql
�[0m16:46:28.941040 [debug] [MainThread]: Parsing macros/schema_tests/table_shape/expect_table_column_count_to_be_between.sql
�[0m16:46:28.942309 [debug] [MainThread]: Parsing macros/schema_tests/column_values_basic/expect_column_values_to_not_be_in_set.sql
�[0m16:46:28.943585 [debug] [MainThread]: Parsing macros/schema_tests/column_values_basic/expect_column_values_to_be_in_set.sql
�[0m16:46:28.944672 [debug] [MainThread]: Parsing macros/schema_tests/column_values_basic/expect_column_values_to_be_increasing.sql
�[0m16:46:28.946236 [debug] [MainThread]: Parsing macros/schema_tests/column_values_basic/expect_column_values_to_be_null.sql
�[0m16:46:28.946834 [debug] [MainThread]: Parsing macros/schema_tests/column_values_basic/expect_column_values_to_be_unique.sql
�[0m16:46:28.947212 [debug] [MainThread]: Parsing macros/schema_tests/column_values_basic/expect_column_values_to_be_between.sql
�[0m16:46:28.948136 [debug] [MainThread]: Parsing macros/schema_tests/column_values_basic/expect_column_values_to_be_decreasing.sql
�[0m16:46:28.949554 [debug] [MainThread]: Parsing macros/schema_tests/column_values_basic/expect_column_values_to_be_in_type_list.sql
�[0m16:46:28.950808 [debug] [MainThread]: Parsing macros/schema_tests/column_values_basic/expect_column_values_to_be_of_type.sql
�[0m16:46:28.951174 [debug] [MainThread]: Parsing macros/schema_tests/column_values_basic/expect_column_values_to_have_consistent_casing.sql
�[0m16:46:28.951733 [debug] [MainThread]: Parsing macros/schema_tests/column_values_basic/expect_column_values_to_not_be_null.sql
�[0m16:46:28.952370 [debug] [MainThread]: Parsing macros/schema_tests/aggregate_functions/expect_column_min_to_be_between.sql
�[0m16:46:28.953336 [debug] [MainThread]: Parsing macros/schema_tests/aggregate_functions/expect_column_unique_value_count_to_be_between.sql
�[0m16:46:28.954305 [debug] [MainThread]: Parsing macros/schema_tests/aggregate_functions/expect_column_quantile_values_to_be_between.sql
�[0m16:46:28.955330 [debug] [MainThread]: Parsing macros/schema_tests/aggregate_functions/expect_column_median_to_be_between.sql
�[0m16:46:28.956299 [debug] [MainThread]: Parsing macros/schema_tests/aggregate_functions/expect_column_proportion_of_unique_values_to_be_between.sql
�[0m16:46:28.957355 [debug] [MainThread]: Parsing macros/schema_tests/aggregate_functions/expect_column_distinct_values_to_equal_set.sql
�[0m16:46:28.958653 [debug] [MainThread]: Parsing macros/schema_tests/aggregate_functions/expect_column_most_common_value_to_be_in_set.sql
�[0m16:46:28.961751 [debug] [MainThread]: Parsing macros/schema_tests/aggregate_functions/expect_column_distinct_values_to_contain_set.sql
�[0m16:46:28.962885 [debug] [MainThread]: Parsing macros/schema_tests/aggregate_functions/expect_column_distinct_count_to_equal.sql
�[0m16:46:28.963692 [debug] [MainThread]: Parsing macros/schema_tests/aggregate_functions/expect_column_distinct_count_to_be_less_than.sql
�[0m16:46:28.964503 [debug] [MainThread]: Parsing macros/schema_tests/aggregate_functions/expect_column_sum_to_be_between.sql
�[0m16:46:28.965496 [debug] [MainThread]: Parsing macros/schema_tests/aggregate_functions/expect_column_stdev_to_be_between.sql
�[0m16:46:28.967497 [debug] [MainThread]: Parsing macros/schema_tests/aggregate_functions/expect_column_mean_to_be_between.sql
�[0m16:46:28.968524 [debug] [MainThread]: Parsing macros/schema_tests/aggregate_functions/expect_column_max_to_be_between.sql
�[0m16:46:28.969483 [debug] [MainThread]: Parsing macros/schema_tests/aggregate_functions/expect_column_distinct_count_to_be_greater_than.sql
�[0m16:46:28.970321 [debug] [MainThread]: Parsing macros/schema_tests/aggregate_functions/expect_column_distinct_values_to_be_in_set.sql
�[0m16:46:28.971383 [debug] [MainThread]: Parsing macros/schema_tests/aggregate_functions/expect_column_distinct_count_to_equal_other_table.sql
�[0m16:46:28.972317 [debug] [MainThread]: Parsing macros/schema_tests/distributional/expect_row_values_to_have_data_for_every_n_datepart.sql
�[0m16:46:28.976841 [debug] [MainThread]: Parsing macros/schema_tests/distributional/expect_column_values_to_be_within_n_moving_stdevs.sql
�[0m16:46:28.984386 [debug] [MainThread]: Parsing macros/schema_tests/distributional/expect_column_values_to_be_within_n_stdevs.sql
�[0m16:46:28.986041 [debug] [MainThread]: Parsing macros/schema_tests/multi-column/expect_select_column_values_to_be_unique_within_record.sql
�[0m16:46:28.988986 [debug] [MainThread]: Parsing macros/schema_tests/multi-column/expect_compound_columns_to_be_unique.sql
�[0m16:46:28.991274 [debug] [MainThread]: Parsing macros/schema_tests/multi-column/expect_multicolumn_sum_to_equal.sql
�[0m16:46:28.992460 [debug] [MainThread]: Parsing macros/schema_tests/multi-column/expect_column_pair_values_to_be_equal.sql
�[0m16:46:28.993259 [debug] [MainThread]: Parsing macros/schema_tests/multi-column/expect_column_pair_values_A_to_be_greater_than_B.sql
�[0m16:46:28.994110 [debug] [MainThread]: Parsing macros/schema_tests/multi-column/expect_column_pair_values_to_be_in_set.sql
�[0m16:46:29.292077 [debug] [MainThread]: 1603: static parser failed on host/bestaende_cleaned.sql
�[0m16:46:29.311976 [debug] [MainThread]: 1602: parser fallback to jinja rendering on host/bestaende_cleaned.sql
�[0m16:46:29.312974 [debug] [MainThread]: 1603: static parser failed on host/rueres.sql
�[0m16:46:29.318669 [debug] [MainThread]: 1602: parser fallback to jinja rendering on host/rueres.sql
�[0m16:46:29.319664 [debug] [MainThread]: 1603: static parser failed on host/fakturierbarer_bestand.sql
�[0m16:46:29.334096 [debug] [MainThread]: 1602: parser fallback to jinja rendering on host/fakturierbarer_bestand.sql
�[0m16:46:29.473460 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'load_project', 'label': '98722af2-3808-4de6-b36c-ca07314d38cb', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x106eaa740>]}
�[0m16:46:29.506899 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'resource_counts', 'label': '98722af2-3808-4de6-b36c-ca07314d38cb', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x106da0b20>]}
�[0m16:46:29.507097 [info ] [MainThread]: Found 3 models, 43 tests, 0 snapshots, 0 analyses, 634 macros, 0 operations, 0 seed files, 2 sources, 0 exposures, 0 metrics
�[0m16:46:29.507222 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'runnable_timing', 'label': '98722af2-3808-4de6-b36c-ca07314d38cb', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1017764d0>]}
�[0m16:46:29.509028 [info ] [MainThread]:
�[0m16:46:29.509367 [debug] [MainThread]: Acquiring new athena connection "master"
�[0m16:46:29.510949 [debug] [ThreadPool]: Acquiring new athena connection "list_awsdatacatalog"
�[0m16:46:29.515251 [debug] [ThreadPool]: Using athena connection "list_awsdatacatalog"
�[0m16:46:29.515369 [debug] [ThreadPool]: On list_awsdatacatalog:
select
distinct schema_name
from awsdatacatalog.INFORMATION_SCHEMA.schemata
[...]
Now I can modify the file [...]/site-packages/dbt/include/global_project/macros/adapters/relation.sql by adding a comment anywhere. The change will get picked up on the next run, note the line Partial parsing: updated file: dbt_athena://macros/adapters/relation.sql:
============================== 2022-12-14 16:47:54.856349 | 237a442b-d8ae-4c9f-8a24-c357ed241617 ==============================
�[0m16:47:54.856353 [info ] [MainThread]: Running with dbt=1.3.1
�[0m16:47:54.856584 [debug] [MainThread]: running dbt with arguments {'debug': True, 'write_json': True, 'use_colors': True, 'printer_width': 80, 'version_check': True, 'partial_parse': True, 'static_parser': True, 'profiles_dir': '/Users/jurgispods/projects/data-importer', 'send_anonymous_usage_stats': True, 'fail_fast': True, 'event_buffer_size': 100000, 'quiet': False, 'no_print': False, 'target': 'prelive', 'indirect_selection': 'eager', 'which': 'test', 'rpc_method': 'test'}
�[0m16:47:54.856734 [debug] [MainThread]: Tracking: tracking
�[0m16:47:54.866647 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x106c654b0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x103a23e20>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x106c652d0>]}
�[0m16:47:54.928801 [debug] [MainThread]: Partial parsing enabled: 0 files deleted, 0 files added, 1 files changed.
�[0m16:47:54.929264 [debug] [MainThread]: Partial parsing: updated file: dbt_athena://macros/adapters/relation.sql
�[0m16:47:54.929419 [debug] [MainThread]: Parsing macros/materializations/seeds/helpers.sql
�[0m16:47:54.932714 [debug] [MainThread]: Parsing macros/materializations/models/incremental/incremental.sql
�[0m16:47:54.940058 [debug] [MainThread]: Parsing macros/materializations/models/table/table.sql
�[0m16:47:54.941759 [debug] [MainThread]: Parsing macros/adapters/relation.sql
�[0m16:47:54.942879 [debug] [MainThread]: Parsing macros/materializations/seeds/helpers.sql
�[0m16:47:54.953379 [debug] [MainThread]: Parsing macros/materializations/seeds/seed.sql
�[0m16:47:54.987937 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'load_project', 'label': '237a442b-d8ae-4c9f-8a24-c357ed241617', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1071418d0>]}
�[0m16:47:54.997579 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'resource_counts', 'label': '237a442b-d8ae-4c9f-8a24-c357ed241617', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1070d0490>]}
�[0m16:47:54.997774 [info ] [MainThread]: Found 3 models, 43 tests, 0 snapshots, 0 analyses, 634 macros, 0 operations, 0 seed files, 2 sources, 0 exposures, 0 metrics
�[0m16:47:54.997904 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'runnable_timing', 'label': '237a442b-d8ae-4c9f-8a24-c357ed241617', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x103a23e20>]}
�[0m16:47:54.999853 [info ] [MainThread]:
�[0m16:47:55.000199 [debug] [MainThread]: Acquiring new athena connection "master"
�[0m16:47:55.001748 [debug] [ThreadPool]: Acquiring new athena connection "list_awsdatacatalog"
�[0m16:47:55.008898 [debug] [ThreadPool]: Using athena connection "list_awsdatacatalog"
�[0m16:47:55.009026 [debug] [ThreadPool]: On list_awsdatacatalog:
select
distinct schema_name
from awsdatacatalog.INFORMATION_SCHEMA.schemata
[...]
and the run will complete successfully,. So it seems to be that the initial parsing of macros is not working correctly, such that the Athena-specific macro is overwritten by the default one. Any idea what could be the reason for this?
from dbt-athena.
nicor88
commented on May 28, 2024
In the adapter, we don't ovewride the default drop macro that could be indeed the issue.
We can try to add a proper macro overwrite
athena__drop_relation that is totally missing.
I believe that as you spotted the issue is due to cascade (used in the default dbt core), that is not implemented in athena.
from dbt-athena.
jurgispods
commented on May 28, 2024
I had the same thought when reading about how dispatch works. When renaming drop_relation to athena__drop_relation, everything works as expected. Do you want me to cook up a PR for prefixing all overriden macros from dbt-core?
from dbt-athena.
jurgispods
commented on May 28, 2024
Seems like all other relevant macros are already prefixed, so drop_relation was the only one that needed to be renamed. Please have a look at my PR.
from dbt-athena.
Jrmyy
commented on May 28, 2024
I removed the athena__drop_relation and replaced it to drop_relation because config is not available in the adapter overidden macro. If this is possible to do with the athena__drop_relation, great, but with Iceberg, Hive table types it was not possible.
from dbt-athena.
nicor88
commented on May 28, 2024
@Jrmyy ah right. But we might need to keep both if we don't find a way to pass the config.
from dbt-athena.
jurgispods
commented on May 28, 2024
I think I found another way to force DBT to find the drop_relation macro from dbt-athena without prefixing it. It involves overriding the drop_relation method in AthenaAdapter and explicitly specifying the project name as an additional parameter to the execute_macro call. Will update my PR accordingly.
from dbt-athena.
jessedobbelaere
commented on May 28, 2024
I'm experiencing the same issue at the moment. I have a custom macro (based on redshift's insert_by_period) and it runs:
{{ adapter.drop_relation(tmp_relation) }}
But when I upgrade from the Tomme dbt-athena adapter to dbt-athena, I also run into this error:
19:41:22.500578 [debug] [Thread-1 (]: On model.search.events: -- /* {"app": "dbt", "dbt_version": "1.3.1", "profile_name": "athena", "target_name": "dev", "node_id": "model.search.events"} */
drop table if exists dev_jessedobbelaere_silver_search.events__dbt_incremental_period0_tmp cascade
Failed to execute query.
...
19:42:37.937626 [error] [MainThread]: Runtime Error
19:42:37.937917 [error] [MainThread]: An error occurred (InvalidRequestException) when calling the StartQueryExecution operation: line 2:103: mismatched input 'cascade'. Expecting: '.', <EOF>
Because the {{ adapter.drop_relation(...) }} command executes the default dbt macro and not the override from dbt-athena (which does not support cascade drops)
from dbt-athena.
jessedobbelaere
commented on May 28, 2024
I ran into this issue again 😕. Steps to reproduce: when I have some models deployed in prod as table. Then changing the materialization to view and redeploying, dbt will execute a drop table if exists dev_jessedobbelaere_silver.mytable cascade and fails.
Any idea how we can overcome the missing athena__drop_relation while still keeping iceberg support?
from dbt-athena.
jurgispods
commented on May 28, 2024
My current workaround is to put a drop_relation macro in my DBT project which takes precedence over everything else:
{% macro drop_relation(relation) -%}
{% if config.get("table_type") != "iceberg" %}
{%- do adapter.clean_up_table(relation.schema, relation.table) -%}
{% endif %}
{% call statement("drop_relation", auto_begin=False) -%}
drop {{ relation.type }} if exists {{ relation }}
{%- endcall %}
{% endmacro %}
But this is very unsatisfying. As discussed in the attached PR, there seems to be an issue in dbt-core, as sometimes the config context is missing when prefixing the macro with default__, which would let it take precedence reliably.
from dbt-athena.
nicor88
commented on May 28, 2024
@jurgispods could you have a look at this PR and test in your setup: #119 ??
from dbt-athena.
nicor88
commented on May 28, 2024
As a note here, @jessedobbelaere confirmed that his issue was solved.
from dbt-athena.
nicor88
commented on May 28, 2024
@jurgispods yup, I had that trailing slash issue too.
For this reason the path on the iceberg tables avoid to use trailing slash...After this issue is fixed, we should consider to add the trailing slack back, as it's a proper convention.
from dbt-athena.
Related Issues (20)
- Add issue templates
- seeds with columns starting with underscore fail HOT 2
- Unable to compile test models in Athena HOT 1
- Add parameter `detailed_table_type` for `list_relations_without_caching` function
- Support column level statistics HOT 4
- Add default_lf_inherited_tags
- When using `force_batch=true` with incremental models, it will fail if there is no data to write HOT 1
- [Communication] Install Slack on the repository to enable release subscription HOT 4
- Prune old table version for incremental models. HOT 1
- Athena partitions limit fix (#360) fails with partitions defined as non-Athena functions HOT 4
- Log the query id HOT 2
- Post hook when too many partitions failed HOT 3
- Allow dbt-athena to grant permissions via lakeformation named resource method
- `external_location` ignored when `table_type='hive'` HOT 5
- Support multi-engine views in Athena
- Fine-Grain Permissions make re-creating tables difficult. HOT 9
- (Potential) bug with incremental iceberg tables HOT 6
- [Feature] set external_location for iceberg table HOT 1
- [Feature] Introduce ha flag for table materialisation and iceberg tables to allow users to control final location
- [Bug] Compilation Error when create view with lf-tags
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dbt-athena.