Separate table type from table format about dbt-athena HOT 4 CLOSED

I like the proposed naming, just some more suggestions:

• table_type: can be external (default, the name external is what is used in hive, we are creating external tables), iceberg (that is managed)
  then the regarding format, this means when we have table_type=iceberg we cannot support the all format, but parquet/orc I guess
  I was also thinking about that when I saw your PR about iceberg.

from dbt-athena.

For the table_type, it is just to stick to the table_type in CTAS definition. So you mean then we will need to map external to hive right ?

from dbt-athena.

How does that affect things is table_type is not set?
Currently we are using format=JSON as a slight hack but easy way to export hourly data (with incremental) for writing data to a partner s3 bucket.
i.e we have our normal DBT models, then we have some partner product models which really would be export views of the last hour/24hours data. This then gets written in JSON via DBT and Athena to their specific bucket.

This has two nice side benefits (if not a bit hacky). Support can use Athena to quickly check what data was sent as it is a registered table and querying the actual data. The engineers only need to worry about DBT for the code which is the same place as the rest of the pipeline.

from dbt-athena.

So you mean then we will need to map external to hive right ?

@Jrmyy So pretty much we have external tables (HIVE) and not external table (ICEBERG). I was not aware that Athena had already the concept of table_type , said so, I agree with you, let's stick with that and with athena values, therefore please ignore my comment about having external as table_type, as using HIVE as table type means already that is an external table.

How does that affect things is table_type is not set?

@pixie79 we will have a default behavior, HIVE, that pretty much leave the actual setup unchanged, that means that you still need to set a format explicitly in your models if you want to use JSON.

from dbt-athena.