Materialized tables creation fails on EMR about dbt-spark HOT 5 CLOSED

Agreed @aaronsteers @Dandandan. I'm going to close this issue, given that those two features will ship in the next release.

from dbt-spark.

@rhousewright You can create a database in advance with
CREATE DATABASE whatever LOCATION 's3://[bucket]/[path]' and point dbt with the database/schema option in the profile settings to this database in Hive so it automatically knows the location. The only problem with EMR, you will need an external hive metastore for persist this database.

from dbt-spark.

Thanks for the report @rhousewright and for some good insight @tromika.

We've designed this plugin to require that the target database is created manually and is initialized with a location. This is analogous to how dbt works on other types of databases - it doesn't create a target database for you, instead operating at the schema-level. It's unclear how this should work on Spark, given that database == schema!

Set a 'root' location in your dbt_project.yml, and have dbt format model names into it
I like this idea! I imagine the location would be specified in the profiles.yml file, given that it will vary between environments, eg:

• dev: location: s3://my-bucket/dbt-drew/models/
• prod: location: s3://my-bucket/prod/models/

Set a specific table location at the table level via config
I think this is a good idea too! Do you think there's any issue with creating every object with the same prefix? I imagined that list calls on S3 would be slow if there were very many models rendered into the same s3 path. Instead, we might be able to generate a prefix by default, like:

s3://my-bucket/env-name/project-name/path/to/model

You'd be able to override part (or all?) of this path in config.

I haven't done any testing, so I actually don't know if that's desirable. What do you guys think?

from dbt-spark.

Just checking in on older issues. Should this still remain open? Seems like it might be resolved by #43 (ability to configure location_root) and/or #40 (support for creating schemas).

from dbt-spark.

For me, an older version was not working, dbt spark master + a location already provided in the glue data catalog (or creating it from dbt with schema creation) does work.
Adding location_root support makes it much more flexible where to store individual tables in EMR.

from dbt-spark.