Issue when parsing dbt models about dbt2looker HOT 12 OPEN

@lewisosborne I was running into the same issue, I bypassed it by deleting that ephemeral tag at the top and then re-running dbt docs generate and then the subsequent dbt2looker command. That gets me passed the initial error highlighted here.

I am running into a different error afterwards where I am seeing not supported for conversion from redshift to looker for all of my data types but maybe we can tackle that one separately.

Let me know if my initial suggestion gets you past the issue with the ephemeral model.

from dbt2looker.

So yesterday I was modding the code and was able to get past the - is the model file empty? error.

Since the parser uses the manifest.json file, a node has its materialization in the config object:

"config": {
    "enabled": true,
    "alias": null,
    "schema": "staging",
    "database": null,
    "tags": [],
    "meta": {},
    "materialized": "ephemeral",
    "persist_docs": {},
    "quoting": {},
    "column_types": {},
    "full_refresh": null,
    "on_schema_change": "ignore",
    "bind": false,
    "post-hook": [],
    "pre-hook": []
},

If the DbtNode changes to:

class DbtNode(BaseModel):
    unique_id: str
    resource_type: str
    config: Dict[str, Any]

then the parse_models() method could be modified to:

def parse_models(raw_manifest: dict, tag=None) -> List[models.DbtModel]:
    manifest = models.DbtManifest(**raw_manifest)
    all_models: List[models.DbtModel] = [
        node
        for node in manifest.nodes.values()
        if node.resource_type == 'model' and node.config['materialized'] != 'ephemeral'
    ]

    # Empty model files have many missing parameters
    for model in all_models:
        if not hasattr(model, 'name'):
            logging.error('Cannot parse model with id: "%s" - is the model file empty?', model.unique_id)
            raise SystemExit('Failed')

    if tag is None:
        return all_models
    return [model for model in all_models if tags_match(tag, model)]

which should ensure the list of models only contains DbtModel instances. This might be overkill as a solution and maybe even not the intended effect but it solve those issues for me.
Also make sure to dbt build any models that are resulting in that error before you run dbt docs generate.

from dbt2looker.

Suggestion for parse_models(), based on @boludo00 's code:

def parse_models(raw_manifest: dict, tag=None) -> List[models.DbtModel]:
    manifest = models.DbtManifest(**raw_manifest)
    all_models: List[models.DbtModel] = [
        node
        for node in manifest.nodes.values()
        if node.resource_type == 'model' and node.config['materialized'] != 'ephemeral'
    ]

    if tag is not None:
        all_models = [model for model in all_models if tags_match(tag, model)]

    # Empty model files have many missing parameters
    for model in all_models:
        if not hasattr(model, 'name'):
            logging.error('Cannot parse model with id: "%s" - is the model file empty?', model.unique_id)
            raise SystemExit('Failed')

    return all_models

from dbt2looker.

Hey I am getting the same issue. Maybe it has to do with the model being ephemeral, since that's what I am experiencing as well.

from dbt2looker.

Perhaps @owlas or @jaypeedevlin may know if this is intentional? Sorry to tag directly, but the issue is getting stale and I would like to be able to generate lkml from my dbt project. Thanks in advance 🙏

from dbt2looker.

Ephemeral models don’t actually exist, they get imported into downstream models as CTEs.

Because of this they can’t be queried directly by dbt2looker (nor looker itself, for that matter). My guess is that dbt2looker needs some special handling of ephemeral models that doesn’t currently exist.

from dbt2looker.

Hey @jaypeedevlin - thanks for the swift response.

I'm not looking to have the ephemeral model generated to lkml. I will only make lkml files out of the estuary "final" dbt models. But it appears that the very existence of an ephemeral model in my dbt project is acting as a blocker for dbt2looker to parse the models - even if that ephemeral model is not tagged with the tag used in the "dbt2looker run".

I will have a try now by tagging only those models that I want to expose as lkml.

from dbt2looker.

Nope, unfortunately that did not have positive affect.

I tagged a single dbt model that I would like to generate lkml for (in model config, tags=["generate_lkml"])
I then ran dbt docs generate
I then ran dbt2looker --tag generate_lkml
Error:
12:15:56 ERROR Cannot parse model with id: "model.smallpdf.brz_exchange_rates" - is the model file empty?

Sigh 😓

from dbt2looker.

Hey, is there a definitive solution for this? Would the solution proposed by @boludo00 be viable to be included in a release?

from dbt2looker.

I also suggest the tag filtering to be done before the loop on all_models

from dbt2looker.

Is it a general rule currently that dbt2Looker does not work when you have ephemeral models?

I also get the error message Cannot parse model with id for every ephemeral model I have in my repo, even if not involved in the models I want to LookMLize

from dbt2looker.

Hi folks, I installed the tool locally after running into this issue with ephemeral models. I was able to get the solution developed by commenters above to work for me. As you can see, I have opened a PR to incorporate this into dbt2looker.

I don't think there's a test suite to run this against, is there? I can tell you that dbt2looker worked as expected for me after I made this change.

from dbt2looker.