[Feature] Improve startup time of typing `dbt` about dbt-core HOT 9 OPEN

I am wondering if some of the imports from that file are causing the delay and if we could move them inside specific functions/classes

import functools
from copy import copy
from dataclasses import dataclass
from typing import Callable, List, Optional, Union

import click
from click.exceptions import (
    Exit as ClickExit,
    BadOptionUsage,
    NoSuchOption,
    UsageError,
)

from dbt.cli import requires, params as p
from dbt.cli.exceptions import (
    DbtInternalException,
    DbtUsageException,
)
from dbt.contracts.graph.manifest import Manifest
from dbt.artifacts.schemas.catalog import CatalogArtifact
from dbt.artifacts.schemas.run import RunExecutionResult
from dbt_common.events.base_types import EventMsg

from dbt-core.

Here is a screenshot of "tuna" gotten with python -X importtime core/dbt/cli/main.py 2> tuna.log followed by tuna tuna.log. It shows the performance of the different imports.

Most of those imports shouldn't be required when running a simple dbt command but I don't know the effort behind not loading those in that case.

from dbt-core.

I just tried to strip quite a bit of code to find if there is a piece to focus on, but even after removing a lot of imports (making dbt work only to show its commands and args), it still takes more than 1 sec.

Instead of moving imports it might be better to have a "click only" flow where we don't call any code from core.dbt and just provide the commands/subcommands/arguments straight away.

from dbt-core.

@b-per Your timing could not be more perfect. I think I can get you a lot of the remaining time back. Check out dbt-labs/dbt-common#98. This should go out with the next release of dbt-common, which I have been told is scheduled for tomorrow, March 26th.

from dbt-core.

Great news!!

And if 0.7s is already much better than 1.5s, I still can't stop thinking that it feels a bit long for something that is just about showing a list of commands and parameters.

from dbt-core.

Agreed. If your changes combined with mine don't get us down to "almost instant" then I can work with you to get us the rest of the way. I'm confident we can do it.

from dbt-core.

My changes are not OK to be merged because they are breaking the normal dbt flow but your change plus this heavyweight removal of imports make dbt run in ~0.45 secs. So, a slight decrease from 0.7 but nothing major either.

from dbt-core.

Makes sense to be able to add sub-command and parameter completion like proposed in dbt-labs/dbt-completion.bash#21

@b-per Two questions for you:

1. It sounds like the poor responsiveness you observed is a known issue with Click. Did you already try this?
2. If the above doesn't help, would this be useful, by any chance?

from dbt-core.