Hi @JerPk and thanks a lot for reporting!
IMO you're right it's a bug. The behaviour with both secrets and env_prefix was not tested so I guess that's why we never saw it.
I chose the 2nd solution on the 3 you suggested because for me the 1st doesn't make sense (it shouldn't use env_prefix when we talk about secrets) and the 3rd doesn't really make sense when you already have folders!
Feedback more than welcome :)
Cheers

from pydantic-settings.

Hi @PrettyWood, I didn't expect such a fast response and implementation.
I completely agree with the solution. (although I will need to change some things in my code when I am updating to this future version.)
Thanks!

from pydantic-settings.

Similar issue exists, if "env" is set for a field.

class Settings(BaseSettings):
    auth_key: str
    api_key: str = Field(..., env="my_api_key")

    redis_dsn: RedisDsn = "redis://user:pass@localhost:6379/1"
    pg_dsn: PostgresDsn = "postgres://postgres:password@localhost:5432/postgres"

    # to override domains:
    # export my_prefix_domains='["foo.com", "bar.com"]'
    domains: Set[str] = set()

    # to override more_settings:
    # export my_prefix_more_settings='{"foo": "x", "apple": 1}'
    more_settings: SubModel = SubModel()

    class Config:
        # env_prefix = "my_prefix_"  # defaults to no prefix, i.e. ""
        fields = {
            "auth_key": {"env": "my_auth_key",},
            "redis_dsn": {"env": ["service_redis_dsn", "redis_url"]},
        }
        secrets_dir = "/var/run"

My secret had to be renamed to my_auth_key to succeed.

from pydantic-settings.

Proposal

Keep the solution as is now - thus use the environment variable name in all situations - not only for trying to read values from env var (and from dotenv file), but also from secrets.

Action needed: update the docs to clarify this rule.

Reasoning

Keep things simple. With unique naming rule for named configuration sources it is much simpler to document the applicaiton - simply give single set of names and add, that they can be provided via env vars, dotenv file or via secrets in a directory.

Another advantage: all existing deployment will work.

from pydantic-settings.

To quote @JerPk we have two solutions

1. Make it clear in the documentation that env_prefix is also used for secrets
2. Do not use the env_prefix for the secrets

I thought option 2 made more sense but reading @vlcinsky opinion, I understand option 1 can be desired.
I will also add option 3 just to be sure

3. Try without env_prefix and then with it (or the other way)

This would hence break nothing and allow both behaviours but could have side effects and is definitely less clear...

I add "feedback wanted" label and hope others will share their point of view :)

from pydantic-settings.

Thanks @PrettyWood for reaction.

Here is my feedback:

Following the KISS I would invite not to add any extra option (such as "for secrets use env_prefix yes/no") unless it is really needed.

$ python -c "import this"|grep -E "Simple|Special cases|There should be one|hard to explain"
Simple is better than complex.
Special cases aren't special enough to break the rules.
There should be one-- and preferably only one --obvious way to do it.
If the implementation is hard to explain, it's a bad idea.

from pydantic-settings.

But, if pydantic/pydantic#2190 get merged, we won't be able to transparently prefix secrets? My stack has many services, and I would like to store secrets as /run/secrets/my_project_api_key, but I really don't want to add my_project_ to all field names. Maybe we should have a secrets_prefix in Config??

from pydantic-settings.

Proposal

Keep the solution as is now - thus use the environment variable name in all situations - not only for trying to read values from env var (and from dotenv file), but also from secrets.

Action needed: update the docs to clarify this rule.

Agree. But if we go this way, then aliases passed via Field(env=...) must honor env_prefix too. Because now they don't.
After that we will get a simple consistent rule: "env_prefix affects every setting source by default. Don't use it or create your own settings source if you want different behavior."

from pydantic-settings.

The workaround to ignore the env_prefix for secrets is adding this method to your settings class:

@classmethod
def settings_customise_sources(
    cls,
    settings_cls: Type[BaseSettings],
    init_settings: PydanticBaseSettingsSource,
    env_settings: PydanticBaseSettingsSource,
    dotenv_settings: PydanticBaseSettingsSource,
    file_secret_settings: PydanticBaseSettingsSource,
) -> Tuple[PydanticBaseSettingsSource, ...]:
    return init_settings, env_settings, dotenv_settings, SecretsSettingsSource(settings_cls, env_prefix=None)

from pydantic-settings.