env_prefix applies to fields with alias when defined in dotenv file about pydantic-settings HOT 18 CLOSED

Here is the PR to improve the docs #225

I've discussed with the team about changing the behavior of dotenv source to accept extra values in V3 and we decided to keep it like this and don't change it.

from pydantic-settings.

Thanks @gsakkis for reporting this 🙏

My fix for #219 probably will fix this issue as well.
I try to add a test case for this issue as well

from pydantic-settings.

@hramezani thanks! Please add a test case or two where the aliased field is nested settings. I'm seeing discrepancies between env vars and dotenv for this too but it's probably the same cause so I didn't open yet another issue.

from pydantic-settings.

I've created #221 to fix it. it would be great to take a look or test it if you want

Please add a test case or two where the aliased field is nested settings. I'm seeing discrepancies between env vars and dotenv for this too but it's probably the same cause so I didn't open yet another issue.

Please test those cases as well or provide an example then I will add them to the PR

from pydantic-settings.

Thanks for the quick turnaround! As I suspected, unfortunately it doesn't work with nested settings.
MCVE:

from pydantic import Field
from pydantic_settings import BaseSettings, SettingsConfigDict


class NestedSettings(BaseSettings):
    a: str = "a"
    b: str = "b"


class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_prefix="xxx__", env_nested_delimiter="__", env_file="tmp.env")

    foo: str = ""
    bar_alias: str = Field("", alias="xxx__bar")
    nested_alias: NestedSettings = Field(default_factory=NestedSettings, alias="xxx__nested")


print(Settings().model_dump())

$ cat tmp.env
xxx__bar=0
xxx__nested__a=1
xxx__nested__b=2

$ python testsettings.py
Traceback (most recent call last):
...
pydantic_core._pydantic_core.ValidationError: 2 validation errors for Settings
xxx__nested__a
  Extra inputs are not permitted [type=extra_forbidden, input_value='1', input_type=str]
    For further information visit https://errors.pydantic.dev/2.5/v/extra_forbidden
xxx__nested__b
  Extra inputs are not permitted [type=extra_forbidden, input_value='2', input_type=str]
    For further information visit https://errors.pydantic.dev/2.5/v/extra_forbidden

from pydantic-settings.

Thanks @gsakkis

I've updated my PR to consider this case as well.
This is still not complete and for example don't raise a validation error if we have xxx__nested__c(extra value) in dotenv file.
But it needs to loop over all the nested models and I think it is not good from performance pov.

from pydantic-settings.

Thanks @hramezani

This is still not complete and for example don't raise a validation error if we have xxx__nested__c(extra value) in dotenv file.

I'm curious why the dotenv source can't just reuse the logic from regular environment variables; the latter seems to be working correctly for every case I've thrown to it.

But it needs to loop over all the nested models and I think it is not good from performance pov.

I'd expect that performance would be the least of a settings library concerns; settings are typically loaded just once at the beginning of an application. Correctness and development experience are way more important IMO.

For the past day or so I've been struggling to replace a single top-level settings model with its nested children models:

class GlobalSettings(BaseSettings):
    model_config = SettingsConfigDict(env_nested_delimiter="__", env_file=".env")

    section1: Section1Settings = Section1Settings()
    section2: Section2Settings = Section2Settings()
    ...

I'd expect this to be a simple refactoring: just copy the model_config to every child settings and add env_prefix="section1__" for Section1Settings, env_prefix="section2__" for Section2Settings, etc. Sadly I've been hitting one wall after another trying to reverse-engineer the different behavior between env vars and dotenv, fields with and without alias, etc.

from pydantic-settings.

I'm curious why the dotenv source can't just reuse the logic from regular environment variables; the latter seems to be working correctly for every case I've thrown to it.

The case that I mentioned not working on env source as well. pydantic-settings env source loop over model fields and try to collect value from envs. it doesn't care about extra envs(that is reasonable because there are a lot of unrelated envs)

The process is the same in dotenv settings because it inherits from env settings source. but we have an extra step in dotenv for collecting extra values as well

I'd expect that performance would be the least of a settings library concerns; settings are typically loaded just once at the beginning of an application. Correctness and development experience are way more important IMO.

I would suggest creating a new issue for the problem with the nested models.

from pydantic-settings.

The case that I mentioned not working on env source as well.

Have you updated the PR since this comment? Unless you refer to a different case, at least the
don't raise a validation error if we have xxx__nested__c(extra value)
case works correctly for me (raises validation error) for both the env and dotenv source with the latest PR.

from pydantic-settings.

I've just updated the PR one time. if you fetch my latest change you will see it is not raising errors.

This is the code:

from pydantic import Field, BaseModel
from pydantic_settings import BaseSettings, SettingsConfigDict


class NestedSettings(BaseModel):
    a: str = "a"
    b: str = "b"


class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_prefix="xxx__", env_nested_delimiter="__", env_file=".env")

    foo: str = ""
    bar_alias: str = Field("", alias="xxx__bar")
    nested_alias: NestedSettings = Field(default_factory=NestedSettings, alias="xxx__nested")


print(Settings().model_dump())

here is the dotnev file content:

xxx__bar=0
xxx__nested__a=1
xxx__nested__b=2
xxx__nested__c=3

from pydantic-settings.

class NestedSettings(BaseModel):

Ah you extend BaseModel instead of BaseSettings, that's why you don't get an error (extra="ignore" instead of "forbid").

Here's a summary as of this commit:

Since I have settings for different models in the same .env file, yyy__other should not raise validation error. Now I can pass extra="ignore" but this also ignores xxx__baz, which I do want to raise as an error.

Basically I want to say "perform strict validation on any variable that starts with env_prefix and ignore everything else, regardless of whether it comes from an env variable or dotenv". Here's the desired table:

Happy to open a new issue If #221 will not make this possible.

from pydantic-settings.

To elaborate on the previous comment, I can already get the desired behavior if I add an extra top-level GlobalSettings model; in fact that's my starting state and I'm trying to refactor it so that it keeps the same behavior but without defining GlobalSettings:

class NestedSettings(BaseSettings):
    a: str = "a"
    b: str = "b"


class XXXSettings(BaseSettings):
    foo: str = ""
    bar_alias: str = Field("", alias="bar")
    nested_alias: NestedSettings = Field(default_factory=NestedSettings, alias="nested")


class YYYSettings(BaseSettings):
    other: str = ""


class GlobalSettings(BaseSettings):
    model_config = SettingsConfigDict(env_nested_delimiter="__", env_file=".env")

    xxx: XXXSettings = XXXSettings()
    yyy: YYYSettings = YYYSettings()


print(GlobalSettings().model_dump())

from pydantic-settings.

Ah you extend BaseModel instead of BaseSettings, that's why you don't get an error (extra="ignore" instead of "forbid").

This is the right way for nested models. as already suggested in docs.

Since I have settings for different models in the same .env file, yyy__other should not raise validation error. Now I can pass extra="ignore" but this also ignores xxx__baz, which I do want to raise as an error.

I think it should raise a validation error for yyy__other. it is like passing an extra input to a model.

Basically I want to say "perform strict validation on any variable that starts with env_prefix and ignore everything else, regardless of whether it comes from an env variable or dotenv". Here's the desired table

pydantic-settings ignores every extra env in env source but in dotenv it will consider them if you already config extra=forbid(default config)

from pydantic-settings.

This is the right way for nested models. as already suggested in docs.

Are there any other docs than this? I hadn't noticed that most (though not all, e.g. RedisSettings) nested models extend BaseModel rather than BaseSettings but I still don't see any explicit suggestion (let alone justification) in the text.

I think it should raise a validation error for yyy__other. it is like passing an extra input to a model.

Does this imply a 1-1 relationship between dotenv files and Settings models? That would be an odd limitation. It's quite common to have a single .env file instead of many for configuring all/most env vars related to an app, basically the equivalent of a shell script that does a bunch of export FOO=BAR. Or perhaps the suggestion is to have a single Settings model per app and everything else should be nested models under it, as in my GlobalSettings example?

from pydantic-settings.

With the following change to #221, my use case would be addressed:

-            if not env_value:
+            if not env_value or not env_name.startswith(self.env_prefix):
                 continue

from pydantic-settings.

Are there any other docs than this? I hadn't noticed that most (though not all, e.g. RedisSettings) nested models extend BaseModel rather than BaseSettings but I still don't see any explicit suggestion (let alone justification) in the text.

There is a plus sing close to class DeepSubModel(BaseModel): in the first example of this section. by clicking on the plus sing you will find
Sub model has to inherit from pydantic.BaseModel, Otherwise pydantic-settings will initialize sub model, collects values for sub model fields separately, and you may get unexpected results.

But probably we need to move it out from there to make it more obvious.

Does this imply a 1-1 relationship between dotenv files and Settings models? That would be an odd limitation.

Yes, right now there is a 1-1 relation between dotenv file and settings model. you still can use one settings file for multiple models, but in this case you need to set extra='ignore' to let pydantic-settings ignores extra values.

It's quite common to have a single .env file instead of many for configuring all/most env vars related to an app, basically the equivalent of a shell script that does a bunch of export FOO=BAR. Or perhaps the suggestion is to have a single Settings model per app and everything else should be nested models under it, as in my GlobalSettings example?

right now we can change it in V2 because it would be a breaking change. I can discuss it with the team and if they are agree we can implement it for V3.

from pydantic-settings.

With the following change to #221, my use case would be addressed:

-            if not env_value:
+            if not env_value or not env_name.startswith(self.env_prefix):
                 continue

Then ValidationError won't be raised if we have an extra key in dotenv file that does not start with env_prefix. in particular this test fails

from pydantic-settings.

But probably we need to move it out from there to make it more obvious.

Indeed, I've gone through the docs multiple times and never noticed the plus signs 🙂.

Yes, right now there is a 1-1 relation between dotenv file and settings model. you still can use one settings file for multiple models, but in this case you need to set extra='ignore' to let pydantic-settings ignores extra values.

Right, but this also ignores extra values that start with env_prefix, which is not ideal. In any case this is also something worth mentioning in the docs.

right now we can change it in V2 because it would be a breaking change. I can discuss it with the team and if they are agree we can implement it for V3.

I see, that would be a nice change. For now I'll keep the GlobalSettings pattern without setting env_prefix. Thanks for your help!

from pydantic-settings.