Regression in customizing sources in pydantic V2 about pydantic-settings HOT 6 OPEN

Thanks @ziima for reporting this.
I haven't checked your provided example in detail but it seems you are going to load configs from yaml file.

If so, pydantic-settings 2.2 added yaml settings source. please check the docs.

from pydantic-settings.

The problem with YamlConfigSettingsSource here is that you can't read from file with arbitrary name provided by user (let's say by a command line option). There's no way to pass that name into the source.

It was possible to set a private attribute on the BaseSettings. In V1, this instance was passed to custom config source and it could read the value. But now, in V2, the settings instance is not passed down to config source so there's no easy way to pass that value (apart from maybe creating some global variable).

If nothing else, documentation is clearly outdated. Also, I was quite surprised by the fact that if you provide multiple config files, it reads them all and merges the result (instead of just using the first one found). This should also be pointed out in the docs.

from pydantic-settings.

The problem with YamlConfigSettingsSource here is that you can't read from file with arbitrary name provided by user (let's say by a command line option). There's no way to pass that name into the source.

You can implement your custom YamlConfigSettingsSource and override YamlConfigSettingsSource.__init__ and change the value of yaml_file_path there.

If nothing else, documentation is clearly outdated. Also, I was quite surprised by the fact that if you provide multiple config files, it reads them all and merges the result (instead of just using the first one found). This should also be pointed out in the docs.

if you mean dotenv file it is already mentioned in Dotenv support section of doc

If you need to load multiple dotenv files, you can pass multiple file paths as a tuple or list. The files will be loaded in order, with each file overriding the previous one.

from pydantic-settings.

The problem with YamlConfigSettingsSource here is that you can't read from file with arbitrary name provided by user (let's say by a command line option). There's no way to pass that name into the source.

You can implement your custom YamlConfigSettingsSource and override YamlConfigSettingsSource.__init__ and change the value of yaml_file_path there.

Yes, but I need to instantiate this custom subclass of YamlConfigSettingsSource in classmethod settings_customise_sources. How do I get it there? The problem is not in the custom settings source, the problem is how to instantiate it with value that is not known until the user runs the code.

Again, in V1, I could use a PrivateAttr on BaseSettings subclass to pass this information down to settings source (because it received settings instance as an argument). In V2, I don't see any way except for some global antipatterns.

If nothing else, documentation is clearly outdated. Also, I was quite surprised by the fact that if you provide multiple config files, it reads them all and merges the result (instead of just using the first one found). This should also be pointed out in the docs.

if you mean dotenv file it is already mentioned in Dotenv support section of doc

I'm not talking about dotenv files. I'm refering to https://docs.pydantic.dev/latest/concepts/pydantic_settings/#customise-settings-sources

settings_customise_sources takes four callables as arguments and returns
any number of callables as a tuple. In turn these callables are called to build
the inputs to the fields of the settings class.

--> Each callable should take an instance of the settings class as its sole argument and return a dict. <--

And also https://docs.pydantic.dev/latest/concepts/pydantic_settings/#other-settings-source

You can also provide multiple files by providing a list of path:

toml_file = ['config.default.toml', 'config.custom.toml']

This doesn't mention anything about the way multiple files are handled.

from pydantic-settings.

Yeah, you don't have access to the settings model instance in source classes and We can't change it in V2 but it will be an important thing to have in V3.

Regarding multiple files for yaml and toml, it is like dotenv file. files loaded from the list and each file's values override the previous one

from pydantic-settings.

+1 On this issue. Being able to parameterize the file source for yaml/toml/etc configurations feels like it should be a basic feature of these settings, and it's frustrating to have to work around it.

from pydantic-settings.