Add "raw" keyword-only parameter to Loader.get_settings() #37
Conversation
|
Further explanation of the use of the "raw" keyword in my pyramid-cookiecutter-starter fork may be warranted. If a yaml config file is chosen for Pyramid, and sqlalchemy is the db backend, a ini config file must still be generated for alembic. Alembic only takes an ini config file. So, when the regression tests run with a yaml config file the test fixture copies the yaml config programmaticaly and calls the alembic API to give alembic a config to use. The convenience is that the test code need not ever worry about the format of the supplied config. It just always copies whatever config it gets as input and gives it to alembic. The test code could be re-engineered to special-case on the config file format in-use, loading alembic's ini config specially and using 2 config files when Pyramid does not take an ini config format. But that seems clunky and opens the possibility that the 2 config files contain different configurations. This would cause confusion. Best is to be able to cleanly copy configs, without double-interpolation/etc. |
|
I'm trying to follow the things you're saying about this but it's against my original design of plaster to expose these types of details so having trouble wrapping my head around these cases you're describing with the cookiecutter and alembic. If alembic needs an ini and you move to a different plaster backend that is not ini then it seems to me you just need to define a separate alembic.ini. Is that wrong? Is the proposal to do something else to convert the yaml into an alembic.Config object yourself and not have alembic load it from a file? Seems like that is what you are doing in your conftest.py at least. It's just kind of a happy little coincidence right now that alembic is compatible with pastedeploy's ini config. |
|
On Wed, 17 Apr 2024 13:02:30 -0700 Michael Merickel ***@***.***> wrote:
I'm trying to follow the things you're saying about this but it's
against my original design of plaster to expose these types of
details so having trouble wrapping my head around these cases you're
describing with the cookiecutter and alembic.
Maybe what I'm doing is "just wrong" and does not fit with your
design.
If alembic needs an ini and you move to a different plaster backend
that is not yaml then it seems to me you just need to define a
separate alembic.ini. Is that wrong?
Not really. There are some disadvantages but nothing that can't
be lived with. The existing cookiecutter code reads only
a single config file and I tried to continue in that vein without
using duplicative config files in different formats. (Most of the
duplication is in the logging config. Alembic does not require much
itself.)
I am already generating separate alembic.ini files (development,
production, etc.) in the cookiecutter so I'd just need to use the
testing one in the testing code.
Is the proposal to do something
else to convert the yaml into an alembic.Config object yourself and
not have alembic load it from a file? Seems like that is what you are
doing in your conftest.py at least.
Yes, that is the proposal. That configurations be
programmaticly copyable, which means being able to turn interpolation
and such off and on. Otherwise, every copy interpolates (etc.),
escaping becomes impossible or a matter of knowing the
internals of how many copies are made, etc.
However, this may only make real sense when copying between
mechanisms that understand similar markup, do similar
things with defaults, etc. The cookiecutter
fork punts on this and works mostly because the alembic config
files are simple. In the general case there's no way of knowing
what's going to need post-processing (beyond the initial parse
of the underlying data structure) and it is unlikely that the
"raw" format is going to be universally interpret-able.
But this does not mean that "raw" is not useful. My original
code copied from yaml to ini just fine. The problem was
that copying from ini to ini "double interpreted".
(So, I leapt forward and started coding. Huzzah! :-)
Programmatic copying does seem to fit in with mixing
config files and declarative in-code configuration.
But maybe that's only true on the surface.
Given your reticence, I'm starting to think that I should
do the direct and simple thing and use two config files.
There's some duplicate configuration and a little
more code with an explicit special case. Nothing really
wonky. And you can keep plaster simple, at least
until there's a more compelling use-case for copying configs.
Anyway, I'm happy to take your guidance here. (I'm sure I
need it, having worked in something of a vacuum.) Whatever decision
you make will let me move forward with the cookiecutter patch.
Thanks for the help.
Regards,
Karl ***@***.***>
Free Software: "You don't pay back, you pay forward."
-- Robert A. Heinlein
|
|
On Wed, 17 Apr 2024 20:13:49 -0500
"Karl O. Pinc" ***@***.***> wrote:
On Wed, 17 Apr 2024 13:02:30 -0700
Michael Merickel ***@***.***> wrote:
> I'm trying to follow the things you're saying about this but it's
> against my original design of plaster to expose these types of
> details
Maybe what I'm doing is "just wrong" and does not fit with your
design.
How does this plan sound? I'll make another fork of the cookecutter
starter that does not use "raw" or otherwise require changes to
plaster et-al. You can look at the two of them, diff them, etc.,
and decide whether any changes to plaster are warranted.
The cookiecutter starter fork without "raw" should
run and you'll be able to also see all the yaml and ini
config files.
Always nice to have something concrete to look at.
If I don't hear anything back from you I'll proceed with
this plan and let you know when there's something to
look at.
Regards,
Karl ***@***.***>
Free Software: "You don't pay back, you pay forward."
-- Robert A. Heinlein
|
|
Irrespective of pyramid-cookiecutter-starter, there is another
consideration. Without a "raw" get_settings() parameter
plaster cannot read entire configuration files.
One case is the "stock" pyramid ini config files generated by
the cookiecutter.
How relevant this is I can't say. It's unclear how useful
it is to be able to read un-interpolated, undefaulted,
config file sections. It *might* sometimes be useful, I guess.
This code (using the stock plaster and plaster_pastedeploy) fails:
```
import plaster
loader = plaster.get_loader('development.ini')
config = {}
for section in loader.get_sections():
config[section] = loader.get_settings(section)
print(config)
```
The traceback ends with:
```
configparser.InterpolationMissingOptionError: Error in file development.ini: Bad value substitution: option 'format' in section 'formatter_generic' contains an interpolation key 'asctime' which is not a valid option name. Raw value: '%(asctime)s %(levelname)-5.5s [%(name)s:%(lineno)s][%(threadName)s] %(message)s'
```
This code (using the patched plaster and plaster_pastedeploy) works:
```
import plaster
from configparser import InterpolationMissingOptionError
loader = plaster.get_loader('development.ini')
config = {}
raw_config = {}
for section in loader.get_sections():
try:
config[section] = loader.get_settings(section)
except InterpolationMissingOptionError:
raw_config[section] = loader.get_settings(section, raw=True)
print(config)
print()
print(raw_config)
```
I don't want to advocate for anything. It is a little odd
that a config file reader can't always read entire configs.
But it could be that that's a feature.
Regards,
Karl ***@***.***>
Free Software: "You don't pay back, you pay forward."
-- Robert A. Heinlein
|
Done. There's a pyramid-cookiecutter-starter PR #137. |
On reflection, I like the cookiecutter better without the "raw" parameter. It's simpler. |
Hi,
This makes it possible to copy configurations and avoid multiple interpolations, or interpolation errors.
In a pyramid-cookiecutter-starter patch I pull out configuration from a yaml file and load it into an configparser.Config() configuration, or simply pull configuration from an ini file and load it into alembic's configuration. (No reason to write different code for yaml and ini config file sources).:
https://github.com/kpinc/pyramid-cookiecutter-starter/blob/fad9192058e950b17c3e2f0232729e69bdc4984b/%7B%7Bcookiecutter.repo_name%7D%7D/tests/sqlalchemy_conftest.py#L37
https://github.com/kpinc/pyramid-cookiecutter-starter/blob/fad9192058e950b17c3e2f0232729e69bdc4984b/%7B%7Bcookiecutter.repo_name%7D%7D/tests/sqlalchemy_conftest.py#L48
It seems like an important feature, even if pyramid-cookiecutter-starter, enhanced to support yaml config files, might somehow not need to use it.