Config resolution order

[acaruso7]

I am using Python programmatic config to generate a database & schemas which are exact clones of an existing db, but differ only in db name. This results in the creation of new schema tech roles containing the new db name prefix. I would like to assign these new schema roles to an existing business role defined in YAML config, but it seems that SnowDDL doesn't "know" about them during YAML resolution, because it resolves Python configs after YAML configs. Per the docs:

During SnowDDL execution YAML configs are resolved first. After that Python modules are resolved one-by-one in alphanumeric order.

What I really need here is the following resolution sequence:

1. Resolve all YAML configs NOT dependent on any objects generated by Python configs
2. Resolve Python configs
3. Resolve remaining YAML configs which ARE dependent on objects generated by Python configs

Is there are way to "force" a particular resolution order?

Reference Code

Python config for db / schema clone

"""Generates config for RAW_SNAPSHOT db, where all config is an exact clone of RAW db, except for db namespace"""
import copy

from snowddl import (
    SnowDDLConfig,
    DatabaseBlueprint,
    DatabaseIdent,
    SchemaBlueprint,
    SchemaIdent,
)


def handler(config: SnowDDLConfig):
    raw_db_name, raw_db_blueprint = list(
        config.get_blueprints_by_type_and_pattern(DatabaseBlueprint, "RAW").items()
    )[0]

    # use existing raw db blueprint as template and alter as needed
    raw_snapshot_db_blueprint = copy.deepcopy(raw_db_blueprint)
    raw_snapshot_db_blueprint.full_name = DatabaseIdent(
        config.env_prefix, "RAW_SNAPSHOT"
    )
    config.add_blueprint(raw_snapshot_db_blueprint)

    for (
        fully_qualified_raw_db_schema_name,
        raw_db_schema_blueprint,
    ) in config.get_blueprints_by_type_and_pattern(SchemaBlueprint, "RAW.*").items():
        # use existing raw db schema blueprint as template and alter as needed
        raw_snapshot_db_schema_blueprint = copy.deepcopy(raw_db_schema_blueprint)
        raw_snapshot_db_schema_blueprint.full_name = SchemaIdent(
            config.env_prefix,
            "RAW_SNAPSHOT",
            fully_qualified_raw_db_schema_name.split(".")[-1],
        )
        config.add_blueprint(raw_snapshot_db_schema_blueprint)

Attempted application of Python generated schema reader role to existing business role:

transformer:
  schema_read:
    - raw_snapshot.*

^^ This fails with error:

ValueError: No SCHEMAS matched wildcard grant with pattern [raw_snapshot.*]

[acaruso7]

My intuition here is that I must apply these Python-generated schema roles to existing YAML generated business roles in the Python module itself. However this feels like a bit of an antipattern / code maintainability challenge

[acaruso7]

Trying this approach, and I can't seem to find any of the SchemaRoleBlueprints in the config object. config.get_blueprints_by_type(SchemaRoleBlueprint) returns an empty dict, and upon inspecting the config object I can't find any SchemaRoleBlueprint objects, even though I know various schema roles exist

[littleK0i]

SchemaRoleBlueprint(s) are generated dynamically in resolver based on SchemaBlueprint(s):
https://github.com/littleK0i/SnowDDL/blob/master/snowddl/resolver/schema_role.py#L16-L24

This approach helps to avoid weird situations when some schemas are generated without schema roles.

I think you should not have a need to edit schema roles. Basic grant of existing schema roles to business roles should suffice.

[littleK0i]

Pattern matching for grants is just a little syntax sugar to prevent excessive repetition for configs with large number of schemas. It is only applied during parsing.

You may generally find and copy-paste any blueprint generation code from parser. For example, this is where you may see how to generate a grant for schema role: https://github.com/littleK0i/SnowDDL/blob/master/snowddl/parser/business_role.py#L125-L137

Try finding relevant BusinessRoleBlueprint(s) and adding a few grants directly to existing blueprint objects:

grant = Grant(
    privilege="USAGE",
    on=ObjectType.ROLE,
    name=build_role_ident(
        config.env_prefix,
        <database_ident>,
        <schema_ident>,
        "OWNER",
        config.SCHEMA_ROLE_SUFFIX,
    )
)

business_role_bp.grants.append(grant)

[acaruso7]

Got it, this works perfectly, thanks! I just wish I could maintain the role inheritance fully in YAML rather than partially in Python. I suppose I could build some abstractions into the handler to read from a YAML file so users can enter configs there and not have to reason about the Python logic. Open to any suggestions here

Thanks again for the assistance!

[littleK0i]

In theory, you may do some dynamic updates for YAML config.

Before running SnowDDL, you may execute a basic Python script to read and update YAML files in place.

It might be a bit more confusing if you get parsing errors, since errors will refer to updated file instead of original file. But it should work just fine otherwise.