Feature plans

[vtolstov]

thanks for such project, i'm fan of pillarstack (i'm slightly rewrite it for my needs and also fixup top.sls file to have equal directory structure (i have /projects /systems (debian, fedora and so), environment, nodes and inside it some dirs for roles like dns-server mail-server and so).
Main drawback that this can't work in default salt setup, most of the people that see my use-case says, that i'm need to use reclass.
What you can say about reclass? Does it possible to replicate with it something like pillarstack and salt-tower?
Also do you plan maintain this project and how long? (this is long term question, because if you drop it other may not pick up it...)

[jgraichen]

thanks for such project, i'm fan of pillarstack (i'm slightly rewrite it for my needs and also fixup top.sls file to have equal directory structure (i have /projects /systems (debian, fedora and so), environment, nodes and inside it some dirs for roles like dns-server mail-server and so).
Main drawback that this can't work in default salt setup, most of the people that see my use-case says, that i'm need to use reclass.
What you can say about reclass? Does it possible to replicate with it something like pillarstack and salt-tower?

I did not know of reclass before and never used it. I did replace pillarstack with salt-tower completely in a few projects. It works well for small and medium but complex setups.

Do the examples/ help with possible usages?

Also do you plan maintain this project and how long? (this is long term question, because if you drop it other may not pick up it...)

Salt tower plugs into the public salt APIs for external pillars and renderers. It has a good test suite that covers all features and some edge cases. Tests run using salt top level functions, therefore integration and loading via pip package is tested too. It does not use any external dependencies.

I do use salt tower for private projects as well as at work but it is not product. All I can do is releasing it as free source and provide best effort support and work on it when I have time.

[vtolstov]

thanks for answer, do you have some feature plans how to extend salt-tower?
For example provide minion_id like pillarstack?

[jgraichen]

minion_id is available. Salt tower uses salts own renderers, e.g. for jinja, therefore everything from salt.renderers.jinja and all helper and filter functions should be available.

{{ grains["..."] }} is available too.

[jgraichen]

You can take a look at the tests too, they show a lot examples of different things that can be used, e.g. minion_id:

def test_context_minion_id(env):
    env.setup({
        'tower.sls':
            '''
            base:
                - template.sls
            ''',
        'template.sls':
            '''
            minion_id: {{ minion_id }}
            '''
    })

    assert env.ext_pillar() == {'minion_id': 'test_master'}

[vtolstov]

{%- set hostname, domainname = minion_id.split('.',1) -%}

i'm use something like this

[vtolstov]

how to deal with absent files? for example i have

base:
  - nodes/z1_unistack_org/mn01.sls
  - nodes/z1_unistack_org/mn01/*.sls

but for example files in nodes/z1_unistack_org/mn01/*.sls not exists

[vtolstov]

or it simply skipped?

[vtolstov]

{%- set archs = {'amd64': 'x86_64', 'i386': 'x86_32'} -%}
{%- set system = '-'.join([grains['os'], grains['osrelease'].split('.', 1)[0], archs.get(grains['osarch'], grains['osarch'])]).lower() %}
{%- set hostname, domainname = minion_id.split('.',1) -%}
{%- set project = '_'.join(domainname.split('.')[-2:]) -%}
{%- set environment = domainname.replace('.','_') -%}
base:
  - nodes/{{ environment }}/{{ hostname }}.sls
  - nodes/{{ environment }}/{{ hostname }}/*.sls
{%- set roles = pillar.get('nodes:'+minion_id+':roles', []) -%}
{%- set includes = pillar.get('nodes:'+minion_id+':includes', []) -%}
{%- for path in includes %}
  - {{ path }}
{%- endfor %}
{%- for role in roles %}
  - systems/defaults/{{ role }}/*.sls
  - systems/{{ system }}/{{ role }}/*.sls
  - projects/defaults/{{ role }}/*.sls
  - projects/{{ project }}/{{ role }}/*.sls
  - environments/defaults/{{ role }}/*.sls
  - environments/{{ environment }}/{{ role }}/*.sls
{%- endfor %}

can you say why {%- for role in roles %} not works if pillar nodes/{{ environment }}/{{ hostname }}.sls contains roles array in needed path ? so i need something like render all that possible and after that rerender using included variables

[vtolstov]

in pillarstack i'm run ext_pillar twice, as i understand in case of salt-tower this is not needed

[jgraichen]

When using jinja, this is always needed. With jinja, the file is compiled, before any file can be loaded.

When {%- for role in roles %} is executed no file from before has been loaded, because this all happens in the first rendering. That is because jinja is a text template engine. Therefore all the statements, including e.g. pillar.get are run before pillarstack or tower can even run. First the whole file is run through jinja, which is a text-to-text transformation. The output from jinja is parsed as YAML. Only then it can be interpreted by pillarstack or tower. This will always be the case and cannot be done differently. You would need an actual programming language for that.

But with tower the problem can be solved with very few changes:

1. Tower supports includes in all data files, e.g. you can move the minion includes into the nodes/{{ environment }}/{{ hostname }}/*.sls files. Just put an include: key inside:

   # nodes/env/minion.sls
   include:
   - ../../path/to/another/file.sls
   

   With this you do not need to do {%- for path in includes %} at all here.

2. You can use the same mechanism for including the roles, just in another file. Putting this in another file allows tower to first load the minion files, e.g.:

   # tower.sls
   base:
     - nodes/{{ environment }}/{{ hostname }}.sls
     - nodes/{{ environment }}/{{ hostname }}/*.sls
     - roles.sls
   

   # roles.sls
   include:
   {%- for role in pillar.get('nodes:'+minion_id+':roles') %}
     - systems/defaults/{{ role }}/*.sls
     - systems/{{ system }}/{{ role }}/*.sls
     - projects/defaults/{{ role }}/*.sls
     - projects/{{ project }}/{{ role }}/*.sls
     - environments/defaults/{{ role }}/*.sls
     - environments/{{ environment }}/{{ role }}/*.sls
   {%- endfor %}
   

These kind of issues cannot be solved easily because jinja and YAML are no programming language.

[jgraichen]

If you have a limited list of roles you can do it this way too:

# tower.sls
base:
  - common/*.sls
  - minion/{{ minion_id }}/*.sls

  - 'I@roles:loadbalancer':
    - roles/loadbalancer.sls

  - 'I@roles:webserver':
    - projects/{site.project}/webserver.sls
    - roles/webserver.sls

This uses salt compound matchers. Because they are interpreted by tower and not jinja, they are run after the common and minion specific files have been loaded. This way you also can use pillar values to expand the path, e.g. {site.project}. But there are no loops here anymore.

[vtolstov]

good suggestions. as i understand i need to assign in each file variables?
so i need either repeat needed stuff or put in common.sls and include it in all needed places?

[jgraichen]

You can include them into the pillar in a common file e.g.:

# tower.sls
base:
  - common/*.sls

# common/00-variables.sls
{%- set archs = {'amd64': 'x86_64', 'i386': 'x86_32'} -%}
{%- set system = '-'.join([grains['os'], grains['osrelease'].split('.', 1)[0], archs.get(grains['osarch'], grains['osarch'])]).lower() %}
{%- set hostname, domainname = minion_id.split('.',1) -%}
{%- set project = '_'.join(domainname.split('.')[-2:]) -%}
{%- set environment = domainname.replace('.','_') -%}

vars:
  system: {{ system }}
  hostname: {{ hostname }}
  domainname: {{ domainname }}
  project: {{ project }}
  environment: {{ environment }}

You could not use all these as pillar keys, e.g. in tower.sls using late binding, or similar to pillarstack in any data file:

# tower.sls

base:
  - common/*.sls
  # Note the single curly brackets, this is *not* a jinja variable
  - nodes/{vars.environment}/{vars.hostname}.sls
  - nodes/{vars.environment}/{vars.hostname}/*.sls
  - roles.sls

# roles.sls
include:
{%- for role in pillar.get('nodes:'+minion_id+':roles') %}
  - systems/defaults/{{ role }}/*.sls
  - systems/{{ pillar.get('vars:system') }}/{{ role }}/*.sls
  - projects/defaults/{{ role }}/*.sls
  - projects/{{ pillar.get('vars:projects') }}/{{ role }}/*.sls
  - environments/defaults/{{ role }}/*.sls
  - environments/{{ pillar.get('vars:environment') }}/{{ role }}/*.sls
{%- endfor %}

[jgraichen]

I hope this helped you. I have setup an additional group and mailling list that might be better suited for longer discussions.