GitFS backed tower pillar

[jgraichen]

Investigate if and how salt-tower can support gitfs backends as a data source.

Check salts gitfs classes and modules, amendlik/gitstack-pillar might be interesting too.

[raddessi]

FWIW I got around this by naming my gitfs source that the pillar data resides in with a constant name like:

gitfs_remotes:
  - git@github.com:<username>/my-salt-config-repo.git:
    - name: my-salt-config-repo-name

Then I just set the source to point to that directory via:

ext_pillar:
  - tower: /var/cache/salt/master/git_pillar/my-salt-config-repo-name/relative/path/to/tower.sls

I'll spend some time trying to get this working natively when I have free time 🤣 I hope soon

[jgraichen]

I am usually checking out repositories on the master with the salt minion.

State:

/srv/salt/:
  file.directory:
    - makedirs: True

{% for name, conf in salt['pillar.get']('salt:repositories', {}).items() %}
/srv/salt/{{ name }}:
  git.latest:
    - name: {{ conf['url'] }}
    - rev: {{ conf['rev'] }}
    - target: /srv/salt/{{ name }}
    - force_fetch: True
    - force_reset: True
    - require:
        - file: /srv/salt/
    - watch_in:
        - cmd: salt/repositories/sync_all
{% endfor %}

salt/repositories/sync_all:
  cmd.wait:
    - name: salt-run --log-level warning saltutil.sync_all

Pillar example:

salt:
  repositories:
    states:
      url: ssh://.../states.git
      rev: main
    pillar:
      url: ssh://.../pillars.git
      rev: main
    reactors:
      url: ssh://.../reactors.git
      rev: main
    # ...

  master:
    file_roots:
      base:
        - /srv/salt/states

    pillar_roots:
      base:
        - /srv/salt/pillar

    gitfs_remotes:
      - https://github.com/jgraichen/salt-tower.git:
          - base: v1.7.0
          
    ext_pillar:
      - tower: /srv/salt/pillar/tower.sls
    # ...

A small daemon is running there, receiving webhooks when commits are pushed, running salt-call state.apply salt/master/repositories (the state file above) on the salt master. In some cases the webhook daemon is only creating/updating a file in /run which is monitored by the salt minion via a beacon due to privilege separation. This results in usually less than one second until changes are live.

---

I do assume that the biggest problem with "native" git support will be calling the salt renderers and have e.g. JINJA import working correctly.

[raddessi]

I initially tried maintaining local copies of the git repo on the master but I like having my development master refresh its file cache every minute or so for easy testing and I didn't want to have the minion on it always locked running a state.apply, even if you drop to 5min that recurring lock gets somewhat annoying. I'm always on the fence about it however.

I really like the webhook concept coupled with the beacon/reactor state.. I may have to swap over to that. Much less load and even faster update times. Yeah.. this is a great idea, thank you. Even if/when tower gets native gifts backing it wouldn't come close to the update speed of that setup 🚀

When I have time to look in to this I'll check out how gitstack handles it natively.. I only tested with that system for a short time but I think it rendered and handled imports in the expected way at the time.