Allow for storing/using metadata about hosts

[bitprophet]

2018 update: this is an old, old ticket and it is primed to actually be implemented now that 2.0 is available to build features on top of.

The tl;dr is that users need ways to store rich data about their target hosts (and, usually, groups or "roles"), first, and then need a way of addressing that information when doing API or CLI level things, second.

Fabric 2 is built on Invoke which has a powerful configuration system, which is then exposed to tasks via a 'context' object. It seems likely that we will build this feature on those, something like (but not necessarily limited to):

• Standardize on some relatively generic config-style format for representing hosts; basically Connection's parameters (user, hostname, port, connect_kwargs, timeout, etc etc) and probably with a bit more on top
  • or at least the opportunity for users to put arbitrary data on top and have that show up in the objects exposed to the user at runtime
• Update Context so it has a lightweight link to some other class or classes which turn the 'raw' config into usable API objects, based on some query or lookup
  • Open question is whether these show up as actual Connections or if there's an intermediate representational class like Host

For example (again: just an off the cuff example!) perhaps we'll set it up so host data gets its own config-style file that lives alongside the regular config files - say, $PROJECT/hosts.(yml|json|py|etc):

web1:
  host: web1
  # Implicit local user, as with Connection
web2:
  host: web2
  user: admin2
  port: 2223
db:
  # Implicit dict-key-is-the-host-value, i.e. implicit "host: db"
  user: dbadmin

Then perhaps there's something like this (using pure Invoke style tasks for now, though certainly this would want the ability to use -H or decorators to select target hosts to 'wrap' the task, as in v1):

@task
def deploy_webs(c):
  # Assuming auto-creation of Connection objects...
  for cxn in [c.find_host('web1'), c.find_host('web2')]:
    cxn.run("hostname")

There are a whole lot of different ways we could slice and dice this, and a lot of directions that it could be extended in; the emphasis should be on giving users as much power and control as reasonably possible and then getting out of their way. Ideally all we'll do is standardize on some very basic way of shoveling data into Connection objects, and add support to the core CLI exec framework to arrive at something resembling Fabric 1 re: selecting execution targets.

All while exposing these mechanisms publicly so advanced users can take matters into their own hands - again I expect anybody beyond the most basic use cases to be highly likely to fall back on a "regular Invoke style tasks + making the needful API calls within those task bodies" approach.

---

Original description

Right now a "host" is solely limited to user/hostname/port. Would be nice, even just for user fabfiles, to store additional information such as operating system, per-host settings like env.shell, and so forth.

Note that this may (or may not) be a good time to reconsider changing the default value of shell to /bin/sh.

---

Originally submitted by Jeff Forcier (bitprophet) on 2009-07-20 at 05:02pm EDT

Relations

• Duplicated by /bin/bash is not always available – per host shell configuration? #43: /bin/bash is not always available – per host shell configuration?
• Related to In some situations, pressing Enter does *not* reuse the previous password #97: In some situations, pressing Enter does not reuse the previous password
• Related to env.port not honored if host string lacks port specification #138: env.port not honored if host string lacks port specification
• Related to Make use of ssh_config where possible #3: Make use of ssh_config where possible
• Related to Use decorator to define tasks #76: Use decorator to define tasks

[bitprophet]

Silas Sewell (silas) posted:

---

Regarding the default shell, even if you don't change to /bin/sh, it might be good to use /usr/bin/env bash to fix situations where the user installed bash.

Example patch: http://github.com/silas/fabric/commit/6f7d33c1a3180fbba21c89447bb9a32b84e839ba

P.S. I posted here because "Support #43" was marked as a duplicate.

---

on 2009-11-09 at 11:13pm EST

[bitprophet]

Erich Heine (sophacles) posted:

---

I came up with a way of doing this host metadata that works pretty well for the simple cases. The relevant code bits are here http://paste.pocoo.org/show/173964/ however there are some ideas that I didn't have time to implement yet -- namely that this type of interface to host metadata might better be defined as a 'protocol'. By this I mean, have a defined interface on the one end, so that various backends may be implemented, e.g. an ldap client and a redis client, and so in. This would allow people to integrate with existing tools better (e.g. buildbot).

---

on 2010-02-04 at 03:33pm EST

[bitprophet]

This is strongly related to some stuff I was experimenting with in #563, where having a real Host object would help a lot with host_string BS, and also pave the way for this kind of per-host settings overrides.

That has been punted to 2.x so this is too.

[mrmachine]

We currently store additional metadata about the remote host in env.server, that we pull from an inventory of server configs saved on disk as JSON. But this only works when we are running tasks on a single server, or when every task that is expected to be run with multiple servers contains code to update env.server based on the value of env.host_string.

One thing that would make this easier for us is if fabric provided a hook for when env.host_string is changed. Then we could just have one piece of code that hooks into that, and updates env.server with the additional context whenever env.host_string is changed.

Could this be implemented easily, without having to go all out and refactor host_string into a full blown Host object?

[grantjenks]

Copying relevant comments from #1748, @peteruhnak commented:

"""
Hi! Is it possible to specify hosts either in a config file or the fabfile itself?

I know I can provide them via CLI (-H), but if the fabfile is designed to communicate with a specific server, then it just forces the user to do extra stuff for no good reason.

The "best" solution I could figure out was to create the connection by hand, e.g.

@task
def my_ls(c):
conn = Connection('myhost')
conn.run('ls')
but that seems quite dirty as I need to (1) duplicate it everywhere and (2) it makes the context argument pointless.
"""

And I followed up with:

"""
Agreed with all above. I really miss the old env.hosts setting. It was beautifully simple.

-1 on needing another config file like invoke.yaml or whatnot. I would much prefer to monkey-patch or some kind of import hook that let me state the hosts.

It also took me a while to figure out that ctx.local is not present if no hosts are specified :(
"""

I understand Invoke provides rich config support. I would hope that included configuring from Python's fab files.

[kaptajnen]

In fabric1 I added the ability to load hosts lists with --set list=somelistfile.txt. In the fabfile I would then parse the file and modify env.hosts and env.passwords with settings from this file. This allowed me to do e.g fab task --set list=datacenter1.txt. I then had dozens of general purpose tasks I could run against predefined lists, in some cases a list per datacenter or list per logical function. I realize this may be a bit of a hack but it has suited me well for many years in my professional career.

I haven't found a way to do this in fab2 since the hosts are loaded inside the executor. I would very much like to do something similar in fab2. However it seems the paradigm in fab2 requires me to specify the hosts in the tasks. This makes it impossible to have my general purpose tasks and use it on a wide variety of servers (without using fab2 -H).

Having some kind of advanced host configuration file and being able to specify it on the command line I think would be a great addition to fabric2.

[benzkji]

as @grantjenks mentioned, there is advanced config support in invoke: http://docs.pyinvoke.org/en/1.1/concepts/configuration.html - so we would just need to build a yaml file, and there we go. this would/could be extended (by hand, everyone on its own for now), to support something like the env concept, as in fabric one. If one doesnt like yaml, this can be completely done in python code, as I understand.

[bitprophet]

@benzkji Absolutely true; users can, right now, put arbitrary data in their config files (py, yml, json, eventually others) and pull that data out inside their tasks to construct Connections and Groups and whatnot. This is intentional; we don't ever want there to be only one way to handle that sort of setup, which is why the objects are all public and such.

See the very top of the ticket, too; I edited the description with a modern outlook on what this feature might look like in v2, in terms of some basic common "for un-opinionated users" helpers. I'm actually tinkering with that in a private (client-of-fabric, not fork) repo these days to see what patterns emerge.

[thelfter]

Hello!

After a lot of reading I still don't understand how this really works. For example If I have a hosts.yml configuration file like this:

hosts.yml:

server1:
  host: serverip
  user: username

How should I use this to create a Connection? I had to rename the hosts.yml to fabric.yml to get access these data, through the context variable, e.g:

@task
def do(ctx):
    ctx['server1']

And it will give back a DataProxy, that I can't use for create connection, or I just didn't find in the documentation

My another problem: how is it possible to specify these hosts declared in the hosts.yml file with -H toggle? It only works if I create an alias in the ~/.ssh/config file which is not so great at all.

Off-topic: prompt was removed from the api? I didn't find a corresponding method.

[SharkFourSix]

Hello!

After a lot of reading I still don't understand how this really works. For example If I have a hosts.yml configuration file like this:

hosts.yml:

server1:
  host: serverip
  user: username

How should I use this to create a Connection? I had to rename the hosts.yml to fabric.yml to get access these data, through the context variable, e.g:

@task
def do(ctx):
    ctx['server1']

And it will give back a DataProxy, that I can't use for create connection, or I just didn't find in the documentation

My another problem: how is it possible to specify these hosts declared in the hosts.yml file with -H toggle? It only works if I create an alias in the ~/.ssh/config file which is not so great at all.

Off-topic: prompt was removed from the api? I didn't find a corresponding method.

I have the same question. I haven't found a single example or tutorial on how to create and use configuration files. The API documentation is very lacking in that regard.

[benzkji]

fabric 2 is great, but from my end user perspective, it's quite more of an effort to get it up and running (compared to fabric 1), for example as a simple deployment tool for Django webapps (in my case).

Are there any news on this issue? I guess an oppiniated "django/whatever deployment" pypi package would make sense, to tinker with the basic things one would need, so onboarding can be far more easier. Is there anything like this known/in the works?

In any case, I'll need to tinker with fabric 2, and share some thoughts here, if they are worth it ;-)

[ckcollab]

What's the appropriate way to store multiple hosts/types of hosts? I.e.:

gpu_workers:
  - ubuntu@12.34.56.78
cpu_workers:
  - ubuntu@12.34.56.77

fab --FLAG gpu_workers do_some_task

[char101]

We can create a custom task decorator that replaces the context argument:

import fabric
from functools import wraps

def task(f, *args, **kwargs):
    @wraps(f)
    def wrapper(c, *args, **kwargs):
        c = fabric.Connection('host')
        f(c, *args, **kwargs)

    return fabric.task(wrapper, *args, **kwargs)

@task
def deploy(c):
    with c.forward_remote(9418):
        c.run('git pull http://localhost/app')

or put it in an external module

fabutil.py

import inspect
import fabric
from functools import wraps


class Connection(fabric.Connection):
  # add helper methods
  def exists(self, path):
     return not self.run(f'test -e "{path}").failed

def task(f, *args, **kwargs):
    caller = inspect.getmodule(inspect.currentframe().f_back)
    host = caller.HOST
    user = caller.USER

    @wraps(f)
    def wrapper(c, *args, **kwargs):
        c = Connection(host, user=user, connect_kwargs={'key_filename': os.path.expanduser('~/.ssh/id_rsa.pub')})
        f(c, *args, **kwargs)

    return fabric.task(wrapper, *args, **kwargs)

fabfile.py

from fabutil import task

HOST = 'host'
USER = 'user'

@task
def deploy(c):
    with c.forward_remote(9418):
        c.run('git pull http://localhost/app')