How to read results in ThreadingGroup during partial failures?

[shadyabhi]

It will be good to include example of that in docs too. Let's suppose I have this code.

from fabric import ThreadingGroup as Group
from fabric.exceptions import GroupException

hosts = ['web1', 'web2', 'web3']
g = Group(*hosts)

try:
    results = g.run('date', hide='both')
except GroupException as e:
    print e.args # I read from code that arguments are passed to exception but I'm unable to use that. `args` argument in my case is an empty tuple. 

Let's suppose that web1 and web3 succeeded while web2 failed to connect or the command returned with non-zero exit code.

• How do I get those details after the exception is raised?
• Is it guaranteed that the exception will be raised only after all the operations are done?

Any help is appreciated.

[bossjones]

Hey @shadyabhi ,

Not sure if you are trying to catch a specific exception or if you just want to catch all exceptions happening in separate threads. I think the ThreadingGroup object already passes around exception objects using queues and raises if any are encountered during execution. I was able to see stack traces using the following code snippet:

def run_data(c):
    c.run('date', hide='both')

@task
def thread_exception_example(c):
    hosts = ['localhost', 'fakehost', 'localhost']
	
    # python list comprehension
    {c: run_data(c) for c in Group(*hosts)}

On the command-line I ran: fab thread-exception-example

[shadyabhi]

Hi @bossjones,

Thanks for replying. However, I'm not using the fab command but rather executing all this by using fabric as a library. So, an answer is still pending and would be appreciated.

In other words, fab is handling what I want to handle in my code myself. There are reasons I don't want to use fab command directly as I would like to have more flexibility in how I invoke tasks.

Thanks

[shadyabhi]

Hi everyone,

I'm sorry, I overlooked the details. Checked out the fabric code and got details.

The argument with GroupException is a dictionary with the key as fabric.connection.Connection object and value as fabric.runners.Result object.

except GroupException as e:
         for c, r in e.result.items():
             print "Connection: {}, Result: {}".format(c, r)

[akiuni]

Hello,

In addition of shadyabhi's comment, here is a way I use to deal with GroupException.
I hope this will help !!

import logging, socket, paramiko.ssh_exception
from fabric import Connection, Config, ThreadingGroup, exceptions, runners

...

g = ThreadingGroup.from_connections(c)
try:
    result = g.run('hostname' )  # (just a basic command)
except exceptions.GroupException as e:
    i = 0
    for c, r in e.result.items():
        print("Host[{}] = [{}]".format(i,c.host) )
        if isinstance(r,runners.Result) :
            print("ok, fine")
        elif isinstance(r,socket.gaierror) :
            print("Network error")
        elif isinstance(r,paramiko.ssh_exception.AuthenticationException) :
            print("Auth failed")
        else:
            print("something else")
        i+=1

(got if from ActivCloud support )

[bradparks]

Thanks @akiuni for your example! I'm new to Python/Fabric and that helped me understand how to make requests using ThreadedGroup, and collect the results when there was an exception. Here's my expanded version of your example. Much appreciated! For what it's worth, I posted this to Stackoverflow as well.

# requires fabric 2.x - run 'pip install fabric' to install it
import logging, socket, paramiko.ssh_exception
from fabric import Connection, Config, SerialGroup, ThreadingGroup, exceptions, runners
from fabric.exceptions import GroupException


# Note: You need to supply your own valid servers here to ssh to of course!
def main():
    testHosts("All should succeed", "validServer1,validServer2,validServer3")
    testHosts("Some should fail", "validServer1,validServer2,BADSERVER1,validServer3,BADSERVER2")

def testHosts(message, hostsAsString):
    print("")
    print(message)

    # Get list of hosts from somewhere, and convert them to connections
    hosts = hostsAsString.split(",")
    servers = [Connection(host=host) for host in hosts]
        
    # Create a thread group to run requests in parallel
    g = ThreadingGroup.from_connections(servers)
    try:
        command = "df -h / | tail -n1 | awk '{print $5}'"
        results = g.run(command, hide=True)
        for r in results:
            connection = results[r]
            print("{}".format(r.host) )
            print("  SUCCESS, " + connection.stdout.strip())
    except GroupException as e:
        # If an exception occurred, at least one request failed. 
        # Iterate through results here
        for c, r in e.result.items():
            print("{}".format(c.host) )
            if isinstance(r,runners.Result) :
                print("  SUCCESS, " + r.stdout.strip())
            elif isinstance(r,socket.gaierror) :
                print("  FAILED,  Network error")
            elif isinstance(r,paramiko.ssh_exception.AuthenticationException) :
                print("  FAILED,  Auth failed")
            else:
                print("  FAILED,  Something other reason")

main()

[pthomas-cyblue]

Thanks - the documentation is lacking examples like this.
There is a typo in the above / stackoverflow example - otherwise works great so far.

for c, r in e.result.items():
should be "servers" in this example
for servers, r in e.result.items():