Worker Cookbook

• A Simple Plugin Example
• A More Realistic Plugin Example
• Share Data between Stages
• Override one Stage of an Action
• Add to one Stage of an Action
• Run code before an Action
• Run code at the end of an Action
• Act on data before it hits the database
• How does Worker Priority work?
• The Execution Order of Plugin Workers

A Simple Plugin Example

In your deployment.yml:

site_local_files: true
extra_worker_plugins: ['X::Demo']

The following file would live at:

/home/netdisco/nd-site-local/lib/App/NetdiscoX/Worker/Plugin/Demo.pm

package App::NetdiscoX::Worker::Plugin::Demo;

use Dancer ':syntax';
use App::Netdisco::Worker::Plugin;
use aliased 'App::Netdisco::Worker::Status';

register_worker({ phase => 'main' }, sub {
  print "I am a worker.\n";
  return Status->done("I have completed my work!");
});

true;

Then you can run:

$ ND2_LOG_PLUGINS=1 ~/bin/netdisco-do demo -D

Which results in something like:

[33131] 2017-11-30 11:05:37  info App::Netdisco version 2.036012_003 loaded.
[33131] 2017-11-30 11:05:37  info demo:  started at Thu Nov 30 11:05:37 2017
[33131] 2017-11-30 11:05:37 debug loading worker plugin App::NetdiscoX::Worker::Plugin::Demo
[33131] 2017-11-30 11:05:37 debug => running workers for phase: main
[33131] 2017-11-30 11:05:37 debug -> run worker main/_base_/0
I am a worker.
[33131] 2017-11-30 11:05:37  info demo: finished at Thu Nov 30 11:05:37 2017
[33131] 2017-11-30 11:05:37  info demo: status done: I have completed my work!

A More Realistic Plugin Example

package App::NetdiscoX::Worker::Plugin::Discover::Neighbors::Routed;
use Dancer ':syntax';

use App::Netdisco::Worker::Plugin;
use App::Netdisco::Transport::SNMP;
use aliased 'App::Netdisco::Worker::Status';

use App::Netdisco::Util::Device qw/get_device is_discoverable/;
use App::Netdisco::JobQueue 'jq_insert';

register_worker({ phase => 'main', driver => 'snmp' }, sub {
  my ($job, $workerconf) = @_;

  my $device = $job->device;
  return unless $device->in_storage and $device->has_layer(3);
  my $snmp = App::Netdisco::Transport::SNMP->reader_for($device)
    or return Status->defer("discover failed: could not SNMP connect to $device");

  my $ospf_peers = $snmp->ospf_peers || {};

  return Status->info(" [$device] neigh - no OSPF peers")
    unless (scalar values %$ospf_peers);

  my $count = 0;
  foreach my $ip (values %$ospf_peers) {
    my $peer = get_device($ip);
    next if $peer->in_storage or not is_discoverable($peer);
    next if vars->{'queued'}->{$ip};

    jq_insert({
      device => $ip,
      action => 'discover',
      subaction => 'with-nodes',
    });

    $count++;
    vars->{'queued'}->{$ip} += 1;
    debug sprintf ' [%s] queue - queued %s for discovery (peer)', $device, $ip;
  }

  return Status->info(" [$device] neigh - $count peers added to queue.");
});

true;

Share Data between Stages

Use the vars stash variable that Dancer provides. This is a HashRef which can store any data you like for the lifetime of the complete action. It becomes available when you use Dancer ':syntax', and will not persist between actions. For example:

package App::Netdisco::Worker::Plugin::Vlan;
vars->{'port'} = get_port($job->device, $job->port)

or, as in the plugin example above:

package App::Netdisco::Worker::Plugin::Discover::Neighbors;
vars->{'queued'}->{$ip} = true;

# and then

package App::Netdisco::Worker::Plugin::Discover::Neighbors::Routed;
next if vars->{'queued'}->{$ip};

Override one Stage of an Action

You need to:

• Use the same stage namespace or a child namespace

• Use the same phase in the sequence

• Set a higher priority

• Return Status done()

So for example to override the Arpnip::Nodes stage, your package must be named App::NetdiscoX::Worker::Plugin::Arpnip::Nodes (or a child namespace). Your worker config hash must have a higher priority, either using the priority key or a driver key mapping to a higher priority. Be sure to run at the same phase. Return Status→done() from your worker.

package App::NetdiscoX::Worker::Plugin::Arpnip::Nodes::MyOverride;

use Dancer ':syntax';
use App::Netdisco::Worker::Plugin;
use aliased 'App::Netdisco::Worker::Status';

register_worker({ phase => 'main', priority => 1000 }, sub {
  return Status->done("I have completed my work!");
});

true;

Add to one Stage of an Action

You need to:

• Use the same stage namespace or a child namespace

• Use the user phase

• Optionally pin to a specific driver type (eg snmp)

• Return Status info()

package App::NetdiscoX::Worker::Plugin::Arpnip::Nodes::MyExtra;

use Dancer ':syntax';
use App::Netdisco::Worker::Plugin;
use aliased 'App::Netdisco::Worker::Status';

register_worker({ phase => 'user' }, sub {
  return Status->info("I have completed my work!");
});

true;

Run code before an Action

You need to:

• Use the same stage namespace or a child namespace

• Use the early phase

• Optionally pin to a specific driver type (eg snmp)

• Return Status info()

package App::NetdiscoX::Worker::Plugin::Arpnip::MyInjection;

use Dancer ':syntax';
use App::Netdisco::Worker::Plugin;
use aliased 'App::Netdisco::Worker::Status';

register_worker({ phase => 'early' }, sub {
  return Status->info("I will be run just before an arpnip!");
});

true;

Run code at the end of an Action

You need to:

• Use the same stage namespace or a child namespace

• Use the late phase

• Optionally pin to a specific driver type (eg snmp)

• Return Status probably info()

package App::NetdiscoX::Worker::Plugin::Arpnip:MyAddon;

use Dancer ':syntax';
use App::Netdisco::Worker::Plugin;
use aliased 'App::Netdisco::Worker::Status';

register_worker({ phase => 'late' }, sub {
  return Status->info("I will be run after an arpnip!");
});

true;

Act on data before it hits the database

This is a special case and only currently available in the arpnip::nodes action, which caches node data in vars and uses the store phase to write to the database.

This allows you to intercept the data in vars during the user phase, which occurs between the main and store phases.

You can simply read the cache, or even edit it, so long as you maintain the same data structure. Ask the Netdisco devs if you’d like this feature for any other actions.

How does Worker Priority work?

As you can see in the examples in this document, priority may be set explicitly using the priority key or implicitly using the driver key, or not at all (which is actually zero). Workers are run in order from highest to lowest priority and stop running when one priority level happens to return Status done().

The mapping of driver to priority level is:

restconf: 500
netconf:  400
eapi:     300
cli:      200
snmp:     100

Therefore setting a priority such as 1000 would cause a worker to override any built-in driver.

The Execution Order of Plugin Workers

There are several dimensions to this: action, stage, phase, driver, and priority.

Action is the scheduled job or netdisco-do command, and is taken from the Perl Package namespace in which the worker is registered, being the component following Plugin:: in the namespace.

Stage is the Perl Package namespace component(s) following the Action. For example the Arpnip::Nodes worker is stage nodes. Child packages of this namespace get folded up into the same stage (there are no sub-stages).

Phase is one of check, early, main, user, store, or late (in that order). Not all phases are used in all workers.

Driver and Priority are effectively the same - the driver being a shorthand for a given priority number (as documented above). Workers are run in descending order of priority from highest to lowest.

Here is the pseudocode runtime order:

• For each Phase in order: check, early, main, user, store, and late

  • Run each Stage (in asciibetical order), starting with the "root" action namespace

    • Run workers from highest priority to lowest

      • Abort if there are check workers but none has returned Status done()

      • Stop running when the previous higher priority level had a worker return Status done()

This last line can be a little tricky to understand. Let’s say there are workers available for each of the netconf, cli, and snmp drivers; that would be priorities 400, 200, and 100 in practice. If the user has configured device_auth settings for all drivers then Netdisco runs the priority 400 workers (netconf) first, and if none has returned done() runs the priority 200 workers (cli) next, and if none has returned done() runs the priority 100 workers (snmp). The result of the job overall is the best status across all workers, which may be done, info, defer, or error.