Backend Plugins

• Introduction
• Overview
• Registering a Worker
  • Package Naming Convention
  • %workerconf Settings
• Writing Workers
  • Worker Return Status
  • Accessing Transports
  • Database Connections
• Worker Execution Order

Introduction

This document is a technical description of the operation and design of Netdisco’s backend plugin system. As there are new terms to learn, and subtleties in the logic, we have a Cookbook, and you can also look at Netdisco’s core plugins themselves.

Overview

Netdisco’s plugin system allows you to alter, add to, remove, or override, components of the backend daemon’s activity. Plugins can be distributed independently from Netdisco to add local functionality. They also integrate fully with the scheduler and command-line netdisco-do app.

Typically, plugin workers gather information from network devices using transports (such as SNMP, SSH, or HTTPS) and store results in the database. Workers combine transports with relevant application protocols such as SNMP, NETCONF (OpenConfig with XML), RESTCONF (OpenConfig with JSON), eAPI, or even CLI scraping.

The combination of transport and protocol is known as a driver. With familar ACL syntax, workers can be restricted to certain vendor platforms, particular drivers, and specific actions in Netdisco’s backend operation.

When a Netdisco action is run (discover, macsuck, etc), all the relevant plugins are loaded and their workers registered, and then run in a particular order according to simple conventions.

Registering a Worker

A worker is Perl code that is registered to Netdisco from within a plugin package. The worker can do anything you like, and will be run when its parent action (determined by the package name) is invoked by Netdisco’s scheduler or the netdisco-do command-line app.

App::Netdisco plugins must load the App::Netdisco::Worker::Plugin module. This exports a helper subroutine to register your worker(s). Here’s the boilerplate code for our example plugin module:

package App::Netdisco::Worker::Plugin::Discover::Neighbors::Routed;

use Dancer ':syntax';
use App::Netdisco::Worker::Plugin;
use aliased 'App::Netdisco::Worker::Status';

# worker registration code goes here, ** see below **

true;

Use the register_worker helper from App::Netdisco::Worker::Plugin to register a worker:

register_worker( $coderef );
# or
register_worker( \%workerconf, $coderef );

For example (using the second form):

register_worker({
  driver => 'unifiapi',
}, sub { "worker code here" });

The %workerconf hashref is optional, and described below. The $coderef is the main body of your worker. You can register more than one worker in a packge, each is run within a Try::Tiny statement to catch errors, and is passed the following arguments:

$coderef->($job, \%workerconf);

The $job is an instance of App::Netdisco::Backend::Job. Note that this class has a device slot which may be filled, depending on the action, and if the device is not yet discovered then the row will not yet be in storage. The \%workerconf hashref is what was passed in the registration of the worker (if anything), plus some other useful data, and is documented below.

Package Naming Convention

The package name used where the worker is declared is significant. Let’s look at the boilerplate example again:

package App::Netdisco::Worker::Plugin::Discover::Neighbors::Routed;

The package name must contain Plugin:: and the namespace component after that becomes the action. For example, workers registered in the above package will be run during a discover job. You can replace Discover with other actions such as Macsuck, Arpnip, Expire, and Nbtstat, or create your own.

The package namespace component following the action (Neighbors in this example) is one stage of the action. The worker’s code is registered within this stage, and you can override, or add to, any of the core stages in Netdisco.

Looking at the core arpnip action, we have two stages: nodes for gathering ARP tables and subnets to gather directly connected prefixes on the router:

App::Netdisco::Worker::Plugin::Arpnip;
App::Netdisco::Worker::Plugin::Arpnip::Nodes;
App::Netdisco::Worker::Plugin::Arpnip::Subnets;

Any packages in namespaces "below" any stage are folded back up into the parent stage. In the Discover::Neighbors example above, workers in the Routed package would be folded into the Neighbors stage. This is especially useful for local sites wishing to amend actions via the NetdiscoX namespace.

Users can run individual stages of an action either at the command line (netdisco-do) or in their schedule, by specifying the job as, for example, discover::neighbors or arpnip::nodes.

Workers may also be registered directly to the parent action. This namespace is typically reserved for "checking" code that will abort the whole action if certain conditions are not met (such as user configuration blocking the action for the target device), or for very simple single-stage actions such as psql or expire.

%workerconf Settings

Along with your worker $coderef can be a %workerconf HASH reference. All settings are optional:

• Access Control Lists

Workers may have only and no parameters configured which use the standard ACL syntax described in Access Control Lists. The only directive is especially useful as it can restrict a worker to a given device platform or operating system. For example:

register_worker({
  only => ['vendor:cisco', 'os:ios-xr'], # Cisco IOS-XR
}, sub { "worker code here" });

• phase (string)

Running an action is separated by Netdisco into six phases: check, early, main, user, store, and late (in that order). Worker code is registered to a phase, or to the user phase if none is specified.

register_worker({
  phase => 'main',
}, sub { "worker code here" });

For example the check phase would abort a whole action if it is blocked by user configuration. The early phase allows initial setup and data gathering, followed by most work taking place in the main phase. The user phase is not used by Netdisco and is reserved for your own post-processing workers. Some stages hold off writing to the database until the store phase, and so the late phase exists as a second user phase so you have access to data pre and post writing to the database. Any Hooks defined on the action are also run in the late phase.

• driver (string)

The driver is a label associated with a group of workers and typically refers to the combination of transport and application protocol. Examples include snmp, netconf, restconf, eapi, and cli. The convention is for driver names to be lowercase.

register_worker({
  driver => 'snmp',
}, sub { "worker code here" });

Users will bind authentication configuration settings to drivers in their configuration. If no driver is specified when registering a worker, it will be run for every device (such as during Expire jobs).

• priority (integer)

Workers get assigned a priority value so that worker code with the same action, stage (package name), driver, and phase, can be overridden or added to. The priority is also used to pick the best driver if more than one is available.

register_worker({
  priority => 120,
}, sub { "worker code here" });

The priority is set from the driver (Netdisco knows what the best drivers are), or else defaults to zero. The Worker Cookbook gives examples of how to use the priority to add, chain, or override Netdisco actions.

Writing Workers

You can register more than one worker subroutine in a packge, and each is run within a Try::Tiny statement to catch errors.

Worker Return Status

The return value of the worker is significant, as it indicates to Netdisco whether to continue, branch, or abort the running of an action. You should either return nothing, or an instance of the aliased App::Netdisco::Worker::Status helper (loaded as in the boilerplate above):

return Status->done('a success that represents the complete action');
# or
return Status->info('a success but not the primary goal of the action');
# or
return Status->defer('could not connect to device for any reason');
# or
return Status->error('something went wrong');

Accessing Transports

From your worker you will want to connect to a device to gather data. This is done using a transport protocol session (SNMP, SSH, etc). Transports are singleton objects instantiated on demand, so they can be shared among a set of workers that are accessing the same device.

See the documentation for each transport to find out how to access it:

• App::Netdisco::Transport::SSH

• App::Netdisco::Transport::SNMP

Database Connections

The Netdisco database is available via the netdisco schema key, as below. You can also use the external_databases configuration item to set up connections to other databases.

# plugin package
use Dancer::Plugin::DBIC;
my $set = schema('netdisco')->resultset('Devices')
                            ->search({vendor => 'cisco'});

Worker Execution Order

Given all of the above (action, stage, phase, driver, and priority), Netdisco builds an ordered list of workers to be run.

Perl Packages matching the action name in the extra_worker_plugins setting and then the worker_plugins setting are loaded. The order does not affect stages, which are sorted alphabetically, but does affect workers of equal priority within a stage. This is a useful subtlety which is described more in the Worker Cookbook.

The six phases (check, early, main, user, store, and late) are run in turn, with workers from each stage and in descending order of priority. The priority is either given in the worker’s configuration or determined from the driver (each driver maps to a priority value), or is zero.

If there are check phase workers but none of them returns a done() status, then the whole action is aborted. Similarly, when higher priority workers of the same stage return done(), then lower ones are not run. The outcome of the action overall is then the best return value (Status instance) from all the workers that have run.