Skip to content

Debugging With Karl

Jump to bottom Edit New page
James Edmondson edited this page Oct 8, 2018 · 10 revisions

The KaRL Interpreter

When you compile MADARA, you also compile a few useful tools that can help you with common operations such as debugging distributed applications. One such tool is called karl and you can find it at $MADARA_ROOT/bin/karl. This binary is so useful for debugging that many people add $MADARA_ROOT/bin to their PATH environment variable, so they can just call karl from the command line. In this wiki, we'll walk through some of the common usages of karl with examples so you can get started with debugging your MADARA applications.

Table of Contents

Main Features

The karl command-line interpreter is a full-featured process with interactive help (-h | --help), configurable transport, and built-in knowledge base. It includes the following capabilities:

  1. Configurable transport (UDP Unicast, Multicast, Broadcast and ZMQ)
  2. Built-In Knowledge Base
  3. Interactive Knowledge and Reasoning Language Interpreter
  4. Configurable save and load system for checkpointing
  5. Encryption (AES 256 bit) and Compression (LZ4) if appropriate features are compiled
  6. Printing variables in the Knowledge Base
  7. Printing variables as they arrive from the transports

There are dozens of command-line options above-and-beyond those listed above. These are just some of the major ones that may help with debugging. See -h and --help for more options

Connecting to other Agents

The main way to connect to other active agents is to add a transport with the appropriate endpoints. The easiest way to connect to other agents is with a Multicast or Broadcast transport. The harder transports are the unicast transports that must have each endpoint configured.

To listen for 30 seconds on a multicast IP address and print out all variables in the resulting knowledge base every 1 second, do the following:

$MADARA_ROOT/bin/karl -y 1 -ky -t 15 -m 239.255.0.1:4150

Shaping what is Printed

There are three main options for printing in karl. These options are:

  1. -k: printing knowledge at the end, when everything is finished (such as KaRL logic evaluations)
  2. -ky: printing knowledge at a certain period. This must be used in conjunction with a period (-y and a wait period, in seconds--e.g., -y 0.5 for a half-second period between evaluations/prints).
  3. --debug: higher level debugging which includes adding an on-receive filter that shows the full TransportContext information of sender, bandwidth usage, etc. every time a message is received from another KnowledgeBase

In addition, you can tailor what is printed out in -k and -ky by using the -kp option, which adds a knowledge variable prefix that you are interested in. To print out specific variables from a multicast ip every 1 second for 15 seconds, you can do the following:

$MADARA_ROOT/bin/karl -kp 'agent.0' -kp 'agent.1' -y 1 -ky -t 15 -m 239.255.0.1:4150

The above would print out any variable in the knowledge base that begins with agent.0 or agent.1. In this way, you can shape what data you print and how often.

Saving Data

There are various command line arguments for saving data in text and binary formats. Text formats are human-readable but slower to save and load. Binary formats are fast but illegible to humans without loading the formats in karl or any MADARA application through the KnowledgeBase::load_context function.

To save data in binary formats, you have the following options:

  1. -sb: save the full context to a file
  2. -sc: save a diff to a file. This option should be paired with a -0b (see Loading Initial Data to get useful results

To save data in text formats, you have the following options:

  1. -s: save the knowledge base in the KaRL language
  2. -sj: save the knowledge base in JSON

By default all variables are saved. You can shape what data is saved by adding prefixes to the save function with the -scp option. For instance, if you'd like to save only agent.0 and agent.1 prefixes to a binary knowledge base, then you can do the following:

$MADARA_ROOT/bin/karl -scp 'agent.0' -scp 'agent.1' -sb test.kb -k -t 15 -m 239.255.0.1:4150

Loading Data

Data can be loaded from a text-based format or a binary format. Text formats are human-readable but slower to save and load. Binary formats are fast but illegible to humans.

There are several different modes of loading contexts. You can load files initially or you could repeatedly load files (e.g., to evaluate a KaRL script at a certain periodicity). In this section, we'll only discuss data loading.

To load data in binary formats, you can use the following:

  1. -0b: load the results of any -sb or -sc operation from karl or KnowledgeBase::save_context or KnowledgeBase::save_checkpoint

To load data in text formats, you can use the following:

  1. -0f: load the knowledge base in the KaRL language

By default all variables are loaded. You can shape what variable prefixes you want to load with the -lcp option. This option is similar to -scp. As an example, if you'd like to just print the variables that start with agent.0 contained in a saved knowledge base, you could do the following:

$MADARA_ROOT/bin/karl -lcp 'agent.0' -0b test.kb -k

Registering Capnproto Types

karl can be configured to load and register Capnp types for usage in inspecting variables and values. To use these tools for Capnp types, we use the following and in the following order:

-ni <schema directory>:  the schema directory to find a capnp type`
-n <tag:type>:          the tag is the type name as registered in a MADARA knowledge base. The type is the name of the Capnp type as defined in a schema file.
-nf <schema directory>/<type file>: the schema file to load

An example loading:

karl -ni $GAMS_ROOT/src/gams/types -n Imu:Imu -nf $GAMS_ROOT/src/gams/types/Imu.capnp

Executing Logic

KaRL can be used to build a knowledge base or even an executable system from scratch. There are examples of self-stabilizing systems with karl, but most people just use karl to create a Knowledge Base that can then be used to seed information in another application, agent, etc.

By default, all arguments on the command line that do not match reserved command line options are logics. For instance:

$MADARA_ROOT/bin/karl -k "agent.0.location=[32.0, 70.1112, 1000] ; agent.0.orientation=[60.077, 108.5317, 42.9791]"

Will print out:

Knowledge in Knowledge Base:
agent.0.location=32.000000, 70.111200, 1000.000000
agent.0.orientation=60.077000, 108.531700, 42.979100

Now, let's instead save that knowledge as a text file and a binary file. We'll call these new knowledge bases agent_info.mf and agent_info.kb.

$MADARA_ROOT/bin/karl -k "agent.0.location=[32.0, 70.1112, 1000] ; agent.0.orientation=[60.077, 108.5317, 42.9791]" -s agent_info.mf -sb agent_info.kb

See Saving Data for more options on saving knowledge.

You could then load that information later, from any terminal or any agent that has access to these files, with either:

$MADARA_ROOT/bin/karl -k -0f agent_info.mf

or

$MADARA_ROOT/bin/karl -k -0b agent_info.kb

See Loading Data for more options on loading knowledge

Periodically Executing Logic

For persistent monitoring of executing systems, you often want to periodically evaluate a logic to detect failures during knowledge transfer of active systems. Here are some of the relevant options to keep in mind:

  1. Any logics that you specify (a non-reserved keyword on the command line) will be evaluated either once (by default) or periodically by specifying -y some_period_in_s. So, -y 0.5 would mean to evaluate the logic every half second.
  2. You can save logics and use them later by specifying the -i option with a KaRL script. Pairing -i with -y some_period_in_s will evaluate any files and logics you specify on the command line at the period you specify.
  3. If you actually want to check for a condition in the logic (the return value is not zero), then use the -c option.

As an example of how this works, consider the following counting logic.

$MADARA_ROOT/bin/karl -ky -y 1 "++.count ;> .count > 10" -t 15 -c

The result of running this option will be:

Knowledge in Knowledge Base:
.count=1

Knowledge in Knowledge Base:
.count=2

Knowledge in Knowledge Base:
.count=3

Knowledge in Knowledge Base:
.count=4

Knowledge in Knowledge Base:
.count=5

Knowledge in Knowledge Base:
.count=6

Knowledge in Knowledge Base:
.count=7

Knowledge in Knowledge Base:
.count=8

Knowledge in Knowledge Base:
.count=9

Knowledge in Knowledge Base:
.count=10

Knowledge in Knowledge Base:
.count=11

And the evaluation will short-circuit before the timeout of 15 seconds. You can use karl to evaluate knowledge as it comes across the network and can make very complicated logics to check for success or failure conditions. For an example of a more complicated success/fail check, see the line intersection test.

Keeping Track of Originator Stats

The karl command line tool can keep track of variable and originator stats by using the -ps and -py arguments. If either of these are used, an aggregate filter is added to the transport that tracks stats of variable updates by originator. These stats are then printed out at the appropriate, user-specified time.

As an example of how this works, consider the following counting logic.

$MADARA_ROOT/bin/karl -y 1 -py -t 15 -m 239.255.0.1:4150

This would launch karl for 15s and every 1s, it will print out originator and variable stats. For instance, when paired with the Madara File Service (MFS), an example printout for the above is the following:

****Originator Stats****
from: mfs_agent: 15 updates (@1.15hz)(@0.12KB/s)
****Variable Stats****
agent.0.sandbox.files.description:
  from: mfs_agent: 15 updates (@1.15hz)(@0.04KB/s)
agent.0.sandbox.files.file.build_checkpoint.sh.last_modified:
  from: mfs_agent: 15 updates (@1.15hz)(@0.00KB/s)
agent.0.sandbox.files.file.build_checkpoint.sh.size:
  from: mfs_agent: 15 updates (@1.15hz)(@0.00KB/s)
agent.0.sandbox.files.file.init.mf.last_modified:
  from: mfs_agent: 15 updates (@1.15hz)(@0.00KB/s)
agent.0.sandbox.files.file.init.mf.size:
  from: mfs_agent: 15 updates (@1.15hz)(@0.00KB/s)
agent.0.sandbox.files.file.samples/chapter16.mp3.last_modified:
  from: mfs_agent: 15 updates (@1.15hz)(@0.00KB/s)
agent.0.sandbox.files.file.samples/chapter16.mp3.size:
  from: mfs_agent: 15 updates (@1.15hz)(@0.00KB/s)
agent.0.sandbox.files.file.samples/chapter6.mp3.last_modified:
  from: mfs_agent: 15 updates (@1.15hz)(@0.00KB/s)
agent.0.sandbox.files.file.samples/chapter6.mp3.size:
  from: mfs_agent: 15 updates (@1.15hz)(@0.00KB/s)
agent.0.sandbox.files.file.samples/init.mf.last_modified:
  from: mfs_agent: 15 updates (@1.15hz)(@0.00KB/s)
agent.0.sandbox.files.file.samples/init.mf.size:
  from: mfs_agent: 15 updates (@1.15hz)(@0.00KB/s)
agent.0.sandbox.files.file.samples/tpn.mobi.last_modified:
  from: mfs_agent: 15 updates (@1.15hz)(@0.00KB/s)
agent.0.sandbox.files.file.samples/tpn.mobi.size:
  from: mfs_agent: 15 updates (@1.15hz)(@0.00KB/s)
agent.0.sandbox.files.name:
  from: mfs_agent: 15 updates (@1.15hz)(@0.01KB/s)
agent.0.sandbox.projects.description:
  from: mfs_agent: 15 updates (@1.15hz)(@0.04KB/s)
agent.0.sandbox.projects.file.mfs.out.last_modified:
  from: mfs_agent: 15 updates (@1.15hz)(@0.00KB/s)
agent.0.sandbox.projects.file.mfs.out.size:
  from: mfs_agent: 15 updates (@1.15hz)(@0.00KB/s)
agent.0.sandbox.projects.file.new_file.out.last_modified:
  from: mfs_agent: 15 updates (@1.15hz)(@0.00KB/s)
agent.0.sandbox.projects.file.new_file.out.size:
  from: mfs_agent: 15 updates (@1.15hz)(@0.00KB/s)
agent.0.sandbox.projects.file.test_checkpointing.out.last_modified:
  from: mfs_agent: 15 updates (@1.15hz)(@0.00KB/s)
agent.0.sandbox.projects.file.test_checkpointing.out.size:
  from: mfs_agent: 15 updates (@1.15hz)(@0.00KB/s)
agent.0.sandbox.projects.name:
  from: mfs_agent: 15 updates (@1.15hz)(@0.02KB/s)

Reducing Stats Printouts

Printouts can be limited for variable stats by a set of prefixes defined on the command line with the -kp option. For instance, the above printouts can be reduced to just the new_file.out file prefixes by saying -kp agent.0.sandbox.projects.file.new_file.out, like the following:

$MADARA_ROOT/bin/karl -kp agent.0.sandbox.projects.file.new_file.out -y 1 -py -t 15 -m 239.255.0.1:4150

Which results in the example following printout (compare to the above):

****Originator Stats****
from: mfs_agent: 16 updates (@1.14hz)(@0.12KB/s)
****Variable Stats****
agent.0.sandbox.projects.file.new_file.out.last_modified:
  from: mfs_agent: 16 updates (@1.14hz)(@0.00KB/s)
agent.0.sandbox.projects.file.new_file.out.size:
  from: mfs_agent: 16 updates (@1.14hz)(@0.00KB/s)

A note about originator strings

By default, if no host is specified when creating knowledge base transports, the underlying KB will generate a UUID to uniquely identify itself to other KBs. This unique identifier is used to distinguish hosts for the purpose of dropping messages and some authentication activities. These strings can be a max of 64 chars long, and default terminal windows are 80 columns wide. If you are having trouble understanding the output of the Originator Stats printouts, you can either grow your terminal wider than 80 columns, or you can change the hostname of your agents to something short, unique and useful.

For instance, the Madara File Service (MFS) can be started with a hostname via its -o option. A reasonably unique identifier for an instance of MFS might be <servername>.mfs. To create an instance of mfs like this, you can call mfs on the command line with $MADARA_ROOT/bin/mfs -o "server1.mfs"

Argument Parameter Files

Argument parameters can be tedious, especially when needing to load many Capnproto type schemas or when repeating commonly used parameters. The karl command line tool helps to alleviate this through parameter files, which can be passed through the command line with the -cf or --config-file option or by populating a default file at $HOME/.madara/karl.cfg with the parameters of your choosing. The parameter files are separated by newlines for each flag or argument and parameter pairing. For instance, the following is a valid parameter file:

-k
-l 3

Which would instruct karl to print final knowledge and also set the logging level to 3 (LOG_MAJOR). The file could be saved to $HOME/.madara/karl.cfg and these options would always be loaded on every invocation of karl. If you wanted to only load the file conditionally, you would save the file at a preferred location and then pass the configuration file in with -cf or --config-file

Add a custom footer
Add a custom sidebar
Clone this wiki locally