Checkpointing

Overview

Checkpointing is the process of saving state so that it might be rolled back later. MADARA provides very powerful checkpointing features to help systems quickly recover, restart, or replay state that was previously recorded. In this guide, we'll discuss the text and binary checkpointing options.

1. Text Checkpointing

There are two text checkpointing mechanisms with direct support in the KnowledgeBase. The first is KnowledgeBase::save_as_karl, and the other is KnowledgeBase::save_as_json. We're only going to cover the save_as_karl function, but the json functions work similarly except this format is primarily useful for people who want to use JSON reading external tools that do not interact with the KnowledgeBase.

save_as_karl saves into the KaRL format, sometimes called the "MADARA format" because the KaRL language is often saved into files with ".mf" extensions. KaRL stands for the Knowledge and Reasoning Language. It is a powerful, optimized scripting language written for real-time systems. However, it can also be used as an easy-to-read configuration language.

When you call save_as_karl, the KnowledgeBase will save its variables in a var1='var1 value'; format that can be evaluated later in the KnowledgeBase with KnowledgeBase::evaluate or KnowledgeBase::wait. To load this type of text checkpoint, you can use a function like madara::utility::file_to_string to read the text file in, and then feed the contents of the file into KnowledgeBase::evaluate. A code example is below.

Saving and loading the entire KnowledgeBase to/from text file

#include "madara/knowledge/KnowledgeBase.h"
#include "madara/utility/Utility.h"

namespace knowledge = madara::knowledge;
namespace filters = madara::filters;

int main (int argc, char ** argv)
{
  knowledge::KnowledgeBase saver, loader;
  knowledge::CheckpointSettings settings;
  
  // create some variables in the KB we're saving from
  saver.set ("int_var", knowledge::KnowledgeRecord::Integer (15));
  saver.set ("str_var", std::string ("some string"));
  saver.set ("double_var", 3.14159);

  // setup a file to save to
  settings.filename = "my_database.mf";

  // save the context with the settings
  saver.save_as_karl (settings);

  // load the context
  loader.evaluate (madara::utility::file_to_string (settings.filename));
  
  // print the variables in the loaded configuration
  loader.print ();
  
  return 0;
}

We'll discuss the usage of CheckpointSettings later. In the above, we just use the filename member of this class.

2. Binary Checkpointing

There are two binary checkpointing mechanisms with direct support in the KnowledgeBase. The first is KnowledgeBase::save_context. This function saves the entire context into a binary format. The second function for binary saving is KnowledgeBase::save_checkpoint, which saves a diff since the last time save_checkpoint or save_context was called. To load either saved format, the KnowledgeBase provides the KnowledgeBase::load_context function.

A demonstration of this optimized saving and loading format is shown below. save_context first.

Saving and loading the entire KnowledgeBase to/from binary file

#include "madara/knowledge/KnowledgeBase.h"
#include "madara/utility/Utility.h"

namespace knowledge = madara::knowledge;
namespace filters = madara::filters;

int main (int argc, char ** argv)
{
  knowledge::KnowledgeBase saver, loader;
  knowledge::CheckpointSettings settings;
  
  // create some variables in the KB we're saving from
  saver.set ("int_var", knowledge::KnowledgeRecord::Integer (15));
  saver.set ("str_var", std::string ("some string"));
  saver.set ("double_var", 3.14159);

  // setup a file to save to
  settings.filename = "my_database.kb";

  // save the context with the settings
  saver.save_context (settings);

  // load the checkpoint
  loader.load_context (settings.filename);
  
  // print the variables in the loaded configuration
  loader.print ();
  
  return 0;
}

A demonstration of this optimized diff-based saving and loading is shown below. Note that you can layer checkpoints on the same file by simply saving to the same file location, as shown below, or you can also save each diff to a different file. If you save to the same file, you can load a single checkpoint in that file or even a range of checkpoints from the file. We'll discuss the usage of CheckpointSettings later.

Saving and loading diffs to/from binary file

#include "madara/knowledge/KnowledgeBase.h"
#include "madara/utility/Utility.h"

namespace knowledge = madara::knowledge;
namespace filters = madara::filters;

int main (int argc, char ** argv)
{
  knowledge::KnowledgeBase saver, loader;
  knowledge::CheckpointSettings settings;
  
  // create some variables in the KB we're saving from
  saver.set ("int_var", knowledge::KnowledgeRecord::Integer (15));
  saver.set ("str_var", std::string ("some string"));
  saver.set ("double_var", 3.14159);

  // setup a file to save to
  settings.filename = "my_database.kb";

  // save a diff of everything that has changed
  saver.save_checkpoint (settings);

  saver.set ("str_var", std::string ("some other string"));
  
  // save a diff of the changes to the knowledge base (just contains str_var)
  saver.save_checkpoint (settings);

  // load the checkpoint
  loader.load_context (settings.filename);
  
  // print the variables in the loaded configuration
  loader.print ();
  
  return 0;
}

3. Configuring with CheckpointSettings

CheckpointSettings is the preferred configuration mechanism for saving and loading checkpoints. CheckpointSettings has a host of options that are not going to be covered in this section. However, it is worth noting some of the important ones and to also understand that the load_context function will actually fill in useful metadata from the checkpoint into the CheckpointSettings class you pass in. One of the most powerful features is BufferFilters, but that is covered a bit in the Encryption Wiki. The other powerful feature you'll want to know about is CheckpointSettings::prefixes.

The CheckpointSettings::prefixes is an STL vector of strings that holds a list of prefixes that you are interested in saving or loading from the KnowledgeBase. If the prefixes are empty, this means save and load all variables in the context. If the prefixes have a variable prefix, then only variables that begin with that prefix will be saved or loaded to/from checkpoints. This is a powerful feature that allows developers to quickly sculpt data. We have seen MADARA checkpoints with hundreds of thousands of variables in them, and this kind of feature can be invaluable in helping people find knowledge quickly.

To see some helpful examples of ways to use prefixes, check out the test_checkpointing MADARA test.

4. Checkpoint Streaming

Normally, checkpoints are saved only when save_checkpoint is called, and while differential checkpoints are supported, they only save the current values in the KnowledgeBase. To ensure all knowledge that passes through the KnowledgeBase, i.e., even update to every KnowledgeRecord, save_checkpoint would need to be called at least as frequently as the most frequently updated record. As an alternative for this use case, Madara provides a streaming capability to automatically generate comprehensive checkpoints over time which will contain all modifications to records.

To start streaming, call KnowledgeBase::attach_streamer() with a CheckpointStreamer object. A CheckpointStreamer object is constructed with a CheckpointSettings object which configures the checkpointing in same ways as above, except that reset_checkpoint is ignored; streaming acts separately from the differential checkpointing system. The constructor also takes a reference to the KnowledgeBase it will attached to, and a hertz rate. Updates will not be written to the disk immediately. The are batched in-memory, and written at the hertz rate given. If during a period no updates occured, the disk will not be written to at all.

For example:

#include "madara/knowledge/KnowledgeBase.h"
#include "madara/knowledge/CheckpointStreamer.h"
#include "madara/utility/Utility.h"

using knowledge = madara::knowledge;

knowledge::KnowledgeBase kb;
knowledge::CheckpointSettings settings;

settings.filename = "stream.stk";

auto streamer = std::make_unique<knowledge::CheckpointStreamer>(std::move(settings), kb, 100);
kb.attach_streamer(std::move(streamer)); // write up to 100 times per second

kb.set("var1", 1); // will be written to disk within 10 ms
kb.set("var1", 2); // will also be written, in addition to the previous value

5. Reading Checkpoints

The KnowledgeBase itself provides a load_context member function which loads a checkpoint with a variety of settings, including filtering by variable name prefix, and limiting to a time range. This function is implemented using a CheckpointReader class, which may be used directly to iterate over a checkpoint, and handle record updates one-at-a-time. This may be useful, for example, for playing back updates in directly managed way, and converting checkpoint files into other formats.

The CheckpointReader class constructor takes a CheckpointSettings class, and uses its contents just like load_context (except it ignores the clear_knowledge flag, since it doesn't directly interact with KnowledgeBase). It provides a next() member function which returns the next update (not filtered by the settings given), starting from the first written. Note that these are not necessarily in time-of-insertion (to KnowledgeBase) order, unless the checkpoint is from the streaming checkpoint system. The return value is a std::pair<std::string, KnowledgeRecord> containing the variable name, and new value. When no updates remain, this function returns an empty string for the variable name.

For example, to print the contents of a checkpoint, you might use the following:

#include <iostream>
#include "madara/knowledge/KnowledgeBase.h"

namespace knowledge = madara::knowledge;

knowledge::CheckpointSettings settings;
settings.filename = "mycheckpoint.kb";

knowledge::CheckpointReader reader (std::move (settings));

for(;;)
{
  auto cur = reader.next ();
  if (cur.first == "")
    break;

  std::cout << cur.first << ": " << cur.second;
}

This class also provides additional member functions:

start() opens the checkpoint and performs the initialization, but stops before reading any updates. Subsequent calls to next() will return updates. This call is optional, as next() will call it automatically if need be.
get_total_read() returns the total number of bytes read from the checkpoint so far.
get_file_header() returns a pointer to the checkpoint's FileHeader metadata. Must call start() or next() first. If not, returns nullptr.
is_open() returns true if the reader has an open checkpoint file. Open files are closed when the reader is destructed.

6. Debugging Checkpoints

Need to see what you've saved without writing code with KnowledgeBase::load_context. Need to just quickly inspect binary or text checkpointings? The KaRL interpreter has you covered. karl is located in the $MADARA_ROOT/bin directory on your file system, and it's so useful for debugging that we recommend people add $MADARA_ROOT/bin to their PATH variable, just so they can run karl from the command line instead of $MADARA_ROOT/bin/karl (which can be annoying).

Anyway, here's how to load a text checkpoint (the results of save_as_karl) and print out the knowledge.

Loading and Printing Knowledge in a KaRL file

karl -i my_database.mf -k

The above tells the KaRL interpreter to load a text-based KaRL file called my_database.mf and then print the resulting, final knowledge.

Loading and Printing Knowledge in a binary checkpoint

karl -0b my_database.kb -k

The above tells the KaRL interpreter to load an initial binary checkpoint located at my_database.kb and then to print the resulting, final knowledge

This is just the tip of the iceberg with the KaRL interpreter. The interpreter is actually network transport capable, and can be configured with UDP unicast, broadcast, or multicast or even 0MQ to listen to or even send knowledge to other agents using MADARA. It's a great tool for debugging. To see all of the powerful options, simply pass in -h or --help to get detailed usage information.

7. Checkpoint Player

In addition to the ability to read checkpoints and iterate through them as shown earlier, you can also attach a player that will update a knowledge base in accordance to the time differences in a Serialized Temporal Knowledge base (STK).

The following illustrates the usage of knowledge::CheckpointPlayer:

#include "madara/knowledge/CheckpointPlayer.h"
#include "madara/knowledge/KnowledgeBase.h"
#include "madara/utility/Utility.h"

namespace knowledge = madara::knowledge;
namespace utility = madara::utility;

knowledge::CheckpointSettings settings;
knowledge::KnowledgeBase kb;

// configure checkpoint settings
settings.filename = "knowledge_changes.stk";
settings.playback_simtime = true;

// create the CheckpointPlayer
knowledge::CheckpointPlayer player (kb.get_context (), settings);

// start the player thread, which will update the KB in approximate time
player.start();

// print the contents of the kb over 10s
for (int i = 0; i < 10; ++i)
{
  kb.print ();
  utility::sleep (1.0);
}

More Information

If you are looking for code examples and guides, your best bet would be to start with the tutorials (located in the tutorials directory of the MADARA root directory--see the README.txt file for descriptions). After that, there are many dozens of tests that showcase and profile the many functions, classes, and functionalities available to MADARA users.

Users may also browse the Library Documentation for all MADARA functions, classes, etc. and the Wiki pages on this website.

Video Tutorials

First Steps in MADARA: Covers installation and a Hello World program.
Intro to Networking: Covers creating a multicast transport and coordination and image sharing between multiple agents.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly