KnowledgeContainers
C++ Guide Series
Architecture | Knowledge Base | Networking | Containers | Threads | Optimizations | KaRL | Encryption | Checkpointing | Knowledge Performance | Logging
The MADARA KaRL scripting environment is fast and efficient, but it can also be difficult to debug and work with for programmers more familiar with object-oriented programming in their host languages. To address the needs of object-oriented developers, MADARA also provides a set of containers which present facades into the Knowledge Base and mimic the look and feel of STL containers. These object-oriented containers are meant to be user-friendly and intuitive, but some guidance may be useful for users who do not know where they are and how they may be initialized.
- MADARA Containers
- 1. Types of Containers
- 2. Interaction with the Transport Layer
- More Information
In this section, we breakdown the types of containers. For a complete list of containers, see the files located inside of the include/madara/knowledge_engine/containers directory of your source distribution, or see the Library documentation for the Containers namespace.
Instance variables are facades into single variable locations within the Knowledge Base and are the most efficient of the MADARA containers. Values are set with a set function, available through the container, and retrieved using the asterisk operator (*) or with a to_integer (), to_double (), or to_string () function. MADARA currently supports the following instance variables:
Location: madara/knowledge/containers/Integer.h
Documentation: Integer
Integer is a container that provides interfaces for setting and getting an integer value from the Knowledge Base.
// setup includes and namespace alias for convenience
#include "madara/knowledge/containers/Integer.h"
namespace engine = madara::knowledge;
namespace containers = engine::containers;
// create a knowledge base and make a container that points inside it
engine::KnowledgeBase knowledge;
containers::Integer my_id (".id", knowledge);
// set id within the knowledge base to 1
my_id = 1;
...
// read the value later
std::cout << "My id is " << *my_id << std::endl;
Location: madara/knowledge/containers/Double.h
Documentation: Double
Double is a container that provides interfaces for setting and getting a double value from the Knowledge Base.
// setup includes and namespace alias for convenience
#include "madara/knowledge/containers/Double.h"
namespace engine = madara::knowledge;
namespace containers = engine::containers;
// create a knowledge base and make a container that points inside it
engine::KnowledgeBase knowledge;
containers::Double funds (".funds", knowledge);
// set funds to 300.50
funds = 300.50;
...
// read the value later
std::cout << "My funds available are " << *funds << std::endl;
Location: madara/knowledge_engine/containers/String.h
Documentation: String
String is a container that provides interfaces for setting and getting a string value from the Knowledge Base.
// setup includes and namespace alias for convenience
#include "madara/knowledge/containers/Double.h"
namespace engine = madara::knowledge;
namespace containers = engine::containers;
// create a knowledge base and make a container that points inside it
engine::KnowledgeBase knowledge;
containers::String name (".name", knowledge);
// set name to "John Smith"
name = "John Smith";
...
// read the value later
std::cout << "My name is " << *name << std::endl;
Array containers are facades into arrays of variable locations within the Knowledge Base, and provide O(1) access times to these elements. Unlike with instance variables, the name that is set for an array container is the prefix for variable locations instead of an actual variable name. MADARA currently supports the following vector/array containers:
Location: madara/knowledge/containers/IntegerVector.h
Documentation: IntegerVector
IntegerVector, also known as IntegerArray, is a container that provides interfaces for setting and getting an integer value from the Knowledge Base.
// setup includes and namespace alias for convenience
#include "madara/knowledge/containers/Integer_Vector.h"
namespace engine = madara::knowledge;
namespace containers = engine::containers;
// create a knowledge base and make a 10 element array that points inside it
engine::KnowledgeBase knowledge;
containers::IntegerVector my_array (".array", knowledge, 10);
// set index 1 of the array to true (1)
my_array.set (1);
// set index 5 of the array to 30
my_array.set (5, 30);
// set index 0 of the array to 17
my_array.set (0, 17);
...
// read the values of index 0, 1, and 5
std::cout << "my_array[0] is " << my_array[0] << std::endl;
std::cout << "my_array[1] is " << my_array[1] << std::endl;
std::cout << "my_array[5] is " << my_array[5] << std::endl;
Location: madara/knowledge/containers/DoubleVector.h
Documentation: DoubleVector
DoubleVector, also known as DoubleArray, is a container that provides interfaces for setting and getting an array of double values within the Knowledge Base.
// setup includes and namespace alias for convenience
#include "madara/knowledge/containers/DoubleVector.h"
namespace engine = madara::knowledge;
namespace containers = engine::containers;
// create a knowledge base and make a 10 element array that points inside it
engine::KnowledgeBase knowledge;
containers::DoubleArray my_array (".array", knowledge, 10);
// set index 1 of the array to 6.5
my_array.set (1, 6.5);
// set index 5 of the array to 31.3
my_array.set (5, 31.3);
// set index 0 of the array to 15.2
my_array.set (0, 15.2);
...
// read the values of index 0, 1, and 5
std::cout << "my_array[0] is " << my_array[0] << std::endl;
std::cout << "my_array[1] is " << my_array[1] << std::endl;
std::cout << "my_array[5] is " << my_array[5] << std::endl;
Location: madara/knowledge/containers/StringVector.h
Documentation: StringVector
StringVector, also known as StringArray, is a container that provides interfaces for setting and getting an array of string values within the Knowledge Base.
// setup includes and namespace alias for convenience
#include "madara/knowledge/containers/StringVector.h"
namespace engine = madara::knowledge;
namespace containers = engine::containers;
// create a knowledge base and make a 10 element array that points inside it
engine::KnowledgeBase knowledge;
containers::StringArray my_array (".array", knowledge, 10);
// set the indices of the array to characters in Snow White
my_array.set (0, "Snow White");
my_array.set (1, "Doc");
my_array.set (2, "Dopey");
my_array.set (3, "Bashful");
my_array.set (4, "Grumpy");
my_array.set (5, "Sneezy");
my_array.set (6, "Sleepy");
my_array.set (7, "Happy");
my_array.set (8, "Huntsman");
my_array.set (9, "Queen");
...
// print out all elements of the array
for (unsigned int i 0; i < 10; ++i)
std::cout << "array[" << i << "] is " << my_array[i] << std::endl;
Location: madara/knowledge/containers/Vector.h
Documentation: Vector
Vector, also known as Array, is a container that provides interfaces for setting and getting an array of multi-typed values within the Knowledge Base.
// setup includes and namespace alias for convenience
#include "madara/knowledge/containers/Vector.h"
namespace engine = madara::knowledge;
namespace containers = engine::containers;
// create a knowledge base and make a 10 element array that points inside it
engine::KnowledgeBase knowledge;
containers::Array my_array ("some_array", knowledge, 10);
// the array contains five characters from Snow White and their ages
my_array.set (0, "Snow White");
my_array.set (1, "Doc");
my_array.set (2, "Dopey");
my_array.set (3, "Bashful");
my_array.set (4, "Grumpy");
my_array.set (5, 18);
my_array.set (6, 70);
my_array.set (7, 74);
my_array.set (8, 72);
my_array.set (9, 68);
...
// print out the people and their ages
for (unsigned int i 0; i < 5; ++i)
std::cout << my_array[i] << "'s age is " << my_array[i + 5] << std::endl;
Map containers are facades into string-keyed maps of variable locations within the Knowledge Base. Unlike with instance variables, the name that is set for a map container is the prefix for variable locations instead of an actual variable name, and access times are O(log m), where m is the number of items existing in the Map container.
Location: madara/knowledge/containers/Map.h
Documentation: Map
The Map class is one of the most versatile container classes available in MADARA. This class is likely to prove invaluable to developers wanting to emulate classes within the knowledge base.
// setup includes and namespace alias for convenience
#include "madara/knowledge/containers/Map.h"
namespace engine = madara::knowledge;
namespace containers = engine::containers;
typedef engine::KnowledgeRecord::Integer Integer;
// create a knowledge base and make a map
engine::KnowledgeBase knowledge;
containers::Map map ("profile.1", knowledge);
// create profile fields for a man named John Smith at profile.1
map.set ("name", "John Smith");
map.set ("age", Integer (30));
map.set ("funds", 300.50);
map.set ("photo1", "/images/profile/1/headshot.jpg");
// print the profile
std::cout << "Name: " << map["name"] << std::endl;
std::cout << "Age: " << map["age"] << std::endl;
std::cout << "funds: " << map["funds"] << std::endl;
std::cout << "Photo: " << map["photo1"] << std::endl;
Flexible Maps are abstractions intended to support multi-dimensional arrays and maps. These containers override the [] operator for both integer-based index and string-based accesses. These operators return other Flexible Maps which can either be set to an arbitrary value (such as an integer, string, double, byte buffer, etc.) or subindexed with integers or strings.
Location: madara/knowledge_engine/containers/FlexMap.h
Documentation: FlexMap
FlexMap replaces the old VectorN class which was less intuitive and useful. FlexMap is O(1) for subindexing (e.g., [1][2]) and only incurs an O(log n), with n being the number of keys in the knowledge base, lookup on the first access of an element with an accessor or mutator function (e.g. to_string() or setting the value explicitly). Flex_Map essentially always assumes an index is not a variable lookup until you are setting or getting a value explicitly. The keys function is an O(n) operation that must look through all keys in the Knowledge_Base to find all versions with the prefix set in constructor or set_name function.
Though the subindexing is O(1), developers should keep in mind that each index is a concatenation of the container name with a delimiter (by default '.') and the index, which is essentially 2 concatenation operations.
FlexMap includes a to_container operation that can convert the FlexMap into any supported container for fast O(1) changes later to String, Integer, Double, etc.
// setup includes and namespace alias for convenience
#include "madara/knowledge/containers/FlexMap.h"
#include "madara/knowledge/containers/String.h"
namespace engine = madara::knowledge;
namespace containers = engine::containers;
// create a knowledge base and make a multi-dimensional array
engine::KnowledgeBase knowledge;
containers::FlexMap my_array ("cities", knowledge);
containers::String city_name;
// set location cities.1.2.3.4.name to "St. Louis"
my_array[1][2][3][4]["name"] = "St. Louis";
// set location cities.4.2.3.4.name to "San Francisco"
my_array[4][2][3][4]["name"] = "San Francisco";
// output the value at [1][2][3][4]
std::cout << "My current hometown is " << my_array[1][2][3][4]["name"].to_string () << std::endl;
// output the value at [4][2][3][4]
std::cout << "My current hometown is " << my_array[4][2][3][4]["name"].to_string () << std::endl;
// Flex_Map can create containers to its elements for faster access
may_array[1][2][3][4]["name"].to_container (city_name);
city_name = "Ontario";
std::cout << "St. Louis has been replaced by " << *city_name << std::endl;
Counters are abstractions intended to support aggregate counters across multiple participating agents. Counters provide a solution to the problem of the many writers problem for single variable increments/decrements. What a Counter does under the hood is give each agent a copy of the variable, and the agent only increments/decrements/sets their versions of the variable. When another agent wants to know, the aggregate count, they call to_integer() or to_record().
Location: madara/knowledge/containers/Counter.h
Documentation: Counter
Updating the Counter takes O(1) time. Retrieving a value from the Counter requires O(N) time, where N is the number of agents participating in the counting operation.
// setup includes and namespace alias for convenience
#include <iostream>
#include "madara/knowledge/KnowledgeBase.h"
#include "madara/knowledge/containers/Counter.h"
namespace engine = madara::knowledge;
namespace containers = engine::containers;
// create a knowledge base and make a multi-dimensional array
engine::KnowledgeBase knowledge;
containers::Counter entries ("entries", knowledge);
// resize our entries counter to have 2 variables (2 agents),
// with our id as 0 (the other agent would be 1)
entries.resize(0, 2);
// add to the number of entries that you are maintaining
++entries;
entries += 2;
// other agents across the network may have updated their entry in "entries"
std::cerr << "Total entries in network is : " << entries.to_integer() << std::endl;
Location: madara/knowledge/containers/CircularBuffer.h
Documentation: CircularBuffer
Templated Version: madara/knowledge/containers/CircularBufferT.h
Documentation: CircularBufferT
The CircularBuffer container abstracts a fixed size buffer within the knowledge base with an index that moves along the buffer to indicate where the latest record has been inserted. This container readily supports a producer/consumer model for high-performance delivery of items between one thread and another. Most of the iteration functions in CircularBuffer are templated to work with the Any type. Consequently, if you create custom classes with the for_each_field templated function, then you can insert your own classes into the CircularBuffer and retrieve them with templated CircularBuffer functions. We also provide a CircularBufferT class which is templated to a certain type for the whole class, which provides a bit more type safety.
For lots of examples on usage, see the test_circular and test_circular_any functions in test_karl_containers.cpp.
// setup includes and namespace alias for convenience
#include <vector>
#include "madara/knowledge/containers/CircularBuffer.h"
namespace engine = madara::knowledge;
namespace containers = engine::containers;
typedef engine::KnowledgeRecord KnowledgeRecord;
// create a knowledge base and make a 10 element array that points inside it
engine::KnowledgeBase knowledge;
containers::CircularBuffer producer ("buffer", knowledge, 100);
const containers::CircularBuffer consumer ("buffer", knowledge);
// create 3 records
producer.add (KnowledgeRecord (5));
producer.add (KnowledgeRecord (5.5));
producer.add (KnowledgeRecord ("a string"));
// grab up to 5 of the latest records (will only return 3 with the above)
std::vector <KnowledgeRecord> snapshot = consumer.get_latest (5);
// order of snapshot will be "a string", 5.5, then 5
std::cout << "snapshot contained: \n";
for (auto record: snapshot)
{
std::cout << record << "\n";
}
// grab up to 5 of the earliest records still in the buffer
std::vector <KnowledgeRecord> inorder = consumer.get_earliest (5);
// order of inorder will be 5, 5.5, "a string"
std::cout << "inorder contained: \n";
for (auto record: inorder)
{
std::cout << record << "\n";
}
// resize and then update the consumer
producer.resize (5);
consumer.resize ();
// add some more items to go over the boundary
producer.add (KnowledgeRecord (6));
producer.add (KnowledgeRecord (7));
producer.add (KnowledgeRecord (8));
// grab latest 4
snapshot = consumer.get_latest (4);
// order of snapshot will be 8, 7, 6, "a string"
std::cout << "snapshot contained: \n";
for (auto record: snapshot)
{
std::cout << record << "\n";
}
Location: madara/knowledge/containers/CircularBufferConsumer.h
Documentation: CircularBufferConsumer
Templated Version: madara/knowledge/containers/CircularBufferConsumerT.h
Documentation: CircularBufferConsumerT
The CircularBufferConsumer container consumes items from a CircularBuffer producer. This consumption process is entirely virtual and many consumers can iterate over the same CircularBuffer. The class contains an internal, local iterator and does not interfere with other instances of CircularBufferConsumer. Most of the iteration functions in CircularBufferConsumer are templated to work with the Any type. Consequently, if you create custom classes with the for_each_field templated function, then you can insert your own classes into the CircularBuffer and retrieve them with CircularBufferConsumer. We also provide a CircularBufferConsumerT class which is templated to a certain type for the whole class, which provides a bit more type safety.
For lots of examples on usage, see the test_circular_consumer and test_circular_consumer_any functions in test_karl_containers.cpp.
// setup includes and namespace alias for convenience
#include <vector>
#include "madara/knowledge/containers/CircularBuffer.h"
#include "madara/knowledge/containers/CircularBufferConsumer.h"
namespace engine = madara::knowledge;
namespace containers = engine::containers;
typedef engine::KnowledgeRecord KnowledgeRecord;
// create a knowledge base and make a 10 element array that points inside it
engine::KnowledgeBase knowledge;
containers::CircularBuffer producer ("buffer", knowledge, 100);
const containers::CircularBufferConsumer consumer ("buffer", knowledge);
// create 3 records
producer.add (KnowledgeRecord (5));
producer.add (KnowledgeRecord (5.5));
producer.add (KnowledgeRecord ("a string"));
// grab up to 5 of the earliest records still in the buffer
std::vector <KnowledgeRecord> inorder = consumer.consume_earliest (5);
// order of inorder will be 5, 5.5, "a string"
std::cout << "inorder contained: \n";
for (auto record: inorder)
{
std::cout << record << "\n";
}
// add same three 3 records again
producer.add (KnowledgeRecord (5));
producer.add (KnowledgeRecord (5.5));
producer.add (KnowledgeRecord ("a string"));
// grab up to 5 of the latest records (will only return 3 with the above)
std::vector <KnowledgeRecord> snapshot = consumer.consume_latest (5);
// order of snapshot will be "a string", 5.5, then 5
std::cout << "snapshot contained: \n";
for (auto record: snapshot)
{
std::cout << record << "\n";
}
// resize and then update the consumer
producer.resize (5);
consumer.resize ();
// add some more items to go over the boundary
producer.add (KnowledgeRecord (6));
producer.add (KnowledgeRecord (7));
producer.add (KnowledgeRecord (8));
// grab latest 4 (actually only 3 returned)
snapshot = consumer.consume_latest (4);
// order of snapshot will be 8, 7, 6
std::cout << "snapshot contained: \n";
for (auto record: snapshot)
{
std::cout << record << "\n";
}
// grab latest 4. This will actually return 4 because peek looks at the producer's buffer index
snapshot = consumer.peek_latest (4);
// order of snapshot will be 8, 7, 6, "a string"
std::cout << "snapshot contained: \n";
for (auto record: snapshot)
{
std::cout << record << "\n";
}
Madara's KnowledgeRecord can hold an internal circular buffer. It can be configured with a finite history capacity, and will then keep all updates, local or remote, to it over time, up to that capacity. This feature provides, inherently, most of the features of CircularBuffer and CircularBufferConsumer, however, to provide the independent consumer capability, Madara provdes the NativeCircularBufferConsumer. This class works much like CircularBufferConsumer, but provides only the consume methods for accessing data, and operates on a single KnowledgeRecord, rather than a set of them.
KnowledgeBase kb;
kb.set("foo", 1);
kb.set_history_capacity("foo", 10);
kb.set("foo", 2);
kb.set("foo", 3);
assert(kb.get_newest("foo") == 3);
assert(kb.get_oldest("foo") == 1);
assert(kb.get_newest("foo", 2)[0] == 2);
NativeCircularBufferConsumer consumer1("foo", kb);
NativeCircularBufferConsumer consumer2("foo", kb);
assert(consumer1.consume() == 1);
assert(consumer2.consume() == 1); // Each consume independently
assert(consumer1.consume() == 2);
assert(consumer2.consume() == 2);
assert(kb.get_oldest("foo") == 1); // Underlying record isn't changedWhen creating containers, each constructor supports a madara::knowledge::KnowledgeUpdateSettings instance that provides control over the updating and retrieving of variables from the context. However, no function in the container classes directly interacts with the transport layer.
Container classes are intended to provide fast, efficient, and convenient facades into the Knowledge Base to support object-oriented programming. In order to send these updates to other reasoning agents in the network, functions will need to be called on the Knowledge_Base. In this section of the Wiki, we'll discuss how the containers are intended to be used and when interactions over the network occur.
If the developer does not intend to use the evaluate or wait function on the Knowledge Base and intends to explicitly use MADARA containers, then the main option for communicating changes over the network is with the send_modifieds function. After all relevant changes have been made through the container classes, simply call send_modifieds.
To showcase this concept, we'll use a complete program that sends a profile over multicast.
#include "madara/knowledge/KnowledgeBase.h"
#include "madara/knowledge/containers/Map.h"
// setup convenience aliases
namespace engine = madara::knowledge;
namespace containers = engine::containers;
typedef engine::KnowledgeRecord::Integer Integer;
int main (int argc, char ** argv)
{
// setup a Multicast transport at 239.255.0.1:4150
madara::transport::QoSTransportSettings settings;
settings.hosts.push_back ("239.255.0.1:4150");
settings.type = madara::transport::MULTICAST;
// use the transport constructor
engine::KnowledgeBase knowledge ("", settings);
// lock the context from any external updates until we're done
knowledge.lock ();
// make changes to a
containers::Map map ("profile.1", knowledge);
// create profile fields for a man named John Smith at profile.1
map.set ("name", "John Smith");
map.set ("age", Integer (30));
map.set ("funds", 300.50);
map.set ("photo1", "/images/profile/1/headshot.jpg");
// unlock the context to allow the network to update our context again
knowledge.unlock ();
/**
* send the changes we've made to profile.1 over the network
* These will show up as updates to profile.1.name, profile.1.age,
* profile.1.funds, and profile.1.photo1
*/
knowledge.send_modifieds ();
return 0;
}
The more recommended way to use containers is within functions called from an evaluate or wait statement. There are several reasons for doing this:
- Functions called from an evaluate are atomic and the context is already locked from outside updates
- An evaluate call, by default, sends all modified variables over the attached network transports
- MADARA containers allow for attaching a context through either the
VariablesorKnowledgeBaseclasses, so there is nothing gained by using containers outside of function calls from evaluate
In the next example, we create a function called set_profile that is called from an evaluate statement within the main function.
#include "madara/knowledge/KnowledgeBase.h"
#include "madara/knowledge/containers/Map.h"
#include "madara/utility/Utility.h"
// setup convenience aliases
namespace engine = madara::knowledge;
namespace containers = engine::containers;
typedef engine::KnowledgeRecord::Integer Integer;
// a Map container to reference profile information
containers::Map profile;
engine::Knowledge_Record
set_profile (engine::FunctionArguments & args, engine::Variables & vars)
{
Madara::Knowledge_Record result;
// set profile information for John Smith
profile.set ("name", "John Smith");
profile.set ("age", Integer (30));
profile.set ("funds", 300.50);
profile.set ("photo1", "/images/profile/1/headshot.jpg");
return result;
}
int main (int argc, char ** argv)
{
// setup a Multicast transport at 239.255.0.1:4150
madara::transport::QoSTransportSettings settings;
settings.hosts.push_back ("239.255.0.1:4150");
settings.type = madara::transport::MULTICAST;
// use the transport constructor
madara::knowledge::KnowledgeBase knowledge ("", settings);
// attach the profile Map container to the knowledge base
profile.set_name ("profile.1", knowledge);
// define the set_profile function within the knowledge base
knowledge.define_function ("set_profile", set_profile);
// call the set_profile function and immediately send updates
knowledge.evaluate ("set_profile ()");
/**
* In a real application, we would probably have a control loop
* that the evaluate is called in until some condition is met.
*
* Here we just sleep for a few seconds to make sure the transport
* is given enough time to send out all of the updates
**/
madara::utility::sleep (3);
return 0;
}
If you are looking for code examples and guides, your best bet would be to start with the tutorials (located in the tutorials directory of the MADARA root directory--see the README.txt file for descriptions). After that, there are many dozens of tests that showcase and profile the many functions, classes, and functionalities available to MADARA users.
Users may also browse the Library Documentation for all MADARA functions, classes, etc. and the Wiki pages on this website.
First Steps in MADARA: Covers installation and a Hello World program.
Intro to Networking: Covers creating a multicast transport and coordination and image sharing between multiple agents.
C++ Guide Series
Architecture | Knowledge Base | Networking | Containers | Threads | Optimizations | KaRL | Encryption | Checkpointing | Knowledge Performance | Logging