Python Custom KnowledgeRecord Types
As for 3.3.0, this feature is deprecated. We specifically removed this due to complications with Boost long term support and the requirement of boost filesystem, which was causing issues with moving to UE4 as a simulation environment for GAMS. If you are interested in this feature, let us know, and we will try to prioritize its reinclusion. However, there is no planned resurrection for this tool at the moment.
This feature lives in v3.2.3, at the latest.
Madara's KnowledgeBase and KnowledgeRecord support several useful types natively: integers, doubles, vectors of both, strings, and blobs. Where appropriate, it is best to use those types. They will be accessible by any KnowledgeBase, and a variety of useful functions will work with them, including overloaded math operators and indexing operations. For types which cannot easily be represented by those native types, KnowledgeRecord, and thus KnowledgeBase, can store an Any type, provided by Madara, which can in turn store custom types directly.
- Supported Types
- The Any Class
- Accessing Cap'n Proto message Any Objects
- Manipulating Native C++ Any Objects
- Any with
KnowledgeRecordandKnowledgeBase
There are two main kinds of objects supported by Any: native C++ types, and
Cap'n Proto messages. Supported native C++ types
must be default constructible, copy constructible, and support serialization
via either the Cereal Library, or via
Madara's translation system to a compatible
Cap'n Proto message struct. Note that this translation system is an alternative
to using Cap'n Proto generated types and schemas directly.
Types must be registered with Madara using a string "tag" which identifies them. This tag should be used on all systems and platforms to refer to the same kind of data, but each might register different types, as long as they have the same serialized format.
To be used from Python, a native C++ custom type must be
registered within C++ code,
or with the predefined Any.register* functions provided in the Python port
for various commonly used types.
Cap'n Proto message schemas can be registered within Python code, and thus do not rely on C++ registration code. Madara uses the pycapnp Python port of Cap'n Proto. Be sure to familiarize yourself with its documentatation, and follow its installation instructions.
Native C++ custom types may only be defined in C++, but can be made available to Python via a shared library. To load a shared library, use the ctypes.cdll.LoadLibrary() or ctypes.windll.LoadLibrary calls. To ensure successful linking, ensure madara.so is loaded first. For example:
from madara.knowledge import *
import ctypes;
ctypes.cdll.LoadLibrary("libdatatypes.so"); # replace with your library's name
Note that if Python is used in a process that uses native C++ types backed by Cap'n Proto serialization, Python code should use the native C++ type support, not attempt to use Cap'n Proto directly.
To store and load Cap'n Proto messages to and from Any objects in Python, they
must be registered with the Any.register_class() static method. It takes
two arguments. The first is is a string "tag" which will be used to portably
identify the type across all systems using it. The second is the Cap'n Proto
generated type, which can be obtained by calling capnp.load() on a schema
file. For example:
using os
using capnp
geo = capnp.load(os.environ["MADARA_ROOT"] + "/tests/capnfiles/Point.capn")
Any.register_class("Point", geo.Point)
Any.register_class("Pose", geo.Pose)
Note that you must register the python class even if the tag is registered in C++ or Java already. You will not be able to use any Cap'n Proto message in Python with Madara that hasn't been registered in Python.
The Python port provides a version of the C++ implementation of Any.
To create an Any holding a Cap'n Proto message, construct it with a pycapnp
builder, which you can obtain using init_message(), and modify with accessors:
msg = geo.Point.new_message();
msg.x = 2
msg.y = 4
msg.z = 6
any = Any(msg)
To create an Any holding a native C++ type, construct it with the name of a registered class:
/**** C++ code ****/
struct Point {
double x, y, z;
};
// Some details omitted, see C++ version documentation for more.
// Register the above
Any::register_type<Point>("point");
/**** Python code ****/
// Register C++ type std::map<std::string, std::string> with tag "smap"
Any.register_string_string_map("smap")
point = new Any("point")
map = new Any("smap")
These Any objects now hold ownership of a default constructed C++ object.
All Any objects will be garbage collected automatically by Python. No explicit
memory management is required (in contrast to the Java port).
The Any object provides a reader() method which will return a pycapnp
reader object for the held object. If the held object is not a registered
Cap'n Proto message, an exception will be thrown. For example:
msg = geo.Point.new_message();
msg.x = 2
msg.y = 4
msg.z = 6
any = Any(msg)
reader = any.reader()
assert (reader.y == 4)
To access fields and contained elements of the held object, Any implements the __getattr__ special function, allowing access like any other attribute:
xref = point.x;
In the above example, xref is an instance of the AnyRef class. The AnyRef acts much like Any itself, and has most of the same methods, but doesn't own the object it points to. It also does not extend the lifetime of that object. It acts like a C pointer. If the Any it was constructed from is destroyed, it will be left dangling, so be careful keeping these objects long-term.
You can modify the data held by the C++ object pointed to by an AnyRef using the assign() method. You can also set fields like attributes:
xref.assign(2);
point.y = 3.5;
An AnyRef (or Any itself) can be converted to various types, the ones supported natively by KnowledgeRecord. Typically, you'll use to_integer(), to_double(), or to_string():
double x = xref.to_double(); // == 2
double y = point.y.to_double(); // == 3.5
These are also accessible with the int(), float(), and str() standard python functions.
You can use normal indexing to access elements of the held type, if it supports indexing by integer or string:
fooRef = map["foo"]; // Creates a new entry in the map, fooRef holds an AnyRef
map["bar"] = "world";
fooRef.assign("hello");
foo = str(fooRef); // == "Hello"
bar = map["bar"].to_string(); // == "World"
You can also call size() to call that method on the held object, if it supports it, and list_fields() to get a list of field names supported by ref().
You can store Any within KnowledgeRecord, and in turn into KnowledgeBase. To store an Any in a KnowledgeRecord, simply construct the record with the Any object:
pointRecord = KnowledgeRecord(point);
If you have a KnowledgeRecord, you can get its contents as an Any using the to_any() method:
p = pointRecord.to_any();
This will work on any KnowledgeRecord. If it contains an Any, you will get a copy of it. If it doesn't, you will get a copy of the stored data held within a new Any. To access that data, be sure to register the appropriate types using the Any.register* static methods.
You can store an Any into a KnowledgeBase using the set() method:
kb = new KnowledgeBase;
kb.set("pointKey", point);
To get an Any from the KnowledgeBase, use the get() method to retrieve a KnowledgeRecord as usual, then call to_any() on it.