parameter observers

Parameters' Observers (Developer guide)

Summary

We wanted to add to Shogun the ability to observe classes' member variables and to record their value. This is a handy feature because it would permit to users to get more fine-grained information about what is going on inside specific algorithms. For instance, one would like to observe how the weights of a LARS algorithm change over the training phase, instead of just having the final ones.

We also added the possibility, for some type of data, to serialize the measurements to a Tensorflow event file. These files can be then used together with Tensorboard to generate cool graphs.

The architecture was built using RxCpp.

Implementation

Observed value's structure

Since Shogun's classes contain many variables of many different types, we needed a structure capable of wrapping them. The class ObservedValue was developed to satisfy this need. Information is sent to observers by means of this class. It consists of different information:

• Name of the parameter it contains;
• Type of the parameter it contains (useful only with respect to observers. It is not the type of the contained parameter, for instance, int, float etc.);
• An Any object which stores the actual value in a type-agnostic way;
• A step variable which stores an increasing numerical "index" (it is used for data visualization with Tensorboard);

Parameters' observers

To analyze the data contained in the ObservedValue instance, a series of parameter's observers were built. The base class is ParameterObserverInterface, which contains the virtual methods needed by an actual observer. The most important are:

• on_next(): this is the main method that needs to be implemented in all the observer's subclasses. Here the actual analysis of the ObservedValue happens;
• filter(): the action which will select which ObservedValue will be received by its parameter's observer. The default implementation is to check if the name of the ObservedValue is present inside a list passed to the ParameterObserver constructor;

All the parameter's observers contain also a member variable called m_type. It is an enumeration which identifies which types of ObservedValue they can accept. At the moment, the only types defined are: TENSORBOARD and CROSSVALIDATION. If an ObservedValue has type TENSORBOARD, it means that its Any object will contain data which can be serialized to Tensorflow event files, therefore it can be parsed only by a ParameterObserver which can read TENSORDBOARD objects (for instance, ParameterObserverScalar).

SGObject

Base class CSGObject was also extended with an observable and a custom method to emit ObservedValue objects.

void CSGObject::observe(const ObservedValue value)
{
	m_subscriber_params->on_next(value);
}

Basically, when we want to make a measurement of a value, we have just to build the ObservedValueobject and then to pass it to the observe() method. RxCpp will take care of sending it to all the observers.

To append a ParameterObserver instance to a CSGObject derived class the user has just to call the method subscribe_to_parameters(ParameterObserverInterface * obs) which will subscribe the ParameterObserver instance to the parameters' observable.

void CSGObject::subscribe_to_parameters(ParameterObserverInterface* obs)
{
	auto sub = rxcpp::make_subscriber<TimedObservedValue>(
	    [obs](TimedObservedValue e) { obs->on_next(e); },
	    [obs](std::exception_ptr ep) { obs->on_error(ep); },
	    [obs]() { obs->on_complete(); });

	// Create an observable which emits values only if they are about
	// parameters selected by the observable.
	auto subscription = m_observable_params
	                        ->filter([obs](ObservedValue v) {
		                        return obs->filter(v.get_name());
		                    })
	                        .timestamp()
	                        .subscribe(sub);
}

Since there can be many parameters emitted by a single CSGObject object, we provide also a method to list all the observable parameters.

void CSGObject::list_observable_parameters();

Example

Please give a look at whichever implementation present into src/shogun/lib/parameter_observers to see how to create your own parameter's observer. A simple example could the ParameterObserverCV.h class which is used to observe CrossValidation parameters.