IntrospectionServer and PicklingError: explicitly exclude variables from being pickled

[sotte]

I have some variables in a state which should not be pickled by the IntrospectionServer. This is of interest for me, first of all, because I get an PicklingError right now (it seems that SwigPyObjects can't be pickled), but more generally, because they are too big and I don't want them to be serialized.

I guess I'm looking for something like this:

sis = smach_ros.IntrospectionServer(
    'sis', sm, '/SM_ROOT', 
    exclude={
        BlaState: ["variable1", "variable2"],
        FooState: ["variableX"]
    }
)

Is there a way to do this? If not, could we implement something like this? I'd be happy to help!

[jbohren]

Ah interesting. Yeah, there's no current way to do this. I like your idea, but let me think about which class should be responsible for checking this.

I think the first thing to do, is catch the error, and display a non-breaking error message. I can patch that when I get a minute (the IntrospectionServer is really simple), or you're welcome to submit a pull request.

I like the idea of being able to exclude UserData keys from being pickled, also, since type objects are first-class citizens in python, you could also exclude objects based on their datatype. Even further, you could set a maximum data size for the introspection status messages.

The pickling happens in the IntrospectionServer, but it pickles the whole UserData dict, itself, so pulling apart individual keys would require either copying the structure without the keys, iterating through the keys and pickling them individually, or overriding UserData's __getstate__() method (doc here) and then just pickle the UserData object directly.

What I don't like is specifying state names and excluded UserData keys at the level of the IntrospectionServer. I think if any userdata keys should be excluded from pickling, they should be excluded at the Container (StateMachine, Concurrence, etc.) level (since that's where the UserData actually lives). Otherwise, you end up having to maintain facets of your plan structure in multiple places.

So then the change that I think would make the most sense would be to add a new optional argument to the Container classes (StateMachine, Concurrence, etc), maybe "private_keys" which could then be passed to each container's UserData structure. Then to modify UserData's __getstate__() method to exclude the private keys when it's being pickled.

Something else that could be done, is to overload UserData's __getstate__() so that it can pickle just the part of the userdata structure which is pickle-able automatically.

If you're interested in doing this, it'd be a great addition!

As an aside, the IntrospectionServer is pretty inefficient, and I think it could be improved a lot, in general (see #5).

[sotte]

The pickling happens in the IntrospectionServer, but it pickles the whole UserData dict, itself, so pulling apart individual keys would require either copying the structure without the keys, iterating through the keys and pickling them individually, or overriding UserData's getstate() method (doc here) and then just pickle the UserData object directly.

Either of the pickling method you suggested would be fine. I like the __getstate__() approach, simply because I've never used it yet :)

What I don't like is specifying state names and excluded UserData keys at the level of the IntrospectionServer. I think if any userdata keys should be excluded from pickling, they should be excluded at the Container (StateMachine, Concurrence, etc.) level (since that's where the UserData actually lives). Otherwise, you end up having to maintain facets of your plan structure in multiple places.

The counterargument is, that the State does not need to know about the introspection of the introspection server.
The state would have information (what variable is private) which is only used by the IntrospectionServer.
Is the UserData important for anything else besides the IntrospectionServer (and of course the exchange of data between states)? If not, then I'd say that private variables are only interesting for the introspection server.

But on the other hand, you're right. You have to maintain the states in multiple places.

Something else that could be done, is to overload UserData's getstate() so that it can pickle just the part of the userdata structure which is pickle-able automatically.

How can we know if something is pickle-able?

I like your idea of excluding certain types and limit the size! This feature, I think, should be part of the IntrospectionServer for the same reason as mentioned above. The IntrospectionServer should know about the serialization, not the state.

For this example, I'll just stick to my previous suggestion, just because...

sis = smach_ros.IntrospectionServer(
    'sis', sm, '/SM_ROOT', 
    exclude_userdata={
        BlaState: ["variable1", "variable2"],
        FooState: ["variableX"]
    },
    exclude_types=[ReallyBigClass, int],
    max_size=1000
)

What do you think?

[sotte]

You're not, by any chance, at the RosCon 2013 right now?

[jbohren]

Haha yes, I am.

-j

On Sat, May 11, 2013 at 5:37 AM, Stefan Otte notifications@github.comwrote:

You're not, by any chance, at the RosCon 2013 right now?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/18#issuecomment-17757101
.

Jonathan Bohren
Laboratory for Computational Sensing and Robotics
http://dscl.lcsr.jhu.edu/People/JonathanBohren

[sotte]

so how should we proceed with this?

[jbohren]

I just got back to the states. Have you tried making the modification we discussed above yet?