-
Notifications
You must be signed in to change notification settings - Fork 374
callbacks
Significant parts of our stack use callbacks, a pattern which is not well defined or standardized for use with Python. In order to maintain consistency, we enforce the following conventions on callbacks:
As callbacks are not a widely recognized Python pattern, we do not wish to expose them to users. Furthermore, we do not wish to consume and execute ANY user-provided code for safety reasons. Thus, all user-facing APIs must be callback free. In order to present an asynchronous API, use the asyncio library in Python 3.5+. We do not support asynchronous APIs for Python 2.7 at this time.
def foo(callback):
# do something
try:
callback()
except: # noqa: E722 do not use bare 'except'
pass
def bar():
def on_complete():
print("completed")
foo(callback=on_complete)
if __name__ == '__main__':
bar()In the above example, the function foo takes a callback as a parameter, and then calls the callback upon completion of foo.
The callback is executed in foo within a try/except block, because foo is executing code unknown to it, which could throw an exception, and we must be able to recover from that (or choose to raise an exception in response).
Furthermore, we must break Python best-practices and use a bare except statement, because foo has no knowledge of what types of exceptions could be thrown by the callback. Thus we add the following comment to suppress our linter:
# noqa: E722 do not use bare 'except'If the callback is optional, as in the following example, the callback must be checked prior to invocation:
def foo(callback=None):
# do something
if callback:
try:
callback()
except: # noqa: E722 do not use bare 'except'
raise FooError("callback raised exception")
else:
pass
def bar():
def on_complete():
print("completed")
foo(callback=on_complete)
if __name__ == '__main__':
bar()This check, following a "Look Before You Leap" (LBYL) pattern, is a necessary diversion from the more Pythonic "Easier to Ask Forgiveness than Permission" (EFAP) pattern. The reason is, if we wollowed EFAP and barrelled forth with trying to execute callback without checking it, a TypeError would be raised. However, we can't just except the TypeError and then pass, since if callback could also potentially raise a TypeError, and in that case we want to raise a FooError. Thus, there is no way to effectively use the EFAP pattern, and we must instead use LBYL to check whether or not the callback is set before trying to execute it.
Callbacks should generally have the format on_<some_event>, e.g. on_publish, on_message_received, on_complete, etc.
Note well that the function provided as a callback to foo, on_complete does not contain the word "callback".
By convention the term "callback" only applies to a consumed function, NOT a provided one.
Thus, foo refers to the function it consumes as a "callback", but bar does NOT refer to the function it provides to foo as a "callback".
The rationale is to avoid confusion in the following, more complex case:
def foo(callback):
# do something
try:
callback()
except: # noqa: E722 do not use bare 'except'
pass
def bar(callback):
def on_complete():
try:
callback()
except: # noqa: E722 do not use bare 'except'
pass
foo(callback=on_complete)
def buzz():
def on_complete():
print("completed")
bar(callback=on_complete)
if __name__ == '__main__':
buzz()As you can see in this example, bar both consumes a callback function, and provides a function for foo to use as a callback. If both were referred to as a "callback" this would be very confusing.
Sometimes, a callback must be set as an attribute on an object rather than provided as an argument to a function/method. Paramater callbacks are limited in that they only work when being set for a given initiated action with a function/method. But sometimes, we need to handle external events. Thus, the need for handler callbacks.
class MessageBox(object):
def __init__(self):
self.on_message_received_handler = None
self.messages = []
def add(message):
self.messages.append(message)
if self.on_message_received_handler:
try:
self.on_message_received_handler()
except: # noqa: E722 do not use bare 'except'
raise MessageError("callback raised exception")
else:
pass
def on_message_received():
print("message received!")
if __name__ == '__main__':
message_box = MessageBox()
message_box.on_message_received_handler = on_message_received
threading.Timer(5.0, message_box.add("hello!")).start()In this example, a message of "hello!" is added to the MessageBox instance every five seconds. When a message is added, via the add method, it calls the on_message_received_handler attribute set on the instance of MessageBox.
This pattern is more complex, and should be avoided whenever possible. Even in the above example, a simpler implementation would have been to add a callback paramater to the add method. This is likely only ever truly necessary for events triggered by external libraries, which have APIs that limit your ability to add callback parameters (e.g. Paho).
As a naming convention, we use the term "handler" instead of "callback" under this model to differentiate the two approaches.
All other rules and conventions that apply to parameter callbacks also apply here with handler callbacks.
A good rule-of-thumb is: If you're writing a function that responds to a single user-initiated action, it should be a callback.
This pattern happens in too many APIs and should be avoided:
class MessageSender(object):
def __init__(self):
self.on_message_sent_handler = None
def send(message):
# does something to send the message
# calls self_on_message_send_handler when the message has been sentThe reason that this is an anti-pattern is because this forces the calling code to have a single function which handles completion of any and all send operations. If a call to send is added to the code, then the on_message_sent_handler needs to be updated to handle completion of that send operation (in addition to all the other send completions it already handles). If many messages or many types of message can be sent, this function could become quite complex.
A better pattern is to make the on_message_sent_handler functionality into a callback:
class MessageSender(object):
def __init__(self):
pass
def send(message, callback):
# does something to send the message
# calls callback when the message has been sentThis way, the calling code can specify a piece of code that runs only when a single message has been sent, and that code doesn't have to worry about any other messages that might be sent.
Another good rule-of-thumb: Don't assume that a handler is only going to be called once.
Consider this case:
class ReconnectingSender(MessageSender):
"""
Object that adds reconnect functionality to the MessageSender class. If the connection drops, this object will re-connect using some internal retry rules.
"""
def __init__(self):
# handler that gets called whenever the network gets connected
self.on_network_connected_handler = None
def connect(self):
# connects the network
def send(self):
# sends a messageThe following code would illustrate this anti-pattern:
class Application(object):
def __init__(self):
self.sender = ReconnectingSender()
def connect_and_send(self):
def on_connected():
# send a message after the network is connected
sender.send("Application Started")
self.sender.on_network_connected_handler = on_connected
self.sender.connect()If the application writer intended the "Application Started" message to be sent once, and only once (when connect-and-send is called), this code does not work. Rather, this code would send out the "Application Started" message every time the network gets connected, whether from a call to connect_and_send or from a re-connect that the ReconnectingSender object initiates as the result of a dropped connection.