Unify reactive APIs across HoloViz

[philippjfr]

Across HoloViz we have been slowly converging on a powerful set of reactive constructs that make it possible to build reactive data pipelines and UI components. In particular we have a set of three core reactive objects that we support.

Reactive objects

Reactive objects represent a reference to a value that can be updated dynamically and then drive reactive computations. Currently we have three such core concepts:

• param.Parameter: A parameter is the basis for all reactive computations and represents a reference to a value that can change and notify downstream consumers of that value.
• param.bind (formerly panel.bind): A function or generator that can update dynamically. I propose we call these reactive functions/generators.
• param.reactive (formerly hvplot.interactive): A proxy for an object that can update dynamically. I propose we call these reactive expressions.

In addition to these core constructs we treat certain other objects as dynamic references.

• panel.Widget: A widget is treated as equivalent to it's value parameter
• ipywidgets.Widget: An ipywidget is wrapped in a Parameterized which reflects it's value traitlet.

Reactive objects are useful on their own and can be used to drive computation and logic OR they can be used as inputs to a visual pipeline to be rendered by Panel or HoloViews.

Renderers

Panel and HoloViews are powerful rendering engines for plots and UI elements that (imperfectly) support dynamic references. In particular we have a variety of ways to take the reactive constructs and turn those into displayable components. Before we cover all the ways let's summarize the potential consumers of our reactive objects:

• HoloViews DynamicMap: A DynamicMap can consume the output of a function and render HoloViews objects rendered by it. They can be generated as follows.
• Panel components: A Panel component now supports accepting a reactive object as input to its parameters, meaning that they automatically update the parameter and therefore the visual representation when the provided reference changes.
• Panel Function/Method Wrappers: Panel also offers a set of components that wrap a reactive function and replace (or update) the UI when the function's output changes.

The problem we face here is that these objects are pretty inconsistent in the way they handle the reactive objects that can serve as their inputs.

HoloViews

Let's start for example with the DynamicMap. It long preceeds most of the reactive concepts and therefore has multiple ways of binding a dynamic reference:

Streams

HoloViews Streams preceed all reactive components in the HoloViz ecosystem. They hold parameters and have a way of notifying subscribers of the changes. At this point this is extremely duplicative of all the other mechanisms we have in HoloViz and does not play well with any of it. To construct a DynamicMap from a stream you do the following:

def function(x, y):
    return hv.Points([(x, y)])

hv.DynamicMap(function, streams=[Tap()])

This binds the parameters of the Tap stream to the keyword arguments of the function.

bind

A much more explicit way to bind parameters to a DynamicMap is bind:

def function(x, y):
    return hv.Points([x, y])

tap = Tap()
hv.DynamicMap(param.bind(function, x=tap.param.x, y=tap.param.y))

This is much closer to the way the rest of HoloViz works.

Element.apply

HoloViews also offers a .apply method which allows taking a static Element and dynamically applying options or other operations to it, e.g.:

size = pn.widgets.IntSlider(start=10, end=100)

hv.Points(...).apply.opts(size=size)

Creates a DynamicMap to dynamically apply the size.

hvplot.interactive (param.reactive)

hvPlot .interactive or param.reactive pipelines can terminate in a DynamicMap by calling .holoviews().

Summary

There are a number of ways of creating reactively updating objects (DynamicMap) in HoloViews. However there is little consistency:

• streams: Old API completely unrelated to everything else we do in HoloViz
• bind: Good that HoloViews can consume these functions but why can't we create a DynamicMap directly from a parameter or a param.reactive pipeline.
• .apply: This is great, we can directly consume any reactive objects as arguments.
• hvplot.interactive: Terminating methods are confusing and we should decide whether we prefer hv.DynamicMap(<reactive_object>) or a terminating method (or maybe both).

Panel

Panel now has two main ways to respond to reactive changes:

1. Passing a reference to the constructor
   • pn.<component>(<ref>)
   • pn.<component>(parameter=<ref>)
2. Wrapping a bound function in a ParamFunction wrapper
   • pn.panel(param.bind(...)) (or pn.param.ParamFunction(param.bind(...)))

This is more consistent than HoloViews but there are still gaps here.

Summary

We have a number of ways of constructing reactively updating UI components and we are not consistent about which we support:

Passing by reference

Passing a reactive object by reference is probably the cleanest and most readable approach, i.e. pn.pane.Markdown(str_param, height=int_param) is very readable, similarly hv.Curve(...).apply.opts(line_width=widget) is too. The only thing missing here is the ability to update a reference if needed, e.g. panel_obj.param.update(width=new_widget) should unbind any old references and bind the new reference.

Terminating methods vs Constructors

We are highly inconsistent in our support for terminating methods and passing of references to a constructor, e.g. pn.panel(param.reactive(...)) and param.reactive(...).panel() will both be supported but hv.DynamicMap(param.reactive(...)) or param.bind(...).panel() will not as of today.

Actions

We need to come up with a way forward here that unifies our stories and makes the APIs symmetric. Some important questions to address before then are:

• What do we call the "reactive objects" described above?
• Do we like/prefer "terminating" methods and is that even the right terminology?
• Do we want all dynamic renderers (i.e. DynamicMap and Panel objects) to accept all reactive objects?

@holoviz-developers please think about this very deeply because we are fairly close to finally unifying our "reactive" story but there's a number of clear gaps that stop us from telling the story cleanly.

[maximlt]

Thanks for that post, thinking about all of that is timely! I haven't come up with any solution yet, still thinking (it's a lot!), I just have for now a couple of precision/suggestions.

Reactive objects

param.Parameter is indeed the basis of all of the reactivity, I read somewhere that sort of thing can be called reactive dependency which I quite liked. I also like to think about it as an input to a computation graph.

There are a few differences in how updates are handled:

• param.Parameter: updating the value doesn't trigger by default, you need to register callbacks (with e.g. .param.watch, @param.depends)
• param.bind and @param.depends: updating reactive dependencies doesn't trigger by default, you need watch=True
• param.reactive (formerly hvplot.interactive): updating reactive dependencies triggers by default, there's no way (yet? is that even desirable?) to make it just a "declarative reactive" (yuk) expression

So param.bind and @param.depends are purely declarative by default, they just express that a function/method depends on some other reactive inputs (they are really meant to be leveraged by some other system, e.g. a GUI library like Param, and we should probably document how). Without watch=True, can these be even called reactive? They don't react much! With watch=True (I dislike pn.bind(..., watch=True)) they effectively become reactive.

I feel that there's a need to align somehow @param.depends, param.bind and param.reactive on this front. .param.watch is low-level and clear enough to me.

Renderers

The only think I'd like to say for now is that we have seen that the gap between Panel and HoloViews/hvPlot is big enough for users to have a hard time understanding, or just getting to know about, DynamicMaps, leading to sub-optimal data viz apps and spreading of an anti-pattern. So whatever we do should not make that gap bigger, or make it easier for Panel users to do the wrong thing.

[jlstevens]

Good summary of the situation but I don't see how we can come up with yet another (supposedly more unified) API without making the situation worse. Plenty of code exists out there using all the different approaches and often mixing them together and so we can't get rid of anything without breaking a lot of existing code. While it would be nice to have one way to do everything, in practice I expect this would just add one more way of doing things to the mix.

For now I would point out that in your example above, I would use the following which is also supported:

def function(x, y):
    return hv.Points([x, y])

tap = Tap()
hv.DynamicMap(function, streams=dict(x=tap.param.x, y=tap.param.y))

I would also say that I agree that passing by reference is probably the best approach - of course that wasn't an option when a lot of these APIs first evolved which is why things became so confusing. And of course, passing by reference is also tricky at times e.g having to use strings to reference parameters in param decorators.

[droumis]

It's encouraging that this discussion is happening and I hope it leads to cleaning up the reactive story. Creating reactive dependencies always trips me up, especially when I'm working across packages.

Here's what I think should be possible at a high level with pseudocode:

I want to naturally use a widget-like object as an input, without fumbling with streams, DynamicMap, apply, opts, watch, bind, 🤯 . Additionally, I want to use valid interactions on one object as dynamic and explicit input to another object, and I don't want to have to use any 'terminating' methods.

slider  = Slider(start=1, end=10)
scatter = Scatter(data, marker_size=slider)
table = Table(reactive(data).sel(x=scatter.tap.x))

The story should be that everything is widget-like - interactive and subscribable.

[jlstevens]

In that example marker_size is not something that would ever be supported by hv.Scatter. However, this suggestion is something that I think would be entirely appropriate for hvplot to handle.

Instead of trying to have a single story for reactivity everywhere, I would focus on having a single, powerful story for hvplot if we want it to be the entry point for most people (at least when it comes to plotting).

[maximlt]

hvPlot supports passing widgets as arguments. Although I'm pretty sure months ago we said that it should be deprecated (yet another way...): https://hvplot.holoviz.org/user_guide/Widgets.html#using-widgets-as-arguments

Also works if you first make it interactive, in which case the widget is displayed automatically (yet another way!):

I liked Demetris' example, I think we should come up with many more examples and see how we'd like to write them (@droumis in that particular example, I wonder .sel(x=scatter.tap.x would work, you need to define a hit radius right?).

[droumis]

@maximlt.. hmm maybe instead of tap, a more useful example would be the following, where the .selection.x is the x index of the data currently selected with a selection-type tool.

slider  = Slider(start=1, end=10)
scatter = Scatter(data, marker_size=slider)
table = Table(reactive(data).sel(x=scatter.selection.x))

[maximlt]

I'm sure there are a million other ways to write this:

import pandas as pd
import numpy as np

import hvplot.pandas
import holoviews as hv
import panel as pn

pn.extension('tabulator')

df = pd.DataFrame(np.random.random((10, 3)), columns=list('abc'))

slider = pn.widgets.IntSlider(value=1, start=1, end=30, step=5)
scatter = df.hvplot.scatter('a', 'b', tools=['box_select'])
pn.Column(
    slider,
    scatter.apply.opts(size=slider),
    df.interactive().iloc[hv.streams.Selection1D(source=scatter).param.index]
)

Not so many lines of code compared to Demetris' example. Probably quite representative of normal users code, mixing concepts from panel/holoviews/hvplot and having to visit all these sites and copy/paste stuff to get something working.

[philippjfr]

@jbednar and I have discussed the idea of accessing streams via an accessor on HoloViews components for a long time, i.e. <element>.events.selection.indexes would automatically create a Selection1D stream attach it on the element and then make it available going forward. At some point I even had a prototype. I'm strongly in favor of this and it would be a very small amount of effort.

[philippjfr]

As for the overall discussion about HoloViews and reactivity, I do think to some extent that getting rid of DynamicMap and making individual elements reactive is only way we're going to get to a sensible "reactive" story in HoloViews. Except I may tweak your suggestion to keep the options separate: hv.Scatter(<reactive_data>).opts(size=widget). My feeling on DynamicMap is simply that it's an additional, very confusing layer that gets in the way of having a declarative and reactive specification of a plot. Writing callbacks (even if they are declarative/reactive callbacks as opposed to imperative callbacks) is a major barrier to readability and while hvPlot can largely abstract away the API, you still end up returning a DynamicMap to the user which is very opaque and confusing in most cases. This certainly would be quite a long term goal though.

[philippjfr]

hvPlot supports passing widgets as arguments. Although I'm pretty sure months ago we said that it should be deprecated (yet another way...)

The main concern there was that passing widgets could change the return type from a HoloViews component to a Panel component. This indeed should not be allowed, i.e. we should only allow passing widgets for options that do not require Panel to fully re-render the element type.

[ahuang11]

Just going to drop this here if it's helpful; my attempt in enumerating all the ways to interact with HoloViz objects:

• hv.DynamicMap
  • streams
  • bind
• apply.opts
• pn.bind
• param.watch
• jslink / link
• link_selections
• param/pn.depends
• hvplot.interactive

[jbednar]

I would focus on having a single, powerful story for hvplot if we want it to be the entry point for most people (at least when it comes to plotting).

I don't think that necessarily addresses anything; hvPlot returns a DynamicMap in many cases, and if people need to work with DynamicMaps, unless the story around DynamicMaps and streams is cleaned up, the complexity will leak through to the end-user's abstraction level.

[ahuang11]

Once this gets implemented, it'd would be a nice test to rewrite the following notebook to get a feel for the new implementation:
https://pydeas.readthedocs.io/en/latest/holoviz_interactions/tips_and_tricks.html

The notebook uses a variety of interactions through DynamicMap, streams, bind, param.watch, apply.opts.