Idea: synth from 2d whiteboard

[kolibril13]

I recently played with the tldraw whiteboard canvas, and found quite a few use cases for the bidirectional communication from the whiteboard to python, e.g. a plot annotation tool, strokes based on python calculations and an image segmentation tool:

Repo here: https://github.com/kolibril13/jupyter-tldraw

Next idea: How about making a 2d whiteboard based synthesizer with ipytone?
E.g. the x coordinate of a stroke could be connected to one slider, and the y coordinate to another slider.
Here is an example code:

import ipyreact
from traitlets import Unicode,observe
from IPython.display import display
class TldrawWidget(ipyreact.ReactWidget):
    my_coordinates = Unicode("Hello World").tag(sync=True)
    
    @observe("my_coordinates")
    def _observe_my_coordinates(self, change):
      print("Stroke:" , self.my_coordinates)
    _esm = """
    import { TDShapeType, Tldraw } from "@tldraw/tldraw";
    import * as React from "react";

    export default function App({ my_text, set_my_coordinates }) {
      const handleChange = (e) => {
        let my_current_stroke = e.selectedIds[0];
        let my_object = e.getShape(my_current_stroke);

        if (my_object === undefined) {
          console.log("no object selected");
        }
        if (my_object !== undefined) {
          let my_points = my_object.points;
          if (my_points !== undefined){
            console.log(my_points.length);
            set_my_coordinates(my_points.toString())
          }
        }
      };

      return (
        <div
          style={{
            position: "relative",
            width: "800px",
            height: "350px",
          }}
        >
          <Tldraw  onChange={handleChange} />
        </div>
      );
    }

    """


tldraw = TldrawWidget()
tldraw

and the stroke coordinates can be accessed in a next cell by typing

tldraw.my_coordinates

[benbovy]

Yes I agree there's an infinite number of possibilities worth exploring in making sound & music with a 2d whiteboard / canvas! Using jupyter-tldraw with ipytone has great potential!

XY pads are often used as UI elements in audio/music software or even as physical devices (the kaoss pad is one of the most popular).

I did some experiments using ipycanvas, like this (silly) example: https://twitter.com/benbovy/status/1365251000643293185

There's also more advanced things we could do with a 2d whiteboard I think, e.g., inspired by beatsurfing. I've made a simple example at the bottom of this notebook (based on ipycanvas too): https://github.com/benbovy/ipytone-jupytercon2023/blob/main/02-Songification.ipynb

[kolibril13]

Great to see that you already had thought into that direction!
Thanks for the links, especially Beatsurfing sounds super cool.
I also thought about the theremin in that context

[kolibril13]

one more inspiration the Seaboard:
https://www.youtube.com/watch?v=6SCug5kUsBs