a bit of coloring

In a bit of thai tunes, an algorave event at Museum Siam that featured algorithmic art and live coding, with visuals projected onto the museum’s facade, we implemented an interactive experiment that allowed visitors to color the museum facade using their mobile phones.

Mobile phone	Projection

This show has 3 phases. In the first phase, visitors are invited to color the museum facade.

phase1.mp4

In the second phase, visitors can choose an icon to show, and the previously-selected icon will pop out. In the final phase, each color corresponds to a note. A mallet sound will be played based on the color of the facade, accompanied by me playing the piano (using a computer keyboard).

phase3.mp4

Tech stack

This project took me (@dtinth) about 6 hours to make. It was created on the same day as its premiere. Without these tools, this project would not have been possible in such a short time frame:

• Firebase Realtime Database for the backend. This saves a lot of time as this project is implemented entirely on the client-side, with no server-side code. Firebase SDK also manages auto-reconnection (on unreliable internet connections) and conflicting updates, which would otherwise require significant effort to implement manually using WebSockets.

• GitHub Copilot for AI-assisted code completions. GitHub Copilot completed most of my code. When I want to do something, I already know the high-level details, but I didn’t remember the exact API signature. Without Copilot, it would cost a lot of time having to look up the correct API signature in the docs. For example, I entered const lpf =, and it knows I need a low-pass filter, so it completed the remaining code: const lpf = ctx.createBiquadFilter(); lpf.type = 'lowpass';.

• Remix (SPA mode) for an easy-to-use React app setup with file-based routing.

• Vite for a remarkable hot-reloading experience, tightening the feedback loop.

• Nano Stores for state management. It’s easy to use and doesn’t get in the way.

• Tailwind CSS for styling. It helps me to iterate very quickly on the visual aspects of the project.

• Figma for creating image assets and measuring the sizes and positions of elements.

This is a screenshot of my Figma file at the end of the project:

Projection

Museum Siam’s Wikipedia page has a photo of the facade. The first thing I did when creating the scene was to create a debug mode (toggled with a shortcut key) that would overlay the facade image on top of the visuals (using mix-blend-mode, simulating a projector). This step helped me get visual alignment with the facade. This debug mode later became useful when aligning visuals on the actual facade. It helps the projectionist adjust the projector without having to guess where the visuals should be.

Debugging off	Debugging on

The projector accepts a 1920x1080 (16:9) signal via HDMI, however, the facade has a different aspect ratio of 36:25. The projector squishes the image to fit the facade. To compensate for this, my visuals have to be stretched to 1920x1080. This is done using CSS transforms.