sonyx (Sound-based tOols for uNderstanding of software sYstems through eXploration) is a toolkit for exploring software systems through sonification in Squeak/Smalltalk. It has initially been developed within the course "Sonic Thinking - Methods of Working with Sound" and has been evaluated and extended within the course "Neurodesign Lecture - Designing for Empathy in Business Contexts" offered by Julia von Thienen from the neurodesign group @ HPI. For more information, please refer to theoretical background.
The main idea of sonyx is to empower developers to understand software systems by listening to particular interesting aspects in the source code. To do this, developers can create sound probes on the fly for any expression in any method in the system. Whenever this expression is reached during the system execution, a user-defined sound is played. Developers can customize and combine these sounds or even configure them to dynamically reflect the state or result of the expression.
This project is based on Babylonian Programming/Smalltalk and Sandblocks.
Make sure to turn your sound on!
SonyxDemoMorph.mp4
 |
 |
Browse all screenshots and screencasts
A ready-to-use Squeak image is available in the latest release.
Extract the zip file and execute the squeak.bat file on Windows, the squeak.sh on Linux, or run the app bundle on macOS.
If you wish, you can also set up sonyx yourself:
-
Install the latest Trunk updates for Squeak.
-
Open a workspace and evaluate the following:
Metacello new baseline: 'Sonyx'; repository: 'github://LinqLover/sonyx:master'; load.
If you are new to Squeak/Smalltalk, I recommend you learn a few basics about it before exploring this project. Squeak is a programming environment that is quite different from many other contemporary solutions. Some aspects of its main philosophy are that everything is an object, including the environment, the compiler, and even all tools in the system, as well as that the user can always interact with or modify all of these objects. As a consequence, you will need to have a basic knowledge of the Squeak tools in order to understand this project or to browse its implementation.
The book Squeak by Example provides a pretty good introduction to Squeak (for this project, you should refer to the Trunk version/SBE-Trunk.pdf from the latest preprint).
It is recommended that you skim through the sections in chapter 1 ("A quick tour of Squeak") but at the very least read section 1.6 ("The system browser") and section 1.7 ("Finding classes").
In the ready-to-use image (see download and installation above), you will find a link to the SonyxDemos class.
When browsing this class, you can select any of the demo messages, select the last line of the method comment (which is denoted by the quotes), and press Cmd + d to run this demo.
To inspect how an example works, select a class or method in its example of interest (for instance, SonyxDemoStream or sorted2) and press Cmd + b resp. Cmd + m to browse it resp. to look up its implementation.
You can define your own sound probes in any method you want.
For example, after opening the demo morph example, browse to SonyxDemoMorph >> #mouseEnter:, select an expression such as self, do a yellow-button click (usually, this corresponds to your right mouse button), and choose "add sonyx probe" from the menu.
Now a probe annotation is displayed that you can use to configure the sound.
Use the prehear icon to listen to it.
Whenever you now invoke this method (in this example, move your mouse over the demo morph to do that), this sound is played.
To customize the sound, type into the probe annotation or click on any part of it to change it.
For instance, click on the default squeakSound (#horn) to replace it with another sound (maybe #croak).
Afterward, press Cmd + s to save the sound probe and trigger it again.
You can add further parameters to the sound by selecting the entire SonyxSound expression with a click and pressing Ctrl + i (insert).
Click into the msg field to choose a parameter key (for instance, balance:).
In the next field, you can enter a value for this parameter (e.g., 0).
Save the probe again and trigger the sound to listen to the difference.
Your sound can also dynamically reflect the context of the method invocation.
To do so, replace the parameter value with any Smalltalk expression that refers to the method context (e.g., anEvent position x from: self left to: self right).
If you now trigger the sound probe again, you can hear the difference based on the circumstances of the method invocation (e.g., at which position your mouse cursor has entered the demo morph).
To retrieve an overview of all currently playing sounds, open the Sonyx Monitor from the main docking bar on top of the screen in the Apps menu. It displays a list of all sound probes that are defined in the system. Watch it while triggering some sounds. You can select any sound probe to read the sound definition, browse the method where the probe is defined, disable/enable the probe, remove it, or prehear the sound.
At the current stage of this project, occasional hick-ups cannot be precluded. In the event that you should stumble into any problems, next to contacting me an option is to check the following guide:
- To stop all running sounds, press Cmd + dot.
- If the sound monitor highlights no longer playing sounds, try the fixup options from the "fixup (in case of trouble)" button on the top.
- If you cannot hear any sounds, try to stop all runnings sounds first. If this does not help, open the sonyx monitor, press the fixup button, and choose "Forget all errors".
- If any other terrible things have happened that stop you from using sonyx as expected, just close Squeak without saving the changes and try your luck again.
In any case, if you find any defect that you are able to reproduce, I will appreciate your bug report.
The package structure of this project is organized as follows:
| Package | Description |
| BaselineOfSonyx | Package metadata. Contains scripts to set up the deploy image and fill it with an initial welcome text. |
| Sonyx-Core | Contains core functionality of sonyx and integration into the Squeak system. Provides the sound API and a set of sound samples. |
| Sonyx-Demo | Contains several examples that demonstrate the features of sonyx.
Browse SonyxDemos to see an overview of all of them. |
| Sonyx-Tests | End-to-end tests for sonyx. |
| Sonyx-UI | View logic for the facilities implemented in the Core package. Includes sound annotations, a sound monitor, and integrates Sandblocks into the probe annotations. |
| Sonyx-Study-Base | Framework for running the user study. |
| Sonyx-Study-RVV | LED visualizations for a visual control condition in the study, based on Runtime Value Visualizations. |
| Sonyx-Study-Solutions | Model solutions to the programming tasks for the user study. |
| Sonyx-StudyTasks | Materials for the programming tasks for the user study. |
We have conducted a controlled experiment with 6 partipants to quantitatively evaluate the use of auditory display in exploratory programming. All details on the design and the outcomes of the study are described here.
There is so much more to tell about this project! For all the other details, including important related work and underlying goals of sonyx, please refer to the wiki. There are also acknowledgments to many kind people. Special thanks go to Julia von Thienen for the statistical evaluation of the user study as well as Tom Beckmann (@tom95) and Patrick Rein (@codeZeilen) for their unremitting support and close cooperation to get sonyx integrated into Babylonian and Sandblocks!
Please also refer to UPSTREAM.md for a list of further implementational work that has been done in the context of this project.
This is only a prototype, and there are still plenty of other ideas that have not yet been tried out. You can find a selection of small and big ideas in the project board.
Some known technical limitations to date are:
- Limited mouse controls of probe expressions. While intended to be completely configurable as GUI, nested sounds and additional parameters still require keyboard control at the moment.
- No compatibility with vanilla Babylonian - inserting examples in a method will play unexpected sounds.
Please do not hesitate to create an issue if you have further ideas or troubles!