Video sonification in the browser

There were several presentations on the sonification of human body movement at MOCO. This, combined with my recent successes in developing browser versions of both VideoViz and VideoScrub, led me to attempt to revive another old Max patch. Motiongram-based sonification There are many ways of sonifying body motion. Nowadays, we often work with skeleton or marker-based models and advanced synthesis engines. However, two decades ago, I began exploring how to turn motiongrams into sound, treating the motiongram as if it were a spectrogram. I also published a paper on the method in 2012. ...

April 26, 2026 · 3 min · 478 words · ARJ
The four use cases considered in this article (from left): reactive paintings, a birdbox, self-playing guitars, and an interactive music ball.

Inverse and indirect mappings

I am excited to announce the publication of a new paper that explores several themes I have been interested in recently: inverse and indirect mappings, embodied AI, and indoor environments. The paper, titled Inverse and indirect mappings in embodied AI systems in everyday environments, is written by PhD fellow Maham Riaz together with my former PhD fellow Cagri Erdem and myself. Embodied AI The paper is written for a special issue on “embodied AI”, a topic I have been interested in for many years. AI research has long focused largely on the “brain” of machines. While that has been successful, we will not be able to progress unless we also consider that machines need to have a body that can sense and act. ...

January 15, 2026 · 4 min · 673 words · ARJ