There were several presentations on the sonification of human body movement at MOCO. This, combined with my recent successes in developing browser versions of both VideoViz and VideoScrub, led me to attempt to revive another old Max patch.

Motiongram-based sonification

There are many ways of sonifying body motion. Nowadays, we often work with skeleton or marker-based models and advanced synthesis engines. However, two decades ago, I began exploring how to turn motiongrams into sound, treating the motiongram as if it were a spectrogram. I also published a paper on the method in 2012.

Implementation

Web Video Sonifyer is a browser-based motion-to-audio synthesiser. You open the page, enable your webcam, and the app continuously:

  • Computes frame difference (how much each pixel changes between consecutive frames)
  • Compresses those differences into a motiongram (a 1D motion profile)
  • Converts that motion profile into an audio spectrum
  • Runs an inverse FFT (iFFT) to generate a short audio frame
  • Uses overlap-add to stitch frames into continuous sound

It also derives a second motion profile across the other image axis and uses it to control left/right panning, so motion on the left side of the camera view tends to “pull” sound left, and motion on the right pulls it right.

How to play it

It is surprisingly expressive. Slow movement can become evolving drones while quick actions become brighter/noisier sounds. Moving on one side of the frame shifts the stereo image. I find it fun to play!

The app gives you some additional controls that let you shape the behaviour:

  • Frequency range: constrain output to a lower band to reduce harshness
  • Silence threshold: make it quiet when you are still
  • Diff gain / noise floor: adjust sensitivity to motion and lighting flicker
  • Smoothing: calm the motion signal for less jittery sound

In practice, I recommend starting with a narrower band, such as 80–800 Hz, then increasing loudness and differential gain until movement is clearly audible.

Web Video sonifyer interface

Technical explanation

For those interested, this is how it works:

  1. Take a 1D motion vector (values in ([0,1]))
  2. Use it as magnitudes for a selected FFT bin range (with a slight low-frequency tilt)
  3. Assign random phase to avoid static tonal artefacts
  4. Build a conjugate-symmetric spectrum (to get real-valued audio)
  5. iFFT → window → overlap-add

The motiongram itself is derived from per-pixel frame difference. To save some CPU, the app analyses a downscaled video frame, which is usually enough for stable interactive results.

Try it yourself

You can try Web Video Sonifyer directly in the browser, and the source code is available on github.com/alexarje/webvideosonifyer. Because it runs on Web Audio and getUserMedia, it works best in a modern Chromium browser. If you try it and have ideas for mappings or sound goals, I would be happy to hear them, ideally through a comment in the GitHub issue tracker.

Thanks to Cursor and Copilot for help with coding and drafting this post.