I have long been fascinated by the similarities and differences between auditory and visual perception and by the way time comes into play. This week, I participated in the Soundshed workshop, where we explored soundscapes and creative approaches to experiencing sound. My contribution was not a fixed media piece but rather the realisation of something I have been thinking about for a long time: a web app called AudioVideoSoup.
Long ’exposure’ of audio and video
Long-exposure photography turns motion into trails while reverb smears a dry sound into space. AudioVideoSoup is based on both at the same time. You load a video recording (with audio) and press play. The picture begins to smear into a running average of everything it has seen. The audio blurs into an ambient wash built from its own spectrum.
Visually, each new frame is folded into a rolling memory of every pixel that has passed through. Aurally, the signal is analysed into frequency bins, and those magnitudes are smoothed over time while the phase is allowed to wander. The result is resynthesised and mixed back with the dry source. Hit “Reset soup” to clear the memory.
Video processing
The video processing is simple. The accumulated video image is:
accum = accum × decay + frame × (1 − decay)
A higher frame memory means a slower fade. The soup blend controls how much of that accumulated image you see versus the current live frame. At 0%, you get the raw video. At 100%, only the average image.
Audio processing
The audio path runs in an AudioWorklet, a separate real-time thread in the Web Audio API, isolated from the main UI to keep processing smooth. The processor performs a short-time Fourier transform (STFT) on incoming audio (FFT = 2048 samples, Hop size = 512 samples, Hann window).
Each frequency bin’s magnitude is exponentially smoothed, while phase is smoothed with a small random drift to keep the resynthesis from sounding frozen or metallic.
The smoothed spectrum is converted back to the time domain via inverse FFT. As for the video processing, you can choose the mix with the dry signal.
Try it
You can try AudioVideoSoup in most modern browsers. At least, Firefox and Chrome work well. Also check out the source code on github.com/alexarje/audiovideosoup. I have had great fun playing around with different recordings and testing the settings. It gives a different experience of both the audio and video content.
Thanks to Cursor for running Claude Sonnet and helping realise this project. I have been thinking about it for at least 15 years, but I have now finally got around to implementing it properly.