I am at the Movement Computing Conference (MOCO) in Montpellier and have been discussing motion capture and analysis all day. Someone asked about my work on video visualisation, and this reminded me about a long wish for creating a browser-based visualiser. I have been developing video visualisation tools for more than two decades. In the beginning, I did it with Max/MSP/Jitter, then moved on to Matlab, and later Python. Until recently, web technologies were not advanced enough, and hardware was not powerful enough to do anything in the browser in real time. And, I didn’t have the programming skills to make it work. Now, with the help of CoPilot, I have finally made it happen: VideoViz is here!

Developing a web app

A web app is not only just a HTML-based web app. The HTML code is only a wrapper for a JavaScript-based app that uses various specialised web technologies. I asked CoPilot to use Python code from the Musical Gestures Toolbox (MGT) to ensure that it is “compatible” with the other workflows I have.

VideoViz is not meant as a full-fledged visualisation tool, so you should use MGT instead. However, I often need to explain how video visualisation works, and then it is much easier to point to a web page than to ask people to install Python and MGT. I have only tested it in Firefox and Chrome, but since it is built on standard Web APIs, it should also work with other modern browsers.

Functionality

VideoViz shows a 2×2 grid of synchronised views:

  • Camera: a centre-cropped square view of your webcam
  • Videogram (Horizontal Avg): a scrolling history built from row-averaged pixels
  • Videogram (Vertical Avg): a scrolling history built from column-averaged pixels
  • Self-similarity: a similarity matrix over recent frames

Together, these displays show various spatial and temporal “features” of whatever is in front of the camera.

VideoViz regular view

I have also included motion-based versions of these displays. These are based on frame-differencing, subtracting subsequent frames in the video stream, a standard method in computer vision. This renders a motion video, two motiongrams (horizontal and vertical), and a self-similarity matrix (SSM).

VideoViz motion view

It is really cool that I managed to get the SSM to work as well. If you are not used to watching such displays, the diagonal shows “now vs now,” and it can be used to identify recurring motion patterns.

There is also a very basic filtering threshold and a normalise button that can help display motion more clearly. Check the readme and Wiki for details.

AI agents to help

I was very impressed to see that a GitHub Copilot Agent (running Claude) managed to make a fully functioning version on the first attempt. However, there were many small issues to resolve, so I had to make more than 50 new commits to get the app to function as I wanted. It helps that I have been developing video-based workflows for 20 years, which enables me to instruct agents quite clearly. I am impressed by what they can do, but they are still not at a point where they hit the mark on the first attempt.

Thanks to CoPilot for building the initial version and to Cursor for assisting with fixing various issues.