I have just finished an intensive stretch of work on the Musical Gestures Toolbox for Python (MGT-python). Or, to be more precise, various AI agents have done most of the work for me, although heavily supervised by me. It started as “let me fix a few things” and turned into a near-complete modernisation. Here I describe how the toolbox went from v1.3.3 to v1.6.9 within a couple of weeks, including a pile of bug fixing, lots of optimisation, and a pile of new features.

Modernising the codebase

The 1.4.0 release rebuilt the plumbing under the toolbox:

  • Modern packaging with pyproject.toml, Python 3.10+, and a CI matrix testing Linux, macOS, and Windows across Python 3.10–3.12.
  • Optional dependency extras — musicalgestures[pose], [ml], [cli], [full] — so you only install what you need.
  • Typed enums and exceptions, a real logging setup, and ruff/mypy checks in CI.
  • New data structures for research workflows: MgFeatures (a named time-series container with NumPy/DataFrame/JSON export), MgVideoReader (a lazy, low-memory streaming frame iterator), an MgPipeline that chains processing steps scikit-learn–style, and MgDataset/MgCorpus for labelled collections of media files.
  • A command-line interface (musicalgestures info, motion, videograms, …) for quick jobs without writing Python.

These were all things we had discussed over the years, but none of us in the team had sufficient software engineering skills to pull it off on our own.

New visualisations and analyses

Over the years, I have collected lots of ideas for things to implement. Many of them I had implemented in Max and Matlab (or even the experimental FFmpeg-based terminal scripts) but they had never reached the Python branch. Now I finally had the opportunity to get them implemented in one large sweep:

  • motiontempo() — the dominant movement tempo from the quantity of motion, in Hz and BPM.
  • motiondescriptors() — higher-level scalar measures of how something moves: motion energy, smoothness (SPARC), entropy (complexity), and spectral descriptors (dominant movement frequency + spectral centroid).
  • heatmap() — where in the frame motion concentrates, accumulated across the whole video.
  • eulerian() — Eulerian Video Magnification (Wu et al., 2012) to amplify changes that are otherwise invisible, like breathing or a pulse (mode='color') or tiny movements (mode='motion').
  • sonomotiongram() — hear your motiongram: it treats the motiongram as a spectrogram and resynthesises it to audio.
  • motionvectors() — draws the motion vectors that video codecs already compute internally.
  • Space-time displays — stroboscope() (chronophotography), silhouette_waterfall(), the Motion History Image (motionhistory()), and a 3D spacetime_volume() point cloud.
  • Self-similarity matrices from motion features, including a combined display that folds both axes of motion into one map.

The new motiondescriptors() summarises a clip’s movement in a single figure — the quantity of motion over time, its power spectrum, and the energy/smoothness/entropy/spectral numbers:

Motion descriptors of a dance clip

And the space-time displays turn a whole sequence into a single still, here a stroboscope (chronophotography):

Stroboscope of a dance clip

A rebuilt pose pipeline

Pose estimation was never done well in MGT and relied on some old libraries. Now, MediaPipe is the default backend, detecting 33 body landmarks. It is fast on a plain CPU, and does not rely on CUDA-enabled OpenCV builds (which is arguably a pain!). OpenPose (body_25/coco/mpi) is still there for multi-person scenes and CUDA setups, and model weights now download automatically on first use.

Beyond the skeleton video, pose() produces an average-pose image and an all-trajectories image, with each marker coloured by its average quantity of motion and dominant movement frequency (plus a per-marker stats CSV).

Average pose with per-marker motion

Sound and sound–motion analyses

The audio side of MGT grew a fuller analysis set based on Librosa: MFCCs, tempo and beat tracking, beat statistics, chromagrams, and improved tempograms. The new thing is that they all share a cached audio decode so repeated analyses are cheap.

The main point is to enable a complete audio–video analysis suite for studying how a single performer’s sound and motion relate:

  • tempo_similarity() — sound tempo vs movement tempo (ratio, nearest harmonic, cross-correlation, lag).
  • phase_synchrony() — phase-locking between the sound and movement rhythms.
  • structure_comparison() — sound self-similarity vs movement self-similarity, with a difference map.
  • body_audio_coupling() — which body parts track the music.
  • dynamics_coupling() — sound loudness vs quantity of motion.

Audio vs. movement tempo similarity

And resample() lets you retime a loaded video (by target fps, speed factor, or frame decimation) and get back a new object, with audio kept in sync.

Faster and leaner

Performance got a lot of attention:

  • Start-up time dropped from ~1.5 s to ~0.5 s, by lazy-loading heavy dependencies (SciPy, IPython) and deferring the numba/LLVM import until a compiled kernel is actually used.
  • Caching throughout makes chained analyses don’t re-read the whole file each time.
  • The toolbox now counts decoded packets, which is accurate across formats (and no slower).
  • GPU acceleration actually works now, when available
  • Phone/portrait videos with a rotation flag are normalised at load, so nothing comes out sideways.

The agents also discovered and fixed a number of bugs along the way, including a critical one that had silently broken motion analysis entirely.

Built to build on

Another cool things is that the public API is now fully type-hinted and the package ships a py.typed marker, so editors and type checkers give you real autocomplete and checking. The display model is explicit and predictable — analysis methods return result objects, and you call .show() when you want to see something.

This is all written up in a proper API reference, a structured user guide, a visual example gallery, animated GIFs of the video outputs, an updated Jupyter/Colab tutorial, and wiki.

Dense optical flow of a dance clip

Try it

Everything is on PyPI and can be installed like this:

pip install --upgrade musicalgestures

Then you can run it like this:

import musicalgestures as mg

mv = mg.MgVideo(mg.examples.dance)
mv.motion().show()
mv.motiondescriptors().show()
mv.pose().show()

Please add bug reports and feature ideas on the issue tracker. Happy analysing!

Thanks to CoPilot, Cursor, and Claude for help with fixing various things in the toolbox and for making the first draft for this blog post.