Over the last few days, we have run a toolbox hackathon between the three music-related research centres of excellence in the Nordic countries (MIB, MMBB, and RITMO). Researchers across all centres are actively developing various toolboxes, including myself and the Musical Gestures Toolbox (MGT). Due to all the MishMash things happening lately, I was unable to spend much time myself on hacking. Instead, I thought of the opportunity to test how GitHub Agents could assist.

There were 27 open issues in the GitHub tracker for MGT: some small bug fixes, and others larger, more complex reorganisation issues. I saw that it was possible to assign an issue to an agent, and thought about giving it a try.

alt text

It worked remarkably well, and within a few days, I had agents help close 8 issues, some of them quite large. I also asked CoPilot to write up a summary of what changed. It politely suggests that “we” did the job, which is not entirely wrong, given that I assigned tasks, reviewed the result, and made the merges and commits. However, the GitHub Agents did the bulk of the work!

1. Packaging and CI reliability improvements

We fixed packaging metadata/build issues that were causing CI failures in modern environments. CI (Continuous Integration) is a software development practice where code changes are automatically built, tested, and validated whenever they are committed to a repository. This helps catch bugs early and ensures code quality before deployment.

Key outcomes:

  1. improved compatibility with modern setuptools/PEP 517 behavior
  2. cleaner metadata generation during install
  3. CI pipeline updated for current action/runtime expectations
  4. reduced risk from deprecated action runtime dependencies

Packaging is now reliable across Python versions, and dependency management is more robust.

2. GPU acceleration is now more practical and safer

We expanded optional CUDA usage to more processing paths while preserving strong CPU fallback behaviour. CUDA (Compute Unified Device Architecture) enables parallel computing on NVIDIA GPUs, significantly accelerating compute-intensive tasks.

The toolbox now supports:

  1. optional GPU acceleration for optical flow paths
  2. optional GPU acceleration for face-blurring workflows
  3. shared CUDA availability helper exposed for easier checks
  4. robust fallback to CPU when CUDA or required OpenCV CUDA modules are not available

Importantly, the changes are optional, so most people use a CPU on their laptops but have access to a GPU if running more powerful machines.

3. Adding Mediapipe support

A major feature addition was optional MediaPipe support in MGT workflows. MediaPipe provides robust, real-time body and hand landmark tracking, which is especially useful for gesture analysis in music performance and movement studies.

Main improvements include:

  1. Added a MediaPipe-based pose and hand tracking path as an optional backend in relevant analysis workflows.
  2. Preserved backward compatibility by keeping existing OpenCV-centric paths as the default.
  3. Implemented graceful fallback behaviour when MediaPipe is not installed, so users can still run standard pipelines.
  4. Improved dependency handling by treating MediaPipe as an optional extra instead of a mandatory install.

This lets users start with the core toolbox setup and enable MediaPipe only when they need advanced landmark-based tracking.

The whole thing started from a single short request:

alt text

4. Test coverage and test stability upgrades

We added targeted tests for new GPU code paths and improved test resilience when environment-specific media metadata differs slightly.

This includes:

  1. expanded tests around GPU/CPU path behaviour and fallback
  2. tolerance improvements for platform-sensitive frame count and duration assertions
  3. better cross-platform reliability for utility tests that depend on ffprobe rounding

This results in fewer false negatives in CI and clearer signals when real regressions happen.

5. Documentation and citation consistency

Last year, I created automated documentation for the toolbox, but never cleaned up the Wiki that previously served as the documentation. Unfortunately, this leads to duplication that will drift over time: defaults change in code, docs are updated, and older wiki text can become stale.

Now, with the help of CoPilot, we made a cleaner separation of more general documentation and explanation in the Wiki:

  1. Defined clear responsibilities:

    • Wiki is now positioned as a practical onboarding layer: navigation, concepts, walkthroughs, and quick examples.
    • Documentation is the canonical technical source: API signatures, default values, installation specifics, and reference details.

  2. Added canonical pointers across wiki chapters:

    • We updated all major wiki chapters to include explicit links to the corresponding authoritative docs pages.
    • This makes it easy for users to start in the Wiki and quickly jump to the latest exact parameter behaviour.

  3. Reduced future maintenance risk:

    • By avoiding dual “source of truth” pages, future updates only need to change one canonical place for technical details.
    • This should significantly reduce stale examples and conflicting instructions.

  4. Aligned citation consistency:

    • We also synchronised citation wording so the SMC 2021 reference is consistently represented where appropriate.

Succesful cleanup

I am impressed by how much we (to put it in the agents’ words) managed to do within a few days. Some of the issues I assigned to the agents stalled, probably because the descriptions were too imprecise. However, they successfully solved 8 longstanding issues (almost) without my assistance. In that sense, the hackathon was a big success for me, and hopefully for others who use MGT.

Thanks to GitHub Agents for assisting with all the bug fixes and improvements, CoPilot Agents for additional fixing and writing a first draft of this post, and Grammarly for final grammar control.