New publication: Some video abstraction techniques for displaying body movement in analysis and performance

leonardo-2013Today the MIT Press journal Leonardo has published my paper entitled “Some video abstraction techniques for displaying body movement in analysis and performance”. The paper is a summary of my work on different types of visualisation techniques of music-related body motion. Most of these techniques were developed during my PhD, but have been refined over the course of my post-doc fellowship.

The paper is available from the Leonardo web page (or MUSE), and will also be posted in the digital archive at UiO after the 6 month embargo period.

A. R. Jensenius. Some video abstraction techniques for displaying body movement in analysis and performance. Leonardo, 46(1):53–60, 2013.

This paper presents an overview of techniques for creating visual displays of human body movement based on video recordings. First a review of early movement and video visualization techniques is given. Then follows an overview of techniques that the author has developed and used in the study of music-related body movements: motion history images, motion average images, motion history keyframe images and motiongrams. Finally, examples are given of how such visualization techniques have been used in empirical music research, in medical research and for creative applications.


   Author = {Jensenius, Alexander Refsum},
   Journal = {Leonardo},
   Number = {1},
   Pages = {53--60},
   Title = {Some video abstraction techniques for displaying body movement in analysis and performance},
   Volume = {46},
   Year = {2013}}

Open lab

We have slowly been moving into our new lab spaces over the last weeks. The official opening of the labs is scheduled for Friday 26 September, but we had a pre-opening “Open lab” for the new music students last week, and here are some of the pictures research coordinator Anne Cathrine Wesnes shot during the presentation.

Here I am telling the students a little about our new research group, and showing the main room:

Showing some realtime video analysis tools, including motion history images and motiongrams:
Video analysis
Demonstrating our new Optitrack motion capture system:
Motion capture
Kristian Nymoen showing the “self-playing piano”, a Disklavier controlled by the movements of two Polhemus electromagnetic trackers.



Traditional keyframe displays of videos are not particularly useful when studying single-shot studio recordings of music-related movements, since they mainly show static postural information and no motion.

Using motion images of various kinds helps in visualizing what is going on in the image. Below can be seen (from left): motion image, with noise reduction, with edge detection, with “trails” and added to the original image.


Making Motiongrams

We are used to visualizing audio with spectrograms, and have been exploring different techniques for visualizing music-related movements in a similar manner. Motiongrams are made by calculating the means of the rows and columns of the motion image (difference between consecutive frames) and plotting them over time.

No motion tracking or other computer vision techniques are applied. A motiongram is simply a reduction of the video stream and is thus a good starting point for further quantitative and qualitative analysis.



Using Motiongrams

Motiongrams allow for quick navigation in video material and for comparative analysis of motion qualities. Although quite rough, it is easy to see differences in the quantity of motion and similarities in upward/downward patterns between motion sequences.

Below is a motiongram of a five minute video of free dance movements to music. The dancer moved to five different musical excerpts (marked a-e) and each excerpt was repeated three times (marked 1-3).

We use motiongrams in comparative studies. Below are motiongrams of three dancers moving freely to the same musical excerpts.

If we zoom into the image and look at the first 40 seconds of the sequence displayed above, it is possible to follow the trajectories of the hands (because of the yellow and read gloves) and head (pink due to saturation), as well as the body (appears blue due to the background).



Future Work

A number of issues will have to be adressed in future research:

  • 3D motiongrams showing both horizontal and vertical motion.
  • Combined displays with audio, video and sensor information.
  • Improve efficiency.


Rolf Inge Godøy and Marcelo M. Wanderley for valuable feedback and support. This research is funded by the Norwegian Research Council.



This post was first presented as a web page of the Musical Gesture group at University of Oslo, in connection to the following publication:

  • Jensenius, A. R. (2006). Using motiongrams in the study of musical gestures. In Proceedings of the 2006 International Computer Music Conference, 6-11 November, New Orleans. [PDF] [Poster]

and has retroactively been moved to this blog so that the content won’t be lost.