Motiongrams
Motiongrams are a visual display of motion over time, somehow similar to how we can visualize audio with spectrograms. No motion tracking or other computer vision techniques are applied. A motiongram is simply a reduction of the video stream and is thus a good starting point for further quantitative and qualitative analysis.
Challenge
Traditional keyframe displays of videos are not particularly useful when studying single-shot studio recordings of music-related movements, since they mainly show static postural information and no motion.
Overview
The process of creating a motiongram starts by calculating the motion image (frame difference image). Motiongrams are then made by calculating the means of the rows and columns of the motion image and plotting them over time:
Using Motiongrams
Motiongrams allow for quick navigation in video material and for comparative analysis of motion qualities. Although quite rough, it is easy to see differences in the quantity of motion and similarities in upward/downward patterns between motion sequences.
Below is a motiongram of a five-minute video of free dance movements to music. The dancer moved to five different musical excerpts (marked a-e) and each excerpt was repeated three times (marked 1-3).
We use motiongrams in comparative studies. Below are motiongrams of three dancers moving freely to the same musical excerpts.
If we zoom into the image and look at the first 40 seconds of the sequence displayed above, it is possible to follow the trajectories of the hands (because of the yellow and red gloves) and head (pink due to saturation), as well as the body (appears blue due to the background).
Sonification of motiongrams
Here is an example of how a motiongram can be used for sonification:
Software
Motiongrams have been implemented in the Musical Gestures Toolbox. There are also two standalone applications for OSX and Windows, built on the basis of the toolbox:
- VideoAnalysis (non-realtime)
- AudioVideoAnalysis (realtime)
Publications
The motiongram technique was first published in this paper:
- Jensenius, A. R. (2006). Using motiongrams in the study of musical gestures. In Proceedings of the 2006 International Computer Music Conference, 6-11 November, New Orleans.
The method was refined further and presented in more detail in:
- Jensenius, A. R. (2007). Action - Sound: Developing Methods and Tools to Study Music-related Body Movement. Ph.D. thesis, University of Oslo.
Acknowledgments
This research is funded by the Research Council of Norway.