I participated in the conference The Aesthetics of Absence in Music of the Twenty-First Century at the Department of Musicology the last couple of days. Judith Lochhead started her keynote lecture with a clip from the movie The art of flying by Jan van Ijken. This is a beautiful short film based on clips of flocking birds:
The art of flying from Jan van IJken on Vimeo.
Of course, I wanted to see how some video visualizations would work, so I reached for the Musical Gestures Toolbox for Python. After purchasing and downloading the video, I run a version of ipython, load the toolbox and import the video file:
import musicalgestures as mg
video = mg.MgVideo('the_art_of_flying.mp4')
Since I knew that this is a video with changing scenes and camera angles, I began by generating a grid image:
video.grid()
By default, this creates a 3x3 grid of evenly distributed frames from the video file:
This gives us a sense of what is going on. Out of curiosity, I also made an average image of the video file:
video.average()
This function can be useful when working with videos recorded with a static camera, but for this video, it mainly shows us a blurred version of the horizon:
Then it is more interesting to look at the videograms:
video.videograms()
The horizontal videogram shows the spatiotemporal development of the entire film, with time running from left to right:
I find it fascinating to look at the complexity revealed in the vertical videogram (click on image to get high-resolution version):
A self-similarity matrix of the motiongrams can be calculated like this:
video.ssm(features='motiongrams')
It reveals some of the structural patterns in the video:
When we get the new sonification module implemented properly in the toolbox, I will try to sonify these visualizations to explore various translations from the spatiotemporal visual domain to the auditory.