Calculating duration of QuickTime movie files

I have been doing video analysis on quicktime movie files for several years, but have never really had the need to use the time information of the movie files. For a project, I now had the need for getting the timecode in seconds out of the files, and this turned out to be a little more tricky than first expected. Hence this little summary for other people that may be in the same situation.

It turns out that QuickTime uses something called time units for the the internal representation of time, and this is also what is output in Jitter when run my analyses on the files. The time unit is not very meaningful for humans, as it is a combination of frames and a timescale that defines the actual length of each time unit. Apple has posted some more technical information about Timecode Media Handler Functions, but it didn’t really help me solve the problem easily.

Fortunately, there are a few threads on this topic on the Cycling ’74 forum, including one on relating sfplay time to quicktime time, and another on converting quicktime Units into time code. These threads helped me realise that calculating the duration of a movie file in seconds, is as easy as dividing the duration in frames by the timescale. And, by knowing the total number of frames it is then also possible to calculate the frames per second of the file. Now that I know this, it is obvious, but I post the patch here in case there are others looking for this information.

----------begin_max5_patcher----------
1334.3oc6Zt0iiZCE.94jeEV7TqzTJ9B25Sspu0+BUqF4.NY7rDHEblY1tZ+
uWegbcGLPVvyLRUq1DgCfOmOet4imutbg2ppWXMdfeC72fEK95xEKzCoFXQ6
0K71ReIqf1nuMurpsaYkBu6L+lf8hPO9eRKx1WPELv1pm3LP99ZpfWUBnk4f
06Z.7RvewEBV8gGccUong+uL0iCQ9AsCWtea0dQASnmuCittdyJ80GuucTQ1
C7xM2WyxDFM.GI+U.BY9LV+k7A.ep8Q34ZQsZ0i+BJ5bwnjtUKFd+QMmV3cR
R3kGDDnZrusbo5i69AYUVAO6y.QEH6HzDboDLefAgUvHQSFHhXkLn2PxbvpY
9PAJIxOINTp+QZZDCsACX7XgwgYtl0HUPsxb9rmF5qmbrdAIn8CvmlPFttVJ
lYU6O8Sy.EgQ9CGhQSLDgo5oy3tOKLT4N1H8NmQeRHQaBzRvH6DjLsDDFFXH
X7rRv4CdwomwNiuTmrC6f3YxWvpQlYC0s1EYRckn0qHhOJzh5A8N9hTN9xDr
2yJoqJXW.0IwrIE4C0xR3UQuZeLi1I9xNl4983JCCf2JZ4Fuax5ZZ4JD2F8c
PjM0cfEJKVQyUxGRthBH9CFqINDqRCV85MI8iIWiwFTZ.KL.Ykrwy.YmdDsZ
uPbp.uKYAzBKRwl51RBNDsuaRDMNc5GX8l8rb999jefUl+MJKAx.zdXfYONQ
V0ej2DsZqIycc78jWu.3mZjEtVl27yyXYWlReHw5ufPqbjj5fZGVWTIeGSVL
CoFoBRPL6+Mzt9k3hPFREjJlGql06ZlwM4DE5ehj16m.wE8SninM+J.5OJJ.
627AmpSYhQ9VR3PFabFjcSjI0z3XCnbPPEbPa3YbT5PBq.+3EVAg8OSAiisV
IBN5CRbEW3Qcfb3j98nvouC7n1xZZnaXeGU1HmfapuHVnSHlzVXioWDwZHAS
5.OjoBO2FYlVeJy17aqDoOOJz+6QcE2vCHCEF+NvepCjTKmCSi+AGcq.mZdK
3l5EdXucUpsaYo1ODffQxsvNczt6p+O0gjtw1caw9BSRuHlTRrsXRvz21XRV
PyMYAYALDR7kAqgVAS36VvL5lSOjFSzBEyNt5BJuwIv5HTztZdGtO3dJO901
gsoDQTf15n8LYgcXi3fBg2UP+xJZs2XSOm.8O0uIn8CnAebizOvyY0Oud84V
MuZxW0MVTUsyZBYUDS0Ryj5.Tyn4iZwF12QtaXjw9Wc3bu5Rcv6RS+G4B++Q
3a9iV32o6EUMBZs.DLJBg5iPPS1WXHxl6P3T02NoVc+Vpnl+xsmyUQlcOyKy
qd161T5Pj4bARM6FFR55HrN145UzrOuQVXTYdu9OudkFWno5IyqfWd8ehKZI
VM9kpeS095rCut1i.BbRjyYMBdoIi5o6QUhI.d7ljt04rxyCQlyaT0oq0nfW
ccXnhi5r9Fl7D3D4QUkXuxyUB8rKOvdjmTmINCx5I0YVOICPbTczykKVndjm
DmINwCcwxgzA2i7D6LwIb.NVWEMXNEmnArXgbJb5OJ36qEK2ET9pPJcD2wcN
5jAHNg8HMa446pj0k2bp8+X8V.izcUgnO2H8EmlnISAvCJuRjy.JNYBH5DJN
QCMOmaR6hmACtPBwkFb3gXv4vJGFxBrSkl9RTicq3zav+TmJN8UjGzcAGfoy
Pz+vTG5LBCmdMfDF6RMHXFyWX1xOc2tmX0MsuRsj3sk9XUs5xn6zWxKMWp6m
fWM6I9g6GqGgVm8.WvxD6qMsg4kjHukp44aK+Ob4vDqA
-----------end_max5_patcher-----------

My main problem, though, was that I already had a lot of analysis files (hundreds) with only the QuickTime time unit as the time reference. It was not an option to rerun these analyses (which have taken weeks to finish), so I had to figure out a way of retroactively calculating a more meaningful timecode (in seconds).

After fiddling around with the time series data for a little while, I realised that it is possible to use the difference between two time samples and, knowing the original fps of the movies (using QuickTime Player to check the fps), calculate the correct duration in seconds. For my dataset there turned out to be only five different time unit durations, so it was fairly easy to write a small script for calculating the durations in Matlab. This is the part of the script that handles the time calculation, in which a is a Matlab structure with my data, hence a.time(1) is the time code of the first sample in the dataset.

time_diff=a.time(2)-a.time(1);
switch(time_diff)
    case 1001
        t = (a.time-a.time(1))/(29.97*time_diff); % 29.97 fps
    case 2000
        t = (a.time-a.time(1))/(29.97*time_diff); % 59.94 fps
    case 3731
        t = (a.time-a.time(1))/(24*time_diff);    % 24 fps
    case 3733
        t = (a.time-a.time(1))/(24*time_diff);    % 24 fps
    case 3750
        t = (a.time-a.time(1))/(24*time_diff);    % 24 fps
    otherwise 
        t = (a.time-a.time(1))/(29.97*time_diff); % 29.97 fps
        disp('!!! Unknown timecode.')
end

For new analyses, I will calculate the correct duration in seconds right away, but this hack has a least helped in solve the problem for my current data set.

New publication: Performing the Electric Violin in a Sonic Space

I am happy to announce that a paper I wrote together with Victoria Johnson has just been published in Computer Music Journal. The paper is based on the experiences that Victoria and I gained while working on the piece Transformation for electric violin and live electronics (see video of the piece below).

Citation
A. R. Jensenius and V. Johnson. Performing the electric violin in a sonic space. Computer Music Journal, 36(4):28–39, 2012.

Abstract
This article presents the development of the improvisation piece Transformation for electric violin and live electronics. The aim of the project was to develop an “invisible” technological setup that would allow the performer to move freely on stage while still being in full control of the electronics. The developed system consists of a video-based motion-tracking system, with a camera hanging in the ceiling above the stage. The performer’s motion and position on stage is used to control the playback of sonic fragments from a database of violin sounds, using concatenative synthesis as the sound engine. The setup allows the performer to improvise freely together with the electronic sounds being played back as she moves around the “sonic space.” The system has been stable in rehearsal and performance, and the simplicity of the approach has been inspiring to both the performer and the audience.

PDF
The PDF will be available in the University of Oslo public repository after the 6 month embargo. Until then, it is available through either MIT Press or Project MUSE.

BibTeX entry
@article{Jensenius:2012,
Author = {Jensenius, Alexander Refsum and Johnson, Victoria},
Journal = {Computer Music Journal},
Number = {4},
Pages = {28–39},
Title = {Performing the Electric Violin in a Sonic Space},
Volume = {36},
Year = {2012}}

Video
Video of the piece Transformation.

Audio recordings as motion capture

I spend a lot of time walking around the city with my daughter these days, and have been wondering how much I move and how the movement is distributed over time. To answer these questions, and to try out a method for easy and cheap motion capture, I decided to record today’s walk to the playground.

I could probably have recorded the accelerometer data in my phone, but I wanted to try an even more low-tech solution: an audio recorder.

While cleaning up some old electronics boxes the other day I found an old Creative ZEN Nano MP3 player. I had totally forgotten about the thing, and I cannot even remember ever using it. But when I found it I remembered that it actually has a built-in microphone and audio recording functionality. The recording quality is horrible, but that doesn’t really matter for what I want to use it for. The good thing is that it can record for hours on the 1GB built-in memory, using some odd compressed audio format (DVI ADPCM).

Since I am mainly interested in recording motion, I decided to put it in my sock and see if that would be a good solution for recording the motion of my foot. I imagined that the sound of my footsteps would be sufficiently loud that they would be easily detected. This is a fairly reduced recording of all my motion, but I was interested in seeing if it was relevant at all.

The result: a 35 MB audio file with 2,5 hours of foot sounds! In case you are interested, here is a 2-minute sample of regular walking. While it is possible to hear a little bit of environmental sounds, the foot steps are very loud and clear.

Now, what can you do with a file like this? To get the file useable for analysis, I started by converting it to a standard AIFF file using Perian in QuickTime 7. After that I loaded it into Matlab using the wonderful MIRToolbox, resampling it to 100 Hz (from 8kHz). It can probably be resampled at an even lower sampling late for this type of data, but I will look more into that later.

The waveform of the 2,5 hour recording looks like this, and reveals some of the structure:

But calculating the smoothed envelope of the curve gives a clearer representation of the motion:

Here we can clearly identify some of the structure of what I (or at least my right foot) was doing for those 2,5 hours. Not bad at all, and definitely relevant for macro-level motion capture.

Based on the findings of a 2 Hz motion peak in the data reported my MacDougall and Moore, I was curious to see if I could find the same in my data. Taking the FFT of the signal gives this overall spectrum:

Clearly, my foot motion shows the strongest peaks at 4 and 5 Hz. I will have to dive into the material a bit more to understand more about these numbers.

The conclusion so far, though, is that this approach may actually be a quite good, cheap and easy method for recording long-term movement data. And with 8kHz sampling rate, this method may also allow for studying micro-movement in more detail. More about that later.

AudioAnalysis v0.5

I am teaching a course in sound theory this semester, and therefore thought it was time to update a little program I developed several years ago, called SoundAnalysis. While there are many excellent sound analysis programs out there (SonicVisualiserPraat, etc.), they all work on pre-recorded sound material. That is certainly the best approach to sound analysis, but it is not ideal in a pedagogical setting where you want to explain things in realtime.

There are not so many realtime audio analysis programs around, at least not anyone that looks and behaves similar on both OSX and Windows. One exception that is worth mentioning is the excellent sound tools from Princeton, but they lack some of the analysis features I am interested in showing to the students.

So my update of the SoundAnalysis program, should hopefully cover a blank spot in the area of realtime sound visualisation and analysis. The new version provides a larger spectrogram view, and the option to change various spectrogram features on the fly. The quantitative features have been moved to a separate window, and now also includes simple beat tracking.

Below is a screenshot giving an overview of the new version:

Overview of AudioAnalysis

Other new selling points include a brand new name… I have also decided to rename it to AudioAnalysis, so that it harmonizes with my AudioVideoAnalysis and VideoAnalysis programs.

The program can be found over on the fourMs software page, and here is a short tutorial video:

Please let me know if you find bugs or other strange things in the program, and I will try to fix them as soon as possible (I expect there to be some Win 64-bit issues…).

PD introductions in Norwegian on YouTube

I am teaching two courses this semester:

In both courses I use Pure Data (PD) for demonstrating various interesting phenomena (additive synthesis, beating, critical bands, etc.), and the students also get various assignments to explore such things themselves. There are several PD introduction videos on YouTube in English, but I found that it could be useful to also have something in Norwegian. So far I have made three screencasts going through the basics of PD and sound synthesis: