Audio recordings as motion capture

I spend a lot of time walking around the city with my daughter these days, and have been wondering how much I move and how the movement is distributed over time. To answer these questions, and to try out a method for easy and cheap motion capture, I decided to record today’s walk to the playground.

I could probably have recorded the accelerometer data in my phone, but I wanted to try an even more low-tech solution: an audio recorder.

While cleaning up some old electronics boxes the other day I found an old Creative ZEN Nano MP3 player. I had totally forgotten about the thing, and I cannot even remember ever using it. But when I found it I remembered that it actually has a built-in microphone and audio recording functionality. The recording quality is horrible, but that doesn’t really matter for what I want to use it for. The good thing is that it can record for hours on the 1GB built-in memory, using some odd compressed audio format (DVI ADPCM).

Since I am mainly interested in recording motion, I decided to put it in my sock and see if that would be a good solution for recording the motion of my foot. I imagined that the sound of my footsteps would be sufficiently loud that they would be easily detected. This is a fairly reduced recording of all my motion, but I was interested in seeing if it was relevant at all.

The result: a 35 MB audio file with 2,5 hours of foot sounds! In case you are interested, here is a 2-minute sample of regular walking. While it is possible to hear a little bit of environmental sounds, the foot steps are very loud and clear.

Now, what can you do with a file like this? To get the file useable for analysis, I started by converting it to a standard AIFF file using Perian in QuickTime 7. After that I loaded it into Matlab using the wonderful MIRToolbox, resampling it to 100 Hz (from 8kHz). It can probably be resampled at an even lower sampling late for this type of data, but I will look more into that later.

The waveform of the 2,5 hour recording looks like this, and reveals some of the structure:

But calculating the smoothed envelope of the curve gives a clearer representation of the motion:

Here we can clearly identify some of the structure of what I (or at least my right foot) was doing for those 2,5 hours. Not bad at all, and definitely relevant for macro-level motion capture.

Based on the findings of a 2 Hz motion peak in the data reported my MacDougall and Moore, I was curious to see if I could find the same in my data. Taking the FFT of the signal gives this overall spectrum:

Clearly, my foot motion shows the strongest peaks at 4 and 5 Hz. I will have to dive into the material a bit more to understand more about these numbers.

The conclusion so far, though, is that this approach may actually be a quite good, cheap and easy method for recording long-term movement data. And with 8kHz sampling rate, this method may also allow for studying micro-movement in more detail. More about that later.

AudioAnalysis v0.5

I am teaching a course in sound theory this semester, and therefore thought it was time to update a little program I developed several years ago, called SoundAnalysis. While there are many excellent sound analysis programs out there (SonicVisualiserPraat, etc.), they all work on pre-recorded sound material. That is certainly the best approach to sound analysis, but it is not ideal in a pedagogical setting where you want to explain things in realtime.

There are not so many realtime audio analysis programs around, at least not anyone that looks and behaves similar on both OSX and Windows. One exception that is worth mentioning is the excellent sound tools from Princeton, but they lack some of the analysis features I am interested in showing to the students.

So my update of the SoundAnalysis program, should hopefully cover a blank spot in the area of realtime sound visualisation and analysis. The new version provides a larger spectrogram view, and the option to change various spectrogram features on the fly. The quantitative features have been moved to a separate window, and now also includes simple beat tracking.

Below is a screenshot giving an overview of the new version:

Overview of AudioAnalysis

Other new selling points include a brand new name… I have also decided to rename it to AudioAnalysis, so that it harmonizes with my AudioVideoAnalysis and VideoAnalysis programs.

The program can be found over on the fourMs software page, and here is a short tutorial video:

Please let me know if you find bugs or other strange things in the program, and I will try to fix them as soon as possible (I expect there to be some Win 64-bit issues…).

PD introductions in Norwegian on YouTube

I am teaching two courses this semester:

In both courses I use Pure Data (PD) for demonstrating various interesting phenomena (additive synthesis, beating, critical bands, etc.), and the students also get various assignments to explore such things themselves. There are several PD introduction videos on YouTube in English, but I found that it could be useful to also have something in Norwegian. So far I have made three screencasts going through the basics of PD and sound synthesis:

AudioVideoAnalysis

To allow everyone to watch their own synchronised spectrograms and motiongrams, I have made a small application called AudioVideoAnalysis.

It currently has the following features:

  • Draws a spectrogram from any connected microphone
  • Draws a motiongram/videogram from any connected camera
  • Press the escape button to toggle fullscreen mode

Built with Max/MSP by Cycling ’74 on OS X.5. I will probably make a Windows version at some point, but haven’t gotten that far yet.

A snapshot of the main interface:

The main window of the AudioVideoAnalysis application

The fullscreen can be toggled with the escape button:

Fullscreen mode in the AudioVideoAnalysis application

The are, obviously, lots of things that can and will be improved in future versions. Please let me know of any problems you experience with the application, and if there is anything in particular you think should be included.