GDIF recording and playback

Kristian Nymoen have updated the Jamoma modules for recording and playing back GDIF data in Max 5. The modules are based on the FTM library (beta 12, 13-15 does not work), and can be downloaded here.

We have also made available three use cases in the (soon to be expanded) fourMs database: simple mouse recording, sound saber and a short piano example. See the video below for a quick demonstration of how it works:

Papers at ICMC 2008

Last week I was in Belfast for the International Computer Music Conference (ICMC 2008). The conference was hosted by SARC, and it was great to finally be able to see (and hear!) the sonic lab which they have installed in their new building.

I was involved in two papers, the first one being a Jamoma-related paper called “Flexible Control of Composite Parameters in Max/MSP” (PDF) written by Tim Place, Trond Lossius, Nils Peters and myself. Below is a picture of Trond giving the presentation. The main point of the paper is that we suggest that parameters should have properties and methods. This is both a general suggestion, and a specific one which we have started implementing in Jamoma using OSC.

The second paper was called “A Multilayered GDIF-Based Setup for Studying Coarticulation in the Movements of Musicians” (PDF) and was written by Kristian Nymoen, Rolf Inge Godøy and myself. This was a presentation of how we are currently using the Sound Description Interchange Format (SDIF) for the storage of GDIF data. This helps solve a number of the challenges we have previously experienced in terms of synchronisation of data, audio and video with different (and varying) sampling rates and resolution.

There are lots of more pictures from the conference on Flickr.

Janer’s dissertation

I had a quick read of Jordi Janer’s dissertation today: Singing-Driven Interfaces for Sound Synthesizers. The dissertation presents a good overview of various types of voice analysis techniques, and suggestions for various ways of using the voice as a controller for synthesis. I am particularly interested in his suggestion of a GDIF namespace for structuring parameters for voice control:

/gdif/instrumental/excitation/loudness x
/gdif/instrumental/modulation/pitch x
/gdif/instrumental/modulation/formants x1 x2
/gdif/instrumental/modulation/breathiness x
/gdif/instrumental/selection/phoneticclass x

Here he is using Cadoz’ division of various types of instrumental “gestures”: excitation, modulation and selection, something which would also make sense for describing other types of instrumental actions.

I am looking forward to getting back to working on GDIF again soon, I just need to finish this semester’s teaching + administrative work + moving into our new lab first…

Some thoughts on GDIF

We had a meeting about GDIF at McGill yesterday, and I realised that people had very different thoughts about what it is and what it can be used for.

While GDIF is certainly intended for formalising the way we code movement and gesture information for realtime usage in NIME using OSC, it is also supposed to be used for offline analysis. I think the best way of doing this, is to have a three level approach as sketched here:

gdif-storage.png

The realtime communication is done with OSC, usually over UDP/IP, while we could use SDIF tools available in FTM for storing the streams (it might be better to just use some binary format for storing the OSC streams). Then, after discussing with Esteban and Jordi from Pompeu Fabra, I have been convinced that it is probably a good idea to use XML for creating structured files for offline analysis.

When it comes to what to store, I think it is important to separate the data into different layers to avoid confusion:

gdif-namespace.png

Not all the streams will have to be communicated all the time (which would obviously create quite a lot of overhead), but they could. The raw data level would typically not be useful for most realtime applications, but for analytical purposes it is crucial to be able to get back to the original data.

NIME paper on GDIF

Here is the poster I presented at NIME 2006 in Paris based on the paper Towards a Gesture Description Interchange Format.

The paper was written together with Tellef Kvifte, and the abstract reads:

This paper presents our need for a Gesture Description Interchange Format (GDIF) for storing, retrieving and sharing information about music-related gestures. Ideally, it should be possible to store all sorts of data from various commercial and custom made controllers, motion capture and computer vision systems, as well as results from different types of gesture analysis, in a coherent and consistent way. This would make it possible to use the information with different software, platforms and devices, and also allow for sharing data between research institutions. We present some of the data types that should be included, and discuss issues which need to be resolved.