Unfortunately, while setting up for the workshop I had problems getting Chuck to work on my new laptop. I haven’t had time to figure out what the problem actually is, but I think it may be an issue with OSX Lion. Anyways, since I was short on time I decided that the quickest solution was just to make something similar to Clix in Max. The result is Click-It, and the GUI looks like this:

The inside of the patch is not very advanced either. It is mainly a click~ object controlled by a metro, with some amplitude adjustments and a little bit of delay and feedback.

This may not be so interesting in itself, but the beauty of this thing comes with 10 or so computers playing it at the same time. Unfortunately, I didn’t make a video of the students performing it yesterday, so that will have to wait until next time it is played.

The patch is available from our fourMs web page. I had almost forgot that we have a repository for the Oslo Laptop Orchestra over at Google Code. That repository is severely out of date, though, and needs some cleaning up and updating at some point. The main reason for the sloppy maintenance is that the laptop orchestra has been kind of dormant over the last few years, since most of our energy has been put into Oslo iPhone Ensemble instead. That said, although the iPhone ensemble is a lot fun, laptop orchestras have some other qualities that I would like to get back to exploring more again at some point.

See below for the full paper and video examples.

The Paper

• Download paper (PDF 2MB)

Abstract: The paper presents a method for sonification of human body motion based on motiongrams. Motiongrams show the spatiotemporal development of body motion by plotting average matrices of motion images over time. The resultant visual representation resembles spectrograms, and is treated as such by the new sonifyer module for Jamoma for Max, which turns motiongrams into sound by reading a part of the matrix and passing it on to an oscillator bank. The method is surprisingly simple, and has proven to be useful for analytical applications and in interactive music systems.

Full reference: A. R. Jensenius. Motion-sound interaction using sonification based on motiongrams. In ACHI 2012: The Fifth International Conference on Advances in Computer-Human Interactions, pages 170–175. IARIA, 2012.

@inproceedings{Jensenius:2012d,
    Author = {Jensenius, Alexander Refsum},
    Booktitle = {ACHI 2012: The Fifth International Conference on Advances in Computer-Human Interactions},
    Pages = {170--175},
    Publisher = {IARIA},
    Title = {Motion-sound Interaction Using Sonification based on Motiongrams},
    Year = {2012}}

Video examples

Video 1: A screencast demonstrating the jmod.sonifyer~ module.

Video 2: Examples of sonification of some basic movement patterns: up-down, sideways, diagonal and circular.

Video 3: One attempt at sonifying the two axes at the same time. Here both horizontal and vertical motiongrams are created from the same video recording, and the sonifications of the two motiongrams have been mapped to the left and right audio channel respectively.

Video 4: Examples of the importance of filtering and thresholding of the motion image for the final sounding result. The recordings were done at high-speed (200 fps) and played back at 25 fps.

Video 5: Sonification of a short violin improvisation (courtesy of Victoria Johnson).

Video 6: Sonification of a piece by a French-Canadian fiddler (courtesy of Erwin Schoonderwaldt).

Video 7: Sonification of free dance to music.

Video 8: Soniperforma: Performing with the sonifyer at Biermannsgården in Oslo on 18 December 2010. The performance was improvised and based on applying only video effects to change the sonic quality.

First, there is a getting started with Phidgets in Max video in Norwegian:

And I have also made a video demonstrating the Phidgets2MIDI application that I developed earlier this year:

I am planning to make some videos showing some more musically interesting use of the electronics and software.

See the demonstration video below:

The module is available from the Jamoma source repository, and will probably make it into an official release at some point.

There are not so many realtime audio analysis programs around, at least not anyone that looks and behaves similar on both OSX and Windows. One exception that is worth mentioning is the excellent sound tools from Princeton, but they lack some of the analysis features I am interested in showing to the students.

So my update of the SoundAnalysis program, should hopefully cover a blank spot in the area of realtime sound visualisation and analysis. The new version provides a larger spectrogram view, and the option to change various spectrogram features on the fly. The quantitative features have been moved to a separate window, and now also includes simple beat tracking.

Below is a screenshot giving an overview of the new version:

Other new selling points include a brand new name… I have also decided to rename it to AudioAnalysis, so that it harmonizes with my AudioVideoAnalysis and VideoAnalysis programs.

The program can be found over on the fourMs software page, and here is a short tutorial video:

Please let me know if you find bugs or other strange things in the program, and I will try to fix them as soon as possible (I expect there to be some Win 64-bit issues…).

The first thing I thought about was the text object, which is used for storing the data and write to a text file. So to check the possible limitations of the object, I have made a small patch that writes lines of 60 random values. It turns out, however, that the text object easily handles 1 000 000 lines of random values, and manage to write the file to disk (366.9 MB).

So I continue my quest for the problem. I attach a screenshot of the code below for people that may be interested in this. The code may also be useful if you want to generate huge files with random numbers.

We have also made available three use cases in the (soon to be expanded) fourMs database: simple mouse recording, sound saber and a short piano example. See the video below for a quick demonstration of how it works:

About motiongrams A motiongram is a way of displaying motion (e.g. human motion) in the time-domain, somehow similar to how we are used to working with time-representations of audio (e.g. waveform displays and sono/spectrograms). The method is based on creating a motion image, doing a matrix reduction on it and plotting the resultant 1xn or nx1 matrices over time either horizontally or vertically.

Comparison of motiongrams Below is an image showing three different types of motiongrams:

1. Single line scan based on regular image
2. Average scan based on regular image
3. Average scan based on motion image

I think all of them are interesting, so the use of them will have to be adjusted according to what type of material you are working with.

Working with various types of sensor and motion capture systems, I see the same need to know how much motion there is in the system, independent of the number of variables and dimensions in the system studied. Thus, whether we use a single 1-dimensional MIDI slider or 32 6-dimensional sensors in a motion capture system, we still need to be able to say whether there is any movement in the system, and approximately how much movement there is.

So I have made a small abstraction in Max that sums up all incoming values, divides by the number of values, finds the first derivative and takes the absolute value of this.

I had two optimization questions while working on the patch:

1. Does it matter whether derivation is done before or after summing up the values?
2. Is it more efficient to use Max objects than Jitter objects?

Answers:

1. No, it does not matter.
2. Max objects are ~3 times faster

A screenshot of the efficiency test patch is shown below, and a zip-file of the patches.

First, electronic music pioneer Pauline Oliveros was also performing in the piece. She performed electric accordeon and live electronics, a great combination.

Second, the performance took place in the magnificent foyer of the new Oslo opera house. Even if it is a large space, we managed to fill it up with all the different stations, equipment and instruments.

Third, I got the chance to test my video analysis system in a large space. I placed a Logitech webcam on the balcony approximately 7 meters above the floor. The camera was connected to two 5 meter USB extension cables, and still worked surprisingly well!
  

I had 5 speakers placed on the floor along the ceiling. Originally, I thought about using speakers stands, but considering the beautiful wooden wall they were placed in front of, I thought it was more visually pleasing to put them on the floor.

I used a slightly modified version of the patch I have been working on with artistic research fellow Victoria Johnson, and we both moved in the space, controlling various bell and water sounds that were recorded for the piece.

Most of the patch was using Jamoma modules, the sound engine was CataRT and spatialisation was done with VBAP.