• Lecture: Introduction to motion capture (in music analysis and performance)
• Demo 1: Working with video analysis using the Musical Gestures Toolbox
• Demo 2: The Xsens MVN BioMech mobile mocap suit
• Workshop: Analysis and performance with Wii controllers, Phidgets accelerometers and Kinect

Below are various resources.

Introduction

PDF of the presentation.

Demo 1: Musical Gestures Toolbox

The Musical Gestures Toolbox is a collection of modules and abstractions developed in and for the graphical programming environment Max. The toolbox is currently being developed within the Jamoma open platform for interactive art-based research and performance.

Download: Jamoma + UserLib and Max

The toolbox is probably most useful for people that are already familiar with Max programming. People looking for more easy-to-use solutions can check out some of my standalone applications at the fourMs software page. These applications should work on most versions of OSX, as well as on WinXP. I know that there are various issues with Windows Vista and Windows 7, and will try to get these problems ironed out as soon as possible.

Demo 2: XSens

Xsens MVN BioMech is a mobile mocap suit based on inertial sensors (accelerometer, gyroscopes, magnetometers). It excels over camera-based systems in that it is portable and allows for mobile motion capture. I will show how the system can be used both for analysis and performance:

• Analysis: mocap recordings will be made with the internal Xsens software, and exported to C3D files that will be imported in Matlab using the Mocap Toolbox.

• Performance: I will also show how the system can be used in realtime, passing data to Max within which the packets will be parsed and used to control sound in realtime.

Workshop:

During the hands-on workshop, participants will be able to try out the above mentioned tools and systems, as well as:

• Phidgets USB motion sensors.

  • Download: Phidgets drivers and program examples (and possibly also Jamoma)

• Nintendo Wii controllers, which allow for wireless inertial motion sensing. Here there are several tools available.

  • OSCulator a general purpose tool for reading data from various human interface devices.
  • WiiDataCapture can record data coming from OSCulator, and format them so that they can easily be imported by the MoCap Toolbox
  • junXion is an application for passing on either OSC or MIDI from Wii controllers and other human interface devices.

• Microsoft Kinect sensor, which allows for inexpensive “3D” motion capture using depth-cameras.

  • Wiki page describing how to work with Kinect in Max

In the middle of all of this, I hear the word transdisciplinarity more and more frequently. Trying to find a proper definition of what this actually means, I came across Marilyn Stember’s 1990 paper Advancing the social sciences through the interdisciplinary enterprise, in which she offers the following overview of different levels of disciplinarity (my summary of her points):

• Intradisciplinary: working within a single discipline.
• Crossdisciplinary: viewing one discipline from the perspective of another.
• Multidisciplinary: people from different disciplines working together, each drawing on their disciplinary knowledge.
• Interdisciplinary: integrating knowledge and methods from different disciplines, using a real synthesis of approaches.
• Transdisciplinary: creating a unity of intellectual frameworks beyond the disciplinary perspectives.

Based on this, I have added two elements (inter and trans) to my former sketch of the different disciplinarities (originally based on Zeigler (1990)):

I am still not entirely sure that I understand the difference between interdisciplinary and transdisciplinary, but I guess that the latter is one more step towards full integration. That is why I have drawn the centre circles so that they almost overlap, but not entirely. I would imagine that when/if full integration of disciplines actually occurs, you are back to a single discipline again, so I have added that to the figure as well.

In her paper Stember argues that many people believe they work interdisciplinary, while in fact it is more common to work multidisciplinary.

For myself, I think I work on the edge between multidisciplinarity and interdisciplinarity. I do most certainly integrate knowledge and methods from different disciplines (mainly music, informatics, psychology, movement science), and try to create a holistic perspective based on this. However, I often feel that I have to choose approach when presenting my work for different (disciplinary) groups. Then I feel like a music researcher when talking to technologists, and as a technologist when talking to music people. This could mean that I have not been able to develop my ideas into a truly interdisciplinary approach, yet. I am not sure I will ever get to transdisciplinarity, and I am not even sure that that would be an interesting goal to work for either. After all, many of the interesting things I come across are based on the “friction” I encounter when working between the different disciplines.

On Thursday we are organising the first Norwegian Championship of standstill at University of Oslo. This is part of the University’s Open Day, a day when potential new students can come and see what happens on campus.

Besides the competitive part, the championship is (of course) a great way to gather more data about how people stand still. The art of standing still is something that has been a great interest of mine for the last year or so, and I have been carrying out different types of smaller experiments to understand more about the micromovements observed when standing still.

For the championship we are not going down the route of asking people to stand still for as long as possible, as they do in the world record for motionlessness (the record is a little more than 30 hours). Rather, we will look at how still people can stand for 6 minutes, measured in the average speed of a motion capture marker placed on the head. The unofficial Norwegian record is 3.8 mm/s, and on Thursday we will see if anyone beats that.

The championship is open for everyone, so do come by if you are in Oslo on Thursday. The poster for the event can be seen below:

Unfortunately, while setting up for the workshop I had problems getting Chuck to work on my new laptop. I haven’t had time to figure out what the problem actually is, but I think it may be an issue with OSX Lion. Anyways, since I was short on time I decided that the quickest solution was just to make something similar to Clix in Max. The result is Click-It, and the GUI looks like this:

The inside of the patch is not very advanced either. It is mainly a click~ object controlled by a metro, with some amplitude adjustments and a little bit of delay and feedback.

This may not be so interesting in itself, but the beauty of this thing comes with 10 or so computers playing it at the same time. Unfortunately, I didn’t make a video of the students performing it yesterday, so that will have to wait until next time it is played.

The patch is available from our fourMs web page. I had almost forgot that we have a repository for the Oslo Laptop Orchestra over at Google Code. That repository is severely out of date, though, and needs some cleaning up and updating at some point. The main reason for the sloppy maintenance is that the laptop orchestra has been kind of dormant over the last few years, since most of our energy has been put into Oslo iPhone Ensemble instead. That said, although the iPhone ensemble is a lot fun, laptop orchestras have some other qualities that I would like to get back to exploring more again at some point.

Fortunately, I have discovered that it is possible to use LaTeX fonts in OSX. Apparently, this is now included as a feature in the latest version(s) of the MacTeX distribution (?), but I also discovered that it is possible to just download the fonts (as OTF files) and install them directly:

1. Download the latest Computer Modern (Latex) Unicode fonts, the ones with *otf.tar.xz extension, from sourceforge

2. Uncompress the archive

3. Open the OSX application “Font Book”

4. Drag all the OTF files onto the Font Book

The end result is that the fonts show up in all OSX applications. All the font names start with CMU, so it is easy to find them when opening the font dialogue in your application.

See below for the full paper and video examples.

The Paper

• Download paper (PDF 2MB)

Abstract: The paper presents a method for sonification of human body motion based on motiongrams. Motiongrams show the spatiotemporal development of body motion by plotting average matrices of motion images over time. The resultant visual representation resembles spectrograms, and is treated as such by the new sonifyer module for Jamoma for Max, which turns motiongrams into sound by reading a part of the matrix and passing it on to an oscillator bank. The method is surprisingly simple, and has proven to be useful for analytical applications and in interactive music systems.

Full reference: A. R. Jensenius. Motion-sound interaction using sonification based on motiongrams. In ACHI 2012: The Fifth International Conference on Advances in Computer-Human Interactions, pages 170–175. IARIA, 2012.

@inproceedings{Jensenius:2012d,
    Author = {Jensenius, Alexander Refsum},
    Booktitle = {ACHI 2012: The Fifth International Conference on Advances in Computer-Human Interactions},
    Pages = {170--175},
    Publisher = {IARIA},
    Title = {Motion-sound Interaction Using Sonification based on Motiongrams},
    Year = {2012}}

Video examples

Video 1: A screencast demonstrating the jmod.sonifyer~ module.

Video 2: Examples of sonification of some basic movement patterns: up-down, sideways, diagonal and circular.

Video 3: One attempt at sonifying the two axes at the same time. Here both horizontal and vertical motiongrams are created from the same video recording, and the sonifications of the two motiongrams have been mapped to the left and right audio channel respectively.

Video 4: Examples of the importance of filtering and thresholding of the motion image for the final sounding result. The recordings were done at high-speed (200 fps) and played back at 25 fps.

Video 5: Sonification of a short violin improvisation (courtesy of Victoria Johnson).

Video 6: Sonification of a piece by a French-Canadian fiddler (courtesy of Erwin Schoonderwaldt).

Video 7: Sonification of free dance to music.

Video 8: Soniperforma: Performing with the sonifyer at Biermannsgården in Oslo on 18 December 2010. The performance was improvised and based on applying only video effects to change the sonic quality.

My solution to the problem of minor (and larger) interruptions has come out of a need of getting more concentrated time to focus on working in the lab, programming, writing papers, etc. It is very difficult to keep focus when new e-mails are popping up all the time. One thing is the interruption time, another thing is the recovery time. In the paper Understanding Email Interaction Increases Organizational Productivity, Jackson et al. show that the recovery time after an e-mail interruption is a little more than a minute, on average. This is based on detailed analyses of a group of people over several working days. They conclude:

All would see a new email-arrived icon appear in the system tray when new email arrived, and 57% would also see a new email-arrived pop-up dialogue box appear. It took each of them an average of 1 minute 44 seconds to react to a new email notification by activating the email application — 70% within six seconds of their arrival and 85% within two minutes of arrival. We found the time it took them to recover from email interrupts and return to their work at the same work rate at which they left it was on average 64 seconds.

An earlier study by DeMarco and Lister (Peopleware: Productive Projects and Teams), show that for phone calls it is much worse: the average recovery time is 15 minutes on average.

Even though these studies are a few years old, I don’t think we have been any better at multitasking in the last years. It may even be worse, with the other types of time-consuming things that we engage in (Twitter, Facebook, etc.). So in the future I will work even harder to minimize interruption time and the recovery time.

To also add some content to this blog entry, here is a short video I shot at Shiphol airport in Amsterdam yesterday. Walking through the otherwise busy airport, I noticed that they had set up a small artificial “park” in the middle of all the shops and cafeterias. There were some visual content, trees and posters, but the most interesting was the audible part: park-like sounds with birds, etc. The visual part was ok, but I think the sound design really made the installation work well.

I have to register all my research activities (publications, lectures, interviews, etc.) in the Norwegian report system Cristin. While they manage to extract data from Cristin on my personal UiO page, the code is not (yet?) available to integrate with other web pages.

That means that I will have to figure out another way to get to the data as well. Since I already use BibTeX for handling my reference database, including my own publications, this would be the best way to start. For NIME 2011 I used the bibtex2web script to generate the proceedings listing. That worked fine, when I finally figured out all the different options. Creating a static set of HTML files is not a problem for a conference. However, my own publication list changes quite often, and then it makes more sense to have a dynamic solution. So I have looked at a few different solutions. The main goal has been to find something that minimizes the time needed to both maintain the script itself as well as updating the BibTeX content. So far I am testing two different solutions.

Since my personal web page is using WordPress, it was easy to start with the bibtex2html WordPress plugin. It is really easy to use: upload the php files to the WP plugin folder, and copy BibTeX code directly into any post/page. The parsed result can be seen on my publication list. I only did one minor tweak to the plugin, removing the two dots (..) that appeared after each link in the bibtex2html.php script.

However, since we also need to create a dynamic solution for our fourMs group, I have also tested bibtexbrowser. This is still just a php script, but it has some more formatting options, and can be more easily and flexibly embedded in other web pages. So this might be a good solution for our group. A few tests can be found here, here and here. A positive thing about the bibtexbrowser is that it also integrates Google scholar metadata. Unfortunately, though, I cannot get the UTF-8 support to work, so all Norwegian characters look funny. Another issue is that it creates its own publication categories based on the titles of publisher, e.g., it has created a separate “workshop” category. So I guess I will have to tweak this plugin quite a bit for it to be useful.

Conclusion: I like the simplicity and dynamic approach of both bibtex2thml and the bibtexbrowser, but since they do not work perfectly out of the box I will have to tweak them to get what I want. Also, none of these two php-based solutions support the display of abstracts. So all in all I think the bibtex2web script that I used for the NIME 2011 proceedings looks better. For now I will try the bibtex2html WP plugin on my personal web page, but I will probably suggest that we use the bibtex2web script for our group page.