See below for the full paper and video examples.

The Paper

• Download paper (PDF 2MB)

Abstract: The paper presents a method for sonification of human body motion based on motiongrams. Motiongrams show the spatiotemporal development of body motion by plotting average matrices of motion images over time. The resultant visual representation resembles spectrograms, and is treated as such by the new sonifyer module for Jamoma for Max, which turns motiongrams into sound by reading a part of the matrix and passing it on to an oscillator bank. The method is surprisingly simple, and has proven to be useful for analytical applications and in interactive music systems.

Full reference: A. R. Jensenius. Motion-sound interaction using sonification based on motiongrams. In ACHI 2012: The Fifth International Conference on Advances in Computer-Human Interactions, pages 170–175. IARIA, 2012.

@inproceedings{Jensenius:2012d,
    Author = {Jensenius, Alexander Refsum},
    Booktitle = {ACHI 2012: The Fifth International Conference on Advances in Computer-Human Interactions},
    Pages = {170--175},
    Publisher = {IARIA},
    Title = {Motion-sound Interaction Using Sonification based on Motiongrams},
    Year = {2012}}

Video examples

Video 1: A screencast demonstrating the jmod.sonifyer~ module.

Video 2: Examples of sonification of some basic movement patterns: up-down, sideways, diagonal and circular.

Video 3: One attempt at sonifying the two axes at the same time. Here both horizontal and vertical motiongrams are created from the same video recording, and the sonifications of the two motiongrams have been mapped to the left and right audio channel respectively.

Video 4: Examples of the importance of filtering and thresholding of the motion image for the final sounding result. The recordings were done at high-speed (200 fps) and played back at 25 fps.

Video 5: Sonification of a short violin improvisation (courtesy of Victoria Johnson).

Video 6: Sonification of a piece by a French-Canadian fiddler (courtesy of Erwin Schoonderwaldt).

Video 7: Sonification of free dance to music.

Video 8: Soniperforma: Performing with the sonifyer at Biermannsgården in Oslo on 18 December 2010. The performance was improvised and based on applying only video effects to change the sonic quality.

My solution to the problem of minor (and larger) interruptions has come out of a need of getting more concentrated time to focus on working in the lab, programming, writing papers, etc. It is very difficult to keep focus when new e-mails are popping up all the time. One thing is the interruption time, another thing is the recovery time. In the paper Understanding Email Interaction Increases Organizational Productivity, Jackson et al. show that the recovery time after an e-mail interruption is a little more than a minute, on average. This is based on detailed analyses of a group of people over several working days. They conclude:

All would see a new email-arrived icon appear in the system tray when new email arrived, and 57% would also see a new email-arrived pop-up dialogue box appear. It took each of them an average of 1 minute 44 seconds to react to a new email notification by activating the email application — 70% within six seconds of their arrival and 85% within two minutes of arrival. We found the time it took them to recover from email interrupts and return to their work at the same work rate at which they left it was on average 64 seconds.

An earlier study by DeMarco and Lister (Peopleware: Productive Projects and Teams), show that for phone calls it is much worse: the average recovery time is 15 minutes on average.

Even though these studies are a few years old, I don’t think we have been any better at multitasking in the last years. It may even be worse, with the other types of time-consuming things that we engage in (Twitter, Facebook, etc.). So in the future I will work even harder to minimize interruption time and the recovery time.

To also add some content to this blog entry, here is a short video I shot at Shiphol airport in Amsterdam yesterday. Walking through the otherwise busy airport, I noticed that they had set up a small artificial “park” in the middle of all the shops and cafeterias. There were some visual content, trees and posters, but the most interesting was the audible part: park-like sounds with birds, etc. The visual part was ok, but I think the sound design really made the installation work well.

I have to register all my research activities (publications, lectures, interviews, etc.) in the Norwegian report system Cristin. While they manage to extract data from Cristin on my personal UiO page, the code is not (yet?) available to integrate with other web pages.

That means that I will have to figure out another way to get to the data as well. Since I already use BibTeX for handling my reference database, including my own publications, this would be the best way to start. For NIME 2011 I used the bibtex2web script to generate the proceedings listing. That worked fine, when I finally figured out all the different options. Creating a static set of HTML files is not a problem for a conference. However, my own publication list changes quite often, and then it makes more sense to have a dynamic solution. So I have looked at a few different solutions. The main goal has been to find something that minimizes the time needed to both maintain the script itself as well as updating the BibTeX content. So far I am testing two different solutions.

Since my personal web page is using WordPress, it was easy to start with the bibtex2html WordPress plugin. It is really easy to use: upload the php files to the WP plugin folder, and copy BibTeX code directly into any post/page. The parsed result can be seen on my publication list. I only did one minor tweak to the plugin, removing the two dots (..) that appeared after each link in the bibtex2html.php script.

However, since we also need to create a dynamic solution for our fourMs group, I have also tested bibtexbrowser. This is still just a php script, but it has some more formatting options, and can be more easily and flexibly embedded in other web pages. So this might be a good solution for our group. A few tests can be found here, here and here. A positive thing about the bibtexbrowser is that it also integrates Google scholar metadata. Unfortunately, though, I cannot get the UTF-8 support to work, so all Norwegian characters look funny. Another issue is that it creates its own publication categories based on the titles of publisher, e.g., it has created a separate “workshop” category. So I guess I will have to tweak this plugin quite a bit for it to be useful.

Conclusion: I like the simplicity and dynamic approach of both bibtex2thml and the bibtexbrowser, but since they do not work perfectly out of the box I will have to tweak them to get what I want. Also, none of these two php-based solutions support the display of abstracts. So all in all I think the bibtex2web script that I used for the NIME 2011 proceedings looks better. For now I will try the bibtex2html WP plugin on my personal web page, but I will probably suggest that we use the bibtex2web script for our group page.

• title
• abstract
• content

But for the ERC application I had to handle four levels:

• Title
• Summary (2000 characters)
• Synopsis (5 pages)
• Proposal (15 pages)

While working on the application, I started thinking about my old fascination of hypertext theory. One concept I found (and still find) interesting here is Ted Nelson’s idea of stretchtext. Stretchtext can be seen as text that can literally be “stretched” to any desired length (see, for example, this example). Conceptually this makes sense. After all, we as humans are able to do such stretching fairly easily, always trying to maximize our content to the limitations we may have. For example, I have no problems talking about my current research project for 1 minute, 5 minutes, 20 minutes or 45 minutes, it is just about “interpolating” the content over the required timeframe. The challenge, of course, is to balance the content in such a way that it makes sense for different durations or number of pages.

But how do you go about when having to write 5 and 15 pages about the same thing. Should you write 15 pages first, and then cut it down to 5? Or is it better to start with 5 pages and then “interpolate” it to 15? My approach this time was not particularly structured, and I constantly found myself moving back and forth between the two documents. This was perhaps not the most ideal solution, since I often found myself making the same changes twice.

The strategy I ended up with, and that I would probably start out with If I were to do such a thing again, was to use the commenting function of LaTeX more actively. In regular word processing software (MS Word, OpenOffice, etc.) there is no easy way to include or remove content from the document easily. The text in the document is there, and if you remove it, it is gone. In LaTeX it is possible to comment out blocks of text by just typing the % sign in front of the line. This makes it easy to “turn off” whole blocks of text. As such, my final 5 page synopsis document contained more or less the same stuff as the full 15 page document, but with large parts of the text commented out.

It would have been nice if LaTeX had had the opportunity to define levels of text. Then I could have chosen to write only one document, and defined which parts should be at level 1, 2, 3, etc. This could then have been used to output the different levels more or less automatically. Such an approach could perhaps be done done with a text outliner (e.g., OmniOutliner), and I am curious to test this out at some point.

However, the biggest challenge of writing a stretchtext is probably not the software being used. It is rather to figure out what content to include, and make it work linguistically at the different levels. In the end, you might end up with writing two separate documents after all…

The video is from Victoria’s final performance as part of her research fellowship in the arts (PhD-equivalent), which happened Monday 28 March 2011 at the Norwegian Academy of Music.

As I wrote earlier this year:

Transformation a piece where we are using video analysis to control sound selection and spatialisation. We have been developing the setup and piece during the last couple of years, and performed variations of the piece at MIC, the Opera house and at the music academy last year.

Flash Player is dead. Its time has passed. It’s buggy. It crashes a lot. It requires constant security updates. It doesn’t work on most mobile devices. It’s a fossil, left over from the era of closed standards and unilateral corporate control of web technology. Websites that rely on Flash present a completely inconsistent (and often unusable) experience for fast-growing percentage of the users who don’t use a desktop browser. It introduces some scary security and privacy issues by way of Flash cookies. Flash makes the web less accessible. At this point, it’s holding back the web.

Since I am starting with a clean system Flash is not installed in the first place, so it is an easy start to the campaign. Unfortunately, Firefox complains about the missing Flash plugin every time I come to a web page with Flash content. This can be changed, although it is one of those “hidden” features of Firefox. I came across a web page describing how to disable the plugin warning when opening about:config in Firefox. However, the variable has a different name on my system. The correct name is “plugins.hide_infobar_for_missing_plugin”, and it should be changed to “true”, as shown in this screenshot.

Two of the main sites that I would need Flash for are YouTube and Vimeo. Fortunately, Youtube has an HTML5-player available that seems to work fine. Vimeo, on the other hand, only has HTML5 support in Safari and Chrome. Too bad, but hopefully a Firefox version will come soon.

1. The record is for continuously standing motionless.
2. You must stand: sitting is not allowed.
3. No facial movements are allowed other then the involuntary blinking of the eye.
4. Deep breathing is permitted provided it does not involve observable movement notably greater than that in normal breathing.
5. No rest breaks are allowed at any point during the event.
6. The venue for such an event should be such that the general public can view.

But from my point of view, being interested in micromovements, I would be very curious to see how still these record holders actually were.

At the ArtsIT conference next month I will present the results of a study on standstill that I have conducted together with Kari Anne Bjerkestrand. I have given a sneak peek of the data earlier, and below is another figure with plots of motion capture data from the study. The plots show data of a marker placed on the neck, from six different 10-minute long standstill recordings of myself and Kari Anne. It is only the vertical position of the marker that is plotted.

From the plots we can see that the running marker displacement was at the scale of only a few millimeters, with a maximum displacement of less than 10mm. It can be argued that this is not much, but it certainly is not absolutely still.

One thing is the quantitative data, another is our subjective experience of standing still. Even though we tried our best to stand physically still, we could easily notice how we were swaying back and forth, doing postural adjustments, etc. Observing the video recordings of ourselves afterwards, it is also possible to see these micromovements through visual inspection only.

Based on these findings, I would be very curious to see how still a person can actually stand, not only measured in hours and minutes, but also in millimeters. So to any aspiring world record breakers: please come and do your next attempt in our lab!

First I found a suggestion to use the setspace function, but it turns out that it is much easier to just use the bibsep option to natbib. Just add the following to the preamble:

\usepackage{natbib}
\setlength{\bibsep}{0.0pt}

and you will get something like this:

Up until now I have been using things like the rather ugly \vspace{-7pt} command to remove space between list items. Now I finally decided to figure out a better solution. While there are many different package to help with this, it seems like the enumitem package is the newest and most comprehensive solution for making compact lists.

It is possible to change the settings of individual lists, but it may be easier to change the settings globally. Then you need to add this to your preamble:

\usepackage{enumitem}
\setitemize{noitemsep,topsep=0pt,parsep=0pt,partopsep=0pt}

And you can then use the regular itemize function in LaTeX. The above example will then look like this:

This way you can remove the space between each item, between the list and the surrounding paragraphs, and the indention of the list items.