NIME Publication Ecosystem Workshop

During the NIME conference this year (which as run entirely online due to the coronavirus crisis), I led a workshop called NIME Publication Ecosystem Workshop. In this post, I will explain the background of the workshop, how it was run in an asynchronous+synchronous mode, and reflect on the results.

If you don’t want to read everything below, here is a short introduction video I made to explain the background (shot at my “summer office” up in the Hardangervidda mountain range in Norway):

Background for the workshop

The idea of the NIME Publication Ecosystem Workshop was to continue community discussions started in the successful NIMEHub workshop in Brisbane in 2016 and the Open NIME workshop in Porto Alegre in 2019. Besides, comes discussions about establishing a NIME journal, as well as better solutions to archive various types of NIME-related activities.

The term “publication” should in this context be understood in a broad sense, meaning different types of output of the community, including but not limited to textual productions. This is particularly important at NIME since this community consists of people designing, building, and performing new musical interfaces.

When I gathered a workshop team and proposed the topic back in January, this was mainly coming out of the increasing focus on Open Research. Please note that I use “open research” here, not “open science”, a significant difference that I have written about in a previous blog post. The focus on more openness in research has recently received a lot of political attention through the Plan S initiative, The Declaration on Research Assessment (DORA), EU’s Horizon Europe, funder’s requirements of FAIR data principles, and so on.

Of course, the recent coronavirus crisis has made it even more necessary to develop open research strategies, as well as finding alternative ways of communicating our research. This also includes rethinking the format of conferences. The need to travel less is not something that will go away after the coronavirus crisis calms down, however. Long-term change is necessary to reduce problems with climate change. While such a move may feel limiting to some of us that could travel to international conferences every year, it also opens possibilities for many others to participate. The topic of this year’s NIME conference was “accessibility,” and it turned out that the virtual conference format was, indeed, one that promoted accessibility in many ways. I will write more on that in another blog post.

When it comes to openness in research, this is something the NIME community has embraced since the beginning. The paper proceedings, for example, have been freely available online all the time. Also, the database of the archive has been made available as a collection of BibTeX files. Some people don’t understand why we do this, but opening up also the metadata for the archive makes it much more flexible to integrate with other data sources. It also makes it much easier to research the community’s output.

Despite these efforts, there are also several things about the NIME conference that we have not been able to make openly available, such as hardware designs, code, empirical data, music performances, installations, and so on. This is not that we don’t want to, but it has proven hard to find long-term solutions that are maintainable by a volunteer-driven community. People in the community have different interests and skills, so it is essential to find solutions that are both innovative and user-friendly at the same time. The longevity of chosen solutions is also important since NIME is central to an increasing number of people’s careers. Hence, we need to balance the exploration of new solutions with the need for preservation and stability. 

In addition to finding solutions for the NIME conference itself, the establishment of a NIME journal has been discussed for several years. This discussion has surfaced again during the testing of a new paper template for the conference. But rather than thinking about the conference proceedings and a journal as two separate projects, one could imagine a broader NIME publication ecosystem that could cover everything from draft manuscripts, complete papers, peer-reviewed proceedings papers, and peer-reviewed journal papers. This could be thought of as a more “Science 2.0”-like system in which the entire research process is open from the beginning.

The aims of the workshop were therefore to:

  1. discuss how a broader publication ecosystem built around (but not limited to) the annual conference could work
  2. brainstorm and sketch concrete (technical) solutions to support such an idea
  3. agree on some concrete steps on how to proceed with the development of such ideas the coming year

Workshop format

We had initially planned to have a physical workshop in Birmingham but ended up with an online event. To make it as accessible as possible, we decided to run it using a combination of asynchronous and synchronous delivery. This included the preparation of various types of introductory material by the organizing committee and some participants. All of this material was gathered into a pre-workshop padlet, which was sent to the participants some days before the online workshop.

The synchronous part of the workshop was split over two-hour-long time slots. We ended up doing it like this to allow people from all time zones to participate in at least one of the workshops. Since most of the organizers were located in Europe, and the conference itself was scheduled around UK time, we ended up with one slot in the morning (9-10 UK time) and one in the afternoon (17-18 UK time). The program for each of the slots was the same so that everyone would feel that they participated equally in the event.

Around 30 people showed up for each time slot, with only a few participating in both. Since preparatory material was distributed beforehand, most of the online workshop time consisted of discussions in breakout rooms with 5-6 people in each group. The groups wrote their feedback into separate padlets and also reported back in a short plenary session at the end of the hour-long session.

A post-workshop padlet was created to gather links after the workshop. The topic was also lively discussed in separate threads on the Slack channel that was used during the conference. After the conference, and as a result of the workshop, we have established a forum on nime.org, with a separate ecosystem thread.

All the pre- and post-workshop material from the workshop has been archived in Zenodo.

Conclusions

It is, of course, impossible to conclude such a vast topic after one workshop. But what is clear is that there is an interest in the community to establish a more extensive ecosystem around the NIME conference. The establishment of a forum to continue discussion is one concrete move ahead. So is the knowledge gained from running a very successful online conference this year. This included pre-recorded talks, written Q&A in Slack channels, plenary sessions, and breakout rooms. A lot of this can also be archived and be part of an extended ecosystem. All in all, things are moving in the right direction, and I am very excited to see where we end up!

New publication: Headphones or Speakers? An Exploratory Study of Their Effects on Spontaneous Body Movement to Rhythmic Music

After several years of hard work, we are very happy to announce a new publication coming out of the MICRO project that I am leading: Headphones or Speakers? An Exploratory Study of Their Effects on Spontaneous Body Movement to Rhythmic Music (Frontiers Psychology).

From the setup of the experiment in which we tested the effects of listening to headphones and speakers.

This is the first journal article of my PhD student Agata Zelechowska, and it reports on a standstill study conducted a couple of years ago. It is slightly different than the paradigm we have used for the Championships of Standstill. While the latter is based on single markers on the head of multiple people, Agata’s experiment was conducted with full-body motion capture of individuals.

The most exciting thing about this new study, is that we have investigated whether there are any differences in people’s micromotion when they listen through either headphones or speakers. Is there a difference? Yes, it is! People move (a little) more when listening through headphones.

Want to know more? The article is Open Access, so you can read the whole thing here. The short summary is here:

Previous studies have shown that music may lead to spontaneous body movement, even when people try to stand still. But are spontaneous movement responses to music similar if the stimuli are presented using headphones or speakers? This article presents results from an exploratory study in which 35 participants listened to rhythmic stimuli while standing in a neutral position. The six different stimuli were 45 s each and ranged from a simple pulse to excerpts from electronic dance music (EDM). Each participant listened to all the stimuli using both headphones and speakers. An optical motion capture system was used to calculate their quantity of motion, and a set of questionnaires collected data about music preferences, listening habits, and the experimental sessions. The results show that the participants on average moved more when listening through headphones. The headphones condition was also reported as being more tiresome by the participants. Correlations between participants’ demographics, listening habits, and self-reported body motion were observed in both listening conditions. We conclude that the playback method impacts the level of body motion observed when people are listening to music. This should be taken into account when designing embodied music cognition studies.

Method chapter freely available

I am a big supporter of Open Access publishing, but for various reasons some of my publications are not openly available by default. This is the case for the chapter Methods for Studying Music-Related Body Motion that I have contributed to the Springer Handbook of Systematic Musicology.

I am very happy to announce that the embargo on the book ran out today, which means that a pre-print version of my chapter is finally freely available in UiO’s digital repository. This chapter is a summary of my experiences with music-related motion analysis, and I often recommend it to students. Therefore it is great that it is finally available to download from everywhere.

Abstract

This chapter presents an overview of some methodological approaches and technologies that can be used in the study of music-related body motion. The aim is not to cover all possible approaches, but rather to highlight some of the ones that are more relevant from a musicological point of view. This includes methods for video-based and sensor-based motion analyses, both qualitative and quantitative. It also includes discussions of the strengths and weaknesses of the different methods, and reflections on how the methods can be used in connection to other data in question, such as physiological or neurological data, symbolic notation, sound recordings and contextual data.

Pixel array images of long videos in FFmpeg

Continuing my explorations of FFmpeg for video visualization, today I came across this very nice blog post on creating “pixel array” images of videos. Here the idea is to reduce every single frame into only one pixel, and to plot this next to each other on a line. Of course, I wanted to try this out myself.

I find that creating motiongrams or videograms is a good way to visualize the content of videos. They are abstract representations, but still reveal some of what is going on. However, for longer videos, motiongrams may be a bit tricky to look at, and they also take a lot of time to generate (hours, or even days). For that reason I was excited to see how pixel array images would work on some of my material.

First I tried on my “standard” dance video:

which gives this pixel array image:

Pixel array image (640 pixels wide) of the dance video above.

Yes, that is mainly a blue line, resulting from the average colour of the video being blue throughout the entire video.

Then I tried with one of the videos from the AIST Dance Video Database:

Which results in this pixel array image:

Pixel array image (640 pixels wide) of the dance video above.

And, yes, that is mainly a gray line… I realize that this method does not work very well with single-shot videos.

To try something very different, I also decided to make a pixel array image of Bergensbanen, a 7-hour TV production of the train between Oslo and Bergen. I made videograms of this recording some years ago, which turned out to be quite nice. So I was excited to see how a pixel array image would work. The end result looks like this (1920 pixels wide):

Pixel array image (1920 pixels wide) of the 7-hour TV production Bergensbanen

As you see, not much is changing, but that also represents the slowness of the train ride. While I originally thought this would be a smart representation, I still think that my videograms were more informative, such as this one:

Bergensbanen
Videogram of Bergensbanen

The big difference between the two visualizations, is that each frame is represented with vertical information in the videogram. The pixel array image, on the other hand, only displays one single pixel per frame. That said, it took only some minutes to generate the pixel array image, and I recall spending several days on generating the videogram.

To sum up, I think that pixel array images are probably more useful for movies and video material in which there are lots of changes throughout. They would be better suited for such a reduction technique. For my videos, in which I always use single-shot stationary cameras, motiongrams and videograms may still be the preferred solution.

Convert MPEG-2 files to MPEG-4

Image result for Canon XF-105
Canon XF105

This is a note to self, and could potentially also be useful to others in need of converting “old-school” MPEG-2 files into more modern MPEG-4 files using FFmpeg.

In the fourMs lab we have a bunch of Canon XF105 video cameras that record .MXF files with MPEG-2 compression. This is not a very useful format for other things we are doing, so I often have to recompress them to something else.

Inspecting one of the files, I just also discovered that they record the audio onto two mono channels:

Stream #0:0: Video: mpeg2video (4:2:2), yuv422p(tv, bt709, top first), 1920x1080 [SAR 1:1 DAR 16:9], 50000 kb/s, 25 fps, 25 tbr, 25 tbn, 50 tbc

Stream #0:1: Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s

Stream #0:2: Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s

So I also want to merge these two mono tracks (which are the left and right inputs of the camera) to a stereo track. FFmpeg comes in handy (as always), and I figured out that this little one-liner will do the trick:

ffmpeg -i input.mxf -vf yadif -vcodec libx264 -q:v 3 -filter_complex "[0:a:0][0:a:1]amerge,channelmap=channel_layout=stereo[st]" -map 0:v -map "[st]" output.mp4

An explanation of some of these settings:

  • yadif: this is for deinterlacing the video
  • libx264: this is probably unnecessary, but forces to use the better MPEG-4 compressor
  • q:v 3: I find this to be a good setting for the video compressor
  • filter_complex: this complex string (courtesy of reddit) does the merging of the two mono sources

Will probably try to add it to MGT-terminal at some point, but this blog post will suffice for now.