New online course: Motion Capture

After two years in the making, I am happy to finally introduce our new online course: Motion Capture: The art of studying human activity.

The course will run on the FutureLearn platform and is for everyone interested in the art of studying human movement. It has been developed by a team of RITMO researchers in close collaboration with the pedagogical team and production staff at LINK – Centre for Learning, Innovation & Academic Development.

Motivation

In the past, we had so few users in the fourMs lab that they could be trained individually. With all the new exciting projects at RITMO and an increasing amount of external users, we realized that it was necessary to have a more structured approach to teaching motion capture to new users.

The idea was to develop an online course that would teach incoming RITMO students, staff, and guests about motion capture basics. After completing the online course, they would move on to hands-on training in the lab. However, once the team started sketching the content of the course, it quickly grew in scope. The result is a six-week online course, a so-called massive open online course (MOOC) that will run on the FutureLearn platform.

People talking in lab
From one of the early workshops with LINK, in which I explain the basics of a motion capture system (Photo: Nina Krogh).

MOOC experience

Developing a MOOC is a major undertaking, but we learned a lot when we developed Music Moves back in 2015-2016. Thousands of people have been introduced to embodied music cognition through that course. In fact, we will run it for the seventh time on 24 January 2022.

Motion capture is only mentioned in passing in Music Moves. Many learners ask for more. Looking around, we haven’t really found any general courses on motion capture. There are many system-specific tutorials and courses, but not any that introduce the basics of motion capture more broadly. As I have written about in the Springer Handbook of Systematic Musicology (open access version), there are many types of motion capture systems. Most people think about the ones where users wear a suit with reflective markers, but this is only one type of motion capture.

From biomechanics to data management

In the new Motion Capture course, we start with teaching the basics of human anatomy and biomechanics. I started using motion capture without that knowledge myself and have later realized that it is better to understand a bit about how the body moves before playing with the technology.

People talking in front of a whiteboard
RITMO lab engineer Kayla Burnim discusses the course structure with Audun Bjerknes and Mirjana Coh from LINK (Photo: Nina Krogh).

The following weeks in the course contain all the information necessary to conduct a motion capture experiment: setting up cameras, calibrating the system, post-processing, and analysis. The focus is on infrared motion capture, but some other sensing technologies are also presented, including accelerometers, muscle sensors, and video analysis. The idea is not to show everything but to give people a good foundation when walking into a motion capture lab.

The last week is dedicated to data management, including documentation, privacy, and legal issues. These are not the most exciting topics if you want to motion capture. But they are necessary if you’re going to research according to today’s regulations.

From idea to course

Making a complete online course is a major undertaking. Having done it twice, I would compare it to writing a textbook. It helps with prior experience and a good team, but it is still a significant team effort.

We worked with UiO’s Centre for Learning, Innovation and Academic Development, LINK, when developing Music Moves, and I also wanted to get them on board for this new project. They helped structure the development into different stages: ideation, development of learning outcomes, production planning, and production. It is tempting to start filming right away, but the result is much better if you plan properly. The last time we made the quizzes and tests last, and this time, I pushed to make them first to know the direction we were heading.

People talking in front of a table
Mikkel Kornberg Skjeflo from LINK explains how the learning experience becomes more engaging by using different learning activities in the course (Photo: Nina Krogh).

Video production

In Music Moves, we did a lot of “talking head” studio recordings, like this one:

It works in bringing over content, but I look uncomfortable and don’t get through the content very well. I find the “dialogue videos” much more engaging:

Looking at the feedback from learners (we have had around 10 000 people in Music Moves over the years!), they also seem to engage more with less polished video material. So for Motion Capture, we decided to avoid “lecture videos”. Instead, we created situations where pairs would talk about a particular topic. We wrote scripts first, but the recordings were spontaneous, making for a much more lively interaction.

The course production coincided with MusicTestLab, an event for testing motion capture in a real-world venue. The team agreed to use this event as a backdrop for the whole course, making for a quote chaotic recording session. Filming an online course in parallel to running an actual experiment that was also streamed live was challenging, but it also gives the learners an authentic look into how we work.

Musicians on stage with motion capture equipment.
Audun Bjerknes and Thea Dahlborg filming a motion capture experiment in the foyer of the Science Library.

Ready for Kick-off

The course will run on FutureLearn from 24 January 2022. In the last months, we have done the final tweaking of the content. Much effort has also been put into ensuring accessibility. All videos have been captioned, images have been labelled, and copyrights have been checked. That is why I compare it to writing a textbook. Writing the content is only part of the process. Similarly, developing a MOOC is not only about writing texts and recording videos. The whole package needs to be in place.

Music Moves has been running since 2016 and is still going strong. I am excited to see how Motion Capture will be received!

Try not to headbang challenge

I recently came across a video of the so-called Try not to headbang challenge, where the idea is to, well, not to headbang while listening to music. This immediately caught my attention. After all, I have been researching music-related micromotion over the last years and have run the Norwegian Championship of Standstill since 2012.

Here is an example of Nath & Johnny trying the challenge:

As seen in the video, they are doing ok, although they are far from sitting still. Running the video through the Musical Gestures Toolbox for Python, it is possible to see when and how much they moved clearly.

Below is a quick visualization of the 11-minute long sequence. The videogram (similar to a motiongram but of the original video) shows quite a lot of motion throughout. There is no headbanging, but they do not sit still.

A videogram of the complete video recording (top) with a waveform of the audio track. Two selected frames from the sequence and “zoomed-in” videograms show the motion of specific passages.

There are many good musical examples listed here. We should consider some of them for our next standstill championship. If corona allows, we plan to run a European Championship of Standstill in May 2022. More information soon!

New Book Chapter: Gestures in ensemble performance

I am happy to announce that Cagri Erdem and I have written a chapter titled “Gestures in ensemble performance” in the new book Together in Music: Coordination, Expression, Participation edited by Renee Timmers Freya Bailes, and Helena Daffern.

Video Teaser

For the book launch, Cagri and I recorded a short video teaser:

Abstract

The more formal abstract is:

The topic of gesture has received growing attention among music researchers over recent decades. Some of this research has been summarized in anthologies on “musical gestures”, such as those by Gritten and King (2006), Godøy and Leman (2010), and Gritten and King (2011). There have also been a couple of articles reviewing how the term gesture has been used in various music-related disciplines (and beyond), including those by Cadoz and Wanderley (2000) and Jensenius et al. (2010). Much empirical work has been performed since these reviews were written, aided by better motion capture technologies, new machine learning techniques, and a heightened awareness of the topic. Still there are a number of open questions as to the role of gestures in music performance in general, and in ensemble performance in particular. This chapter aims to clarify some of the basic terminology of music-related body motion, and draw up some perspectives of how one can think about gestures in ensemble performance. This is, obviously, only one way of looking at the very multifaceted concept of gesture, but it may lead to further interest in this exciting and complex research domain.

Ten years after Musical Gestures

We began writing this ensemble gesture chapter in 2020, about ten years after the publication of the chapter Musical gestures: Concepts and methods in research. That musical gesture chapter has, to my surprise, become my most-cited publication to date. When I began working on the topic of musical gestures with Rolf Inge Godøy back in the early 2000s, it was still a relatively new topic. Most music researchers I spoke to didn’t understand why we were interested in the body.

Fast forward to today, and it is hard to find music researchers that are not interested in the body in one way or another. So I am thrilled about the possibility of expanding some of the “old” thoughts about musical gestures into the ensemble context in the newly published book chapter.

Rigorous Empirical Evaluation of Sound and Music Computing Research

At the NordicSMC conference last week, I was part of a panel discussing the topic Rigorous Empirical Evaluation of SMC Research. This was the original description of the session:

The goal of this session is to share, discuss, and appraise the topic of evaluation in the context of SMC research and development. Evaluation is a cornerstone of every scientific research domain, but is a complex subject in our context due to the interdisciplinary nature of SMC coupled with the subjectivity involved in assessing creative endeavours. As SMC research proliferates across the world, the relevance of robust, rigorous empirical evaluation is ever-increasing in the academic and industrial realms. The session will begin with presentations from representatives of NordicSMC member universities, followed by a more free-flowing discussion among these panel members, followed by audience involvement.

The discussion was moderated by Sofia Dahl (Aalborg University) and consisted of Nashmin Yeganeh (University of Iceland), Razvan Paisa (Aalborg University), and Roberto Bresin (KTH).

The challenge of interdisciplinarity

Everyone in the panel agreed that rigorous evaluation is important. The challenge is to figure out what type(s) of evaluation is useful and plausible within sound and music computing research. This was efficiently illustrated in a list of the different methods that are employed by the researchers at KTH.

A list of methods in use by the sound and music computing researchers at KTH.

Roberto Bresin had divided the KTH list into methods that they have been working with for decades (in red) and newer methods that they are currently exploring. The challenge is that each of these methods requires different knowledge and skills, and they all have different types of evaluation.

Although we have a slightly different research profile at UiO than at KTH, we also have a breadth of methodological approaches in SMC-related research. I pointed to a model I often use to explain what we are doing:

A simplified model of explaining my research approach.

The model has two axes. One shows a continuum between artistic and scientific research methods and outputs. Another is a continuum between performing research on natural and cultural phenomena. In addition, we develop and use various types of technologies for all of these.

The reason I like to bring up this model is to explain that things are connected. I often hear that artistic and scientific research are completely different things. Sure, they are different, but there are also commonalities. Similarly, there is an often unnecessary divide between the humanities and the social and natural sciences. True, they have different foci but when studying music we need to take all of these into account. Music involves everything from “low-level” sensory phenomena to “high-level” emotional responses. One can focus on one or the other, but if we really want to understand musical experiences – or make new ones for that matter – we need to see the whole picture. Thus, evaluations of whatever we do also need to have a holistic approach.

Open Research as a Tool for Rigorous Evaluation

My entry into the panel discussion was that we should use the ongoing transition to Open Research practices as an opportunity to also perform more rigorous evaluations. I have previously argued why I believe open research is better research. The main argument is that sharing things (methods, code, data, publications, etc.) openly forces researchers to document everything better. Nobody wants to make sloppy things publicly available. So the process of making all the different parts of the open research puzzle openly available is a critical component of a rigorous evaluation.

In the process of making everything open, we realize, for example, that we need better tools and systems. We also experience that we need to think more carefully about privacy and copyright. That is also part of the evaluation process and lays the ground for other researchers to scrutinize what we are doing.

Summing up

One of the challenges of discussing rigorous evaluation in the “field” of sound and music computing is that we are not talking about one discipline with one method. Instead, we are talking about a set of approaches to developing and using computational methods for sound and music signals and experiences. If you need to read that sentence a couple of times, it is understandable. Yes, we are combining a lot of different things. And, yes, we are coming from different backgrounds: the arts and humanities, the social and natural sciences, and engineering. That is exactly what is cool about this community. But it is also why it is challenging to agree on what a rigorous evaluation should be!

MusicLab Copenhagen

After nearly three years of planning, we can finally welcome people to MusicLab Copenhagen. This is a unique “science concert” involving the Danish String Quartet, one of the world’s leading classical ensembles. Tonight, they will perform pieces by Bach, Beethoven, Schnittke and folk music in a normal concert setting at Musikhuset in Copenhagen. However, the concert is nothing but normal.

Live music research

During the concert, about twenty researchers from RITMO and partner institutions will conduct investigations and experiments informed by phenomenology, music psychology, complex systems analysis, and music technology. The aim is to answer some big research questions, like:

  • What is musical complexity?
  • What is the relation between musical absorption and empathy?
  • Is there such a thing as a shared zone of absorption, and is it measurable?
  • How can musical texture be rendered visually?

The concert will be live-streamed (on YouTube and Facebook) and it will also be aired on Danish radio. There will also be a short film documenting the whole process.

Researchers and staff from RITMO (and friends) in front of the concert venue.

Real-world Open Research

This concert will be the biggest and most complex MusicLab event to date. Still, all the normal “ingredients” of a MusicLab will be in place. The core is a spectacular performance. We will capture a lot of data using state-of-the-art technologies, but in a way that is as little obtrusive as possible for performers and the audience. After the concert, both performers and researchers will talk about the experience.

Of course, being a flagship Open Research project, all the collected data will be shared openly. The researchers will show glimpses of data processing procedures as part of the “data jockeying” at the end of the event. However, it is first when all data is properly uploaded and pre-processed that data processing can start. All the involved researchers will dig into their respective data. But since everything is openly available, anyone can go in and work on the data as they wish.

Proper preparation

Due to the corona situation, the event has been postponed several times. That has been unfortunate and stressful for everyone involved. On the positive side, it has also meant that we have been able to rehearse and prepare very well. Already a year ago we ran a full rehearsal of the technical setup of the concert. We even live-streamed the whole preparation event, in the spirit of “slow TV”:

I am quite confident that things will run smooth during the concert. Of course, there are always obstacles. For example, one of our eye-trackers broke in one of the last tests. And it is always exciting to wait for Apple and Google to approve updates of our MusicLab app in their respective app stores.

Want to see how it went. Have a look here.