Create timelapse video from images with FFmpeg

I take a lot of timelapse shots with a GoPro camera. Usually, I do this with the camera’s photo setting instead of the video setting. That is because I find it easier to delete unwanted pictures from the series that way. It also simplifies selecting individual photos when I want that. But then I need a way to create a timelapse video from the photos easily.

Here is an FFmpeg one-liner that does the job:

ffmpeg -r 10 -pattern_type glob -i "*.JPG" -s 1920x1440 -vcodec libx264 output.mp4

To break down the different parameters a little:

  • -r 10″: the framerate (fps)
  • “-pattern_type glob”: to allow for selecting all JPGs using “*.JPG”
  • “-s 1920×1440”: downscales the images to a pseudo-like HD format
  • “-vcodec libx264”: force to use this codec

New paper: Who Moves to Music? Empathic Concern Predicts Spontaneous Movement Responses to Rhythm and Music

A few days after Agata Zelechowska defended her PhD dissertation, we got the news that her last paper was finally published in Music & Science. It is titled Who Moves to Music? Empathic Concern Predicts Spontaneous Movement Responses to Rhythm and Music and was co-authored by Victor Gonzalez Sanchez, Bruno Laeng, Jonna Vuoskoski, and myself.

The paper is based on Agata’s headphones-speakers experiment. We have previously published a paper showing that people move more when listening on headphones. This, however, the focus was on the data gathered on individual differences. Many variables were tested, but it was only empathic concern that turned out to be a motion predictor.

Here is a short video teaser about the article:

And here is the abstract:

Moving to music is a universal human phenomenon, and previous studies have shown that people move to music even when they try to stand still. However, are there individual differences when it comes to how much people spontaneously respond to music with body movement? This article reports on a motion capture study in which 34 participants were asked to stand in a neutral position while listening to short excerpts of rhythmic stimuli and electronic dance music. We explore whether personality and empathy measures, as well as different aspects of music-related behaviour and preferences, can predict the amount of spontaneous movement of the participants. Individual differences were measured using a set of questionnaires: Big Five Inventory, Interpersonal Reactivity Index, and Barcelona Music Reward Questionnaire. Liking ratings for the stimuli were also collected. The regression analyses show that Empathic Concern is a significant predictor of the observed spontaneous movement. We also found a relationship between empathy and the participants’ self-reported tendency to move to music.

And the full reference is:

Zelechowska, A., Gonzalez Sanchez, V. E., Laeng, B., Vuoskoski, J. K., & Jensenius, A. R. (2020). Who Moves to Music? Empathic Concern Predicts Spontaneous Movement Responses to Rhythm and Music. Music & Science, 3, 2059204320974216. https://doi.org/10.1177/2059204320974216

The different parts of the experiment (from left to right): preparation, first listening session, first set of questionnaires, second listening session, second set of questionnaires.

We are working on preparing the data of the experiment for sharing. It will be shared as part of the Oslo Standstill Database.

Running a Hybrid Disputation

Yesterday, I wrote about Agata Zelechowska’s disputation. We decided to run it as a hybrid production, even though there was no audience present. It would, of course, have been easier to run it as an online-only event. However, we expect that hybrid is the new “normal” for such events, and therefore thought that it would be good to get started exploring the hybrid format right away. In this blog post, I will write up some of our experiences.

The setup in the hall, with the candidate presenting to the left and the disputation leader to the right.

The disputation was run in Forsamlingssalen, a nice lecture room in Harald Schjelderups hus, where RITMO is located. We had seen the need for recording lectures in the hall and had even before corona installed two PTZ cameras and a video mixer in the hall. This setup was primarily intended for recording lectures and secondary for streaming on YouTube. We actually never got around to use the new system before corona closed down the university in the spring. So the disputation was a good chance for getting the system up and running.

Forsamlingssalen, seen from the lecture podium. One PTZ camera is placed on the left wall and one in the back next to the projector. LED lights help illuminate the speakers.

What turned out to be the most challenging part of the setup was to figure out how to interface the system with Zoom. We quickly decided to use a Zoom Webinar instead of a Zoom Room. The Webinar solution is better for public events where you want to control the “production”. It is also safer since only invited panellists are allowed to show their camera and to speak.

Zoom (both the Webinar solution and regular Rooms) is in many ways a small video production tool in its own right. However, we realised that it is quite challenging to use in a more traditional multi-camera setup. Its strength is on allowing multiple people with single-camera setups to interact. We did make it work in the end, but it was quite a puzzle to get right.

People and PCs

There were four people visible in the disputation: the candidate and disputation leader were present in Forsamlingssalen and the two opponents that joined remotely. The two opponents used the normal Zoom client on their PCs, so their part was easy enough (of course, we ensured that they had good audio and video quality).

The second opponent projected on the wall during the disputation.

For the setup in Forsamlingssalen, the candidate was standing at the podium with a desktop PC with two screens and cabled ethernet. Her presentation was shown on the right screen and her notes on the left. We played with the idea of showing her presentation as a video stream through the video mixer but ended up using Zoom’s screen sharing function. That meant that we had to run Zoom on the PC, and then we could also add her image from a web camera sitting on top of the left screen.

The speaker desk, with two screens, a desk microphone, and a web camera above the screen to the left.

The image from that PC is also what goes to the projected screen in the hall. That was not so important now, since there was no audience. But for future hybrid disputations (and other events) we need to also think about people in the hall.

The hall has a nice microphone setup, with a swan neck microphone next to the PC screens and several wireless microphones. We ended up equipping both the candidate and disputation leader with wireless clip-on mics to ensure that they had a good sound coming through. The mixed microphone signal was then fed into the lecture PC to be shared through Zoom.

Two wireless microphones were used, one for the candidate and the other for the disputation leader. These microphones were connected to the PA system in the hall and sent as inputs to the desk PC to be included in the Zoom stream.
The disputation leader only had an empty lecturer desk. His image was captured from one of the PTZ cameras in the back of the hall, and his sound was captured through a clip-on mic that was connected to the sound mixer.

Cabling

The cabling in the room is set up so that there is a combined HDMI signal being sent from the podium to the video mixer. This signal contains the main image from the PC, which is also projected on the wall. It also contains the combined audio signal coming from the microphones and the PC, played over the hall’s loudspeakers. As such, we can easily tap the same signal that an audience in the hall will experience. Also, the two PTZ cameras’ signals in the hall go into separate channels on the video mixer. Below is a sketch of the cabling in the room.

The AV routing in the hall.

The original plan was to run productions from the video mixer, which can stream directly to various servers and also record on an SD card. Since we used Zoom for the disputation, however, we hooked up a separate PC in the control room, to which we fed the mixed video signal through a video grabber. This PC was then connected to Zoom, and we could switch between the two PTZ cameras in the live stream.

Research assistant Aleksander Tidemann controlling the two PTZ cameras, video mixer, and Zoom PC in the control room.

Lessons learned

Many things worked well, but we also learned some lessons.

  1. Running a hybrid event is (much) more difficult than doing a physical or online-only event. It is challenging to create something that works well both in the room and with an online audience.
  2. Having good audio is imperative. This is particularly tricky in a hybrid situation, in which you can easily get into feedback problems. Fortunately, we have a very robust PA system available with high-quality clip-on microphones.
  3. Combining Zoom with a multi-camera production pipeline is challenging. Zoom is good for connecting multiple people with one PC+camera+microphone each. Adding in a multi-camera video in one Zoom channel worked, but it is difficult to mix video for a window size you don’t know. Zoom, the viewer can choose the size and position of windows. Doing picture-in-picture on the video mixer, for example, may lead to video images that are too small to watch if the viewer is using “gallery” mode.
  4. It is challenging to be the main “producer” of a hybrid Zoom event. I have run many Zoom Room meetings and also several Webinars. But this was the first time I tried a large-scale Webinar event with a hybrid setup. It worked ok in the end, but it was tricky when I did not have access to the necessary “tools” to actually control what was going on. For example, as a host, you are allowed to turn people’s video and audio off. But you cannot turn them on again. I had originally planned to turn the camera and microphone on the lecture podium off during breaks, and then turn them back on again when we started each session. The candidate should not have to think about such things, but this meant that I had to physically go over to the machine to turn things on when we should start. That also meant that I had to sit in the hall because it would be too far to get to the control room one floor up.
A view of the hall from the control room, one floor up from the hall.

I think the final result was fine, as can be seen in a recording of the event:

In the future, however, I would probably not run such an event as a Zoom Webinar. Given that we have a nice multi-camera setup in the hall, it would be better to run it as a regular video stream. Then we would have full control over the production. The candidate could stand at the podium and focus on giving her presentation, and we could mix the audio and video in the back.

However, the challenging part with such a setup would be to figure out how to best add in the opponents in the mix. I would probably opt to connect on a separate PC (trough a Zoom Room) that would be shown separately from the presentation. Exactly how to do that will be an experiment for our next disputation!

PhD disputation of Agata Zelechowska

I am happy to announce that Agata Zelechowska yesterday successfully defended her PhD dissertation during a public disputation. The dissertation is titled Irresistible Movement: The Role of Musical Sound, Individual Differences and Listening Context in Movement Responses to Music and has been carried out as part of my MICRO project at RITMO.

A “zoom-style” disputation photo. From top left: Maria Witek (second opponent), Rolf Inge Godøy (committee leader), Marc Leman (first opponent), Agata Zelechowska (candidate), Peter Edwards (disputation leader), me (main supervisor), Bruno Laeng (co-supervisor), and Jonna Vuoskoski (co-supervisor, “photo-shopped” in).

The dissertation is composed of five papers and an extended introduction. The abstract reads:

This dissertation examines the phenomenon of spontaneous movement responses to music. It attempts to grasp and illustrate the complexity of this behaviour by viewing it from different perspectives. Unlike most previous studies on music and body movement, this dissertation places the focus on barely visible manifestations of movement, such as those that may occur when listening to music while standing still. The point of departure is a reflection on movement responses to music and why such responses are considered universal among humans. This is followed by a discussion on the different approaches to studying how music ‘inspires’ movement, and an overview of the different factors that can potentially contribute to the emergence of movement responses to music. The first goal of the empirical research was to verify the common conception that ‘music makes us move’ and examine whether such movement responses can be involuntary. Three of the five included papers show that music can, indeed, make people move, even when they try to stand as still as possible. The second goal is to explore different factors that contribute to movement responses to music. Throughout the included papers, several topics are examined, including rhythmic complexity, tempo, music genres, individual differences and playback systems. The theoretical chapters show how these topics fit into three broader components of the music experience: music, listener and context. Overall, the results suggest that several factors seem to increase movement responses to music: the clear underlying pulse in the sound stimuli, the rhythmic complexity, a tempo of around 120 beats per minute, listening on headphones rather than speakers and high empathy of the listener. All in all, this dissertation contributes to bridging several gaps in the literature on music-related body movement. It also broadens the perspective on why, how and when music moves us.

Abstract of Agata Zelechowska’s dissertation

Congratulations to Agata on the dissertation and the defense!

Opportunities and Challenges with Citizen Science

Citizen Science is on everyone’s lips these days, at least on the lips of people working in research administration, funding agencies and in institutional leadership. As a member of the EUA Expert Group on Open Science/Science 2.0, I am also involved in ongoing discussions on the topic.

Yesterday, I took part in the workshop Citizen Science in an institutional context organized by EUA and OpenAire. A recording of my talk is available here:

Video is good for many things, but textual information may be easier to search for and skim through, so in this blog post, I will summarize some of the points from my talk.

As always, I started by briefly explaining my reasoning for talking about Open Research instead of Open Science. This is particularly important for people working within the arts and humanities, for whom “science” may not be the best term.

Defining Citizen Science

There are lots of definitions of citizen science, such as this one:

Citizen science […] is scientific research conducted, in whole or in part, by amateur (or nonprofessional) scientists.

Wikipedia

That is fine on a general level, but it is more unclear what it means in practice. In my experience, many people think of citizen science as primarily involving citizens in the data collection. Then citizen science is just one building block in the (open) research ecosystem:

A narrow definition of citizen science.

Another, more open, definition of citizen science, focuses on the inclusion of citizens in all parts of the research process. This is the approach that I think is most interesting, and is the one I am focusing on.

A broader definition of citizen science.

Opportunities of Citizen Science

I have never thought of myself as a “citizen science researcher”. Some people build all their research on such an approach. For me, it has more been an add-on to other research activities. Still, I have done several citizen science-like projects over the years. In the talk, I presented two of these briefly: Hjernelæring and MusicLab. I will present these briefly in the following.

Case 1: Hjernelæring

The first case is my collaboration with Hjernelæring, a Norwegian company producing educational material for schools. I was challenged to create an exercise that could be used in classrooms, which could also be used to collect research data. My main research project at the moment (MICRO) is focusing on music-related micromotion, so it was natural to build an exercise around this. We have done several studies in the lab over the years, including the Norwegian Championship of Standstill. The latter has been an efficient way of attracting many participants to the study. We also try to give something back. All participants get a chance to download plots of their own micromotion, and we make the data available in the Oslo Standstill Database. So they are free to use and analyze the data if they wish.

Since we couldn’t rely on any particular technology in the classrooms, I ended up making an exercise where the kids would stand still with and without music, and then draw their experiences on a piece of paper. The teacher would then scan these drawings and send to me for analysis. I think this is a nice example of how to get involved with schools. It is research dissemination, because the kids learn about the research we are doing, why we do it, and what can come out of it. And it is data collection, since the teachers provide us with research data.

Case 2: MusicLab

The second case I presented was on MusicLab. This is an innovation project at the University of Oslo, where we at RITMO collaborate with the Science Library in exploring an extreme version of Open Research. Each MusicLab event is organized around a concert. The idea is to collect various types of data (motion capture, physiology, audio, video, etc.) of both performers and audience members that can be used in studying the event. There is usually also a workshop on a topic related to the concert, a panel discussion, and a data jockeying session in which some of the data is analyzed on the fly. As such, we try to open the entire research process to the public, and also include everyone in the data collection and analysis.

The main parts of a MusicLab event.

Challenges of Citizen Science

The last part of the presentation was devoted to presenting some of the challenges of Citizen Science. These were particularly focused on institutional challenges. There are, obviously, also many research challenges. Still, at the moment, I think it is important to help institutions to develop policies and tools for helping researchers to run Citizen Science projects.

My list of challenges include the need for (more):

  • technical infrastructure for data collection, handling, storage, and archiving. Many institutions have built up internal systems for data flows, but it is usually difficult to share data openly. I also see that IT departments are usually involved in handling storage solutions, while libraries are involved in archiving. This creates an unfortunate gap between the two (storage and archiving).
  • channels for connecting to citizens. Working with an external partner is usually a good strategy for connecting with citizens. Still, it also means that the researchers (and institutions) have to rely on a third party in communication with citizens. Some universities have built up their own Citizen Science centres, which may help with facilitating communication.
  • legal support (privacy+copyright). All the “normal” challenges of GDPR, copyright, etc., become even more difficult when involving citizens at all stages of the research process. Clearly, there is a need for more support to solve all the legal issues involved.
  • data management support. It is both a skill and a craft to collect data, handle data, equip data with metadata, store it, and archive it properly. Researchers need to learn all of these to some extent, but we also need more professional data managers to help at all stages. I think libraries’ future will largely be connected to the data management of various kinds.
  • strategies for avoiding bias and pressure from citizens. One of the big criticisms/scepticisms of Citizen Science is that it may lead to all sorts of unfortunate effects. Research is under pressure many places, and by involving more people in the research process, this may also lead to several challenges. I believe that more openness is the answer to this problem. Transparency at all levels will help expose whatever goes on in the data collection and analysis phase. This may mitigate potential challenges arising from people trying to push the research in one way or another. This, of course, requires the development of solid infrastructures, proper metadata, persistent IDs, version control, etc.
  • incentives and rewards for researchers (and institutions). Citizen Science is still new to many. As for anything else, if we want to make a change, it is necessary to support people interested in trying things out.
A sketch of how our new MusicLab app allows for secure data collection to UiO servers. The challenging part is to figure out how to give citizens access to the data, and how to handle the data archiving.

Conclusion

In sum, I believe there is a huge potential in citizen science. After thinking about it for a while, I think that more focus on Citizen Science is a natural extension of current Open Science initiatives. For that to happen, however, we need to solve all of the above. A good starting point is to develop policies from a top-down perspective. Equally important is to give researchers time (and some money) to set up pilot projects to try things out. After all, there will be no Citizen Science, if there are no researchers to initiate it in the first place.