Kayaking motion analysis

Like many others, I bought a kayak during the pandemic, and I have had many nice trips in the Oslo fiord over the last year. Working at RITMO, I think a lot about rhythm these days, and the rhythmic nature of kayaking made me curious to investigate the pattern a little more.

Capturing kayaking motion

My spontaneous investigations into kayak motion began with simply recording a short video of myself kayaking. This was done by placing an action camera (a GoPro Hero 8, to be precise) on my life vest. The result looks like this:

In the future, it would be interesting to also test with a proper motion capture system (see this article for an overview of different approaches). However, as they say, the best motion capture system is the one you have at hand, and cameras are by far the easiest one to bring around.

Analysing kayaking motion

For the analysis, I reached for the Musical Gestures Toolbox for Python. It has matured nicely over the last year and is also where we are putting in most new development efforts these days.

The first step of motion analysis is to generate a motion video:

From the motion video, MGT will also create a motiongram:

From the motiongram, it is pretty easy to see the regularity of the kayaking strokes. This may be even easier from the videogram:

We also get information about the centroid and quantity of motion:

The quantity of motion can be used for further statistical analysis. But for now, I am more interested in exploring how it is possible to better visualise the rhythmic properties of the video itself. It was already on the list to implement directograms in MGT, and this is even higher on the list now.

The motion average image (generated from the motion video) does not reveal much about the motion.

It is generated by calculating the average of all the frames. What is puzzling is the colour artefacts. I wonder whether that is coming from some compression error in the video or a bug somewhere in MGT for Python. I cannot see the same artefacts in the average image:

Analysing the sound of kayaking

The video recording also has sound, so I was curious to see if this could be used for anything. True, kayaking is a quiet activity, so I didn’t have very high hopes. Also, GoPros don’t have particularly good microphones, and they compress the sound a lot. Still, there could be something in the signal. To begin with, the waveform display of the sound does not tell that much:

The spectrogram does not reveal that much either, although it is interesting to see the effects of the sound compression done by the GoPro (the horizontal lines from 5k and upward).

Then the tempogram is more interesting.

It is exciting to see that it estimates the tempo to be 122 BPM, and this resonates with theories about 120 BPM being the average tempo of moderate human activity.

This little investigation into the sound and video of kayaking made me curious about what else can be found from such recordings. In particular, I will continue to explore approaches to analysing the rhythm of audiovisual recordings. It also made me look forward to a new kayaking season!

Some thoughts on microphones for streaming and recording

Many people have asked me about what types of microphones to use for streaming and recording. This is really a jungle, with lots of devices and things to think about. I have written some blog posts about such things previously, such as tips for doing Skype job interviews, testing simple camera/mic solutions, running a Hybrid Disputation, and how to work with plug-in-power microphones.

Earlier today I held a short presentation about microphones at RITMO. This was during our informal Food & Paper lunch seminar, where people eat their lunch while listening to presentations about different topics (usually something academic, but sometimes also other things). Here is a cut-down version of the presentation:

The presentation starts by drawing up the main things to think about: microphones and speakers and the environments that people use these devices within. When we stream or record, we don’t really control other people’s speakers and environments. So the two things we should think about are (1) the microphone we use and (2) the environment we are in.

The make a long story short, here are my general advice:

• Place yourself in a “dry” and quiet space, if possible. A small room with carpets and curtains is much better than a big and empty space.
• A headset with a boom microphone will usually give the best sound overall, without feedback, and allow you to move your head around. I have many USB headsets from Logitech, Jabra, and Poly, and all of them are fine. The more expensive ones are more comfortable to wear, but the sound quality doesn’t really differ that much. I generally try to avoid Bluetooth headsets since they need to be charged and paired to function. If you can live with a cable, you will get better sound for a lower price.
• A “podcast-style” condenser microphone will give a more pleasant and radio-like sound. You can also avoid sitting with headphones on all the time, which is very tiresome after some hours. However, condenser microphones are usually relatively large, need a stand, and you may get into feedback problems. There are many options here, but I have been very positively surprised by this cheap Marantz USB microphone.
• A lavalier microphone is the best choice for making video recordings. They are small, pick up sounds nicely, and some (like the Røde Smartlav+) can be connected directly to a mobile phone or laptop.

There are always better, more expensive, and more complicated solutions out there. However, I am very impressed by some of the newest products that have arrived on the market. The products highlighted above are reasonably priced and will greatly improve the audio of both streaming and recording.

Convert MPEG-2 files to MPEG-4

This is a note to self, and could potentially also be useful to others in need of converting “old-school” MPEG-2 files into more modern MPEG-4 files using FFmpeg.

In the fourMs lab we have a bunch of Canon XF105 video cameras that record .MXF files with MPEG-2 compression. This is not a very useful format for other things we are doing, so I often have to recompress them to something else.

Inspecting one of the files, I just also discovered that they record the audio onto two mono channels:

```Stream #0:0: Video: mpeg2video (4:2:2), yuv422p(tv, bt709, top first), 1920x1080 [SAR 1:1 DAR 16:9], 50000 kb/s, 25 fps, 25 tbr, 25 tbn, 50 tbc

Stream #0:1: Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s

Stream #0:2: Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s```

So I also want to merge these two mono tracks (which are the left and right inputs of the camera) to a stereo track. FFmpeg comes in handy (as always), and I figured out that this little one-liner will do the trick:

`ffmpeg -i input.mxf -vf yadif -vcodec libx264 -q:v 3 -filter_complex "[0:a:0][0:a:1]amerge,channelmap=channel_layout=stereo[st]" -map 0:v -map "[st]" output.mp4`

An explanation of some of these settings:

• yadif: this is for deinterlacing the video
• libx264: this is probably unnecessary, but forces to use the better MPEG-4 compressor
• q:v 3: I find this to be a good setting for the video compressor
• filter_complex: this complex string (courtesy of reddit) does the merging of the two mono sources

Will probably try to add it to MGT-terminal at some point, but this blog post will suffice for now.

Simple tips for better video conferencing

Very many people are currently moving to video-based meetings. For that reason I have written up some quick advise on how to improve your setup. This is based on my interview advise, but grouped differently.

Network

The first important thing is to have as good a network as you can. Video conferencing requires a lot of bandwidth, so even though your e-mail and regular browsing works fine, it may still not be sufficient for good video transmission.

• Cabled network: If you are able to connect with an Ethernet cable to your router, that would usually always be the best and most solid solution.
• Wireless network: If cable won’t work for you (it is also difficult logistically in my own apartment), try to get as close as possible to your wi-fi router.

Audio

I would argue that improving the audio is more important than the video for video conferencing. Most video conferencing systems (Skype, Zoom, etc.) will prioritize the audio channel, which means that the video may stutter while the audio is passing through fine.

The main trick is to aim for separating the “foreground” as much as possible from the “background”. There are some very basic audio principles to follow:

• Use a headset: The best way to get decent sound for video conferencing, is to move the microphone as close as possible to your mouth. Headsets with a microphone boom in front of your face are the best, but a regular mobile phone headset (the one that came with your mobile phone, for example) would still be better than nothing.
• Use headphones: If you for some reason do not have a headset with built-in microphone, using a regular pair of headphones is still better than using the speakers on your computer. With this setup you use the microphone on the computer, which may not be ideal, but at least you won’t get feedback problems.
• Avoid reverberant rooms: If you aim for clarity in conversation, it is typically better to sit in a smaller and more damped room than a large one. That means that a bedroom is typically better than a larger living room. If you use a headset this is less important, but particularly if you only use the built-in microphone and speakers on a laptop, this could make a huge difference in how your voice gets through.
• Mute yourself: In most system there is a button to mute yourself. If you are not talking all the time, it helps to mute yourself from the discussion. Just remember to unmute when you want to say something!

Video

The same principle of separating “foreground” from “background” applies to the video.

• Lighting: To obtain the best possible video image, think about your placement with respect to lighting. It is, for example, not ideal to sit in front of a window, since a bright light in the background will make it difficult to see your face.
• Background: The best is to sit in front of a plain wall. If that is not possible, consider whether the background of your image is what you want to show to your fellow students/colleagues.
• Video angle: If you are using the built-in camera on your computer you may not have too many options for how to place the camera. But you may still consider shifting the camera position so that you and your surroundings look as good as possible.

Summing up

There are, of course, many ways to improve your video conferencing setup. Many people believe that you need to invest in expensive equipment to get good results. But even cheap consumer products are very capable of producing decent results these days. So it is more a matter of optimizing what you have. Good luck!

Testing simple camera and microphone setups for quick interviews

We just started a new run of our free online course Music Moves. Here we have a tradition of recording wrap-up videos every Friday, in which some of the course educators answer questions from the learners. We have recorded these in many different ways, from using high-end cameras and microphones to just using a handheld phone. We have found that using multiple cameras and microphones is too time-consuming, both in setup and editing. Using only a mobile phone is extremely easy to set up, but we have had challenges with the speech’s audibility. Before recording this semester’s wrapup videos, I, therefore, decided to test out some solutions based on the equipment I had lying around:

• Sony RX100 V
• GoPro Hero 7 w/o audio connector
• Zoom Q8
• Samsung Galaxy Note 8
• Røde Smartlav+ lavalier microphone
• DPA Core 4060 lavalier microphone

In the following I will show some of the results of the testing.

Sony RX 100 V

First out was the Sony RX 100 V, which is my preferred stills camera. It is small, yet has a large image chip. This makes it able to capture much nicer photos than my phone. The biggest problem with the Sony camera is that it does not have a separate microphone input. So I had to rely on the sound from the built-in microphones. As can be imagined, the image quality is good, although the focus is slightly off in this recording. The audio is ok, but the built-in microphone picks up a lot of ambient sounds. It works, but it is clearly not a good solution for this type of setup.

GoPro Hero 7

Then I moved on to the GoPro Hero 7 with just the built-in microphone. This worked much better than expected. The audio is quite clear, and it is easy to hear what I am saying. The colours of the video are vivid, but the image is compressed quite a bit. The video is very wide-angled, which is super-practical for such an interview setting, although it looks slightly skewed on the edges. But overall this was a positive surprise.

Connecting a Røde Smartlav+ to the GoPro results in an immaculate sound. In fact, this could have been a very nice setup, had it not been for some challenges with placing the camera. That is because the audio dongle for the GoPro is (1) bent downwards and (2) this makes it impossible to use the housing needed to put it on a tripod (as can be seen in the picture to the right). This makes it super-clumsy to use this setup in a real-life situation. I hear rumours about a new audio add-on for new GoPro cameras, which may be very interesting to check out.

Zoom Q8

My next device is Zoom Q8. This is actually a sound recorder with a built-in camera, so one would expect audio to be the main priority. This is also the case. The video is quite noisy, but the sound quality is much better than with the GoPro. Still, I find that the microphone picks up quite a bit of the room. This is good for music recordings, but not so good when the focus is on speech quality.

Hooking up a DPA 4060 lavalier microphone to the Zoom Q8 definitely helps. This is a high-quality microphone, and it needs phantom power (which the Zoom Q8 can deliver). As expected, this gives great sound, loud and clear. The downside is that it requires bringing an extra XLR cable together with the microphone and camera since the DPA cable is too short for such an interview setup. I like the wide-angle of the video, but the quality of the video is not very good.

Samsung Galaxy Note 8

Mobile phones are becoming increasingly powerful, and I also had to try the camera of my Samsung Galaxy Note 8. I have a small Manfrotto mobile phone stand, making it possible to place it on a tripod at a suitable distance. After recording, I realized how much less wide-angle the phone image is than the GoPro and Zoom cameras, leaving my head cut off in the shots. This doesn’t matter for the testing here, however. The first video is using the built-in microphone of the mobile phone. I am very positively surprised about how crisp and clear my voice is coming through here. In fact, it is quite similar to the GoPro. The video quality is also very good—clearly, the runner-up after the Sony camera.

And, finally, I connected the SmartLav+ lavalier microphone to the Samsung phone. Here the sound is, of course, very similar to the GoPro recordings.

Conclusion

It is not entirely straight forward to conclude from this testing, but here are some of my thoughts after this very rapid and not very systematic testing:

• Using on-body microphones (lavalier) greatly improves the audibility as compared to using built-in microphones.
• The DPA 4060 is great, but the Smartlav+ is more than good enough for interviews.
• The GoPro could have been a great device for such interviews, had it not been for the skewed image and the clumsiness of the audio adaptor.
• Unfortunately, even though the Zoom Q8 is the best audio device (as it should!), its video is too bad to be used for such recordings.
• All in all, I think that the easiest and best solution is the Samsung phone with Smartlav+.