Creating multi-exposure keyframe image displays with FFmpeg and ImageMagick

While I was testing visualization of some videos from the AIST database earlier today, I wanted to also create some “keyframe image displays”. This can be seen as a way of doing multi-exposure photography, and should be quite straightforward to do. Still it took me quite some time to figure out exactly how to implement it. It may be that I was searching for the wrong things, but in case anyone else is looking for the same, here is a quick write up.

The current procedure is done using a combination of two very handy command line tools: FFmpeg and ImageMagick. I would like to add it to both the Matlab and Python versions of the Musical Gestures Toolbox as well, but will need to figure that out another time.

In this example I will use a hip-hop dance video from the AIST database:

The first step is to extract keyframes from the video file using this one-liner ffmpeg command:

ffmpeg -skip_frame nokey -i *.mp4 -vsync 0 -r 30 -f image2 t%02d.tiff

This will use the keyframes from the MP4 file, which should be faster than doing a new analysis of the file. It could, of course, also be possible to sample the video at regular intervals, but the keyframes seem to work fine for my usage. I also choose to save the exported keyframes as TIFF files to avoid running multiple rounds of compression on the files. The end result is a bunch of keyframe images that can be used for further processing.

Automagically exported keyframe images.

In my search for a solution, I tried a lot of complex things. But it turned out to be super-simple to get what I wanted:

convert *.tiff -background white -compose darken -flatten keyframes.jpg

Here we use the convert function of ImageMagick to add all the exported keyframes together to one combined image:

Keyframe image display of hip-hop video.

Since the dancer was moving in more or less the same place all the time, it is quite compact. Running the same functions on another video of a contemporary dancer, on the other hand, shows some of the potential of this visualization method. Here is the video:

Which results in this keyframe display image:

Besides being cool to look at, it is also quite informative when it comes to telling what is going on in the video. You get information about the temporal and spatial movement of the dancer, although it is difficult to understand exactly when she was moving where.

Next is to also include these methods in the Musical Gestures Toolbox.

Converting MXF files to MP4 with FFmpeg

We have a bunch of Canon XF105 at RITMO, a camera that records MXF files. This is not a particularly useful file format (unless for further processing). Since many of our recordings are just for documentation purposes, we often see the need to convert to MP4. Here I present two solutions for converting MXF files to MP4, both as individual files and a combined file from a folder. These are shell scripts based on the very useful FFmpeg.

Convert individual MXF files to individual MP4 files

The first solution is based on converting a bunch of MXF files to individual MP4 files. This is practical if there are multiple, individual shots.

Save the script above as mxf2mp4.sh, make it executable, with a command like:

chmod u+x mxf2mp4.sh

and run the file:

./mxf2mp4.sh

Convert a folder of MXF files to one MP4 file

The second solution is when we have made one long recording, which is split up into individual MXF files of 1.9 GB size (the maximum size of FAT32-formatted drives) in the camera. Then the aim is to merge all of these together to one MP4 file. This script will do the trick:

Do the same as above to run the script.

Trim video files using FFmpeg

This is a note to self, and hopefully to others, about how to easily and quickly trim videos without recompressing the file.

I often have long video recordings that I want to split or trim (side note: sometimes people call this “cropping”, but in my world cropping is to cut out parts of the image, that is, a spatial transformation. Splitting and trimming are temporal transformations).

You can split and trim files in most video editing software, but these will typically also recompress the file on export. This reduces the quality of the video, and it also takes a long time. A much better solution is to perform “lossless” trimming, and fortunately there is a way to do this with the wonder-tool FFmpeg. Being a command line utility (available on most platforms) it has a ton of different options, and I never remember these. So here it goes, this is what I use (on Ubuntu) to trim out parts of a long video file:

ffmpeg -i input.mp4 -ss 01:19:27 -to 02:18:51 -c:v copy -c:a copy output.mp4

This will cut out the section from about 1h19min to 2h18min, and will only take a few seconds to run. If you instead want to specify a fixed duration, you can use:

ffmpeg -i input.mp4 -ss 00:01:10 -t 00:01:05 -c:v copy -c:a copy output.mp4

This will extract 1min5sec starting from 1min10sec in the file.