Creating multi-exposure keyframe image displays with FFmpeg and ImageMagick

While I was testing visualization of some videos from the AIST database earlier today, I wanted to also create some “keyframe image displays”. This can be seen as a way of doing multi-exposure photography, and should be quite straightforward to do. Still it took me quite some time to figure out exactly how to implement it. It may be that I was searching for the wrong things, but in case anyone else is looking for the same, here is a quick write up.

The current procedure is done using a combination of two very handy command line tools: FFmpeg and ImageMagick. I would like to add it to both the Matlab and Python versions of the Musical Gestures Toolbox as well, but will need to figure that out another time.

In this example I will use a hip-hop dance video from the AIST database:

The first step is to extract keyframes from the video file using this one-liner ffmpeg command:

ffmpeg -skip_frame nokey -i *.mp4 -vsync 0 -r 30 -f image2 t%02d.tiff

This will use the keyframes from the MP4 file, which should be faster than doing a new analysis of the file. It could, of course, also be possible to sample the video at regular intervals, but the keyframes seem to work fine for my usage. I also choose to save the exported keyframes as TIFF files to avoid running multiple rounds of compression on the files. The end result is a bunch of keyframe images that can be used for further processing.

Automagically exported keyframe images.

In my search for a solution, I tried a lot of complex things. But it turned out to be super-simple to get what I wanted:

convert *.tiff -background white -compose darken -flatten keyframes.jpg

Here we use the convert function of ImageMagick to add all the exported keyframes together to one combined image:

Keyframe image display of hip-hop video.

Since the dancer was moving in more or less the same place all the time, it is quite compact. Running the same functions on another video of a contemporary dancer, on the other hand, shows some of the potential of this visualization method. Here is the video:

Which results in this keyframe display image:

Besides being cool to look at, it is also quite informative when it comes to telling what is going on in the video. You get information about the temporal and spatial movement of the dancer, although it is difficult to understand exactly when she was moving where.

Next is to also include these methods in the Musical Gestures Toolbox.

Converting MXF files to MP4 with FFmpeg

We have a bunch of Canon XF105 at RITMO, a camera that records MXF files. This is not a particularly useful file format (unless for further processing). Since many of our recordings are just for documentation purposes, we often see the need to convert to MP4. Here I present two solutions for converting MXF files to MP4, both as individual files and a combined file from a folder. These are shell scripts based on the handy FFmpeg.

Convert individual MXF files to individual MP4 files

The first solution is based on converting a bunch of MXF files to individual MP4 files. This is practical if there are multiple, single shots.

Save the script above as mxf2mp4.sh, make it executable, with a command like:

chmod u+x mxf2mp4.sh

and run the file:

./mxf2mp4.sh

Convert a folder of MXF files to one MP4 file

The second solution is when we have made one long recording, which is split into individual MXF files of 1.9 GB size (the maximum size of FAT32-formatted drives) in the camera. Then the aim is to merge all of these to one MP4 file. This script will do the trick:

Do the same as above to run the script.

Trim video files using FFmpeg

This is a note to self, and hopefully to others, about how to easily and quickly trim videos without recompressing the file.

I often have long video recordings that I want to split or trim. On a side note, sometimes people call this “cropping”, but in my world, cropping is to cut out parts of the image, that is a spatial transformation. Splitting and trimming are temporal transformations. I have another blog post on cropping video files using FFmpeg.

You can split and trim files in most graphical video editing software, but these will typically also recompress the export file. This reduces the quality of the video, and it also takes a long time. A much better solution is to perform “lossless” trimming. Fortunately, there is a way to do this with the beautiful command-line utility FFmpeg. It is available for most platforms and has a ton of different options. So to remember how it is done, here is a simple one-liner:

ffmpeg -i input.mp4 -ss 01:19:27 -to 02:18:51 -c:v copy -c:a copy output.mp4

This will cut out the section from about 1h19min (after the -ss command) to 2h18min (after the -to command). The copy parts of the command are meant to copy both the original audio and video content without recompressing. This means that the above command only takes a few seconds to run.

You may instead want to specify a fixed duration to extract, in which case you can use:

ffmpeg -i input.mp4 -ss 00:01:10 -t 00:01:05 -c:v copy -c:a copy output.mp4

This will extract 1min5sec (using the -t flag) starting from 1min10sec (the -ss flag) in the file. Happy trimming!