Some thoughts on data signal processing in Max

We are having a Jamoma workshop at the fourMs lab this week. Most of the time is being spent on making Jamoma 0.5 stable, but we are also discussing some other issues. Throughout these discussions, particularly about how to handle multichannel audio in Max, I have realised that we should also start thinking about data signals as a type in itself.

Jamoma is currently, as is Max, split into three different “types” of modules and processing: control, audio and video. Due to the nature of digital audio and video, they have received special attention when it comes to signal flow and operation in Jamoma. Data signals, on the other hand, are still currently being treated together with control data (e.g. parameter and message communication between the modules). This makes sense in some cases, e.g. when the data signals being passed around are of a time-discrete or time-limited nature. However, when working with continuous data signals, and particularly with data coming from various types of sensor interfaces and motion capture devices, I start to see that our current approach is not adequate.

My first ideas about how to handle this in Jamoma is based on making a separation between control data and continuous signals:

  • Control data:
    • Discrete: single events
    • Continuous: events with a defined beginning and end, e.g. a ramp
  • Continuous signals:
    • Data
    • Audio
    • Video

Even though there are a myriad of ways of handling digital audio and video signals, they are usually well defined when it comes to resolution, speed, etc. Data signals, on the other hand, typically vary much more:

  • Resolution: typically ranging from 2-16 bit
  • Speed: typically ranging from 10-1000 Hz
  • Data types: integers and floats
  • Dimensions: single values (1x1), lists (1xN) and vectors (MxN)

When it comes to working with data signals in Max, there are several possibilities:

  • Max: this is the standard way of dealing with data signals in Max.
    • Pros: Easy to work with. There are a number of standard logical and mathematical operations, and some list processing tools.
    • Cons: Limited by the Max scheduler. Poor handling of massive amounts of data. No good visualisation tools. No advanced data signal processing tools.
  • MSP: it is easy to convert data signals into audio signals using sig~.
    • Pros: Use standard DSP tools, including filters, poly, etc. Work with data in the frequency domain (doing FFT and iFFT).
    • Cons: Audio is typically handled at fixed sampling/bit rates, which may cause a lot of overhead when it comes to working with data signals.
  • Jitter: although it is often being used only for video, Jitter is in fact conceived of as a more general data processing tool.
    • Pros: Lots of matrix processing tools. Possible to visualise data using OpenGL.
    • Cons: Although Jitter is supposed to be a generic matrix processing tool, many objects are still focusing mainly on video processing. Creating visualisations in OpenGL is not straightforward. Exporting files to other software (e.g. Matlab) is not straightforward.
  • FTM and MnM: a standalone library for Max (and PD) developed at IRCAM.
    • Pros: Lots of good and more advanced tools for handling data signals. Supports SDIF read/write which is supported in Matlab.
    • Cons: Still only runs in beta in Max 4, Max 5 version not coming before 2009?

As this list shows, all approaches have positive and negative implications. I will spend some time testing the various approaches and see which ones may work better.