My introductory lectures went through some of the theory of an embodied understanding of the experience of music. One aspect of this theory that I find very relevant for the development of interactive works is what I call action-sound relationships. By this I mean that we have an intuitive understanding of how certain actions may produce certain sounds. This is the cognitive basis for the fact that we can “hear” an action we only see, and “see” the action of a sound we can only hear. These ideas are presented and discussed more thoroughly in my PhD dissertation.
Examples of realtime motion capture
I went through a number of examples of how to use motion capture in musical contexts. Here are but a few of the examples:
Transformation is an improvisation piece for electric violin and live electronics. It is based on the idea of letting the performer control a large collection of sound fragments while moving around on stage. The technical setup for the piece is based on a video-based motion tracking system developed in Jamoma, coupled to the playback of sounds using concatenative synthesis in CataRT. Transformatillon is described more thoroughly in a paper in the upcoming issue of Computer Music Journal (winter 2012).
The SoundSaber motion capture instrument is tracking the position of a rod in space using an infrared marker-based motion capture system. The setup is described in more detail in this NIME 2011 paper.
Dance Jockey is a setup/piece in which the Xsens MVN inertial motion capture suit is used to control both sample playback and synthesis using a combination of posture and action recognition of the full body of the performer. It is described in this NIME 2012 paper.
Maxobjects.com is a database with pointers to a number of third party externals for Max. It is a great resource to find whatever you need.
Jamoma is a large collection of modules and externals, including video analysis and mapping tools
CNMAT depot is a large collection of externals, tutorials, and example patches.
The Wii controllers are a great way of getting started to use accelerometer data in Max. They are wireless (Bluetooth), and once you figure out the pairing with your computer they work quite well. There are several ways of using the with Max, amongst others:
ajh.wiimote is an external for getting data from the Wii controllers into Max.
OSCulator is a multipurpose control and mapping tool working with different types of controllers, including the Wii. Even though it may be a little more work passing OSC messages into Max from a separate application, OSCulator is what I find the most stable way of working with Wii controllers in Max.
Junxion is another multipurpose control and mapping tool, developed at STEIM. Works with lots of controllers + it also does video tracking.
The Phidgets kits are an inexpensive, user-friendly and soldering-free way of getting started working with musical electronics. The kits come with a sensor interface and a number of different types of sensors to test out.
The driver is necessary to get the data into your computer.