HandMod turns camera landmarks into MIDI control. Raise a hand to open a filter, tilt your head to add vibrato, or use the distance between your hands to shape a larger change.
The chain is simple enough to understand, and predictable enough to learn as an instrument.
How it works
A camera-modulation instrument is a chain of four plain steps.
First, the camera. The browser asks permission, you say yes, and frames start arriving about thirty times a second.
Second, tracking. Each frame goes through a landmark model that finds points on you: twenty-one per hand, a mesh over your face, and joints of your body. This runs entirely on your machine. Frames and landmarks stay in the browser, keeping the control loop local and responsive.
Third, distillation. Raw points become named quantities with physical meaning, each normalized to a 0-1 range: hand roll, pinch, mouth openness, hand distance, smile, and more. HandMod derives about forty-five of these. Touch your thumbs and index fingers together to frame a four-cornered shape in the air; its area, position, angle, and twist state all become usable signals.
Fourth, mapping. Any signal can be pointed at any destination through a lane: pick the signal, choose the CC number, set the range, choose a response curve, and add smoothing. That lane is ordinary MIDI. A synth in the tab or a connected MIDI device sees a controller.
The mapping stays continuous from movement to control value, so the same gesture produces a repeatable response.
The wah, since you asked
The first mapping to try is mouth openness to filter frequency. A wah pedal follows one dimension of movement, and your jaw can control the same kind of sweep.
In HandMod, pick a lane, set its source to Mouth Openness, and aim it at the built-in synth's cutoff. Add a little smoothing so small movements stay stable. Play a chord and speak: closed mouth, closed filter; open vowels, open sweep.
Eight mapping ideas
- Head tilt → vibrato depth. Lean into a note the way you already do involuntarily.
- Smile → reverb mix. Deadpan is dry. Grin and the room blooms. Ridiculous, and completely addictive.
- Pinch → delay feedback. Squeeze a repeat until it self-oscillates, release to let it die.
- Two-hand distance → anything big. The theremin gesture. Wide arms for wide sound.
- Hand roll → a knob you no longer own. Rolling your wrist is the same movement as turning a pot, minus the pot.
- The two-hand shape → three parameters at once. Area to cutoff, angle to pan, height to reverb: now you are steering a sound by resizing and tilting a frame in the air.
- The twist → a switch. The shape crossing into two triangles is a clean on/off you can throw with a flick of one wrist.
- Eyebrows → accent. Raise them on the downbeat. You were going to anyway.
Latency and tracking limits
The signal runs on the video clock, so expect thirty to sixty milliseconds between a move and the change. That works for sweeps, sends, and morphs, but feels late for drum-like triggers. More smoothing steadies slow gestures and delays fast ones. Tracking also needs clear light and contrast.
Play one, then build one
HandMod is playable on this site in a browser tab. Camera permission is optional, and processing runs on your device.
The underlying parts live in the monorepo as the vision-control package: camera permissions, tracking, signal math, and mapping. That makes HandMod useful as both an instrument and a starting point for new camera-driven ideas.
