← All modules
AvailableMIDI & Modulation

HandMod

Map hand gestures to MIDI control.

Engine preview

Not built for macOS, Windows or Linux

Frames are analysed in this tab and discarded. No frame, still, landmark or metric ever leaves your browser, only the CC numbers you mapped.

About

Every modulation source on a computer is either a shape you drew in advance or a knob you have one spare hand for. HandMod is the third option, and unlike CamMod's deliberately dumb frame-differencing it knows what a hand is: twenty-one landmarks per hand, a face mesh, a body pose, updated every frame. That means a lane can follow the roll of your wrist rather than the movement of a blob, and roll is a thing you can repeat on purpose. A modulation source you cannot repeat is a modulation source you cannot perform with.

Forty-five quantities come out of the tracker, all normalised to 0-1 before a lane touches them: position, roll, pitch, yaw, spread, pinch and area for each hand; the separation between your hands and between your index fingers; mouth openness, brow height, smile and head angles; shoulder, elbow and torso geometry from the pose model. Eight lanes each pick one, scale it into a CC window, bend it through a curve and smooth it. Point two lanes at the same source with different curves and you have a crossfade you conduct.

The two-hand shape mode is the part worth trying first. Touch thumbs and index fingers on both hands and HandMod reads the quadrilateral you are framing, its area, its centre, its angle, as four more sources. Twist one hand and the quad crosses itself into two triangles, which the module reports as a discrete twist state you can map to something switch-like. It is a control surface you make out of the air and then deform, and there is no hardware that does this.

Privacy, plainly. The models are fetched once from a CDN and then everything runs in the page: frames go from the camera into a canvas, landmarks come out, and both are discarded before the next frame. Nothing is uploaded, not frames, not stills, not landmarks, not a summary, because there is no server in this loop at all. The browser's camera indicator is on while HandMod runs and goes off the moment you stop it. Turn off your Wi-Fi after the models load and the whole module works exactly the same; that is the test we would want to run, so run it.

Camera access is a request, not a requirement. Decline it and the module still renders, explains what it would have done and stays inspectable, the same posture the microphone modules take. A host without a camera API at all, which today means every native build, reports that honestly rather than showing a dead button; the capture path is behind an interface so a standalone shell can implement it over its own bridge later without any of the mapping code changing.

Build evidence

Every row below is a recorded build row or the recorded absence of one. Nothing here is inferred from what this module is meant to support.

Module
music.codex.handmod v2.0.0
Built at
no recorded row carries a build time; the browser lanes record none
Closure hash
not a Processor — no lowering closure
Toolchain
  • scriptc 64acce1ae3b8…
  • moduleToolchainDist c621ffab721d…
  • pluginShellKitRuntime bca003784692…
  • faceShellGeneration 8533207ab81f…
  • uiPreservation cd28dbd7e3d6…
  • packagerVersion 3
Build result and download for each format
FormatResultDownload
Browser package (JS)Not built by this module's lane

this module has no release.json; its packager lane records no browser program rows

Not built for the browser
Browser package (WASM)Not built by this module's lane

this module has no release.json; its packager lane records no browser program rows

Not built for the browser
VST3Not built — the native export is blocked

native-export.json records nativeEligibility "blocked"

Not built for macOS, Windows or Linux
Audio UnitNot built — the native export is blocked

native-export.json records nativeEligibility "blocked"

Not built for macOS
CLAPNot built — the native export is blocked

native-export.json records nativeEligibility "blocked"

Not built for macOS, Windows or Linux
Standalone appNot built — the native export is blocked

native-export.json records nativeEligibility "blocked"

Not built for macOS, Windows or Linux

Parity

No lane recorded an artifact; there is no parity evidence.

Presets

Start from a named sound, then adjust the instrument to make it yours.

InitClassic ModThereminFilter SweepFace ControlConductorQuad Shape

How it works.

A getUserMedia stream feeds MediaPipe's tasks-vision landmarkers, running on the GPU through WebGL where the browser allows it. Hands come back as 21 points each, the face as 478, the pose as 33, all in normalised frame coordinates. Each set is smoothed with a one-pole filter across frames before anything reads it, because raw landmark output jitters by a pixel or two even from a perfectly still hand and that jitter would otherwise arrive as audible noise on every lane.

The metric layer is pure maths over those points and is the part that is unit-tested: wrist-to-knuckle vectors become roll, pitch and yaw through a small orientation solver ported from the original tool; fingertip distances become spread and pinch; the convex extent becomes area. Shape mode takes four fingertips, orders them into a quad, and runs a shoelace area, a centroid, a long-axis angle and a segment-intersection test - that last one is what reports the twist, splitting the quad into two triangles and summing their areas instead.

Each enabled lane then reads one normalised value, applies its curve, scales it into its Min-Max window, smooths it again at lane level and emits a CC only when the rounded value actually changed, so a still hand does not spam identical bytes. Every event goes through the shared Audio & MIDI session to the selected destination, while a local parameter mirror keeps the page's controls visually in step without opening a second MIDI connection.

None of this is audio DSP, so HandMod has no kernel and no parity claim. It runs on the camera's frame clock rather than the audio block clock, and expect roughly 40-80 ms between a physical move and an audible change - most of it camera capture and model inference, not the mapping. That is fine for sweeps, swells and shapes, and it is not fine for triggering notes, which is why HandMod sends no notes.

Goes well with.

Preview

CamMod

Map camera motion to MIDI CC.

PreviewNot built for macOS, Windows or Linux
Preview

OSC Mini

A compact subtractive synth for learning the signal path.

Preview
Preview

Chordex

Generate a voiced chord from one note.

PreviewNo build recorded