The picture above is a rack. Left: UpesLoop, a generative MIDI module, writing a drum pattern with a transformer running inside the browser tab. Right: UpesBox, a sixteen-pad drum machine, playing that pattern with sounds rendered by Stable Audio, one triangle morphing the whole kit between three worlds. Below: Guitarabin, a pedalboard, colouring the result. The whole thing is a URL: /rack/upes.
We built the two generative modules in two days at the Montreal Music Hackspace hackathon in August, on a platform that is older than the weekend. This post is about the weekend: what we built, what was fast, what broke.
Who we are
UPES, pronounced oops: researchers, artists, sound engineers and tech nerds who met at the Hackspace. Alphabetically:
- Jacinthe Carlier — UX and strategy
- Kris Grant — sample creation, musicology, ambassador (website incoming)
- Keon-Ju Maverick Lee — dataset research, AI model parameter tweaking, curation and sampling
- Felix Menard — product engineering
- Michael Zajner — hardware support research, strategy
The brief
Not a demo of generative AI. Two instruments you would keep in a session, that talk to each other, finished by Sunday evening.
UpesLoop: a loop you steer
Most generative MIDI tools are slot machines: pull, get a loop, pull again. We wanted a drummer who listens: change something and the part evolves instead of restarting.
UpesLoop runs MIDI-GPT (Metacreation Lab, Yellow Small checkpoint), exported to ONNX as a prefill graph and a decode graph and executed in the browser with onnxruntime-web. No server. About 158 MB of weights stream into the tab, cached after the first load; inference runs on your machine.
Four decisions turned the model into an instrument:
- Reslop regenerates from the take you are looking at. New slop starts from a seed flavor, Random by default.
- A constrained note space. Hard scale masking plus a soft pitch-histogram bias keep the model inside the frame you set, so temperature reads as creativity rather than noise.
- Auto mode regenerates a second after your last control change: turn a knob, hear the answer.
- Melody, rhythm, harmony and multi-track modes, each with its own defaults, and MIDI drag-out to a DAW track.
UpesBox: one triangle, sixteen sounds, three worlds
The question: can a drum kit morph the way a single sample does? Not sixteen independent morphs; one gesture that moves the whole kit.
The mechanism is corner alignment. Every pad is a three-corner morph triangle, and the corners line up across the kit: corner one of the kick, snare and hat all belong to the same theme. One global triangle position pulls all sixteen sounds toward the same world, and the waveforms redraw as you drag.
The sounds are generated. Corner material is rendered with Stable Audio 3 from prompts, one per theme per drum role. Instead of crossfading three WAVs, we interpolate in the model's latent space: a 28-node barycentric lattice across the triangle is decoded into a bank, and each pad resolves its triangle position to the nearest cells and blends them. The point between "city" and "jungle" is a new sound, not two sounds at half volume.
Three kits ship. Grey Studio (Analog / Digital / Room) is fully synthesized, the baseline. Field Menagerie carries eight real latent-morph banks on a synthesized backbone; play this one to hear the morph. Siam Triptych (City / Jungle / Temple) is the full idea with synthesized stand-ins in the corners; filling it with generated material (sixteen roles × three themes, dozens of candidates per role) is next. A missing bank falls back to synth, so a pad can lose its material without going silent.
Guitarabin closes the chain to make one point: generated material is a signal like any other. Compressor, boost, drive, tone, tremolo, vibrato, delay, reverb. Turn the echo up and the generated kit sits in a room.
What broke, what ran out of time
Hardware pedals. The platform cross-builds for embedded Linux, so we published packs for two targets: Elk Stomp (STM32MP157F, Cortex-A7, 32-bit ARM) and Lydia (Raspberry Pi 5, 64-bit). It was not easy and it was full of peculiarities. At one point the Elk pack was labelled for the wrong CPU architecture; we caught it, rebuilt and re-stamped every artifact. Today the packs exist, pass structural checks (right ISA, right exports, kernel symbols resolve), and nobody has run them on a board. It is a promising export target we intend to automate in the compiler. We will claim it when a pedal makes a sound.
LoRA training. Stable Audio 3 takes small adapters (about 18 MB, trained on top of the frozen model) that pull generation toward a sound you taught it, behind a trigger token and a strength knob. The pipeline was staged and we started exploring it. It would have added a real layer of control to the sampler: train on your sounds, then generate the kit's corners from them at a chosen strength. We ran out of weekend. It stays high on the list.
UpesLoop stayed in the browser. The model is big, and the right native story is "ship the plugin with its model". The VST3 build is bring-your-own-model today. Correct for a two-day build; not where it should stay.
Why two days was enough
Almost none of it was built in two days. The weekend went to the generative layer and the module faces; everything underneath already existed, and three things were fast because of it:
- The sampler. The three-face bank format and the lattice decode already existed (Tetra Sampler, eight morph banks). The weekend's work was sixteen pads, corner alignment, the global triangle and the Stable Audio corners, not sample playback.
- Piping MIDI into the sampler. UpesLoop emits MIDI, UpesBox consumes it, Guitarabin takes audio. Dropping the three onto the /patch canvas made the rack; no glue code.
- Working in a DAW. UpesBox and Guitarabin export as VST3, CLAP, AU and macOS standalone from the same TypeScript source, and the UpesBox plugins are gated by rendering plugin and offline reference and comparing bytes. Nobody wrote C++ over the weekend.
That last one is the holy grail of audio development: one codebase, one language, running everywhere, browser tab and DAW alike, and it was the easy part of the weekend.
Where it is
Everything runs on the web today. UpesBox and Guitarabin also ship as VST3, CLAP, AU and macOS standalone, with Elk and Lydia packs built but not yet run on hardware; UpesLoop is web only, with a bring-your-own-model VST3. The browser faces of UpesLoop and UpesBox run on Web Audio while the native UpesBox runs the compiled kernel; moving the browser face onto that kernel is next.
Next
- UpesLoop and UpesBox as downloadable plugins with their models included.
- The full generative fill of UpesBox kits, then the LoRA layer on top.
- Elk and Lydia verified on real boards, with hardware export automated in the compiler.
- UpesLoop and UpesBox browser faces on the compiled kernel.
Come build one
Try the rack at /rack/upes, wire the modules with others on /patch, publish yours. If you were at the Hackspace with us: thank you. If not, the next one is where to find us.
