Working with Multi-Modal Voice-Visual Installation Systems
Most people approaching voice-visual synthesis systems start with the assumption that the hard part is getting audio and video to play together. It isn't. The hard part is making them mean anything to each other. I learned this the hard way after spending roughly eight months on what I thought would be a straightforward mapping project in early 2023. The installation I was building for a small gallery space used a condenser microphone feeding into a Max/MSP patch that drove a TouchDesigner network projecting onto a draped fabric surface. It sounded perfect on paper. It looked like every bad demo reel from 2015 when I actually ran it. Lily Dolores Harris's approach to voice-visual work fundamentally reframed how I think about this entire category. She didn't treat voice as a control signal. Voice was one thread in a larger textile, and the visual layer had its own generative autonomy. That distinction changes everything about how you design the system architecture. When you build a system where voice directly controls visual parameters in real time—pitch maps to position, amplitude maps to scale—you're building a puppet rig. The result is predictable within about three minutes of viewing. Viewers figure out the code. Once they figure out the code, they stop seeing the work. Harris's method avoids this by introducing a separation between the voice input and the visual generator. The voice becomes a probability field, not a deterministic controller. This is harder to build. It is also the only thing that sustains attention beyond the novelty phase.
Let me walk through how this actually works in practice. The system I ended up building had four distinct stages, and understanding the relationship between them is more important than any single software choice.
Stage One: Voice Capture and Preprocessing
You need a clean signal. This sounds obvious, but most tutorials skip past this entirely and assume your audio input is ready to go. It isn't. I use a Shure SM7B into a Weatherhead preamp, running into a RME Fireface UCX. The RME's ADAT inputs give me sample-accurate timing across the entire chain, which matters more than you'd expect when you're eventually syncing this to a visual engine. The preprocessing stage does three things: high-pass filtering at 80Hz to remove mic rumble and HVAC noise, a gentle compressor at a 3:1 ratio with a -24dB threshold, and a noise gate set to close below -48dBFS. The gate is critical. Without it, your system will generate visual output from room noise, which makes everything look sloppy within seconds of someone walking into the frame. I learned that lesson when a visitor's footsteps triggered a cascading visual glitch during a live demonstration. The audience laughed. It wasn't funny.
Get the Full Details
Stage Two: Feature Extraction
This is where people pick the wrong tool. There are approximately fourteen different ways to extract features from an audio signal, and the ones most tutorials recommend—basic FFT, RMS energy, zero-crossing rate—are the wrong ones for generative visual work. They give you data. They don't give you musical data. I extract four features and nothing more: spectral centroid, attack time, fundamental frequency (using autocorrelation, not FFT—FFT gives you pitch but autocorrelation gives you perceived pitch, which is different), and a transient density measure computed from the difference signal between consecutive frames. These four features map cleanly onto visual parameters without requiring complex normalization or scaling. One thing that took me weeks to figure out: the spectral centroid doesn't behave linearly across the human voice range. A soprano's centroid shift from low to high register produces a different numerical range than a bass singing the same interval. If you're building a system intended for multiple voices or variable sources, you need per-speaker normalization. I solved this by tracking the running median of each speaker's spectral centroid over the first ten seconds of input and using that as the baseline offset. Any deviation from that baseline becomes your control value. The system stays stable regardless of voice type.
Stage Three: The Translation Layer
This is the stage where 90% of projects fail, and I've seen it fail in professional settings. The translation layer is the function that maps your audio features to visual parameters. A linear mapping is almost always wrong. Human perception is logarithmic. Visual systems are typically driven by exponential or sinusoidal curves depending on the parameter. I use a combination of tanh() compression for position-based parameters and power curves (exponent of 0.6) for size and opacity. The result is that subtle vocal variations produce visible but not overwhelming visual changes, while strong vocal events produce dramatic but not violent responses. The curve you choose determines whether the work feels organic or mechanical. This is not a minor detail. It is the single most important design decision in the entire system. I should mention a practical problem I encountered here. When I first deployed the system with a tanh() mapping on position, the visuals would occasionally hit the boundary of the projection surface and the performer would see the image bouncing off the edges of the fabric. It looked like a screensaver. The fix was implementing soft clipping at 95% of the boundary rather than hard clipping at 100%. The remaining 5% margin is imperceptible to the audience but prevents the jarring edge-bounce effect that breaks immersion immediately.
Stage Four: Visual Generation
For the visual engine, I settled on TouchDesigner for the projection work and a separate generative sketch running in p5.js for the standalone version. TouchDesigner handles the projection mapping and real-time compositing. The p5.js version runs in a browser for simpler deployments where projection equipment isn't available. The visual output itself is a combination of particle systems and layered noise fields. The particles respond to transient density and fundamental frequency. The noise fields respond to spectral centroid and attack time. Both layers run at 60fps on an RTX 3080 with the RME interface and a BenQ LU955 projector. Latency from mic input to projected output is approximately 35ms, which is below the threshold where most viewers can detect the delay. Some sensitive observers reported noticing a slight lag at the 30ms mark during our testing. Pushing latency below 25ms would require a GPU upgrade or reducing the resolution to 1080p, neither of which I was willing to do for the final installation.
A Problem You Won't Find in Documentation
Here is a specific issue that cost me three days of work and isn't covered in any tutorial: thermal throttling on the GPU under sustained load. The TouchDesigner network I built stayed stable during testing, but during the actual installation the projector housing warmed the ambient temperature in the display area by approximately 4°C. After about forty minutes of continuous operation, the RTX 3080 began throttling, causing frame drops that were visible as micro-stutters in the projection. The audio continued running normally, so the voice and visuals slowly drifted out of alignment. Not enough to be consciously noticeable, but enough to make the whole thing feel slightly wrong. The workaround was twofold. First, I added a thermal monitoring script in Python that reads the GPU temperature via nvidia-smi and automatically reduces the renderer resolution from 1920x1080 to 1600x900 when the GPU hits 78°C. This maintains frame rate at the cost of some resolution. Second, I repositioned the projector to reduce heat recirculation. The combination kept the system stable for eight-hour runs without intervention.
What This Approach Does Poorly
I want to be clear about the limitations. This system does not work well with speech that contains significant consonant noise. Plosives like "p" and "t" create transient spikes that can overwhelm the visual generator if the attack time detection isn't properly filtered. I use a de-esser in the signal chain before feature extraction, but it's a bandaid. The fundamental problem is that human speech contains both tonal and non-tonal components, and separating them cleanly in real time is still an unsolved problem in audio engineering. For pure vocal tones and singing, the system works very well. For spoken word with heavy articulation, it degrades noticeably. Another limitation: the system requires a quiet environment. Any background noise above approximately 35dB SPL will be picked up by the microphone and processed as voice input. In a gallery setting with twenty-five viewers, this is manageable. In a public space with ambient noise, you need directional microphone techniques or a boundary microphone array. I haven't solved the public-space deployment problem well. I tried a Sennheiser MKH 30 shotgun mic, but it introduced phase issues with the projection timing that I couldn't resolve without expensive DSP hardware. If you're looking for a turnkey solution that handles these edge cases automatically, this approach isn't it. The system I described requires deliberate setup, tuning, and ongoing maintenance. It's not beginner-friendly, and it shouldn't pretend to be. The payoff is a work that lasts longer than the novelty period, which is more than most voice-visual installations achieve.
Getting Started Without Overcomplicating It
If you want to try this yourself, start with the simplest possible version. Use FreeSound or a recording of your own voice as the input instead of a live microphone. Run it through a basic FFT analysis in anyone's favorite environment—Python with librosa, Pure Data, even Ableton's built-in audio analysis. Map one feature to one visual parameter. Get the mapping curve right before you add complexity. A single well-mapped parameter will feel more alive than seven poorly mapped ones. The file structure I recommend is straightforward. One directory for the audio preprocessing scripts, one for the feature extraction module, one for the translation/mapping functions, and one for the visual output. Keep them separate. When the projection mapping breaks at 2am before a showing—and it will break—you need to be able to isolate which layer is failing without digging through a monolithic script. There's no download link for a complete working system because this kind of work is entirely dependent on your hardware, your space, and your artistic intent. The architecture is. The specifics are personal. That's the point Harris was making, and it's the point that matters if you're actually building something rather than collecting techniques.
