hal-voice

I watched 2001: A Space Odyssey and wanted to know how close I could get to the voice.

It turned into something useful because Hal — a local Claude Code setup with an Obsidian vault as memory — was already sitting there wanting a better interface than a terminal. Once it worked, talking to it genuinely beat typing for quick questions. hal-voice runs no model of its own: it is a voice on the front of a Claude Code session, so whatever that session can already do — your tools, your MCP servers, your project’s memory — it can now do out loud.

A local model generates slowly, and speech makes that unbearable. The naive pipeline — capture, transcribe, generate the whole reply, synthesise, play — puts several seconds of complete silence in the middle. Silence is much worse in a voice interface than on screen: you can watch text stream, but you can’t watch nothing happen, and after two seconds you assume it’s broken.

So the reply is streamed and pipelined. Claude Code runs headless for structured incremental output, the output is split into sentences as it arrives, sentence one goes to the synthesiser while the model is still writing sentence two, and audio starts as soon as that first sentence is synthesised. First audio lands about as fast as the model can finish one sentence rather than a whole reply. That single change is the difference between the project feeling broken and feeling alive.

Barge-in matters for the same reason — a machine that can’t be interrupted is irritating to use. And speech recognition runs on CPU deliberately, so the whole GPU stays free for the language model and the synthesiser.

GPT-SoVITS fine-tuned on supplied clips, then a client-side FX chain: EQ voicing, a small dry cabin reverb, loudness matching. The clone gets timbre; it doesn’t get the room, the pacing or the weight.

Three of the FX parameters ended up at zero, and finding that out took longer than building the chain. The one worth reading about is pitch_flatten. It reconstructs audio from the Hilbert analytic signal — which is only valid for narrowband signals. Speech is broadband, so the resynthesised phase threw artefacts across the whole spectrum. Everything sounded fried, and I spent a while assuming the voice clone itself was bad. Synthesising one line with the FX bypassed took thirty seconds and settled it.

In the middle panel below, the vertical streaks across every band are those reconstruction artefacts: spectral flatness rises from 0.0006 to 0.0010 and the harmonic-to-percussive energy ratio falls from 0.46 to 0.29. pitch_shift_semitones and drive went to zero for related reasons — both were compensating for a problem that wasn’t there. I now reach for the bypass test first. HAL’s monotone comes from the clone and a slow speed setting, not from post-processing.

Works: the full push-to-talk loop, streaming synthesis, barge-in and the eye; MCP server mode, giving speak and listen inside an ordinary Claude Code session; and a doctor command, which catches the audio-device problems that cause most failures.

Deliberately deferred: the wake word is off by default. The code is there and “Hey HAL” is configurable, but detection is unreliable, so its dependencies aren’t installed either. Push-to-talk is the supported way in.

Known rough edges: it’s still slow — streaming hides a great deal, but there is a floor to how much it can hide, and on the CPU-only profile synthesis runs slower than real time, so HAL pauses before answering. Barge-in can clip the first syllable of your next sentence. The voice effects are tuned to one voice, so treat the defaults as a starting point. Linux only.

HAL was performed by the late Douglas Rain, who did not consent to and could not have consented to this. This is non-commercial fan work, and it ships the pipeline rather than the performance — no training clips, no transcripts and no trained weights are distributed, and the fetch script deliberately refuses to pull audio from public hosts, because moving a recording to a different URL doesn’t change who owns it. In most jurisdictions a person’s voice is protected, and cloning one you have no rights to is not okay, whatever the tooling makes easy.