An AI Assistant with Walter Sobchak's temper
“Hey Walter,” and a gravel-voiced Vietnam veteran tells me I’m out of my element. He won’t turn off my lights. He’d rather explain that there are rules, and that he didn’t watch his friends die face down in the muck to be a glorified home assistant.
Here’s the voice.
The character is a prompt, not a fine-tune
My instinct was to reach for fine-tuning. Collect a dataset, rent a GPU, train weights. But for a character layer, you can take shortcuts. Walter Sobchak already exists in the base model’s training data, and the film’s dialogue is right there to anchor the voice. You don’t need new weights. You need a good prompt.
Walter is an Ollama model built from a Modelfile, with a base model, a system prompt that lays out the character (Vietnam vet, bowling league, converted to Judaism, treats every rules infraction as a moral emergency), and a handful of real lines from the film as few-shot examples, each paired with the situation that provoked it. That last part did the heavy lifting. Showing the model “here is the kind of thing that sets Walter off, and here is exactly what he says” sharpened the responses far more than any amount of adjective-piling.
Me: I think we should just let this one go, man. Walter: You’re out of your goddamn mind! This is NOT ‘Nam, it’s a goddamn bowling score. Mark it ZERO.
Cloning the voice on an 8GB card
A text Walter is fun, but Walter needs a voice. It had to be John Goodman’s, running locally on a graphics card I already owned, an 8GB RTX 2070 Super. The first engine I tried, XTTS, got the words right and the person wrong. Clean, generic, not John Goodman. F5-TTS zero-shot cloning got the likeness from a short reference clip (the voice scripts will be public soon). The catch is memory. An 8GB card can’t hold a chat model and a voice model at once, so the voice model lives in regular RAM and only moves onto the GPU for the half-second it takes to speak.
The surprisingly hard part
Here’s where the afternoon project became a week. I wanted Walter on the physical voice satellite in my house, summoned by name, not just in a browser tab.
A custom wake word is its own small model that runs on the microcontroller in the speaker. Rather than pay a service, I built a self-hosted trainer (the training scripts will be public soon) that generates thousands of synthetic “Hey Walter” samples and trains against a pile of negative audio. It hit about 97% recall at under one false trigger an hour. The model was never the hard part. The week went to driver and library versions fighting each other, and a firmware toolchain that had auto-updated out of sync with the open-source project. Pin the toolchain back, build, flash over the air, and the speaker came up with a new word in its vocabulary.
Making him properly Walter
With it working, the personality wanted sharpening.
First, speed. The voice loop has to feel like a conversation, so Walter dropped from hermes3:8b to hermes3:3b, the same Hermes family, still uncensored and role-play-tuned, about twice as fast, so the character survived the cut.
Second, he had to swear. The model was politely sanitizing him, and a Walter who says “fudge” is not Walter. The fix was making explicit profanity non-negotiable in the prompt and un-censoring the example lines.
Third, when he doesn’t understand you, he doesn’t admit it. He launches into a war story. A small model gets confused more often than a big one, so I turned the failure mode into the character. Ask him something off-script and you get deflection, not an error.
Me: Turn off the kitchen lights. Walter: There are rules, Dude. Don’t fuck around with shit you can’t explain. That’s like asking what happened in Vietnam.
He won’t run my house, which is by design. Home control lives on a separate wake word (“Okay Nabu”). “Hey Walter” is purely for talking to Walter, and his confusion turns into Vietnam instead of an error message.
My Takeaway
The character cost an afternoon, the voice an evening, and the wake word a week, and almost none of that week was the interesting machine-learning part. The model was never the hard part. There are rules, and these days the rules are mostly about dependency versions.