Voice Agents: Unpacking the Pipeline

When most people think about AI, they picture chatbots, coding copilots, or predictive models. But what if your systems could talk — and listen — like a human?

At IndyPy’s January 2025 meetup, Research Engineer Aaron Soellinger pulled back the curtain on the world of voice agents. His presentation offered a practical look at what it really takes to build a responsive, voice-driven assistant using open tools and Python-based architecture.

The biggest takeaway? Voice tech may sound seamless — but building something that actually works takes real engineering.

A Hands-On Demo with Pirate Prompts

Aaron began with a familiar tech story: a folder of code, a free account, some hacked-together environment variables, and a goal — to build a working voice agent with speech recognition, natural language processing, and a human-like voice.

Using open source tools like VAD (Voice Activity Detection), ASR (Automatic Speech Recognition), and TTS (Text-to-Speech), he stitched together a Python-based stack. His live demo featured an AI receptionist who could schedule a haircut or give advice in a pirate accent.

The setup was scrappy but effective — and repeatable by any developer who wants to tinker.

The Hidden Complexity of Voice Interfaces

If you're considering voice interfaces — whether to streamline support, create new products, or explore hands-free workflows — the challenge isn’t vision. It’s execution.

The pieces are available. APIs exist. But integrating them into a system that responds quickly, works in noisy environments, and feels natural to users? That’s where teams often stumble.

Aaron didn’t sugarcoat it. Latency, background noise, brittle pipelines — these are the real-world hurdles standing between you and a production-ready voice assistant.

5 Lessons for Building Voice-First Systems

1. Voice is a Pipeline, Not a Feature: Each component — transport, VAD, ASR, TTS, LLM — has to be tuned and tested in context. Voice doesn’t come “out of the box.”
2. Speed Makes or Breaks Trust: One second of silence feels like forever. Prioritize low-latency performance, especially in noisy or unpredictable environments.
3. Human Voice Matters: Even when users know it’s a bot, they want it to sound human. Tools like ElevenLabs can dramatically improve perception and usability. It’s not just about function; it’s about feeling.
4. LLMs Add Power, But Also Complexity: Prompting gets you started. But real applications evolve toward Retrieval-Augmented Generation (RAG), structured context, and iteration over time.
5. Start Small, But Design for Growth: Even basic agents reveal architectural needs quickly. Invest early in a flexible, scalable foundation to avoid painful rewrites.

Why It Matters — and What to Do Next

Voice is becoming a real option for customer service, internal tools, scheduling, fieldwork, and more. And the technology is within reach.

But it’s also demanding. Unlike web or chat interfaces, voice has no buffer. No progress bar. When a user speaks, they expect an answer — fast. That means real-time performance, robust architecture, and thoughtful design.

Aaron’s talk didn’t promise shortcuts. It laid out the path: voice agents are absolutely buildable — but only when you understand the moving parts and plan accordingly.

So ask yourself:

• What would a voice interface unlock for your users?
• Are you ready to support low-latency, multi-component systems?
• Can your team design for real-world challenges like noise, speed, and reliability?

If those questions spark ideas, you’re in the right place. Voice tech is no longer experimental — it’s actionable. You just need the right strategy.

Explore Six Feet Up’s AI services to see what’s possible.

Watch the Full Presentation