Voice agents — pipeline, engine, and realtime

Pipeline: STT → LLM → TTS (provider-agnostic)

pipelineVoice({ stt, llm, tts }) returns a VoiceProvider composable with any of the 8 text Providers. The LLM is just a Provider — swap anthropic() for openai() and nothing else changes. Tools, guardrails, sessions, and tracing all flow through the same conversation engine that powers realtime.

import { pipelineVoice } from "samai-sdk/voice";
import { deepgramSTT } from "samai-sdk/voice";
import { elevenLabsTTS } from "samai-sdk/voice";
import { defineVoiceAgent } from "samai-sdk/voice";
import { anthropic } from "samai-sdk";

// Any text Provider works as the LLM — swap anthropic() for openai() etc.
const voiceProvider = pipelineVoice({
  stt: deepgramSTT({ apiKey: process.env.DEEPGRAM_KEY }),
  llm: anthropic({ apiKey: process.env.ANTHROPIC_KEY }),
  tts: elevenLabsTTS({ apiKey: process.env.ELEVEN_KEY }),
});

const agent = defineVoiceAgent({
  name: "concierge",
  instructions: "You are a helpful, concise voice assistant.",
  model: "claude-sonnet-4-5",
  voice: { interruption: "confidence-gated", backchannel: true },
});

const session = await voiceProvider.connect({ agent, session });
session.on("user-speech-ended", (e) => console.log("user:", e.transcript));
session.on("interruption", () => console.log("barge-in — TTS cancelled"));
session.sendAudio(micChunk); // ArrayBuffer from your mic
session.interrupt();         // explicit barge-in
await session.close();

Conversation engine & turn-taking

ConversationEngine is a deterministic state machine — idle → listening → thinking → speaking → interrupted → idle — reused by both pipeline and realtime. Barge-in is confidence-gated via VoiceActivityDetector + InterruptionController and cancels the in-flight LLM+TTS with an AbortController; low-confidence blips (coughs, TV) are ignored.

import { ConversationEngine } from "samai-sdk/voice";

const engine = new ConversationEngine({ agent, session, trace });
// engine.handleUserSpeechEnded(transcript, confidence) -> clarification or null
// engine.handleBargeIn({ confidence, durationMs })     -> true if barge-in accepted
// engine.getState() // "idle" | "listening" | "thinking" | "speaking" | "interrupted"

Behavior: goals, clarification, and shaping

IntentTracker persists goal state via the same Session store used for text chat (so voice + text share a store); ClarificationPolicy asks for missing/low-confidence slots before calling the LLM; ResponseShaper trims replies for voice (short inputs → short answers) and optionally emits backchannels on long turns.

STT/TTS adapters (optional peers)

deepgramSTT() and elevenLabsTTS() are thin adapters over @deepgram/sdk and elevenlabs — both optional, lazily imported. Nothing in samai-sdk core requires them to install or build. In tests or without a key they fall back to a mock transport so the pipeline stays testable.

WebRTC transport

WebRTCVoiceTransport wraps the browser RTCPeerConnection with a Node-compatible mock fallback; webrtc-signaling.ts exposes createOffer / handleAnswer / handleOffer / addIceCandidate so you can wire any signaling channel (WebSocket, etc.).

Realtime as a VoiceProvider

openaiRealtime() (in samai-sdk/voice) is a thin VoiceProvider over the existing createRealtimeSession() WebSocket — same VoiceSession surface as the pipeline, so agents can switch transports without changing call sites.

React hook

useVoiceAgent() from samai-sdk/react-voice mirrors useAgent from samai-sdk/react — connect() / disconnect() / sendAudio() / interrupt() plus isSpeaking / isListening / transcript state.

import { useVoiceAgent } from "samai-sdk/react-voice";
import { pipelineVoice } from "samai-sdk/voice";

function VoiceButton({ provider, agent }) {
  const { isConnected, isSpeaking, transcript, connect, disconnect, sendAudio, interrupt } =
    useVoiceAgent(provider, agent);
  return (
    <button onClick={isConnected ? disconnect : connect}>
      {isSpeaking ? "Speaking…" : isConnected ? "Listening" : "Connect"}
    </button>
  );
}

Testing & observability

createMockSTTProvider() / createMockTTSProvider() / createMockVoiceTransport() from samai-sdk/voice/testing let you drive the full pipeline deterministically — no audio deps, no network. Every VoiceAgentEvent also records into RunTrace as voice-turn / interruption / clarification / goal-update so exportRunTraceToOtel() and renderTraceHTML() work unchanged.

Legacy: TTS / transcription REST

import { generateSpeech, transcribeAudio } from "samai-sdk";
import { writeFile, readFile } from "node:fs/promises";

const { audio } = await generateSpeech({ input: "Hello there!", voice: "nova" });
await writeFile("out.mp3", audio);

const { text } = await transcribeAudio({ audio: await readFile("recording.mp3"), filename: "recording.mp3" });

Legacy: raw realtime WebSocket

import { createRealtimeSession } from "samai-sdk";

const session = createRealtimeSession({
  instructions: "You are a helpful, concise voice assistant.",
  voice: "alloy",
  tools: [getWeatherTool],
});

session.on((event) => {
  if (event.type === "audio.delta") playAudioChunk(event.audio);
  if (event.type === "speech_started") stopSpeakerPlayback();
});

await session.connect();
session.sendText("What's the weather in Tokyo?");
session.interrupt();
await session.close();

Handles the network/protocol side only — pairing it with actual mic capture and speaker playback is up to your app. On Node < 22, or for header-based auth (recommended), install the optional ws peer dependency; without it, connections fall back to OpenAI's documented subprotocol-based auth.