Grok Voice 3.0: Build a Speech-to-Speech Agent (and When Not To)

1. The launch facts

xAI’s announcement calls Grok Voice 3.0 “the world’s most intelligent speech-to-speech model”, which is positioning, not evidence. What is concrete: one model that takes audio in and returns audio out over a WebSocket, with server-side voice activity detection, tool calling, 20 listed language codes and billing per minute of conversation.

Microsoft’s October 1 launch has a different shape: MAI-Transcribe-2-Streaming (speech to text), MAI-Voice-2.1 and MAI-Voice-2.1-Flash (text to speech). None of them is a speech-to-speech model, so they are building blocks for the STT → LLM → TTS route. Microsoft Learn labels the streaming transcription model a public preview without an SLA.

2. What improved vs Grok Voice 2.0

xAI’s post compares 3.0 with 2.0 using numbers collected by Artificial Analysis on October 13, 2026. The Speech-to-Speech Index goes from 81.3 to 86.9, Big Bench Audio (speech reasoning) from 97.2% to 98.5%, and the Speech Agent Arena Elo from 1011 to 1070. Task success barely moves: 94.6% to 94.8%.

The jumps that matter for agents are in conversation mechanics. Full Duplex Bench, which tests pauses, turn-taking and interruptions, goes from 95.1% to 97.0%. τ-Voice, support-call scenarios with accents, background noise and interruptions, goes from 56.5% to 71.6%. The price stayed flat, so on paper it is a free upgrade.

3. Evidence, and who measured it

Unlike many launch posts, the headline numbers here are not xAI grading itself: they come from Artificial Analysis, an independent benchmarking site with a public leaderboard. What xAI did choose is the comparison set and the metrics it highlights, so I read the leaderboard row by row on October 14 to see the full picture.

Grok Voice 3.0 leads the Index and time to first audio in that table, but it does not win every column: GPT-Live-1 (Astra, medium) scores higher on τ-Voice, Gemini 3.8 Live has the highest Arena Elo of the five, and GPT-Live-1 (Sol, low) edges it on Full Duplex Bench.

4. Speech-to-speech or STT → LLM → TTS?

A speech-to-speech model listens and speaks in one step, so turn-taking, interruptions and prosody live inside the model. A pipeline chains three services (transcribe, think, synthesize) and you own the glue: voice activity detection, barge-in, buffering and the text in the middle.

That text in the middle is the real trade. With a pipeline you can log it, redact it, run guardrails on it and swap the LLM whenever a cheaper one appears. With speech-to-speech you write less orchestration and, per Artificial Analysis, get a fast first audio, but you have less to inspect unless you turn on transcription events.

How I would choose

5. Quickstart: a minimal client

Everything below is adapted from xAI’s voice docs and verified against them on 2026-10-14. I have not run it against a live key: treat it as a text-in, audio-out smoke test you can extend with a microphone.

Install the websockets package, export XAI_API_KEY, run the script and open reply.wav. The audio is linear16 PCM at 24 kHz; I assume mono because the docs do not state a channel count.

import asyncio
import base64
import json
import os
import wave

import websockets

URL = "wss://api.x.ai/v1/realtime?model=grok-voice-latest"
HEADERS = {"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"}


async def main():
    async with websockets.connect(URL, additional_headers=HEADERS) as ws:
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "voice": "eve",
                "instructions": "You are a helpful customer support agent.",
                "turn_detection": {"type": "server_vad"},
                "audio": {
                    "input": {"format": {"type": "audio/pcm", "rate": 24000}},
                    "output": {"format": {"type": "audio/pcm", "rate": 24000}},
                },
            },
        }))
        await ws.send(json.dumps({
            "type": "conversation.item.create",
            "item": {
                "type": "message",
                "role": "user",
                "content": [{"type": "input_text", "text": "Hello!"}],
            },
        }))
        await ws.send(json.dumps({"type": "response.create"}))

        pcm = bytearray()
        async for message in ws:
            event = json.loads(message)
            if event["type"] == "response.output_audio.delta":
                pcm += base64.b64decode(event["delta"])
            elif event["type"] == "response.done":
                break

    with wave.open("reply.wav", "wb") as f:
        f.setnchannels(1)   # mono (assumed, not stated in the docs)
        f.setsampwidth(2)   # 16-bit linear PCM
        f.setframerate(24000)
        f.writeframes(bytes(pcm))


asyncio.run(main())

# 1) microphone -> model (with server_vad, append is the only audio event you need)
async def send_audio(ws, chunks):  # chunks: raw PCM16 at 24 kHz
    async for chunk in chunks:
        await ws.send(json.dumps({
            "type": "input_audio_buffer.append",
            "audio": base64.b64encode(chunk).decode(),
        }))

# 2) a function tool, declared inside session.update -> session.tools
tools = [{
    "type": "function",
    "name": "get_order_status",
    "description": "Look up an order by id.",
    "parameters": {
        "type": "object",
        "properties": {"order_id": {"type": "string"}},
        "required": ["order_id"],
    },
}]

# 3) when the model calls it, return the output, then ask for the spoken reply
if event["type"] == "response.function_call_arguments.done":
    result = lookup_order(**json.loads(event["arguments"]))  # your code
    await ws.send(json.dumps({
        "type": "conversation.item.create",
        "item": {
            "type": "function_call_output",
            "call_id": event["call_id"],
            "output": json.dumps(result),
        },
    }))
    await ws.send(json.dumps({"type": "response.create"}))

# server: mint a short-lived token (never ship XAI_API_KEY to the client)
curl -X POST https://api.x.ai/v1/realtime/client_secrets \
  -H "Authorization: Bearer $XAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"expires_after": {"seconds": 300}}'

// browser: pass the token as a WebSocket subprotocol
const ws = new WebSocket("wss://api.x.ai/v1/realtime",
  ["xai-client-secret." + token]);

{
  "type": "session.update",
  "session": {
    "input_audio_transcription": {"model": "mai-transcribe-2-streaming"},
    "modalities": ["text", "audio"],
    "instructions": "You are a helpful assistant.",
    "voice": {"name": "en-US-Harper:MAI-Voice-2-Flash", "type": "azure-standard"},
    "turn_detection": {"type": "azure_semantic_vad_multilingual"}
  }
}

6. Honest limits and what to do this week

Verdict

Grok Voice 3.0 is a credible upgrade: the same $0.08 per minute, better conversation mechanics on an independent leaderboard, and a protocol simple enough to smoke-test in about 50 lines of Python. It is not the cheapest route per minute, and it does not win every benchmark column.

My rule: prototype with speech-to-speech because it gets you to a working conversation in an afternoon, then price a pipeline once you know your real characters and tokens per minute. If you need text-level control or the cheapest minute, Microsoft’s MAI components are worth a look as soon as they leave preview.