Add Qwen3-TTS streaming endpoint for voice synthesis (#40)

* Task completion checkpoint

* Task completion checkpoint

* Task completion checkpoint

* Add Qwen3-TTS research document for live TTS replacement

Research findings for replacing Chatterbox TTS with Qwen3-TTS-12Hz-0.6B-Base:

- Current TTS: Chatterbox-Turbo-ONNX with batch-only generation, no streaming
- Qwen3-TTS: 97ms end-to-end latency, streaming support, 3-second voice cloning
- Voice cloning: Requires 3s reference audio + transcript (Makima voice planned)
- Integration: Python service with WebSocket bridge (no ONNX export available)
- Languages: 10 supported including English and Japanese

Document includes:
- Current architecture analysis (makima/src/tts.rs)
- Qwen3-TTS capabilities and requirements
- Feasibility assessment for live/streaming TTS
- Audio clip requirements for voice cloning
- Preliminary technical approach with architecture diagrams

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 03:11:15 UTC

* Add Qwen3-TTS research documentation

Comprehensive research on replacing Chatterbox TTS with Qwen3-TTS-12Hz-0.6B-Base:

- Current TTS implementation analysis (Chatterbox-Turbo-ONNX in makima/src/tts.rs)
- Qwen3-TTS capabilities: 97ms streaming latency, voice cloning with 3s reference
- Cross-lingual support: Japanese voice (Makima/Tomori Kusunoki) speaking English
- Python microservice architecture recommendation (FastAPI + WebSocket)
- Implementation phases and technical approach
- Hardware requirements and dependencies

Key findings:
- Live/streaming TTS is highly feasible with 97ms latency
- Voice cloning fully supported with 0.95 speaker similarity
- Recommended: Python microservice with WebSocket streaming

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add comprehensive Qwen3-TTS integration specification

This specification document defines the complete integration of
Qwen3-TTS-12Hz-0.6B-Base as a replacement for the existing Chatterbox-Turbo
TTS implementation. The document covers:

## Functional Requirements
- WebSocket endpoint /api/v1/speak for streaming TTS
- Voice cloning with default Makima voice (Japanese VA speaking English)
- Support for custom voice references
- Detailed client-to-server and server-to-client message protocols
- Integration with Listen page for bidirectional speech

## Non-Functional Requirements
- Latency targets: < 200ms first audio byte
- Audio quality: 24kHz, mono, PCM16/PCM32f
- Hardware requirements: CUDA GPU with 4-8GB VRAM
- Scalability: 10 concurrent sessions per GPU

## Architecture Specification
- Python TTS microservice with FastAPI/WebSocket
- Rust proxy endpoint in makima server
- Voice prompt caching mechanism (LRU cache)
- Error handling and recovery strategies

## API Contract
- Complete WebSocket message format definitions (TypeScript)
- Error codes and responses (TTS_UNAVAILABLE, SYNTHESIS_ERROR, etc.)
- Session state machine and lifecycle management

## Voice Asset Requirements
- Makima voice clip specifications (5-10s WAV, transcript required)
- Storage location: models/voices/makima/
- Metadata format for voice management

## Testing Strategy
- Unit tests for Python TTS service and Rust proxy
- Integration tests for WebSocket flow
- Latency benchmarks with performance targets
- Test data fixtures for various text lengths

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add Qwen3-TTS implementation plan

Comprehensive implementation plan for replacing Chatterbox-TTS with
Qwen3-TTS streaming TTS service, including:

- Task breakdown with estimated hours for each phase
- Phase 1: Python TTS microservice (FastAPI, WebSocket)
- Phase 2: Rust proxy integration (speak.rs, tts_client.rs)
- Detailed file changes and new module structure
- Testing plan with unit, integration, and latency benchmarks
- Risk assessment with mitigation strategies
- Success criteria for each phase

Based on specification in docs/specs/qwen3-tts-spec.md

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add author and research references to TTS implementation plan

Add links to research documentation and author attribution.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 03:25:06 UTC

* Add Python TTS service project structure (Phase 1.1-1.3)

Create the initial makima-tts Python service directory structure with:
- pyproject.toml with FastAPI, Qwen-TTS, and torch dependencies
- config.py with pydantic-settings TTSConfig class
- models.py with Pydantic message models (Start, Speak, Stop, Ready, etc.)

This implements tasks P1.1, P1.2, and P1.3 from the Qwen3-TTS implementation plan.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add TTS engine and voice manager for Qwen3-TTS (Phase 1.4-1.5)

Implement core TTS functionality:
- tts_engine.py: Qwen3-TTS wrapper with streaming audio chunk generation
- voice_manager.py: Voice prompt caching with LRU eviction and TTL support

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 03:30:06 UTC

* Add TTS proxy client and message types (Phase 2.1, 2.2, 2.4)

- Add tts_client.rs with TtsConfig, TtsCircuitBreaker, TtsError,
  TtsProxyClient, and TtsConnection structs for WebSocket proxying
- Add TTS message types to messages.rs (TtsAudioEncoding, TtsPriority,
  TtsStartMessage, TtsSpeakMessage, TtsStopMessage, TtsClientMessage,
  TtsReadyMessage, TtsAudioChunkMessage, TtsCompleteMessage,
  TtsErrorMessage, TtsStoppedMessage, TtsServerMessage)
- Export tts_client module from server mod.rs
- tokio-tungstenite already present in Cargo.toml

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add TTS WebSocket handler and route (Phase 2.3, 2.5, 2.6)

- Create speak.rs WebSocket handler that proxies to Python TTS service
- Add TtsState fields (tts_client, tts_config) to AppState
- Add with_tts() builder and is_tts_healthy() methods to AppState
- Register /api/v1/speak route in the router
- Add speak module export in handlers/mod.rs

The handler forwards WebSocket messages bidirectionally between
the client and the Python TTS microservice with proper error handling.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add Makima voice profile assets for TTS voice cloning

Creates the voice assets directory structure with:
- manifest.json containing voice configuration (voice_id, speaker,
  language, reference audio path, and Japanese transcript placeholder)
- README.md with instructions for obtaining voice reference audio

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add Rust-native Qwen3-TTS integration research document

Research findings for integrating Qwen3-TTS-12Hz-0.6B-Base directly into
the makima Rust codebase without Python. Key conclusions:
- ONNX export is not viable (unsupported architecture)
- Candle (HF Rust ML framework) is the recommended approach
- Model weights available in safetensors format (2.52GB total)
- Three components needed: LM backbone, code predictor, speech tokenizer
- Crane project has Qwen3-TTS as highest priority (potential upstream)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 11:21:43 UTC

* [WIP] Heartbeat checkpoint - 2026-01-27 11:24:19 UTC

* [WIP] Heartbeat checkpoint - 2026-01-27 11:26:43 UTC

* feat: implement Rust-native Qwen3-TTS using candle framework

Replace monolithic tts.rs with modular tts/ directory structure:

- tts/mod.rs: TtsEngine trait, TtsEngineFactory, shared types (AudioChunk,
  TtsError), and utility functions (save_wav, resample, argmax)
- tts/chatterbox.rs: existing ONNX-based ChatterboxTTS adapted to implement
  TtsEngine trait with Mutex-wrapped sessions for Send+Sync
- tts/qwen3/mod.rs: Qwen3Tts entry point with HuggingFace model loading
- tts/qwen3/config.rs: Qwen3TtsConfig parsing from HF config.json
- tts/qwen3/model.rs: 28-layer Qwen3 transformer with RoPE, GQA (16 heads,
  8 KV heads), SiLU MLP, RMS norm, and KV cache
- tts/qwen3/code_predictor.rs: 5-layer MTP module predicting 16 codebooks
- tts/qwen3/speech_tokenizer.rs: ConvNet encoder/decoder with 16-layer RVQ
- tts/qwen3/generate.rs: autoregressive generation loop with streaming support

Add candle-core, candle-nn, candle-transformers, safetensors to Cargo.toml.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: integrate TTS engine into speak WebSocket handler

- Update speak.rs handler to use TTS engine directly from SharedState
  instead of returning a stub "not implemented" error
- Add TtsEngine (OnceCell lazy-loaded) to AppState in state.rs with
  get_tts_engine() method for lazy initialization on first connection
- Implement full WebSocket protocol: client sends JSON speak/cancel/stop
  messages, server streams binary PCM audio chunks and audio_end signals
- Create voices/makima/manifest.json for Makima voice profile configuration
- All files compile successfully with zero errors

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add /speak TTS page with WebSocket audio playback

Add a new /speak frontend page for text-to-speech via WebSocket.
The page accepts text input and streams synthesized PCM audio through
the Web Audio API. Includes model loading indicator, cancel support,
and connection status. Also adds a loading bar to the listen page
ControlPanel during WebSocket connection.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

1 files changed, 329 insertions, 0 deletions

diff --git a/makima/frontend/src/hooks/useSpeakWebSocket.ts b/makima/frontend/src/hooks/useSpeakWebSocket.ts
new file mode 100644
index 0000000..3ef8851
--- /dev/null
+++ b/makima/frontend/src/hooks/useSpeakWebSocket.ts
@@ -0,0 +1,329 @@
+import { useState, useCallback, useRef, useEffect } from "react";
+import { SPEAK_ENDPOINT } from "../lib/api";
+
+export type SpeakStatus =
+  | "disconnected"
+  | "connecting"
+  | "connected"
+  | "loading_model"
+  | "speaking"
+  | "error";
+
+export interface SpeakWebSocketState {
+  status: SpeakStatus;
+  error: string | null;
+}
+
+export function useSpeakWebSocket() {
+  const [state, setState] = useState<SpeakWebSocketState>({
+    status: "disconnected",
+    error: null,
+  });
+
+  const wsRef = useRef<WebSocket | null>(null);
+  const audioContextRef = useRef<AudioContext | null>(null);
+  const audioQueueRef = useRef<Float32Array[]>([]);
+  const isPlayingRef = useRef(false);
+  const modelLoadingTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
+  const nextPlayTimeRef = useRef(0);
+
+  // Clean up on unmount
+  useEffect(() => {
+    return () => {
+      if (wsRef.current) {
+        wsRef.current.close();
+        wsRef.current = null;
+      }
+      if (audioContextRef.current) {
+        audioContextRef.current.close();
+        audioContextRef.current = null;
+      }
+      if (modelLoadingTimerRef.current) {
+        clearTimeout(modelLoadingTimerRef.current);
+        modelLoadingTimerRef.current = null;
+      }
+    };
+  }, []);
+
+  const getAudioContext = useCallback((): AudioContext => {
+    if (!audioContextRef.current || audioContextRef.current.state === "closed") {
+      audioContextRef.current = new AudioContext({ sampleRate: 24000 });
+    }
+    return audioContextRef.current;
+  }, []);
+
+  const playAudioQueue = useCallback(() => {
+    if (isPlayingRef.current) return;
+    isPlayingRef.current = true;
+
+    const ctx = getAudioContext();
+
+    function scheduleNext() {
+      const chunk = audioQueueRef.current.shift();
+      if (!chunk) {
+        isPlayingRef.current = false;
+        return;
+      }
+
+      const buffer = ctx.createBuffer(1, chunk.length, 24000);
+      buffer.copyToChannel(chunk, 0);
+
+      const source = ctx.createBufferSource();
+      source.buffer = buffer;
+      source.connect(ctx.destination);
+
+      // Schedule playback at the right time to avoid gaps
+      const now = ctx.currentTime;
+      const startTime = Math.max(now, nextPlayTimeRef.current);
+      source.start(startTime);
+      nextPlayTimeRef.current = startTime + buffer.duration;
+
+      source.onended = () => {
+        if (audioQueueRef.current.length > 0) {
+          scheduleNext();
+        } else {
+          isPlayingRef.current = false;
+        }
+      };
+    }
+
+    scheduleNext();
+  }, [getAudioContext]);
+
+  const connect = useCallback((): Promise<boolean> => {
+    return new Promise((resolve) => {
+      if (wsRef.current?.readyState === WebSocket.OPEN) {
+        resolve(true);
+        return;
+      }
+
+      if (wsRef.current) {
+        wsRef.current.close();
+        wsRef.current = null;
+      }
+
+      setState({ status: "connecting", error: null });
+
+      try {
+        const ws = new WebSocket(SPEAK_ENDPOINT);
+        ws.binaryType = "arraybuffer";
+        wsRef.current = ws;
+
+        ws.onopen = () => {
+          setState({ status: "connected", error: null });
+          resolve(true);
+        };
+
+        ws.onmessage = (event) => {
+          // Binary data = PCM audio chunk
+          if (event.data instanceof ArrayBuffer) {
+            // Clear model loading timer on first audio data
+            if (modelLoadingTimerRef.current) {
+              clearTimeout(modelLoadingTimerRef.current);
+              modelLoadingTimerRef.current = null;
+            }
+
+            // Update status to speaking if not already
+            setState((s) => {
+              if (s.status === "loading_model" || s.status === "connected") {
+                return { ...s, status: "speaking" };
+              }
+              return s;
+            });
+
+            // Convert PCM16 LE to Float32
+            const pcm16 = new Int16Array(event.data);
+            const float32 = new Float32Array(pcm16.length);
+            for (let i = 0; i < pcm16.length; i++) {
+              float32[i] = pcm16[i] / 32768;
+            }
+
+            audioQueueRef.current.push(float32);
+            playAudioQueue();
+            return;
+          }
+
+          // Text data = JSON message
+          try {
+            const message = JSON.parse(event.data);
+
+            switch (message.type) {
+              case "audio_end":
+                // Clear model loading timer
+                if (modelLoadingTimerRef.current) {
+                  clearTimeout(modelLoadingTimerRef.current);
+                  modelLoadingTimerRef.current = null;
+                }
+                // Wait for audio queue to drain, then go back to connected
+                // Use a short delay to let buffered audio finish
+                {
+                  const checkDone = () => {
+                    if (audioQueueRef.current.length === 0 && !isPlayingRef.current) {
+                      setState((s) => {
+                        if (s.status === "speaking" || s.status === "loading_model") {
+                          return { ...s, status: "connected" };
+                        }
+                        return s;
+                      });
+                    } else {
+                      setTimeout(checkDone, 100);
+                    }
+                  };
+                  checkDone();
+                }
+                break;
+
+              case "error":
+                if (modelLoadingTimerRef.current) {
+                  clearTimeout(modelLoadingTimerRef.current);
+                  modelLoadingTimerRef.current = null;
+                }
+                setState({
+                  status: "error",
+                  error: message.message || `Error: ${message.code}`,
+                });
+                break;
+            }
+          } catch {
+            console.error("Failed to parse speak WebSocket message:", event.data);
+          }
+        };
+
+        ws.onerror = () => {
+          setState({
+            status: "error",
+            error: "Failed to connect to speak server",
+          });
+          resolve(false);
+        };
+
+        ws.onclose = (event) => {
+          if (modelLoadingTimerRef.current) {
+            clearTimeout(modelLoadingTimerRef.current);
+            modelLoadingTimerRef.current = null;
+          }
+
+          let errorMessage: string | null = null;
+          if (event.code === 1006) {
+            errorMessage = "Connection failed - server may be unavailable";
+          } else if (event.code !== 1000 && event.code !== 1001) {
+            errorMessage = `Connection closed unexpectedly (code: ${event.code})`;
+          }
+
+          setState((s) => ({
+            status: "disconnected",
+            error: errorMessage || s.error,
+          }));
+          wsRef.current = null;
+        };
+      } catch (err) {
+        const message =
+          err instanceof Error ? err.message : "Failed to create WebSocket connection";
+        setState({ status: "error", error: message });
+        resolve(false);
+      }
+    });
+  }, [playAudioQueue]);
+
+  const speak = useCallback(
+    async (text: string) => {
+      if (!text.trim()) return;
+
+      // Connect if not connected
+      if (!wsRef.current || wsRef.current.readyState !== WebSocket.OPEN) {
+        const connected = await connect();
+        if (!connected) return;
+      }
+
+      // Reset audio state
+      audioQueueRef.current = [];
+      isPlayingRef.current = false;
+      nextPlayTimeRef.current = 0;
+
+      // Resume audio context if suspended (browser autoplay policy)
+      const ctx = getAudioContext();
+      if (ctx.state === "suspended") {
+        await ctx.resume();
+      }
+
+      // Start loading timer - if no audio arrives in 2 seconds, show loading state
+      modelLoadingTimerRef.current = setTimeout(() => {
+        setState((s) => {
+          if (s.status === "connected" || s.status === "connecting") {
+            return { ...s, status: "loading_model" };
+          }
+          return s;
+        });
+        modelLoadingTimerRef.current = null;
+      }, 2000);
+
+      // Send speak request
+      wsRef.current?.send(
+        JSON.stringify({ type: "speak", text })
+      );
+
+      setState((s) => ({ ...s, error: null }));
+    },
+    [connect, getAudioContext]
+  );
+
+  const cancel = useCallback(() => {
+    // Clear audio queue
+    audioQueueRef.current = [];
+    isPlayingRef.current = false;
+    nextPlayTimeRef.current = 0;
+
+    // Clear model loading timer
+    if (modelLoadingTimerRef.current) {
+      clearTimeout(modelLoadingTimerRef.current);
+      modelLoadingTimerRef.current = null;
+    }
+
+    // Send cancel message
+    if (wsRef.current?.readyState === WebSocket.OPEN) {
+      wsRef.current.send(JSON.stringify({ type: "cancel" }));
+    }
+
+    setState((s) => ({
+      ...s,
+      status: wsRef.current?.readyState === WebSocket.OPEN ? "connected" : "disconnected",
+    }));
+  }, []);
+
+  const disconnect = useCallback(() => {
+    // Clear audio queue
+    audioQueueRef.current = [];
+    isPlayingRef.current = false;
+    nextPlayTimeRef.current = 0;
+
+    if (modelLoadingTimerRef.current) {
+      clearTimeout(modelLoadingTimerRef.current);
+      modelLoadingTimerRef.current = null;
+    }
+
+    if (wsRef.current) {
+      // Send stop message before closing
+      if (wsRef.current.readyState === WebSocket.OPEN) {
+        wsRef.current.send(JSON.stringify({ type: "stop" }));
+      }
+      wsRef.current.close(1000, "User disconnected");
+      wsRef.current = null;
+    }
+
+    setState({ status: "disconnected", error: null });
+  }, []);
+
+  return {
+    ...state,
+    isConnected:
+      state.status === "connected" ||
+      state.status === "speaking" ||
+      state.status === "loading_model",
+    isSpeaking: state.status === "speaking",
+    isModelLoading: state.status === "loading_model",
+    speak,
+    cancel,
+    connect,
+    disconnect,
+  };
+}