Add Qwen3-TTS streaming endpoint for voice synthesis (#40)

* Task completion checkpoint

* Task completion checkpoint

* Task completion checkpoint

* Add Qwen3-TTS research document for live TTS replacement

Research findings for replacing Chatterbox TTS with Qwen3-TTS-12Hz-0.6B-Base:

- Current TTS: Chatterbox-Turbo-ONNX with batch-only generation, no streaming
- Qwen3-TTS: 97ms end-to-end latency, streaming support, 3-second voice cloning
- Voice cloning: Requires 3s reference audio + transcript (Makima voice planned)
- Integration: Python service with WebSocket bridge (no ONNX export available)
- Languages: 10 supported including English and Japanese

Document includes:
- Current architecture analysis (makima/src/tts.rs)
- Qwen3-TTS capabilities and requirements
- Feasibility assessment for live/streaming TTS
- Audio clip requirements for voice cloning
- Preliminary technical approach with architecture diagrams

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 03:11:15 UTC

* Add Qwen3-TTS research documentation

Comprehensive research on replacing Chatterbox TTS with Qwen3-TTS-12Hz-0.6B-Base:

- Current TTS implementation analysis (Chatterbox-Turbo-ONNX in makima/src/tts.rs)
- Qwen3-TTS capabilities: 97ms streaming latency, voice cloning with 3s reference
- Cross-lingual support: Japanese voice (Makima/Tomori Kusunoki) speaking English
- Python microservice architecture recommendation (FastAPI + WebSocket)
- Implementation phases and technical approach
- Hardware requirements and dependencies

Key findings:
- Live/streaming TTS is highly feasible with 97ms latency
- Voice cloning fully supported with 0.95 speaker similarity
- Recommended: Python microservice with WebSocket streaming

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add comprehensive Qwen3-TTS integration specification

This specification document defines the complete integration of
Qwen3-TTS-12Hz-0.6B-Base as a replacement for the existing Chatterbox-Turbo
TTS implementation. The document covers:

## Functional Requirements
- WebSocket endpoint /api/v1/speak for streaming TTS
- Voice cloning with default Makima voice (Japanese VA speaking English)
- Support for custom voice references
- Detailed client-to-server and server-to-client message protocols
- Integration with Listen page for bidirectional speech

## Non-Functional Requirements
- Latency targets: < 200ms first audio byte
- Audio quality: 24kHz, mono, PCM16/PCM32f
- Hardware requirements: CUDA GPU with 4-8GB VRAM
- Scalability: 10 concurrent sessions per GPU

## Architecture Specification
- Python TTS microservice with FastAPI/WebSocket
- Rust proxy endpoint in makima server
- Voice prompt caching mechanism (LRU cache)
- Error handling and recovery strategies

## API Contract
- Complete WebSocket message format definitions (TypeScript)
- Error codes and responses (TTS_UNAVAILABLE, SYNTHESIS_ERROR, etc.)
- Session state machine and lifecycle management

## Voice Asset Requirements
- Makima voice clip specifications (5-10s WAV, transcript required)
- Storage location: models/voices/makima/
- Metadata format for voice management

## Testing Strategy
- Unit tests for Python TTS service and Rust proxy
- Integration tests for WebSocket flow
- Latency benchmarks with performance targets
- Test data fixtures for various text lengths

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add Qwen3-TTS implementation plan

Comprehensive implementation plan for replacing Chatterbox-TTS with
Qwen3-TTS streaming TTS service, including:

- Task breakdown with estimated hours for each phase
- Phase 1: Python TTS microservice (FastAPI, WebSocket)
- Phase 2: Rust proxy integration (speak.rs, tts_client.rs)
- Detailed file changes and new module structure
- Testing plan with unit, integration, and latency benchmarks
- Risk assessment with mitigation strategies
- Success criteria for each phase

Based on specification in docs/specs/qwen3-tts-spec.md

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add author and research references to TTS implementation plan

Add links to research documentation and author attribution.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 03:25:06 UTC

* Add Python TTS service project structure (Phase 1.1-1.3)

Create the initial makima-tts Python service directory structure with:
- pyproject.toml with FastAPI, Qwen-TTS, and torch dependencies
- config.py with pydantic-settings TTSConfig class
- models.py with Pydantic message models (Start, Speak, Stop, Ready, etc.)

This implements tasks P1.1, P1.2, and P1.3 from the Qwen3-TTS implementation plan.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add TTS engine and voice manager for Qwen3-TTS (Phase 1.4-1.5)

Implement core TTS functionality:
- tts_engine.py: Qwen3-TTS wrapper with streaming audio chunk generation
- voice_manager.py: Voice prompt caching with LRU eviction and TTL support

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 03:30:06 UTC

* Add TTS proxy client and message types (Phase 2.1, 2.2, 2.4)

- Add tts_client.rs with TtsConfig, TtsCircuitBreaker, TtsError,
  TtsProxyClient, and TtsConnection structs for WebSocket proxying
- Add TTS message types to messages.rs (TtsAudioEncoding, TtsPriority,
  TtsStartMessage, TtsSpeakMessage, TtsStopMessage, TtsClientMessage,
  TtsReadyMessage, TtsAudioChunkMessage, TtsCompleteMessage,
  TtsErrorMessage, TtsStoppedMessage, TtsServerMessage)
- Export tts_client module from server mod.rs
- tokio-tungstenite already present in Cargo.toml

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add TTS WebSocket handler and route (Phase 2.3, 2.5, 2.6)

- Create speak.rs WebSocket handler that proxies to Python TTS service
- Add TtsState fields (tts_client, tts_config) to AppState
- Add with_tts() builder and is_tts_healthy() methods to AppState
- Register /api/v1/speak route in the router
- Add speak module export in handlers/mod.rs

The handler forwards WebSocket messages bidirectionally between
the client and the Python TTS microservice with proper error handling.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add Makima voice profile assets for TTS voice cloning

Creates the voice assets directory structure with:
- manifest.json containing voice configuration (voice_id, speaker,
  language, reference audio path, and Japanese transcript placeholder)
- README.md with instructions for obtaining voice reference audio

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add Rust-native Qwen3-TTS integration research document

Research findings for integrating Qwen3-TTS-12Hz-0.6B-Base directly into
the makima Rust codebase without Python. Key conclusions:
- ONNX export is not viable (unsupported architecture)
- Candle (HF Rust ML framework) is the recommended approach
- Model weights available in safetensors format (2.52GB total)
- Three components needed: LM backbone, code predictor, speech tokenizer
- Crane project has Qwen3-TTS as highest priority (potential upstream)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 11:21:43 UTC

* [WIP] Heartbeat checkpoint - 2026-01-27 11:24:19 UTC

* [WIP] Heartbeat checkpoint - 2026-01-27 11:26:43 UTC

* feat: implement Rust-native Qwen3-TTS using candle framework

Replace monolithic tts.rs with modular tts/ directory structure:

- tts/mod.rs: TtsEngine trait, TtsEngineFactory, shared types (AudioChunk,
  TtsError), and utility functions (save_wav, resample, argmax)
- tts/chatterbox.rs: existing ONNX-based ChatterboxTTS adapted to implement
  TtsEngine trait with Mutex-wrapped sessions for Send+Sync
- tts/qwen3/mod.rs: Qwen3Tts entry point with HuggingFace model loading
- tts/qwen3/config.rs: Qwen3TtsConfig parsing from HF config.json
- tts/qwen3/model.rs: 28-layer Qwen3 transformer with RoPE, GQA (16 heads,
  8 KV heads), SiLU MLP, RMS norm, and KV cache
- tts/qwen3/code_predictor.rs: 5-layer MTP module predicting 16 codebooks
- tts/qwen3/speech_tokenizer.rs: ConvNet encoder/decoder with 16-layer RVQ
- tts/qwen3/generate.rs: autoregressive generation loop with streaming support

Add candle-core, candle-nn, candle-transformers, safetensors to Cargo.toml.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: integrate TTS engine into speak WebSocket handler

- Update speak.rs handler to use TTS engine directly from SharedState
  instead of returning a stub "not implemented" error
- Add TtsEngine (OnceCell lazy-loaded) to AppState in state.rs with
  get_tts_engine() method for lazy initialization on first connection
- Implement full WebSocket protocol: client sends JSON speak/cancel/stop
  messages, server streams binary PCM audio chunks and audio_end signals
- Create voices/makima/manifest.json for Makima voice profile configuration
- All files compile successfully with zero errors

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add /speak TTS page with WebSocket audio playback

Add a new /speak frontend page for text-to-speech via WebSocket.
The page accepts text input and streams synthesized PCM audio through
the Web Audio API. Includes model loading indicator, cancel support,
and connection status. Also adds a loading bar to the listen page
ControlPanel during WebSocket connection.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

1 files changed, 271 insertions, 0 deletions

diff --git a/makima/src/tts/qwen3/config.rs b/makima/src/tts/qwen3/config.rs
new file mode 100644
index 0000000..6fb55d7
--- /dev/null
+++ b/makima/src/tts/qwen3/config.rs
@@ -0,0 +1,271 @@
+//! Qwen3-TTS model configuration.
+//!
+//! Parses config.json from the HuggingFace model repository to configure
+//! the language model, code predictor, and speech tokenizer.
+
+use serde::Deserialize;
+
+use crate::tts::TtsError;
+
+/// Top-level configuration for Qwen3-TTS-12Hz-0.6B-Base.
+#[derive(Debug, Clone, Deserialize)]
+pub struct Qwen3TtsConfig {
+    /// Language model (talker) configuration.
+    #[serde(default = "Qwen3LmConfig::default")]
+    pub lm: Qwen3LmConfig,
+
+    /// Code predictor (multi-token prediction) configuration.
+    #[serde(default = "CodePredictorConfig::default")]
+    pub code_predictor: CodePredictorConfig,
+
+    /// Speech tokenizer configuration.
+    #[serde(default = "SpeechTokenizerConfig::default")]
+    pub speech_tokenizer: SpeechTokenizerConfig,
+}
+
+impl Default for Qwen3TtsConfig {
+    fn default() -> Self {
+        Self {
+            lm: Qwen3LmConfig::default(),
+            code_predictor: CodePredictorConfig::default(),
+            speech_tokenizer: SpeechTokenizerConfig::default(),
+        }
+    }
+}
+
+impl Qwen3TtsConfig {
+    /// Load from a config.json file path.
+    pub fn from_json_path(path: &std::path::Path) -> Result<Self, TtsError> {
+        let content = std::fs::read_to_string(path)
+            .map_err(|e| TtsError::Config(format!("failed to read config: {e}")))?;
+        Self::from_json_str(&content)
+    }
+
+    /// Load from a JSON string.
+    pub fn from_json_str(json: &str) -> Result<Self, TtsError> {
+        // Try to parse the full HuggingFace config.json format first
+        if let Ok(hf_config) = serde_json::from_str::<HfConfig>(json) {
+            return Ok(Self::from_hf_config(&hf_config));
+        }
+        // Fall back to direct deserialization
+        serde_json::from_str(json)
+            .map_err(|e| TtsError::Config(format!("failed to parse config: {e}")))
+    }
+
+    /// Convert from HuggingFace's config.json format.
+    fn from_hf_config(hf: &HfConfig) -> Self {
+        Self {
+            lm: Qwen3LmConfig {
+                hidden_size: hf.hidden_size.unwrap_or(1024),
+                num_hidden_layers: hf.num_hidden_layers.unwrap_or(28),
+                num_attention_heads: hf.num_attention_heads.unwrap_or(16),
+                num_key_value_heads: hf.num_key_value_heads.unwrap_or(8),
+                intermediate_size: hf.intermediate_size.unwrap_or(3072),
+                head_dim: hf.head_dim.unwrap_or(128),
+                vocab_size: hf.vocab_size.unwrap_or(151_936),
+                max_position_embeddings: hf.max_position_embeddings.unwrap_or(32_768),
+                rms_norm_eps: hf.rms_norm_eps.unwrap_or(1e-6),
+                rope_theta: hf.rope_theta.unwrap_or(1_000_000.0),
+                use_sliding_window: hf.use_sliding_window.unwrap_or(false),
+                sliding_window: hf.sliding_window,
+                hidden_act: hf.hidden_act.clone().unwrap_or_else(|| "silu".to_string()),
+            },
+            code_predictor: CodePredictorConfig {
+                hidden_size: hf.code_predictor_hidden_size.unwrap_or(1024),
+                num_layers: hf.code_predictor_num_layers.unwrap_or(5),
+                num_attention_heads: hf
+                    .code_predictor_num_attention_heads
+                    .unwrap_or(16),
+                num_code_groups: hf.num_code_groups.unwrap_or(16),
+                codebook_vocab_size: hf.codebook_vocab_size.unwrap_or(2048),
+                rms_norm_eps: hf.rms_norm_eps.unwrap_or(1e-6),
+            },
+            speech_tokenizer: SpeechTokenizerConfig::default(),
+        }
+    }
+}
+
+/// Language model configuration (28-layer Qwen3 transformer).
+#[derive(Debug, Clone, Deserialize)]
+pub struct Qwen3LmConfig {
+    /// Hidden dimension of transformer layers.
+    pub hidden_size: usize,
+    /// Number of transformer layers.
+    pub num_hidden_layers: usize,
+    /// Number of attention heads.
+    pub num_attention_heads: usize,
+    /// Number of key-value heads (GQA).
+    pub num_key_value_heads: usize,
+    /// Feed-forward intermediate size.
+    pub intermediate_size: usize,
+    /// Dimension per attention head.
+    pub head_dim: usize,
+    /// Text vocabulary size.
+    pub vocab_size: usize,
+    /// Maximum sequence length for RoPE.
+    pub max_position_embeddings: usize,
+    /// RMS normalization epsilon.
+    pub rms_norm_eps: f64,
+    /// RoPE theta parameter.
+    pub rope_theta: f64,
+    /// Whether to use sliding window attention.
+    pub use_sliding_window: bool,
+    /// Sliding window size (if enabled).
+    pub sliding_window: Option<usize>,
+    /// Activation function name.
+    pub hidden_act: String,
+}
+
+impl Default for Qwen3LmConfig {
+    fn default() -> Self {
+        Self {
+            hidden_size: 1024,
+            num_hidden_layers: 28,
+            num_attention_heads: 16,
+            num_key_value_heads: 8,
+            intermediate_size: 3072,
+            head_dim: 128,
+            vocab_size: 151_936,
+            max_position_embeddings: 32_768,
+            rms_norm_eps: 1e-6,
+            rope_theta: 1_000_000.0,
+            use_sliding_window: false,
+            sliding_window: None,
+            hidden_act: "silu".to_string(),
+        }
+    }
+}
+
+impl Qwen3LmConfig {
+    /// Number of key-value head groups for GQA.
+    pub fn num_kv_groups(&self) -> usize {
+        self.num_attention_heads / self.num_key_value_heads
+    }
+}
+
+/// Code predictor (multi-token prediction) configuration.
+#[derive(Debug, Clone, Deserialize)]
+pub struct CodePredictorConfig {
+    /// Hidden size (matches LM hidden size).
+    pub hidden_size: usize,
+    /// Number of predictor transformer layers.
+    pub num_layers: usize,
+    /// Number of attention heads.
+    pub num_attention_heads: usize,
+    /// Number of codebook groups (residual codebooks).
+    pub num_code_groups: usize,
+    /// Vocabulary size per codebook.
+    pub codebook_vocab_size: usize,
+    /// RMS norm epsilon.
+    pub rms_norm_eps: f64,
+}
+
+impl Default for CodePredictorConfig {
+    fn default() -> Self {
+        Self {
+            hidden_size: 1024,
+            num_layers: 5,
+            num_attention_heads: 16,
+            num_code_groups: 16,
+            codebook_vocab_size: 2048,
+            rms_norm_eps: 1e-6,
+        }
+    }
+}
+
+/// Speech tokenizer (ConvNet codec) configuration.
+#[derive(Debug, Clone, Deserialize)]
+pub struct SpeechTokenizerConfig {
+    /// Number of RVQ codebooks.
+    pub num_codebooks: usize,
+    /// Codebook embedding dimension.
+    pub codebook_dim: usize,
+    /// Codebook vocabulary size per layer.
+    pub codebook_size: usize,
+    /// Encoder/decoder hidden channels.
+    pub hidden_channels: usize,
+    /// Output sample rate.
+    pub sample_rate: u32,
+    /// Token frame rate (Hz).
+    pub frame_rate: f32,
+    /// HuggingFace model ID for the speech tokenizer.
+    pub model_id: String,
+}
+
+impl Default for SpeechTokenizerConfig {
+    fn default() -> Self {
+        Self {
+            num_codebooks: 16,
+            codebook_dim: 256,
+            codebook_size: 2048,
+            hidden_channels: 512,
+            sample_rate: 24_000,
+            frame_rate: 12.5,
+            model_id: "Qwen/Qwen3-TTS-Tokenizer-12Hz".to_string(),
+        }
+    }
+}
+
+/// HuggingFace config.json format (partial, fields we need).
+#[derive(Debug, Deserialize)]
+struct HfConfig {
+    hidden_size: Option<usize>,
+    num_hidden_layers: Option<usize>,
+    num_attention_heads: Option<usize>,
+    num_key_value_heads: Option<usize>,
+    intermediate_size: Option<usize>,
+    head_dim: Option<usize>,
+    vocab_size: Option<usize>,
+    max_position_embeddings: Option<usize>,
+    rms_norm_eps: Option<f64>,
+    rope_theta: Option<f64>,
+    use_sliding_window: Option<bool>,
+    sliding_window: Option<usize>,
+    hidden_act: Option<String>,
+    // Code predictor specific fields
+    code_predictor_hidden_size: Option<usize>,
+    code_predictor_num_layers: Option<usize>,
+    code_predictor_num_attention_heads: Option<usize>,
+    num_code_groups: Option<usize>,
+    codebook_vocab_size: Option<usize>,
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_default_config() {
+        let config = Qwen3TtsConfig::default();
+        assert_eq!(config.lm.hidden_size, 1024);
+        assert_eq!(config.lm.num_hidden_layers, 28);
+        assert_eq!(config.lm.num_attention_heads, 16);
+        assert_eq!(config.lm.num_key_value_heads, 8);
+        assert_eq!(config.lm.head_dim, 128);
+        assert_eq!(config.lm.num_kv_groups(), 2);
+        assert_eq!(config.code_predictor.num_layers, 5);
+        assert_eq!(config.code_predictor.num_code_groups, 16);
+        assert_eq!(config.speech_tokenizer.num_codebooks, 16);
+    }
+
+    #[test]
+    fn test_config_from_json() {
+        let json = r#"{
+            "hidden_size": 1024,
+            "num_hidden_layers": 28,
+            "num_attention_heads": 16,
+            "num_key_value_heads": 8,
+            "intermediate_size": 3072,
+            "vocab_size": 151936,
+            "max_position_embeddings": 32768,
+            "rms_norm_eps": 1e-6,
+            "rope_theta": 1000000.0,
+            "hidden_act": "silu"
+        }"#;
+
+        let config = Qwen3TtsConfig::from_json_str(json).unwrap();
+        assert_eq!(config.lm.hidden_size, 1024);
+        assert_eq!(config.lm.num_hidden_layers, 28);
+        assert_eq!(config.lm.vocab_size, 151_936);
+    }
+}