Add Qwen3-TTS streaming endpoint for voice synthesis (#40)

* Task completion checkpoint

* Task completion checkpoint

* Task completion checkpoint

* Add Qwen3-TTS research document for live TTS replacement

Research findings for replacing Chatterbox TTS with Qwen3-TTS-12Hz-0.6B-Base:

- Current TTS: Chatterbox-Turbo-ONNX with batch-only generation, no streaming
- Qwen3-TTS: 97ms end-to-end latency, streaming support, 3-second voice cloning
- Voice cloning: Requires 3s reference audio + transcript (Makima voice planned)
- Integration: Python service with WebSocket bridge (no ONNX export available)
- Languages: 10 supported including English and Japanese

Document includes:
- Current architecture analysis (makima/src/tts.rs)
- Qwen3-TTS capabilities and requirements
- Feasibility assessment for live/streaming TTS
- Audio clip requirements for voice cloning
- Preliminary technical approach with architecture diagrams

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 03:11:15 UTC

* Add Qwen3-TTS research documentation

Comprehensive research on replacing Chatterbox TTS with Qwen3-TTS-12Hz-0.6B-Base:

- Current TTS implementation analysis (Chatterbox-Turbo-ONNX in makima/src/tts.rs)
- Qwen3-TTS capabilities: 97ms streaming latency, voice cloning with 3s reference
- Cross-lingual support: Japanese voice (Makima/Tomori Kusunoki) speaking English
- Python microservice architecture recommendation (FastAPI + WebSocket)
- Implementation phases and technical approach
- Hardware requirements and dependencies

Key findings:
- Live/streaming TTS is highly feasible with 97ms latency
- Voice cloning fully supported with 0.95 speaker similarity
- Recommended: Python microservice with WebSocket streaming

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add comprehensive Qwen3-TTS integration specification

This specification document defines the complete integration of
Qwen3-TTS-12Hz-0.6B-Base as a replacement for the existing Chatterbox-Turbo
TTS implementation. The document covers:

## Functional Requirements
- WebSocket endpoint /api/v1/speak for streaming TTS
- Voice cloning with default Makima voice (Japanese VA speaking English)
- Support for custom voice references
- Detailed client-to-server and server-to-client message protocols
- Integration with Listen page for bidirectional speech

## Non-Functional Requirements
- Latency targets: < 200ms first audio byte
- Audio quality: 24kHz, mono, PCM16/PCM32f
- Hardware requirements: CUDA GPU with 4-8GB VRAM
- Scalability: 10 concurrent sessions per GPU

## Architecture Specification
- Python TTS microservice with FastAPI/WebSocket
- Rust proxy endpoint in makima server
- Voice prompt caching mechanism (LRU cache)
- Error handling and recovery strategies

## API Contract
- Complete WebSocket message format definitions (TypeScript)
- Error codes and responses (TTS_UNAVAILABLE, SYNTHESIS_ERROR, etc.)
- Session state machine and lifecycle management

## Voice Asset Requirements
- Makima voice clip specifications (5-10s WAV, transcript required)
- Storage location: models/voices/makima/
- Metadata format for voice management

## Testing Strategy
- Unit tests for Python TTS service and Rust proxy
- Integration tests for WebSocket flow
- Latency benchmarks with performance targets
- Test data fixtures for various text lengths

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add Qwen3-TTS implementation plan

Comprehensive implementation plan for replacing Chatterbox-TTS with
Qwen3-TTS streaming TTS service, including:

- Task breakdown with estimated hours for each phase
- Phase 1: Python TTS microservice (FastAPI, WebSocket)
- Phase 2: Rust proxy integration (speak.rs, tts_client.rs)
- Detailed file changes and new module structure
- Testing plan with unit, integration, and latency benchmarks
- Risk assessment with mitigation strategies
- Success criteria for each phase

Based on specification in docs/specs/qwen3-tts-spec.md

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add author and research references to TTS implementation plan

Add links to research documentation and author attribution.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 03:25:06 UTC

* Add Python TTS service project structure (Phase 1.1-1.3)

Create the initial makima-tts Python service directory structure with:
- pyproject.toml with FastAPI, Qwen-TTS, and torch dependencies
- config.py with pydantic-settings TTSConfig class
- models.py with Pydantic message models (Start, Speak, Stop, Ready, etc.)

This implements tasks P1.1, P1.2, and P1.3 from the Qwen3-TTS implementation plan.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add TTS engine and voice manager for Qwen3-TTS (Phase 1.4-1.5)

Implement core TTS functionality:
- tts_engine.py: Qwen3-TTS wrapper with streaming audio chunk generation
- voice_manager.py: Voice prompt caching with LRU eviction and TTL support

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 03:30:06 UTC

* Add TTS proxy client and message types (Phase 2.1, 2.2, 2.4)

- Add tts_client.rs with TtsConfig, TtsCircuitBreaker, TtsError,
  TtsProxyClient, and TtsConnection structs for WebSocket proxying
- Add TTS message types to messages.rs (TtsAudioEncoding, TtsPriority,
  TtsStartMessage, TtsSpeakMessage, TtsStopMessage, TtsClientMessage,
  TtsReadyMessage, TtsAudioChunkMessage, TtsCompleteMessage,
  TtsErrorMessage, TtsStoppedMessage, TtsServerMessage)
- Export tts_client module from server mod.rs
- tokio-tungstenite already present in Cargo.toml

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add TTS WebSocket handler and route (Phase 2.3, 2.5, 2.6)

- Create speak.rs WebSocket handler that proxies to Python TTS service
- Add TtsState fields (tts_client, tts_config) to AppState
- Add with_tts() builder and is_tts_healthy() methods to AppState
- Register /api/v1/speak route in the router
- Add speak module export in handlers/mod.rs

The handler forwards WebSocket messages bidirectionally between
the client and the Python TTS microservice with proper error handling.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add Makima voice profile assets for TTS voice cloning

Creates the voice assets directory structure with:
- manifest.json containing voice configuration (voice_id, speaker,
  language, reference audio path, and Japanese transcript placeholder)
- README.md with instructions for obtaining voice reference audio

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add Rust-native Qwen3-TTS integration research document

Research findings for integrating Qwen3-TTS-12Hz-0.6B-Base directly into
the makima Rust codebase without Python. Key conclusions:
- ONNX export is not viable (unsupported architecture)
- Candle (HF Rust ML framework) is the recommended approach
- Model weights available in safetensors format (2.52GB total)
- Three components needed: LM backbone, code predictor, speech tokenizer
- Crane project has Qwen3-TTS as highest priority (potential upstream)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 11:21:43 UTC

* [WIP] Heartbeat checkpoint - 2026-01-27 11:24:19 UTC

* [WIP] Heartbeat checkpoint - 2026-01-27 11:26:43 UTC

* feat: implement Rust-native Qwen3-TTS using candle framework

Replace monolithic tts.rs with modular tts/ directory structure:

- tts/mod.rs: TtsEngine trait, TtsEngineFactory, shared types (AudioChunk,
  TtsError), and utility functions (save_wav, resample, argmax)
- tts/chatterbox.rs: existing ONNX-based ChatterboxTTS adapted to implement
  TtsEngine trait with Mutex-wrapped sessions for Send+Sync
- tts/qwen3/mod.rs: Qwen3Tts entry point with HuggingFace model loading
- tts/qwen3/config.rs: Qwen3TtsConfig parsing from HF config.json
- tts/qwen3/model.rs: 28-layer Qwen3 transformer with RoPE, GQA (16 heads,
  8 KV heads), SiLU MLP, RMS norm, and KV cache
- tts/qwen3/code_predictor.rs: 5-layer MTP module predicting 16 codebooks
- tts/qwen3/speech_tokenizer.rs: ConvNet encoder/decoder with 16-layer RVQ
- tts/qwen3/generate.rs: autoregressive generation loop with streaming support

Add candle-core, candle-nn, candle-transformers, safetensors to Cargo.toml.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: integrate TTS engine into speak WebSocket handler

- Update speak.rs handler to use TTS engine directly from SharedState
  instead of returning a stub "not implemented" error
- Add TtsEngine (OnceCell lazy-loaded) to AppState in state.rs with
  get_tts_engine() method for lazy initialization on first connection
- Implement full WebSocket protocol: client sends JSON speak/cancel/stop
  messages, server streams binary PCM audio chunks and audio_end signals
- Create voices/makima/manifest.json for Makima voice profile configuration
- All files compile successfully with zero errors

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add /speak TTS page with WebSocket audio playback

Add a new /speak frontend page for text-to-speech via WebSocket.
The page accepts text input and streams synthesized PCM audio through
the Web Audio API. Includes model loading indicator, cancel support,
and connection status. Also adds a loading bar to the listen page
ControlPanel during WebSocket connection.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

1 files changed, 261 insertions, 0 deletions

diff --git a/makima/src/tts/qwen3/code_predictor.rs b/makima/src/tts/qwen3/code_predictor.rs
new file mode 100644
index 0000000..0ef8a1d
--- /dev/null
+++ b/makima/src/tts/qwen3/code_predictor.rs
@@ -0,0 +1,261 @@
+//! Multi-Token Prediction (MTP) code predictor.
+//!
+//! After the main LM predicts the zeroth codebook token, this module
+//! predicts the remaining 15 codebook layers in parallel from the
+//! LM's hidden states.
+//!
+//! Architecture:
+//! - 5 transformer layers (same structure as main LM layers)
+//! - 16 output heads, one per codebook (vocab 2048 each)
+//! - Input: last hidden state from main LM + zeroth codebook embedding
+//! - Output: 16 codebook token predictions
+
+use candle_core::{Device, Module, Result, Tensor, D};
+use candle_nn::{embedding, linear_no_bias, rms_norm, Embedding, Linear, RmsNorm, VarBuilder};
+
+use super::config::{CodePredictorConfig, Qwen3LmConfig};
+use super::model::{KvCache, Qwen3Attention, Qwen3Mlp, RotaryEmbedding};
+
+/// A single code predictor transformer layer.
+///
+/// Uses the same pre-norm residual structure as the main LM layers.
+pub struct CodePredictorLayer {
+    self_attn: Qwen3Attention,
+    mlp: Qwen3Mlp,
+    input_layernorm: RmsNorm,
+    post_attention_layernorm: RmsNorm,
+}
+
+impl CodePredictorLayer {
+    pub fn new(config: &CodePredictorConfig, vb: VarBuilder) -> Result<Self> {
+        // Construct a Qwen3LmConfig-like view for the attention/MLP constructors
+        let lm_config = Qwen3LmConfig {
+            hidden_size: config.hidden_size,
+            num_hidden_layers: config.num_layers,
+            num_attention_heads: config.num_attention_heads,
+            num_key_value_heads: config.num_attention_heads, // No GQA in predictor
+            intermediate_size: config.hidden_size * 3, // 3072 for hidden=1024
+            head_dim: config.hidden_size / config.num_attention_heads,
+            rms_norm_eps: config.rms_norm_eps,
+            ..Qwen3LmConfig::default()
+        };
+
+        let self_attn = Qwen3Attention::new(&lm_config, vb.pp("self_attn"))?;
+        let mlp = Qwen3Mlp::new(&lm_config, vb.pp("mlp"))?;
+        let input_layernorm = rms_norm(
+            config.hidden_size,
+            config.rms_norm_eps,
+            vb.pp("input_layernorm"),
+        )?;
+        let post_attention_layernorm = rms_norm(
+            config.hidden_size,
+            config.rms_norm_eps,
+            vb.pp("post_attention_layernorm"),
+        )?;
+
+        Ok(Self {
+            self_attn,
+            mlp,
+            input_layernorm,
+            post_attention_layernorm,
+        })
+    }
+
+    pub fn forward(
+        &self,
+        hidden_states: &Tensor,
+        rope: &RotaryEmbedding,
+        kv_cache: &mut KvCache,
+        attention_mask: Option<&Tensor>,
+    ) -> Result<Tensor> {
+        let residual = hidden_states;
+        let hidden_states = self.input_layernorm.forward(hidden_states)?;
+        let hidden_states =
+            self.self_attn
+                .forward(&hidden_states, rope, kv_cache, attention_mask)?;
+        let hidden_states = (residual + hidden_states)?;
+
+        let residual = &hidden_states;
+        let hidden_states = self.post_attention_layernorm.forward(&hidden_states)?;
+        let hidden_states = self.mlp.forward(&hidden_states)?;
+        let output = (residual + hidden_states)?;
+
+        Ok(output)
+    }
+}
+
+/// Multi-token prediction code predictor.
+///
+/// Takes the hidden states from the main LM and predicts all 16 codebook
+/// tokens. The zeroth codebook is predicted by the main LM head; this
+/// module predicts the remaining 15 residual codebooks.
+pub struct CodePredictor {
+    /// Embedding layer for codebook tokens (shared across groups).
+    code_embeddings: Vec<Embedding>,
+    /// Projection from LM hidden + code embedding to predictor hidden.
+    input_proj: Linear,
+    /// 5 transformer layers.
+    layers: Vec<CodePredictorLayer>,
+    /// Final normalization.
+    norm: RmsNorm,
+    /// Per-codebook output heads (16 heads, each projecting to codebook_vocab_size).
+    output_heads: Vec<Linear>,
+    /// RoPE for the predictor's attention layers.
+    rope: RotaryEmbedding,
+    config: CodePredictorConfig,
+}
+
+impl CodePredictor {
+    pub fn new(
+        config: &CodePredictorConfig,
+        lm_config: &Qwen3LmConfig,
+        vb: VarBuilder,
+    ) -> Result<Self> {
+        let predictor_vb = vb.pp("code_predictor");
+
+        // Code embeddings for each codebook group
+        let mut code_embeddings = Vec::with_capacity(config.num_code_groups);
+        for i in 0..config.num_code_groups {
+            let emb = embedding(
+                config.codebook_vocab_size,
+                config.hidden_size,
+                predictor_vb.pp(format!("code_embeddings.{i}")),
+            )?;
+            code_embeddings.push(emb);
+        }
+
+        // Input projection: LM hidden (1024) + code embedding (1024) -> predictor hidden (1024)
+        let input_proj = linear_no_bias(
+            config.hidden_size * 2,
+            config.hidden_size,
+            predictor_vb.pp("input_proj"),
+        )?;
+
+        // Transformer layers
+        let mut layers = Vec::with_capacity(config.num_layers);
+        for i in 0..config.num_layers {
+            let layer =
+                CodePredictorLayer::new(config, predictor_vb.pp(format!("layers.{i}")))?;
+            layers.push(layer);
+        }
+
+        let norm = rms_norm(
+            config.hidden_size,
+            config.rms_norm_eps,
+            predictor_vb.pp("norm"),
+        )?;
+
+        // Output heads for each codebook
+        let mut output_heads = Vec::with_capacity(config.num_code_groups);
+        for i in 0..config.num_code_groups {
+            let head = linear_no_bias(
+                config.hidden_size,
+                config.codebook_vocab_size,
+                predictor_vb.pp(format!("output_heads.{i}")),
+            )?;
+            output_heads.push(head);
+        }
+
+        // RoPE for predictor attention (uses same theta/dim as main LM but with predictor head_dim)
+        let predictor_head_dim = config.hidden_size / config.num_attention_heads;
+        let rope_config = Qwen3LmConfig {
+            head_dim: predictor_head_dim,
+            rope_theta: lm_config.rope_theta,
+            max_position_embeddings: lm_config.max_position_embeddings,
+            ..Qwen3LmConfig::default()
+        };
+        let rope = RotaryEmbedding::new(&rope_config, vb.dtype(), vb.device())?;
+
+        Ok(Self {
+            code_embeddings,
+            input_proj,
+            layers,
+            norm,
+            output_heads,
+            rope,
+            config: config.clone(),
+        })
+    }
+
+    /// Predict all 16 codebook tokens from the LM hidden state.
+    ///
+    /// `lm_hidden`: [batch, 1, hidden_size] — last hidden state from main LM
+    /// `zeroth_code`: the token predicted by the main LM head (zeroth codebook)
+    ///
+    /// Returns: Vec of 16 token indices (one per codebook), starting with zeroth_code.
+    pub fn predict(
+        &self,
+        lm_hidden: &Tensor,
+        zeroth_code: u32,
+        device: &Device,
+    ) -> Result<Vec<u32>> {
+        let mut all_codes = Vec::with_capacity(self.config.num_code_groups);
+        all_codes.push(zeroth_code);
+
+        // The code predictor iterates through codebook groups.
+        // For each group i (1..16), it:
+        //   1. Embeds the previous codebook token
+        //   2. Concatenates with LM hidden state
+        //   3. Projects through the predictor layers
+        //   4. Predicts the next codebook token via output_head[i]
+        let mut prev_code = zeroth_code;
+
+        for group_idx in 1..self.config.num_code_groups {
+            // Embed the previous codebook token
+            let code_tensor = Tensor::from_vec(
+                vec![prev_code],
+                (1, 1),
+                device,
+            )?;
+            let code_emb = self.code_embeddings[group_idx - 1].forward(&code_tensor)?;
+
+            // Concatenate LM hidden state with code embedding
+            let combined = Tensor::cat(&[lm_hidden, &code_emb], D::Minus1)?;
+
+            // Project to predictor hidden size
+            let mut hidden = self.input_proj.forward(&combined)?;
+
+            // Run through predictor transformer layers (no KV cache needed — single step)
+            let mut kv_caches: Vec<KvCache> =
+                (0..self.config.num_layers).map(|_| KvCache::new()).collect();
+            for (i, layer) in self.layers.iter().enumerate() {
+                hidden = layer.forward(&hidden, &self.rope, &mut kv_caches[i], None)?;
+            }
+
+            hidden = self.norm.forward(&hidden)?;
+
+            // Predict codebook token
+            let logits = self.output_heads[group_idx].forward(&hidden)?;
+
+            // Greedy decode: argmax
+            let logits_flat = logits.squeeze(0)?.squeeze(0)?; // [codebook_vocab_size]
+            let next_code = logits_flat
+                .argmax(0)?
+                .to_scalar::<u32>()?;
+
+            all_codes.push(next_code);
+            prev_code = next_code;
+        }
+
+        Ok(all_codes)
+    }
+
+    /// Number of codebook groups.
+    pub fn num_code_groups(&self) -> usize {
+        self.config.num_code_groups
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_code_predictor_config() {
+        let config = CodePredictorConfig::default();
+        assert_eq!(config.num_layers, 5);
+        assert_eq!(config.num_code_groups, 16);
+        assert_eq!(config.codebook_vocab_size, 2048);
+        assert_eq!(config.hidden_size, 1024);
+    }
+}