diff options
Diffstat (limited to 'docs/plans/qwen3-tts-implementation-plan.md')
| -rw-r--r-- | docs/plans/qwen3-tts-implementation-plan.md | 514 |
1 files changed, 514 insertions, 0 deletions
diff --git a/docs/plans/qwen3-tts-implementation-plan.md b/docs/plans/qwen3-tts-implementation-plan.md new file mode 100644 index 0000000..76ecb33 --- /dev/null +++ b/docs/plans/qwen3-tts-implementation-plan.md @@ -0,0 +1,514 @@ +# Qwen3-TTS Implementation Plan — Pure Rust (Candle) + +**Version:** 2.0 +**Created:** 2026-01-27 +**Status:** Final +**Authors:** makima development team +**Spec Reference:** [docs/specs/qwen3-tts-spec.md](../specs/qwen3-tts-spec.md) +**Research:** [docs/research/rust-native-tts-research.md](../research/rust-native-tts-research.md) + +--- + +## Table of Contents + +1. [Overview](#1-overview) +2. [Task Breakdown](#2-task-breakdown) +3. [File Changes](#3-file-changes) +4. [Phase 1: Candle-Based TTS Module](#4-phase-1-candle-based-tts-module) +5. [Phase 2: WebSocket Handler + Voice Assets](#5-phase-2-websocket-handler--voice-assets) +6. [Phase 3: Optimization + Integration](#6-phase-3-optimization--integration) +7. [Testing Plan](#7-testing-plan) +8. [Risk Assessment](#8-risk-assessment) +9. [Dependencies & Prerequisites](#9-dependencies--prerequisites) +10. [Success Criteria](#10-success-criteria) + +--- + +## 1. Overview + +This plan details the implementation of Qwen3-TTS integration for the makima system as a **pure Rust** solution using the **candle** ML framework. There is no Python microservice and no proxy pattern — the TTS model runs directly inside the main makima process, loading safetensors weights via candle. + +### Key Objectives + +1. Implement Qwen3-TTS model inference natively in Rust using candle +2. Create a `makima/src/tts/` module with TTS trait, Chatterbox adapter, and Qwen3 submodule +3. Update the `/api/v1/speak` WebSocket handler to call the TTS engine directly +4. Enable streaming audio delivery with <200ms time-to-first-audio (TTFA) +5. Support voice cloning with default Makima voice + +### Architecture Summary + +``` +Client Browser + │ + │ WebSocket: /api/v1/speak + ▼ +Makima Server (Rust/Axum) + │ + │ speak.rs handler → TTS Engine (in-process) + │ + │ candle-based Qwen3-TTS inference + │ (safetensors weights loaded directly) + ▼ +Audio Stream back to client +``` + +**Key architectural decisions:** +- **No Python.** All inference runs in Rust via candle. +- **No microservice.** TTS runs in-process, no separate service to deploy. +- **No proxy.** The speak handler calls the TTS engine directly. +- **Lazy loading.** Models loaded on first TTS request (like listen.rs pattern). +- **SafeTensors.** Weights loaded directly — no ONNX conversion needed. + +--- + +## 2. Task Breakdown + +### Phase 1: Candle-Based TTS Module (Priority: Critical) + +| ID | Task | Depends On | Estimated Hours | +|----|------|------------|-----------------| +| P1.1 | Create `makima/src/tts/mod.rs` — TTS trait + factory + types | - | 3 | +| P1.2 | Move existing `tts.rs` to `makima/src/tts/chatterbox.rs` | P1.1 | 2 | +| P1.3 | Create `makima/src/tts/qwen3/config.rs` — Model config parsing | P1.1 | 2 | +| P1.4 | Implement `makima/src/tts/qwen3/model.rs` — 28-layer LM backbone | P1.3 | 12 | +| P1.5 | Implement `makima/src/tts/qwen3/code_predictor.rs` — MTP module | P1.4 | 8 | +| P1.6 | Implement `makima/src/tts/qwen3/speech_tokenizer.rs` — ConvNet codec | P1.3 | 10 | +| P1.7 | Implement `makima/src/tts/qwen3/generate.rs` — Autoregressive generation | P1.4, P1.5, P1.6 | 8 | +| P1.8 | Create `makima/src/tts/qwen3/mod.rs` — Public API | P1.7 | 3 | +| P1.9 | Add candle dependencies to `Cargo.toml` | - | 1 | +| P1.10 | Unit tests for config, model layers, tokenizer | P1.4-P1.6 | 6 | + +**Phase 1 Total: ~55 hours** + +### Phase 2: WebSocket Handler + Voice Assets (Priority: High) + +| ID | Task | Depends On | Estimated Hours | +|----|------|------------|-----------------| +| P2.1 | Rewrite `speak.rs` — Direct TTS handler (remove proxy) | P1.8 | 6 | +| P2.2 | Add TTS models to `SharedState` (lazy loading via `OnceCell`) | P1.8 | 3 | +| P2.3 | Implement voice prompt caching (LRU) | P1.8 | 3 | +| P2.4 | Remove `tts_client.rs` (no longer needed) | P2.1 | 1 | +| P2.5 | Update `state.rs` — Remove TTS proxy fields, add TTS model fields | P2.2 | 2 | +| P2.6 | Update `mod.rs` — Remove `tts_client` module | P2.4 | 0.5 | +| P2.7 | Create voice manifest structure (`models/voices/makima/`) | - | 1 | +| P2.8 | Acquire Makima voice reference audio | - | 2 | +| P2.9 | Test voice cloning quality | P1.8, P2.8 | 2 | + +**Phase 2 Total: ~20.5 hours** + +### Phase 3: Optimization + Integration (Priority: Medium) + +| ID | Task | Depends On | Estimated Hours | +|----|------|------------|-----------------| +| P3.1 | Implement streaming generation (token-by-token waveform decode) | P2.1 | 6 | +| P3.2 | GPU memory optimization (bf16, cache management) | P3.1 | 4 | +| P3.3 | Listen page integration for bidirectional speech | P2.1 | 4 | +| P3.4 | Latency benchmarks | P3.1 | 3 | +| P3.5 | Integration tests (WebSocket end-to-end) | P2.1 | 4 | +| P3.6 | Documentation | P3.5 | 2 | + +**Phase 3 Total: ~23 hours** + +--- + +## 3. File Changes + +### New Files + +``` +makima/src/tts/ +├── mod.rs // TTS trait, factory, shared types +├── chatterbox.rs // Existing ONNX-based Chatterbox (moved from tts.rs) +└── qwen3/ + ├── mod.rs // Qwen3TTS public API + ├── model.rs // Qwen3 LM transformer (28 layers) + ├── code_predictor.rs // MTP module (5 layers, 16 codebooks) + ├── speech_tokenizer.rs // Encoder + Decoder (causal ConvNet + RVQ) + ├── config.rs // Model config from config.json / safetensors + └── generate.rs // Autoregressive generation loop with KV cache +``` + +### Modified Files + +| File | Change Description | +|------|-------------------| +| `makima/src/server/handlers/speak.rs` | Rewrite: direct TTS engine call instead of proxy | +| `makima/src/server/state.rs` | Remove `tts_client`/`tts_config` fields, add `tts_models: OnceCell<TtsModels>` | +| `makima/src/server/mod.rs` | Remove `pub mod tts_client;` | +| `makima/src/server/handlers/mod.rs` | No change (speak already exported) | +| `makima/Cargo.toml` | Add candle-core, candle-nn, candle-transformers; remove tokio-tungstenite if unused | +| `makima/src/lib.rs` or `main.rs` | Add `pub mod tts;` | + +### Deleted Files + +| File | Reason | +|------|--------| +| `makima/src/server/tts_client.rs` | No longer needed — no proxy pattern | +| `tts-service/` (entire directory) | Python service rejected; pure Rust solution | + +--- + +## 4. Phase 1: Candle-Based TTS Module + +### 4.1 TTS Trait and Factory (`tts/mod.rs`) + +```rust +use async_trait::async_trait; + +/// Audio chunk for streaming output. +pub struct AudioChunk { + pub samples: Vec<f32>, + pub sample_rate: u32, + pub is_final: bool, +} + +/// TTS engine trait — implemented by Chatterbox and Qwen3. +#[async_trait] +pub trait TtsEngine: Send + Sync { + /// Generate audio from text. + async fn generate( + &self, + text: &str, + voice_id: &str, + language: &str, + ) -> Result<Vec<AudioChunk>, TtsError>; + + /// Pre-load a voice prompt. + async fn load_voice(&self, voice_id: &str) -> Result<(), TtsError>; + + /// Check if the engine is ready. + fn is_ready(&self) -> bool; +} +``` + +### 4.2 Qwen3 LM Backbone (`tts/qwen3/model.rs`) + +Extend candle-transformers' Qwen2 model implementation: + +- **28 transformer layers** with RoPE, GQA (16 heads, 8 KV heads), head dim 128 +- **Hidden size:** 1024, **intermediate size:** 3072 +- **Input:** text tokens + reference audio codes (concatenated) +- **Output:** zeroth codebook token logits + +**Key implementation detail:** The existing `candle_transformers::models::qwen2` module provides the base attention and MLP layers. We extend this with: +- TTS-specific input embedding (text + audio token embeddings) +- Speaker encoder concatenation +- Code predictor output head (instead of standard LM head) + +### 4.3 Code Predictor (`tts/qwen3/code_predictor.rs`) + +- **5-layer** transformer module +- **Input:** hidden states from the main LM +- **Output:** 16 codebook predictions (vocab size 2048 each) +- After the main LM predicts the zeroth codebook token, this module predicts the remaining 15 codebook layers in parallel + +### 4.4 Speech Tokenizer (`tts/qwen3/speech_tokenizer.rs`) + +Two sub-components: + +**Encoder** (used for voice cloning): +- Causal 1D ConvNet converting reference audio waveform → discrete multi-codebook tokens +- 16-layer RVQ (Residual Vector Quantization) +- First codebook = semantic (WavLM-guided), remaining 15 = acoustic + +**Decoder** (used for audio output): +- Causal 1D ConvNet reconstructing waveforms from discrete codes +- Input: 16 codebook indices → lookup embeddings → ConvNet → waveform +- Output: 24kHz mono audio + +**candle implementation notes:** +- `candle_nn::Conv1d` for all convolution layers +- `candle_nn::Embedding` for codebook lookups +- Weight normalization handled manually + +### 4.5 Autoregressive Generation (`tts/qwen3/generate.rs`) + +```rust +pub async fn generate( + model: &Qwen3Model, + code_predictor: &CodePredictor, + speech_tokenizer: &SpeechTokenizer, + text_tokens: &[u32], + voice_prompt: &VoicePrompt, +) -> Result<Vec<AudioChunk>, TtsError> { + // 1. Encode reference audio → speaker embedding + audio codes + let speaker_emb = speech_tokenizer.encode(&voice_prompt.audio)?; + + // 2. Prepare input: [text_tokens, audio_codes] + let input = prepare_input(text_tokens, &speaker_emb)?; + + // 3. Autoregressive loop with KV cache + let mut kv_cache = KvCache::new(model.num_layers()); + let mut generated_codes = Vec::new(); + + loop { + let logits = model.forward(&input, &mut kv_cache)?; + let next_token = sample_token(&logits); + + if next_token == EOS_TOKEN { break; } + generated_codes.push(next_token); + + // 4. Code predictor: predict remaining 15 codebooks + let all_codes = code_predictor.predict(&model.last_hidden_state(), next_token)?; + + // 5. Decode to audio (can be done incrementally for streaming) + let chunk = speech_tokenizer.decode(&all_codes)?; + // yield chunk for streaming + } +} +``` + +--- + +## 5. Phase 2: WebSocket Handler + Voice Assets + +### 5.1 Speak Handler (Rewritten) + +```rust +// makima/src/server/handlers/speak.rs +// +// Direct TTS handler — no proxy, no external service. + +pub async fn websocket_handler( + ws: WebSocketUpgrade, + State(state): State<SharedState>, +) -> Response { + ws.on_upgrade(|socket| handle_speak_socket(socket, state)) +} + +async fn handle_speak_socket(socket: WebSocket, state: SharedState) { + let session_id = Uuid::new_v4().to_string(); + + // Lazy-load TTS models (like listen.rs does for STT) + let tts = match state.get_tts_models().await { + Ok(tts) => tts, + Err(e) => { + send_error(&mut socket, "MODEL_LOADING", &e.to_string()).await; + return; + } + }; + + // Session loop: parse JSON messages, dispatch to TTS engine + let (mut sender, mut receiver) = socket.split(); + + while let Some(msg) = receiver.next().await { + match parse_client_message(msg) { + ClientMessage::Start(config) => { + // Load voice, send Ready + } + ClientMessage::Speak(text) => { + // Run inference, stream audio chunks + for chunk in tts.engine.generate(&text, voice_id, language).await? { + sender.send(Message::Binary(chunk.to_pcm16())).await?; + } + sender.send(complete_message()).await?; + } + ClientMessage::Stop => break, + ClientMessage::Cancel => { /* abort current generation */ } + } + } +} +``` + +### 5.2 State Changes + +Remove from `AppState`: +- `tts_client: Option<Arc<TtsProxyClient>>` +- `tts_config: Option<TtsConfig>` +- `with_tts()` method +- `is_tts_healthy()` method + +Add to `AppState`: +- `tts_models: OnceCell<TtsModels>` — lazily loaded TTS engine +- `get_tts_models()` method (async, like `get_ml_models()`) + +### 5.3 Voice Assets + +``` +models/voices/makima/ +├── manifest.json # Voice metadata +├── reference.wav # 5-15 second reference audio +└── transcript.txt # Exact transcript of reference audio +``` + +--- + +## 6. Phase 3: Optimization + Integration + +### 6.1 Streaming Generation + +The 12Hz model's causal architecture enables token-by-token waveform generation: +- Each token = ~80ms of audio (12.5 Hz) +- After generating each token, decode immediately and send audio chunk +- Client receives audio before full generation completes + +### 6.2 GPU Memory Optimization + +- Load weights in bf16/f16 (candle supports both) +- Implement KV cache with fixed maximum size +- Clear cache between sessions +- CPU fallback when GPU is unavailable + +### 6.3 Listen Page Integration + +Following the pattern in `listen.rs`: +- TTS model protected behind `tokio::sync::Mutex` +- WebSocket endpoint emits audio chunks as tokens are generated +- Bidirectional: STT (listen) → process → TTS (speak) loop + +--- + +## 7. Testing Plan + +### 7.1 Unit Tests + +| Test Area | Coverage | Key Tests | +|-----------|----------|-----------| +| Config parsing | 100% | Load config from JSON, validate fields | +| Model layers | 80% | Attention, MLP, Conv1d shapes | +| Code predictor | 85% | Multi-codebook output shapes | +| Speech tokenizer | 80% | Encode/decode round-trip | +| Voice cache | 95% | LRU eviction, TTL expiration | +| Message parsing | 100% | All client/server message types | + +### 7.2 Integration Tests + +| Test | Description | +|------|-------------| +| WebSocket flow | Connect → Start → Speak → Audio chunks → Complete → Stop | +| Error handling | Invalid text, unknown voice, model loading failure | +| Cancellation | Cancel mid-generation | +| Voice cloning | Generate with custom reference audio | + +### 7.3 Latency Benchmarks + +| Metric | Target | Acceptable | Warning | +|--------|--------|------------|---------| +| First Audio (short text) | < 150ms | < 200ms | > 300ms | +| First Audio (medium text) | < 200ms | < 300ms | > 500ms | +| First Audio (long text) | < 300ms | < 500ms | > 800ms | +| Inter-chunk latency | < 30ms | < 50ms | > 100ms | +| GPU memory | < 4GB | < 6GB | > 8GB | + +--- + +## 8. Risk Assessment + +### 8.1 Technical Risks + +| Risk | Likelihood | Impact | Mitigation | +|------|------------|--------|------------| +| **Candle implementation takes longer** | Medium | Medium | Reference Crane's Spark-TTS; use qwen3-rs as LM reference | +| **Speech tokenizer ConvNet is complex** | Medium | High | Study PyTorch source; ConvNet layers are simpler than transformers | +| **Model quality differs from PyTorch** | Low | High | Validate with reference audio; ensure bf16 precision | +| **Crane ships Qwen3-TTS first** | Medium | Positive | Adopt their implementation or use as reference | +| **GPU memory issues** | Low | Medium | 0.6B model is small (~2.5GB); fits in 4GB VRAM | + +### 8.2 Contingency Plans + +| Scenario | Response | +|----------|----------| +| Candle implementation blocked | Use Crane crate as dependency if they ship Qwen3-TTS | +| ConvNet decoder too complex | Implement simplified decoder; optimize later | +| Latency exceeds targets | Start with batch mode + chunked delivery (acceptable UX) | +| No GPU available | CPU fallback with candle's MKL support (degraded performance) | + +--- + +## 9. Dependencies & Prerequisites + +### 9.1 Rust Dependencies + +Add to `Cargo.toml`: + +```toml +[dependencies] +candle-core = "0.8" +candle-nn = "0.8" +candle-transformers = "0.8" +# Keep existing: tokenizers, hf-hub, safetensors, ndarray +``` + +### 9.2 Hardware Requirements + +| Component | Minimum | Recommended | +|-----------|---------|-------------| +| GPU | CUDA 4GB VRAM / Metal (macOS) | NVIDIA RTX 3060+ (8GB+) | +| RAM | 8GB | 16GB | +| Storage | 5GB (model weights) | 10GB | + +### 9.3 Voice Asset Prerequisites + +Before Phase 2 voice testing: +1. Makima voice reference audio (5-15 seconds, clean speech) +2. Accurate transcript of reference audio +3. Format: WAV 24kHz mono, 16-bit PCM + +--- + +## 10. Success Criteria + +### 10.1 Phase 1 Completion + +- [ ] candle-based Qwen3 model loads safetensors weights +- [ ] Forward pass produces valid logits +- [ ] Speech tokenizer encodes/decodes audio +- [ ] Code predictor generates 16 codebook layers +- [ ] Unit tests pass with > 80% coverage + +### 10.2 Phase 2 Completion + +- [ ] `/api/v1/speak` endpoint produces audio from text +- [ ] No Python service required +- [ ] Voice cloning works with reference audio +- [ ] Error handling returns appropriate codes +- [ ] speak.rs calls TTS engine directly (no proxy) + +### 10.3 Phase 3 Completion + +- [ ] Streaming generation with < 200ms TTFA +- [ ] GPU memory usage < 6GB +- [ ] Integration tests pass +- [ ] Listen page bidirectional speech works +- [ ] Latency benchmarks documented + +### 10.4 Final Acceptance Criteria + +1. **Functional:** End-to-end TTS streaming via WebSocket, pure Rust, no Python +2. **Performance:** TTFA < 200ms, subsequent chunks < 100ms +3. **Quality:** Synthesized speech is intelligible and recognizable as Makima +4. **Reliability:** Error handling is robust; graceful degradation on GPU failure +5. **Architecture:** Clean `tts/` module with trait-based engine selection + +--- + +## Appendix A: Quick Start Commands + +### Development + +```bash +# Build with candle GPU support +cd makima +cargo build --features cuda # or --features metal for macOS + +# Run server with TTS enabled +TTS_ENGINE=qwen3 TTS_DEVICE=cuda:0 cargo run + +# Run TTS-specific tests +cargo test tts +``` + +### Benchmarks + +```bash +# Run latency benchmarks (requires GPU) +cargo bench --bench tts_latency +``` + +## Appendix B: Reference Implementations + +- [candle-transformers qwen2 model](https://docs.rs/candle-transformers/latest/candle_transformers/models/qwen2/index.html) — base attention/MLP layers +- [qwen3-rs](https://github.com/reinterpretcat/qwen3-rs) — educational Qwen3 in Rust +- [Crane](https://github.com/lucasjinreal/Crane) — Rust TTS engine (Qwen3-TTS on roadmap) +- [docs/research/rust-native-tts-research.md](../research/rust-native-tts-research.md) — full feasibility analysis |