Add Qwen3-TTS research document for live TTS replacement

Research findings for replacing Chatterbox TTS with Qwen3-TTS-12Hz-0.6B-Base:

- Current TTS: Chatterbox-Turbo-ONNX with batch-only generation, no streaming
- Qwen3-TTS: 97ms end-to-end latency, streaming support, 3-second voice cloning
- Voice cloning: Requires 3s reference audio + transcript (Makima voice planned)
- Integration: Python service with WebSocket bridge (no ONNX export available)
- Languages: 10 supported including English and Japanese

Document includes:
- Current architecture analysis (makima/src/tts.rs)
- Qwen3-TTS capabilities and requirements
- Feasibility assessment for live/streaming TTS
- Audio clip requirements for voice cloning
- Preliminary technical approach with architecture diagrams

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

1 files changed, 548 insertions, 0 deletions

diff --git a/docs/research/tts-qwen3-research.md b/docs/research/tts-qwen3-research.md
new file mode 100644
index 0000000..a961b4f
--- /dev/null
+++ b/docs/research/tts-qwen3-research.md
@@ -0,0 +1,548 @@
+# TTS Research: Qwen3-TTS-12Hz-0.6B-Base Integration
+
+## Executive Summary
+
+This document evaluates replacing the current Chatterbox TTS implementation with Qwen3-TTS-12Hz-0.6B-Base for the makima system. The goal is to enable near-real-time voice synthesis with voice cloning capabilities, defaulting to Makima's Japanese voice (Tomori Kusunoki) speaking English.
+
+**Key Findings:**
+- Qwen3-TTS offers superior streaming capabilities (~97ms latency) compared to the current batch-only Chatterbox implementation
+- Voice cloning requires only 3 seconds of reference audio
+- No official ONNX export exists; Python/PyTorch inference required
+- The 0.6B model is optimized for resource-constrained environments
+
+---
+
+## 1. Current TTS Implementation Analysis
+
+### 1.1 Architecture Overview
+
+The current implementation uses **Chatterbox-Turbo-ONNX** from ResembleAI:
+
+```
+Location: makima/src/tts.rs
+Model ID: ResembleAI/chatterbox-turbo-ONNX
+Sample Rate: 24,000 Hz
+```
+
+**Components:**
+| Component | File | Purpose |
+|-----------|------|---------|
+| `speech_encoder.onnx` | ~XX MB | Encodes reference audio to speaker embeddings |
+| `embed_tokens.onnx` | ~XX MB | Token embedding layer |
+| `language_model.onnx` | ~XX MB | Autoregressive text-to-speech token generation |
+| `conditional_decoder.onnx` | ~XX MB | Converts speech tokens to waveform |
+| `tokenizer.json` | ~KB | Text tokenization |
+
+### 1.2 Current API Surface
+
+```rust
+pub struct ChatterboxTTS {
+    speech_encoder: Session,
+    embed_tokens: Session,
+    language_model: Session,
+    conditional_decoder: Session,
+    tokenizer: Tokenizer,
+}
+
+impl ChatterboxTTS {
+    // Load from pretrained models
+    pub fn from_pretrained(model_dir: Option<&str>) -> Result<Self, TtsError>;
+
+    // Generate speech (requires voice reference)
+    pub fn generate_tts(&mut self, _text: &str) -> Result<Vec<f32>, TtsError>;
+
+    // Voice cloning from file path
+    pub fn generate_tts_with_voice(
+        &mut self,
+        text: &str,
+        sample_audio_path: &Path,
+    ) -> Result<Vec<f32>, TtsError>;
+
+    // Voice cloning from raw samples
+    pub fn generate_tts_with_samples(
+        &mut self,
+        text: &str,
+        samples: &[f32],
+        sample_rate: u32,
+    ) -> Result<Vec<f32>, TtsError>;
+}
+```
+
+### 1.3 Current Capabilities
+
+| Feature | Supported | Notes |
+|---------|-----------|-------|
+| Voice Cloning | **Yes** | Required for all synthesis |
+| Streaming | **No** | Batch generation only |
+| Languages | Limited | English-focused |
+| ONNX Runtime | **Yes** | CPU inference via `ort` crate |
+| GPU Acceleration | Partial | ONNX supports CUDA EP |
+| Real-time Factor | Unknown | Not benchmarked |
+
+### 1.4 Integration Points
+
+The TTS module is currently:
+- Exposed as `pub mod tts` in `lib.rs`
+- Used in `main.rs` for testing
+- **Not integrated with the web server** (no `/api/v1/tts` endpoint)
+
+The audio processing infrastructure is shared with the Listen (STT) module:
+- `audio.rs` provides format conversion utilities
+- `symphonia` for decoding various audio formats
+- Resampling to target sample rates (16kHz for STT, 24kHz for TTS)
+
+---
+
+## 2. Qwen3-TTS-12Hz-0.6B-Base Analysis
+
+### 2.1 Model Overview
+
+**Source:** [Hugging Face - Qwen/Qwen3-TTS-12Hz-0.6B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base)
+
+| Specification | Value |
+|---------------|-------|
+| Parameters | 0.6B |
+| Release Date | January 22, 2026 |
+| Architecture | Dual-Track hybrid streaming LM |
+| Tokenizer | Qwen3-TTS-Tokenizer-12Hz |
+| Frame Rate | 12.5 Hz |
+| Output Sample Rate | 24 kHz |
+| Languages | 10 (Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian) |
+
+### 2.2 Key Features
+
+| Feature | Status | Details |
+|---------|--------|---------|
+| **Voice Cloning** | Yes | 3-second minimum reference audio |
+| **Streaming** | Yes | 97ms end-to-end latency |
+| **Real-time** | Yes | First audio packet after single character |
+| **Multilingual** | Yes | 10 languages supported |
+| **Instruction Control** | No | Base model limitation |
+
+### 2.3 Streaming Architecture
+
+The Dual-Track architecture enables:
+1. **Streaming text input** - Processes text incrementally
+2. **Streaming audio output** - Emits audio packets as generated
+3. **Multi-Token Prediction (MTP)** - Enables immediate speech decoding from first codec frame
+
+**Latency Benchmarks:**
+- First token latency: ~97ms (end-to-end)
+- Optimized TTFT: ~170ms on RTX 5090 (community fork)
+- Initial implementations: ~800ms TTFT (before optimization)
+
+### 2.4 Voice Cloning Requirements
+
+| Requirement | Specification |
+|-------------|---------------|
+| Reference Audio Length | **3 seconds minimum** |
+| Audio Format | WAV, MP3, or common formats |
+| Input Methods | File path, URL, base64, numpy array |
+| Reference Text | **Required** (transcript of reference audio) |
+| X-Vector Only Mode | Optional (speaker embedding only, lower quality) |
+
+### 2.5 Python API
+
+```python
+from qwen_tts import Qwen3TTSModel
+
+# Load model
+model = Qwen3TTSModel.from_pretrained(
+    "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
+    device_map="cuda:0",
+    dtype=torch.bfloat16,
+    attn_implementation="flash_attention_2",
+)
+
+# Voice cloning
+wavs, sr = model.generate_voice_clone(
+    text="Hello, this is a test.",
+    language="English",
+    ref_audio="reference.wav",
+    ref_text="Original speaker text from reference",
+)
+
+# Reusable prompt (efficient for multiple generations)
+prompt = model.create_voice_clone_prompt(
+    ref_audio="reference.wav",
+    ref_text="Reference transcript",
+)
+
+wavs, sr = model.generate_voice_clone(
+    text="New text",
+    language="English",
+    voice_clone_prompt=prompt,
+)
+```
+
+### 2.6 Dependencies
+
+```
+pip install -U qwen-tts
+pip install -U flash-attn --no-build-isolation  # Optional, recommended
+```
+
+**Requirements:**
+- Python 3.12 recommended
+- CUDA-capable GPU (for optimal performance)
+- FlashAttention 2 compatible hardware
+- PyTorch with bfloat16 support
+
+---
+
+## 3. Feasibility Assessment
+
+### 3.1 Streaming/Live TTS Feasibility
+
+**Assessment: FEASIBLE with caveats**
+
+| Factor | Current State | Path Forward |
+|--------|---------------|--------------|
+| Streaming API | Experimental (community fork) | Use [dffdeeq/Qwen3-TTS-streaming](https://github.com/dffdeeq/Qwen3-TTS-streaming) or wait for official support |
+| Latency Target | 97ms advertised | Achievable with optimization |
+| First Token | ~170ms optimized | Acceptable for conversational use |
+| Audio Stability | First 1-2s may have timbre issues | Known limitation, may need buffering |
+
+**Streaming Implementation Status:**
+- Official repository: Streaming documented but **not released**
+- Community fork: Working implementation with ~170ms TTFT
+- vLLM-Omni: Offline inference only (online serving planned)
+
+### 3.2 Voice Cloning for Makima
+
+**Assessment: FULLY FEASIBLE**
+
+Requirements for Makima voice cloning:
+1. **3+ seconds of clean audio** - Tomori Kusunoki (Japanese VA) speaking
+2. **Transcript of the audio** - Required for full quality
+3. **Audio format** - WAV/MP3 acceptable
+
+**Audio Sources:**
+- Chainsaw Man anime clips
+- Voice actress promotional material
+- Behind The Voice Actors database
+
+**Considerations:**
+- Japanese VA speaking English may work with explicit `language="English"` setting
+- May need English-speaking Makima clips (Suzie Yeung, English dub VA) as fallback
+- X-vector mode available if transcript is difficult to obtain
+
+### 3.3 Integration Complexity
+
+| Component | Complexity | Notes |
+|-----------|------------|-------|
+| Model Loading | Medium | Python subprocess or PyO3 bridge required |
+| Streaming API | High | WebSocket integration needed |
+| Voice Caching | Low | `create_voice_clone_prompt()` supports this |
+| Audio Format | Low | Both use 24kHz output |
+| ONNX Migration | N/A | **No ONNX export available** |
+
+### 3.4 ONNX vs Python Inference
+
+**Current approach (Chatterbox):** Rust + ONNX Runtime
+- Pros: Native Rust, low latency, CPU-friendly
+- Cons: Limited model ecosystem, no streaming
+
+**Required approach (Qwen3-TTS):** Python + PyTorch
+- Pros: Full model access, streaming support, GPU acceleration
+- Cons: Python subprocess overhead, dependency management
+
+**Integration Options:**
+
+1. **Python Subprocess/Service**
+   - Run `qwen-tts` as separate Python service
+   - Communicate via HTTP/WebSocket
+   - Cleanest separation, easiest to implement
+
+2. **PyO3 Bridge**
+   - Embed Python in Rust binary
+   - Higher complexity, tighter integration
+   - May have GIL contention issues
+
+3. **Custom ONNX Export** (Future)
+   - Not currently available
+   - Would require community effort
+   - No timeline from Qwen team
+
+**Recommendation:** Python service with WebSocket streaming
+
+---
+
+## 4. Audio Clip Requirements
+
+### 4.1 For Voice Cloning Setup
+
+| Requirement | Specification |
+|-------------|---------------|
+| Minimum Duration | 3 seconds |
+| Recommended Duration | 5-10 seconds |
+| Format | WAV (preferred), MP3 |
+| Sample Rate | Any (will be resampled) |
+| Content | Clear speech, minimal background noise |
+| Transcript | Required for full quality |
+
+### 4.2 Makima Voice Sources
+
+**Priority 1: Japanese VA (Tomori Kusunoki) speaking Japanese**
+- Source: Chainsaw Man anime
+- Challenge: Need clear dialogue without music/SFX
+- Fallback: May not transfer well to English output
+
+**Priority 2: English VA (Suzie Yeung)**
+- Source: Chainsaw Man English dub
+- Advantage: Native English speaker for English output
+- Trade-off: Different vocal characteristics from Japanese VA
+
+**Recommended Approach:**
+1. Extract 5-10 second clips of both VAs
+2. Test voice cloning quality with each
+3. Select based on English speech naturalness
+4. Store reference audio + transcript in `models/voices/makima/`
+
+### 4.3 Transcript Requirements
+
+For optimal voice cloning:
+```
+ref_audio: "models/voices/makima/makima-reference.wav"
+ref_text: "The exact words spoken in the reference audio"
+```
+
+X-vector fallback (lower quality, no transcript needed):
+```python
+prompt = model.create_voice_clone_prompt(
+    ref_audio="reference.wav",
+    x_vector_only_mode=True,  # No transcript required
+)
+```
+
+---
+
+## 5. Preliminary Technical Approach
+
+### 5.1 Architecture Overview
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    Makima Server (Rust)                     │
+├─────────────────────────────────────────────────────────────┤
+│  ┌─────────────┐  ┌─────────────┐  ┌──────────────────────┐│
+│  │ Listen (STT)│  │ TTS Proxy   │  │ Chat/Contract APIs   ││
+│  │ /api/v1/    │  │ /api/v1/tts │  │ /api/v1/...          ││
+│  │ listen      │  │             │  │                      ││
+│  └──────┬──────┘  └──────┬──────┘  └──────────────────────┘│
+│         │                │                                  │
+│         │         ┌──────▼──────┐                          │
+│         │         │  WebSocket  │                          │
+│         │         │   Bridge    │                          │
+│         │         └──────┬──────┘                          │
+└─────────┼────────────────┼──────────────────────────────────┘
+          │                │
+          │         ┌──────▼──────┐
+          │         │ Python TTS  │
+          │         │   Service   │
+          │         │ (Qwen3-TTS) │
+          │         └─────────────┘
+          │
+   ┌──────▼──────┐
+   │  ML Models  │
+   │ (Parakeet,  │
+   │ Sortformer) │
+   └─────────────┘
+```
+
+### 5.2 Python TTS Service
+
+**Proposed Architecture:**
+
+```python
+# tts_service.py
+import asyncio
+from fastapi import FastAPI, WebSocket
+from qwen_tts import Qwen3TTSModel
+
+app = FastAPI()
+model = None
+voice_prompts = {}
+
+@app.on_event("startup")
+async def load_model():
+    global model
+    model = Qwen3TTSModel.from_pretrained(
+        "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
+        device_map="cuda:0",
+        dtype=torch.bfloat16,
+        attn_implementation="flash_attention_2",
+    )
+
+    # Pre-load Makima voice prompt
+    voice_prompts["makima"] = model.create_voice_clone_prompt(
+        ref_audio="models/voices/makima/reference.wav",
+        ref_text="[Makima reference transcript]",
+    )
+
+@app.websocket("/ws/tts")
+async def tts_stream(websocket: WebSocket):
+    await websocket.accept()
+    while True:
+        data = await websocket.receive_json()
+        text = data["text"]
+        voice = data.get("voice", "makima")
+        language = data.get("language", "English")
+
+        # Generate with streaming (when available)
+        prompt = voice_prompts.get(voice)
+        wavs, sr = model.generate_voice_clone(
+            text=text,
+            language=language,
+            voice_clone_prompt=prompt,
+        )
+
+        # Send audio chunks
+        await websocket.send_bytes(wavs[0].tobytes())
+
+@app.post("/api/tts")
+async def tts_batch(request: TTSRequest):
+    # Batch fallback for non-streaming use cases
+    ...
+```
+
+### 5.3 Rust Integration
+
+**New Endpoint: `/api/v1/tts`**
+
+```rust
+// server/handlers/tts.rs
+pub async fn tts_websocket_handler(
+    ws: WebSocketUpgrade,
+    State(state): State<SharedState>,
+) -> Response {
+    ws.on_upgrade(|socket| handle_tts_socket(socket, state))
+}
+
+async fn handle_tts_socket(socket: WebSocket, state: SharedState) {
+    // Proxy WebSocket to Python TTS service
+    let tts_client = state.tts_client.lock().await;
+
+    let (mut sender, mut receiver) = socket.split();
+
+    while let Some(msg) = receiver.next().await {
+        match msg {
+            Ok(Message::Text(text)) => {
+                // Forward to Python service
+                let response = tts_client.generate(text).await;
+
+                // Stream audio back
+                for chunk in response.audio_chunks {
+                    sender.send(Message::Binary(chunk)).await.ok();
+                }
+            }
+            _ => {}
+        }
+    }
+}
+```
+
+### 5.4 Voice Prompt Caching
+
+```rust
+// Pre-computed voice prompts stored in state
+pub struct TtsConfig {
+    pub default_voice: String,
+    pub voices: HashMap<String, VoicePrompt>,
+}
+
+pub struct VoicePrompt {
+    pub name: String,
+    pub ref_audio_path: PathBuf,
+    pub ref_text: String,
+    pub language: String,
+    // Cached prompt from Python service
+    pub cached_prompt_id: Option<String>,
+}
+```
+
+### 5.5 Message Protocol
+
+**Client -> Server:**
+```json
+{
+  "type": "synthesize",
+  "text": "Hello, I am Makima.",
+  "voice": "makima",
+  "language": "English",
+  "stream": true
+}
+```
+
+**Server -> Client:**
+```json
+// Audio chunk
+{"type": "audio", "data": "<base64 PCM>", "sample_rate": 24000, "final": false}
+
+// Completion
+{"type": "complete", "duration_ms": 1234}
+
+// Error
+{"type": "error", "code": "SYNTHESIS_ERROR", "message": "..."}
+```
+
+---
+
+## 6. Implementation Phases
+
+### Phase 1: Research & Setup (Current)
+- [x] Analyze current TTS implementation
+- [x] Research Qwen3-TTS capabilities
+- [x] Document feasibility and approach
+- [ ] Acquire Makima voice reference clips
+- [ ] Test voice cloning quality
+
+### Phase 2: Python Service
+- [ ] Create Python TTS service with FastAPI
+- [ ] Implement batch TTS endpoint
+- [ ] Implement WebSocket streaming (when available)
+- [ ] Add voice prompt management
+- [ ] GPU optimization with FlashAttention 2
+
+### Phase 3: Rust Integration
+- [ ] Add TTS proxy endpoints to makima server
+- [ ] WebSocket bridge implementation
+- [ ] Voice configuration management
+- [ ] Error handling and fallbacks
+
+### Phase 4: Production Ready
+- [ ] Health checks for Python service
+- [ ] Voice prompt caching optimization
+- [ ] Latency benchmarking
+- [ ] Integration with Listen page
+
+---
+
+## 7. Open Questions
+
+1. **Streaming API Availability**: When will official streaming support be released?
+   - Fallback: Use community fork or batch with chunked responses
+
+2. **Voice Quality**: How well does Japanese VA voice clone to English speech?
+   - Action: Test with both Japanese and English VA samples
+
+3. **GPU Requirements**: What's the minimum VRAM for 0.6B model?
+   - Estimate: ~2-4GB with bf16 quantization
+
+4. **Latency Target**: What's acceptable for "close to live" TTS?
+   - Proposal: <500ms first audio, <100ms subsequent chunks
+
+5. **Transcript Acquisition**: How to obtain accurate transcripts for voice clips?
+   - Options: Manual transcription, Whisper ASR, community resources
+
+---
+
+## 8. References
+
+- [Qwen3-TTS-12Hz-0.6B-Base (Hugging Face)](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base)
+- [Qwen3-TTS GitHub Repository](https://github.com/QwenLM/Qwen3-TTS)
+- [Qwen3-TTS Technical Report (arXiv)](https://arxiv.org/abs/2601.15621)
+- [Streaming Inference Issue #77](https://github.com/QwenLM/Qwen3-TTS/issues/77)
+- [Community Streaming Fork](https://github.com/dffdeeq/Qwen3-TTS-streaming)
+- [Makima Voice Actors](https://www.behindthevoiceactors.com/characters/Chainsaw-Man/Makima/)
+- [Chatterbox-Turbo-ONNX (Current Model)](https://huggingface.co/ResembleAI/chatterbox-turbo-ONNX)