Add Qwen3-TTS streaming endpoint for voice synthesis (#40)

* Task completion checkpoint

* Task completion checkpoint

* Task completion checkpoint

* Add Qwen3-TTS research document for live TTS replacement

Research findings for replacing Chatterbox TTS with Qwen3-TTS-12Hz-0.6B-Base:

- Current TTS: Chatterbox-Turbo-ONNX with batch-only generation, no streaming
- Qwen3-TTS: 97ms end-to-end latency, streaming support, 3-second voice cloning
- Voice cloning: Requires 3s reference audio + transcript (Makima voice planned)
- Integration: Python service with WebSocket bridge (no ONNX export available)
- Languages: 10 supported including English and Japanese

Document includes:
- Current architecture analysis (makima/src/tts.rs)
- Qwen3-TTS capabilities and requirements
- Feasibility assessment for live/streaming TTS
- Audio clip requirements for voice cloning
- Preliminary technical approach with architecture diagrams

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 03:11:15 UTC

* Add Qwen3-TTS research documentation

Comprehensive research on replacing Chatterbox TTS with Qwen3-TTS-12Hz-0.6B-Base:

- Current TTS implementation analysis (Chatterbox-Turbo-ONNX in makima/src/tts.rs)
- Qwen3-TTS capabilities: 97ms streaming latency, voice cloning with 3s reference
- Cross-lingual support: Japanese voice (Makima/Tomori Kusunoki) speaking English
- Python microservice architecture recommendation (FastAPI + WebSocket)
- Implementation phases and technical approach
- Hardware requirements and dependencies

Key findings:
- Live/streaming TTS is highly feasible with 97ms latency
- Voice cloning fully supported with 0.95 speaker similarity
- Recommended: Python microservice with WebSocket streaming

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add comprehensive Qwen3-TTS integration specification

This specification document defines the complete integration of
Qwen3-TTS-12Hz-0.6B-Base as a replacement for the existing Chatterbox-Turbo
TTS implementation. The document covers:

## Functional Requirements
- WebSocket endpoint /api/v1/speak for streaming TTS
- Voice cloning with default Makima voice (Japanese VA speaking English)
- Support for custom voice references
- Detailed client-to-server and server-to-client message protocols
- Integration with Listen page for bidirectional speech

## Non-Functional Requirements
- Latency targets: < 200ms first audio byte
- Audio quality: 24kHz, mono, PCM16/PCM32f
- Hardware requirements: CUDA GPU with 4-8GB VRAM
- Scalability: 10 concurrent sessions per GPU

## Architecture Specification
- Python TTS microservice with FastAPI/WebSocket
- Rust proxy endpoint in makima server
- Voice prompt caching mechanism (LRU cache)
- Error handling and recovery strategies

## API Contract
- Complete WebSocket message format definitions (TypeScript)
- Error codes and responses (TTS_UNAVAILABLE, SYNTHESIS_ERROR, etc.)
- Session state machine and lifecycle management

## Voice Asset Requirements
- Makima voice clip specifications (5-10s WAV, transcript required)
- Storage location: models/voices/makima/
- Metadata format for voice management

## Testing Strategy
- Unit tests for Python TTS service and Rust proxy
- Integration tests for WebSocket flow
- Latency benchmarks with performance targets
- Test data fixtures for various text lengths

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add Qwen3-TTS implementation plan

Comprehensive implementation plan for replacing Chatterbox-TTS with
Qwen3-TTS streaming TTS service, including:

- Task breakdown with estimated hours for each phase
- Phase 1: Python TTS microservice (FastAPI, WebSocket)
- Phase 2: Rust proxy integration (speak.rs, tts_client.rs)
- Detailed file changes and new module structure
- Testing plan with unit, integration, and latency benchmarks
- Risk assessment with mitigation strategies
- Success criteria for each phase

Based on specification in docs/specs/qwen3-tts-spec.md

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add author and research references to TTS implementation plan

Add links to research documentation and author attribution.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 03:25:06 UTC

* Add Python TTS service project structure (Phase 1.1-1.3)

Create the initial makima-tts Python service directory structure with:
- pyproject.toml with FastAPI, Qwen-TTS, and torch dependencies
- config.py with pydantic-settings TTSConfig class
- models.py with Pydantic message models (Start, Speak, Stop, Ready, etc.)

This implements tasks P1.1, P1.2, and P1.3 from the Qwen3-TTS implementation plan.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add TTS engine and voice manager for Qwen3-TTS (Phase 1.4-1.5)

Implement core TTS functionality:
- tts_engine.py: Qwen3-TTS wrapper with streaming audio chunk generation
- voice_manager.py: Voice prompt caching with LRU eviction and TTL support

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 03:30:06 UTC

* Add TTS proxy client and message types (Phase 2.1, 2.2, 2.4)

- Add tts_client.rs with TtsConfig, TtsCircuitBreaker, TtsError,
  TtsProxyClient, and TtsConnection structs for WebSocket proxying
- Add TTS message types to messages.rs (TtsAudioEncoding, TtsPriority,
  TtsStartMessage, TtsSpeakMessage, TtsStopMessage, TtsClientMessage,
  TtsReadyMessage, TtsAudioChunkMessage, TtsCompleteMessage,
  TtsErrorMessage, TtsStoppedMessage, TtsServerMessage)
- Export tts_client module from server mod.rs
- tokio-tungstenite already present in Cargo.toml

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add TTS WebSocket handler and route (Phase 2.3, 2.5, 2.6)

- Create speak.rs WebSocket handler that proxies to Python TTS service
- Add TtsState fields (tts_client, tts_config) to AppState
- Add with_tts() builder and is_tts_healthy() methods to AppState
- Register /api/v1/speak route in the router
- Add speak module export in handlers/mod.rs

The handler forwards WebSocket messages bidirectionally between
the client and the Python TTS microservice with proper error handling.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add Makima voice profile assets for TTS voice cloning

Creates the voice assets directory structure with:
- manifest.json containing voice configuration (voice_id, speaker,
  language, reference audio path, and Japanese transcript placeholder)
- README.md with instructions for obtaining voice reference audio

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add Rust-native Qwen3-TTS integration research document

Research findings for integrating Qwen3-TTS-12Hz-0.6B-Base directly into
the makima Rust codebase without Python. Key conclusions:
- ONNX export is not viable (unsupported architecture)
- Candle (HF Rust ML framework) is the recommended approach
- Model weights available in safetensors format (2.52GB total)
- Three components needed: LM backbone, code predictor, speech tokenizer
- Crane project has Qwen3-TTS as highest priority (potential upstream)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 11:21:43 UTC

* [WIP] Heartbeat checkpoint - 2026-01-27 11:24:19 UTC

* [WIP] Heartbeat checkpoint - 2026-01-27 11:26:43 UTC

* feat: implement Rust-native Qwen3-TTS using candle framework

Replace monolithic tts.rs with modular tts/ directory structure:

- tts/mod.rs: TtsEngine trait, TtsEngineFactory, shared types (AudioChunk,
  TtsError), and utility functions (save_wav, resample, argmax)
- tts/chatterbox.rs: existing ONNX-based ChatterboxTTS adapted to implement
  TtsEngine trait with Mutex-wrapped sessions for Send+Sync
- tts/qwen3/mod.rs: Qwen3Tts entry point with HuggingFace model loading
- tts/qwen3/config.rs: Qwen3TtsConfig parsing from HF config.json
- tts/qwen3/model.rs: 28-layer Qwen3 transformer with RoPE, GQA (16 heads,
  8 KV heads), SiLU MLP, RMS norm, and KV cache
- tts/qwen3/code_predictor.rs: 5-layer MTP module predicting 16 codebooks
- tts/qwen3/speech_tokenizer.rs: ConvNet encoder/decoder with 16-layer RVQ
- tts/qwen3/generate.rs: autoregressive generation loop with streaming support

Add candle-core, candle-nn, candle-transformers, safetensors to Cargo.toml.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: integrate TTS engine into speak WebSocket handler

- Update speak.rs handler to use TTS engine directly from SharedState
  instead of returning a stub "not implemented" error
- Add TtsEngine (OnceCell lazy-loaded) to AppState in state.rs with
  get_tts_engine() method for lazy initialization on first connection
- Implement full WebSocket protocol: client sends JSON speak/cancel/stop
  messages, server streams binary PCM audio chunks and audio_end signals
- Create voices/makima/manifest.json for Makima voice profile configuration
- All files compile successfully with zero errors

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add /speak TTS page with WebSocket audio playback

Add a new /speak frontend page for text-to-speech via WebSocket.
The page accepts text input and streams synthesized PCM audio through
the Web Audio API. Includes model loading indicator, cancel support,
and connection status. Also adds a loading bar to the listen page
ControlPanel during WebSocket connection.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

1 files changed, 351 insertions, 0 deletions

diff --git a/TTS_RESEARCH.md b/TTS_RESEARCH.md
new file mode 100644
index 0000000..da7b8b8
--- /dev/null
+++ b/TTS_RESEARCH.md
@@ -0,0 +1,351 @@
+# TTS Research Document: Qwen3-TTS Integration for Makima
+
+## Executive Summary
+
+This document summarizes research on replacing the existing ChatterboxTTS implementation with Qwen3-TTS for live/streaming TTS with Makima's Japanese voice speaking English.
+
+---
+
+## 1. Current TTS Implementation Analysis
+
+### 1.1 Architecture Overview
+
+The current makima codebase uses **ChatterboxTTS** (ResembleAI/chatterbox-turbo-ONNX) with the following components:
+
+| Component | File | Purpose |
+|-----------|------|---------|
+| TTS Module | `makima/src/tts.rs` | Core TTS inference using ONNX Runtime |
+| Audio Processing | `makima/src/audio.rs` | Audio decoding, resampling (Symphonia) |
+| Library Export | `makima/src/lib.rs` | Exposes `pub mod tts` |
+
+### 1.2 ChatterboxTTS Technical Details
+
+```rust
+// Key constants from tts.rs
+pub const SAMPLE_RATE: u32 = 24_000;
+const MODEL_ID: &str = "ResembleAI/chatterbox-turbo-ONNX";
+const DEFAULT_MODEL_DIR: &str = "models/chatterbox-turbo";
+```
+
+**ONNX Model Files:**
+- `speech_encoder.onnx` - Encodes reference voice audio
+- `embed_tokens.onnx` - Text token embedding
+- `language_model.onnx` - Autoregressive token generation (24 layers, 16 KV heads)
+- `conditional_decoder.onnx` - Decodes speech tokens to waveform
+- `tokenizer.json` - Text tokenization
+
+### 1.3 Current API Surface
+
+```rust
+pub struct ChatterboxTTS {
+    pub fn from_pretrained(model_dir: Option<&str>) -> Result<Self, TtsError>
+    pub fn generate_tts(&mut self, text: &str) -> Result<Vec<f32>, TtsError>  // Returns VoiceRequired error
+    pub fn generate_tts_with_voice(text: &str, sample_audio_path: &Path) -> Result<Vec<f32>, TtsError>
+    pub fn generate_tts_with_samples(text: &str, samples: &[f32], sample_rate: u32) -> Result<Vec<f32>, TtsError>
+}
+pub fn save_wav(samples: &[f32], path: &Path) -> Result<(), TtsError>
+```
+
+### 1.4 Voice Cloning Capabilities (Current)
+
+- **Requires** voice reference audio (returns `VoiceRequired` error without it)
+- Accepts reference audio via file path or raw samples
+- Resamples reference to 24kHz internally
+- Uses speaker embeddings + speaker features for voice cloning
+
+### 1.5 Streaming/Live TTS (Current)
+
+**NOT SUPPORTED** - The current implementation:
+- Generates entire audio in one pass
+- Uses autoregressive token generation with max 1024 tokens
+- No chunked/streaming output capability
+- Full pipeline must complete before audio is available
+
+### 1.6 Server Integration Status
+
+The TTS module is **not currently exposed via HTTP endpoints**. The server (`makima/src/server/mod.rs`) has:
+- `/api/v1/listen` - WebSocket for Speech-to-Text (STT) only
+- No TTS endpoints exist
+
+---
+
+## 2. Qwen3-TTS Model Analysis
+
+### 2.1 Model Specifications
+
+| Attribute | Value |
+|-----------|-------|
+| **Model** | Qwen/Qwen3-TTS-12Hz-0.6B-Base |
+| **Parameters** | 0.6B (also available: 1.7B version) |
+| **Architecture** | Discrete multi-codebook LM (16 codebooks, 2048 size) |
+| **Tokenizer** | Qwen3-TTS-Tokenizer-12Hz |
+| **Sample Rate** | 12 Hz tokenizer (reconstructs to standard rates) |
+| **License** | Apache 2.0 |
+
+### 2.2 Key Capabilities
+
+#### Voice Cloning
+- **3-second rapid voice clone** - Minimal reference audio needed
+- **Flexible input formats**: local files, URLs, base64, (numpy_array, sample_rate) tuples
+- **Reusable voice prompts**: Create once, use for multiple generations
+
+```python
+# Voice cloning example
+model = Qwen3TTSModel.from_pretrained("Qwen/Qwen3-TTS-12Hz-0.6B-Base")
+wavs, sr = model.generate_voice_clone(
+    text="Target text",
+    language="English",
+    ref_audio="reference.wav",
+    ref_text="Reference transcript"
+)
+```
+
+#### Streaming/Live TTS
+- **97ms end-to-end latency** - Ultra-low latency streaming
+- **Dual-track hybrid streaming architecture** - Supports both streaming and non-streaming
+- **First packet after single character** - Immediate response capability
+
+#### Multilingual Support
+10 languages: Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian
+
+#### Quality Metrics
+| Metric | Value |
+|--------|-------|
+| WER (English) | 1.32 |
+| Speaker Similarity (English) | 0.829 |
+| PESQ_WB | 3.21 |
+| STOI | 0.96 |
+| UTMOS | 4.16 |
+
+### 2.3 Requirements
+
+```bash
+# Environment
+Python 3.12 (recommended)
+CUDA-compatible GPU with 8GB+ VRAM
+
+# Installation
+pip install -U qwen-tts
+pip install -U flash-attn --no-build-isolation  # Optional, reduces GPU memory
+```
+
+### 2.4 Current Deployment Options
+
+| Option | Status | Notes |
+|--------|--------|-------|
+| **Python (qwen-tts)** | Stable | Official package |
+| **vLLM-Omni** | Offline only | Online serving coming |
+| **ONNX Export** | Not available | No official support |
+| **Rust Implementation** | Draft PR #8 | Early development |
+| **DashScope API** | Available | Alibaba Cloud hosted |
+
+---
+
+## 3. Makima Voice Audio Clips
+
+### 3.1 Voice Actress Information
+
+| Attribute | Value |
+|-----------|-------|
+| **Character** | Makima (Chainsaw Man) |
+| **Japanese VA** | Tomori Kusunoki (楠木ともり) |
+| **English VA** | Suzie Yeung |
+| **Agency** | Sony Music Artists |
+
+### 3.2 Audio Clip Sources
+
+1. **Chainsaw Man Anime Episodes** - Primary source for Japanese voice
+2. **Behind The Voice Actors** - Character voice samples
+3. **YouTube Clips** - Interview compilations, scene clips
+4. **Official Media** - Promotional videos, trailers
+
+### 3.3 Audio Requirements for Voice Cloning
+
+#### Qwen3-TTS Requirements
+| Parameter | Requirement |
+|-----------|-------------|
+| **Minimum Duration** | 3 seconds (basic quality) |
+| **Recommended Duration** | 10-30 seconds (professional quality) |
+| **Format** | WAV, FLAC, MP3, OGG, AIFF, AAC |
+| **Sample Rate** | 24kHz or above recommended |
+| **Channels** | Mono preferred |
+
+#### Best Practices
+- **Clean audio**: No background music/noise
+- **Single speaker**: Makima's voice only
+- **Consistent tone**: Avoid dramatic variations
+- **Include transcript**: Reference text improves quality
+- **Varied content**: Mix of sentence types for flexibility
+
+#### Recommended Clip Types
+1. Calm, composed dialogue (Makima's signature tone)
+2. Commands/instructions (authoritative delivery)
+3. Questions (natural intonation)
+4. Longer monologues (for voice consistency)
+
+---
+
+## 4. Feasibility Assessment for Live/Streaming TTS
+
+### 4.1 Technical Challenges
+
+| Challenge | Severity | Notes |
+|-----------|----------|-------|
+| No ONNX export | **High** | Current codebase uses ONNX Runtime |
+| Rust implementation | **High** | Only draft PR available |
+| Python dependency | Medium | Would require sidecar service |
+| GPU memory | Medium | 8GB+ VRAM required |
+| Streaming API | Low | Supported in Qwen3-TTS |
+
+### 4.2 Integration Approaches
+
+#### Option A: Python Sidecar Service (Recommended)
+**Architecture**: Rust server + Python TTS service via HTTP/gRPC
+
+**Pros:**
+- Uses official Qwen3-TTS Python package
+- Full streaming support (97ms latency)
+- Simpler maintenance
+
+**Cons:**
+- Additional deployment complexity
+- Inter-process communication overhead
+
+```
+┌─────────────────┐     HTTP/gRPC     ┌─────────────────┐
+│  Makima Server  │ ◄──────────────► │  Qwen3-TTS      │
+│  (Rust/Axum)    │                   │  (Python/FastAPI)│
+└─────────────────┘                   └─────────────────┘
+```
+
+**Available Implementations:**
+- [ValyrianTech/Qwen3-TTS_server](https://github.com/ValyrianTech/Qwen3-TTS_server) - FastAPI server
+- [Qwen3-TTS-Openai-Fastapi](https://github.com/twolven/Qwen3-TTS-Openai-Fastapi) - OpenAI-compatible API
+
+#### Option B: Wait for Rust Implementation
+**Status**: Draft PR #8 in early development
+
+**Pros:**
+- Native Rust integration
+- No Python dependency
+- Matches current architecture
+
+**Cons:**
+- Unknown timeline
+- May require significant adaptation
+
+#### Option C: Hybrid (ChatterboxTTS + Qwen3-TTS)
+Keep ChatterboxTTS for ONNX compatibility, add Qwen3-TTS for streaming
+
+**Pros:**
+- Gradual migration
+- Fallback capability
+
+**Cons:**
+- Dual model maintenance
+- Increased complexity
+
+### 4.3 Recommendation
+
+**Short-term (1-2 weeks)**: Implement **Option A** with Python sidecar
+- Deploy ValyrianTech/Qwen3-TTS_server or similar
+- Add HTTP client in Rust to call TTS service
+- Implement WebSocket endpoint for streaming audio
+
+**Long-term (3-6 months)**: Monitor Rust implementation progress
+- Evaluate draft PR #8 stability
+- Consider contributing to Rust port
+- Migrate to native Rust when mature
+
+---
+
+## 5. Preliminary Technical Approach
+
+### 5.1 Phase 1: Voice Preparation
+
+1. **Collect Makima Audio Clips**
+   - Extract 3-5 clean clips from anime (10-30 seconds each)
+   - Ensure Japanese voice, clear audio, no BGM
+   - Prepare transcripts for each clip
+
+2. **Test Voice Cloning Quality**
+   - Use Qwen3-TTS demo to validate clips
+   - Iterate on clip selection for best results
+
+### 5.2 Phase 2: TTS Service Setup
+
+1. **Deploy Qwen3-TTS Server**
+   ```bash
+   # Using ValyrianTech server
+   docker run --gpus all -p 7860:7860 qwen3-tts-server
+   ```
+
+2. **Configure Voice Clone Profile**
+   - Upload Makima reference audio
+   - Store voice clone prompt for reuse
+
+### 5.3 Phase 3: Makima Integration
+
+1. **Add TTS Client Module**
+   ```rust
+   // New module: makima/src/tts_client.rs
+   pub struct QwenTTSClient {
+       base_url: String,
+       voice_profile: String,
+   }
+
+   impl QwenTTSClient {
+       pub async fn generate_speech(&self, text: &str) -> Result<Vec<u8>, Error>
+       pub async fn generate_speech_streaming(&self, text: &str) -> impl Stream<Item = Vec<u8>>
+   }
+   ```
+
+2. **Add TTS Endpoint**
+   ```rust
+   // In makima/src/server/mod.rs
+   .route("/api/v1/tts", post(tts_handler))
+   .route("/api/v1/tts/stream", get(tts_streaming_handler))
+   ```
+
+3. **WebSocket Integration for Listen Page**
+   - Bidirectional audio: STT input, TTS output
+   - Low-latency streaming for conversational flow
+
+### 5.4 Phase 4: Listen Page Integration
+
+1. **Update Frontend**
+   - Add TTS playback capability
+   - Handle streaming audio chunks
+   - UI for voice response indicators
+
+2. **Orchestration Logic**
+   - STT → LLM → TTS pipeline
+   - Interrupt handling for user speech
+
+---
+
+## 6. Open Questions
+
+1. **Voice Rights**: Are there legal considerations for cloning Tomori Kusunoki's voice?
+2. **GPU Allocation**: Shared GPU for STT + TTS, or separate?
+3. **Latency Budget**: What's acceptable end-to-end latency for Listen page?
+4. **Fallback Strategy**: What happens if TTS service is unavailable?
+5. **Multi-user**: How to handle concurrent TTS requests?
+
+---
+
+## 7. References
+
+- [Qwen3-TTS HuggingFace](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base)
+- [Qwen3-TTS GitHub](https://github.com/QwenLM/Qwen3-TTS)
+- [Qwen3-TTS Technical Report](https://arxiv.org/abs/2601.15621)
+- [ValyrianTech Qwen3-TTS Server](https://github.com/ValyrianTech/Qwen3-TTS_server)
+- [Qwen3-TTS OpenAI-Compatible FastAPI](https://github.com/twolven/Qwen3-TTS-Openai-Fastapi)
+- [Makima Voice Actors](https://www.behindthevoiceactors.com/tv-shows/Chainsaw-Man/Makima/)
+- [ChatterboxTTS Audio Guidelines](https://github.com/resemble-ai/chatterbox/issues/39)
+- [Voice Cloning Best Practices - Resemble AI](https://www.resemble.ai/script-to-read-for-voice-cloning-guidelines/)
+- [Qwen3-rs (Rust LLM implementation)](https://github.com/reinterpretcat/qwen3-rs)
+
+---
+
+*Document created: Research phase for Makima TTS replacement contract*