From eabd1304cce0e053cd32ec910d2f0ea429e8af14 Mon Sep 17 00:00:00 2001 From: soryu Date: Wed, 28 Jan 2026 02:54:17 +0000 Subject: Add Qwen3-TTS streaming endpoint for voice synthesis (#40) * Task completion checkpoint * Task completion checkpoint * Task completion checkpoint * Add Qwen3-TTS research document for live TTS replacement Research findings for replacing Chatterbox TTS with Qwen3-TTS-12Hz-0.6B-Base: - Current TTS: Chatterbox-Turbo-ONNX with batch-only generation, no streaming - Qwen3-TTS: 97ms end-to-end latency, streaming support, 3-second voice cloning - Voice cloning: Requires 3s reference audio + transcript (Makima voice planned) - Integration: Python service with WebSocket bridge (no ONNX export available) - Languages: 10 supported including English and Japanese Document includes: - Current architecture analysis (makima/src/tts.rs) - Qwen3-TTS capabilities and requirements - Feasibility assessment for live/streaming TTS - Audio clip requirements for voice cloning - Preliminary technical approach with architecture diagrams Co-Authored-By: Claude Opus 4.5 * [WIP] Heartbeat checkpoint - 2026-01-27 03:11:15 UTC * Add Qwen3-TTS research documentation Comprehensive research on replacing Chatterbox TTS with Qwen3-TTS-12Hz-0.6B-Base: - Current TTS implementation analysis (Chatterbox-Turbo-ONNX in makima/src/tts.rs) - Qwen3-TTS capabilities: 97ms streaming latency, voice cloning with 3s reference - Cross-lingual support: Japanese voice (Makima/Tomori Kusunoki) speaking English - Python microservice architecture recommendation (FastAPI + WebSocket) - Implementation phases and technical approach - Hardware requirements and dependencies Key findings: - Live/streaming TTS is highly feasible with 97ms latency - Voice cloning fully supported with 0.95 speaker similarity - Recommended: Python microservice with WebSocket streaming Co-Authored-By: Claude Opus 4.5 * Add comprehensive Qwen3-TTS integration specification This specification document defines the complete integration of Qwen3-TTS-12Hz-0.6B-Base as a replacement for the existing Chatterbox-Turbo TTS implementation. The document covers: ## Functional Requirements - WebSocket endpoint /api/v1/speak for streaming TTS - Voice cloning with default Makima voice (Japanese VA speaking English) - Support for custom voice references - Detailed client-to-server and server-to-client message protocols - Integration with Listen page for bidirectional speech ## Non-Functional Requirements - Latency targets: < 200ms first audio byte - Audio quality: 24kHz, mono, PCM16/PCM32f - Hardware requirements: CUDA GPU with 4-8GB VRAM - Scalability: 10 concurrent sessions per GPU ## Architecture Specification - Python TTS microservice with FastAPI/WebSocket - Rust proxy endpoint in makima server - Voice prompt caching mechanism (LRU cache) - Error handling and recovery strategies ## API Contract - Complete WebSocket message format definitions (TypeScript) - Error codes and responses (TTS_UNAVAILABLE, SYNTHESIS_ERROR, etc.) - Session state machine and lifecycle management ## Voice Asset Requirements - Makima voice clip specifications (5-10s WAV, transcript required) - Storage location: models/voices/makima/ - Metadata format for voice management ## Testing Strategy - Unit tests for Python TTS service and Rust proxy - Integration tests for WebSocket flow - Latency benchmarks with performance targets - Test data fixtures for various text lengths Co-Authored-By: Claude Opus 4.5 * Add Qwen3-TTS implementation plan Comprehensive implementation plan for replacing Chatterbox-TTS with Qwen3-TTS streaming TTS service, including: - Task breakdown with estimated hours for each phase - Phase 1: Python TTS microservice (FastAPI, WebSocket) - Phase 2: Rust proxy integration (speak.rs, tts_client.rs) - Detailed file changes and new module structure - Testing plan with unit, integration, and latency benchmarks - Risk assessment with mitigation strategies - Success criteria for each phase Based on specification in docs/specs/qwen3-tts-spec.md Co-Authored-By: Claude Opus 4.5 * Add author and research references to TTS implementation plan Add links to research documentation and author attribution. Co-Authored-By: Claude Opus 4.5 * [WIP] Heartbeat checkpoint - 2026-01-27 03:25:06 UTC * Add Python TTS service project structure (Phase 1.1-1.3) Create the initial makima-tts Python service directory structure with: - pyproject.toml with FastAPI, Qwen-TTS, and torch dependencies - config.py with pydantic-settings TTSConfig class - models.py with Pydantic message models (Start, Speak, Stop, Ready, etc.) This implements tasks P1.1, P1.2, and P1.3 from the Qwen3-TTS implementation plan. Co-Authored-By: Claude Opus 4.5 * Add TTS engine and voice manager for Qwen3-TTS (Phase 1.4-1.5) Implement core TTS functionality: - tts_engine.py: Qwen3-TTS wrapper with streaming audio chunk generation - voice_manager.py: Voice prompt caching with LRU eviction and TTL support Co-Authored-By: Claude Opus 4.5 * [WIP] Heartbeat checkpoint - 2026-01-27 03:30:06 UTC * Add TTS proxy client and message types (Phase 2.1, 2.2, 2.4) - Add tts_client.rs with TtsConfig, TtsCircuitBreaker, TtsError, TtsProxyClient, and TtsConnection structs for WebSocket proxying - Add TTS message types to messages.rs (TtsAudioEncoding, TtsPriority, TtsStartMessage, TtsSpeakMessage, TtsStopMessage, TtsClientMessage, TtsReadyMessage, TtsAudioChunkMessage, TtsCompleteMessage, TtsErrorMessage, TtsStoppedMessage, TtsServerMessage) - Export tts_client module from server mod.rs - tokio-tungstenite already present in Cargo.toml Co-Authored-By: Claude Opus 4.5 * Add TTS WebSocket handler and route (Phase 2.3, 2.5, 2.6) - Create speak.rs WebSocket handler that proxies to Python TTS service - Add TtsState fields (tts_client, tts_config) to AppState - Add with_tts() builder and is_tts_healthy() methods to AppState - Register /api/v1/speak route in the router - Add speak module export in handlers/mod.rs The handler forwards WebSocket messages bidirectionally between the client and the Python TTS microservice with proper error handling. Co-Authored-By: Claude Opus 4.5 * Add Makima voice profile assets for TTS voice cloning Creates the voice assets directory structure with: - manifest.json containing voice configuration (voice_id, speaker, language, reference audio path, and Japanese transcript placeholder) - README.md with instructions for obtaining voice reference audio Co-Authored-By: Claude Opus 4.5 * Add Rust-native Qwen3-TTS integration research document Research findings for integrating Qwen3-TTS-12Hz-0.6B-Base directly into the makima Rust codebase without Python. Key conclusions: - ONNX export is not viable (unsupported architecture) - Candle (HF Rust ML framework) is the recommended approach - Model weights available in safetensors format (2.52GB total) - Three components needed: LM backbone, code predictor, speech tokenizer - Crane project has Qwen3-TTS as highest priority (potential upstream) Co-Authored-By: Claude Opus 4.5 * [WIP] Heartbeat checkpoint - 2026-01-27 11:21:43 UTC * [WIP] Heartbeat checkpoint - 2026-01-27 11:24:19 UTC * [WIP] Heartbeat checkpoint - 2026-01-27 11:26:43 UTC * feat: implement Rust-native Qwen3-TTS using candle framework Replace monolithic tts.rs with modular tts/ directory structure: - tts/mod.rs: TtsEngine trait, TtsEngineFactory, shared types (AudioChunk, TtsError), and utility functions (save_wav, resample, argmax) - tts/chatterbox.rs: existing ONNX-based ChatterboxTTS adapted to implement TtsEngine trait with Mutex-wrapped sessions for Send+Sync - tts/qwen3/mod.rs: Qwen3Tts entry point with HuggingFace model loading - tts/qwen3/config.rs: Qwen3TtsConfig parsing from HF config.json - tts/qwen3/model.rs: 28-layer Qwen3 transformer with RoPE, GQA (16 heads, 8 KV heads), SiLU MLP, RMS norm, and KV cache - tts/qwen3/code_predictor.rs: 5-layer MTP module predicting 16 codebooks - tts/qwen3/speech_tokenizer.rs: ConvNet encoder/decoder with 16-layer RVQ - tts/qwen3/generate.rs: autoregressive generation loop with streaming support Add candle-core, candle-nn, candle-transformers, safetensors to Cargo.toml. Co-Authored-By: Claude Opus 4.5 * feat: integrate TTS engine into speak WebSocket handler - Update speak.rs handler to use TTS engine directly from SharedState instead of returning a stub "not implemented" error - Add TtsEngine (OnceCell lazy-loaded) to AppState in state.rs with get_tts_engine() method for lazy initialization on first connection - Implement full WebSocket protocol: client sends JSON speak/cancel/stop messages, server streams binary PCM audio chunks and audio_end signals - Create voices/makima/manifest.json for Makima voice profile configuration - All files compile successfully with zero errors Co-Authored-By: Claude Opus 4.5 * feat: add /speak TTS page with WebSocket audio playback Add a new /speak frontend page for text-to-speech via WebSocket. The page accepts text input and streams synthesized PCM audio through the Web Audio API. Includes model loading indicator, cancel support, and connection status. Also adds a loading bar to the listen page ControlPanel during WebSocket connection. Co-Authored-By: Claude Opus 4.5 --------- Co-authored-by: Claude Opus 4.5 --- docs/research/TTS_RESEARCH_NOTES.md | 405 ++++++++++++++++++++++++++++++++++++ 1 file changed, 405 insertions(+) create mode 100644 docs/research/TTS_RESEARCH_NOTES.md (limited to 'docs/research/TTS_RESEARCH_NOTES.md') diff --git a/docs/research/TTS_RESEARCH_NOTES.md b/docs/research/TTS_RESEARCH_NOTES.md new file mode 100644 index 0000000..72ac8c6 --- /dev/null +++ b/docs/research/TTS_RESEARCH_NOTES.md @@ -0,0 +1,405 @@ +# TTS Replacement Research Notes + +## Executive Summary + +This document summarizes research on replacing the existing TTS endpoint in makima with Qwen3-TTS-12Hz-0.6B-Base, with the goal of supporting voice cloning using Makima's Japanese voice speaking English, and achieving near-live/streaming TTS capabilities. + +--- + +## 1. Current TTS Implementation Analysis + +### 1.1 Current Model: Chatterbox-Turbo + +The existing TTS implementation in `makima/src/tts.rs` uses **ResembleAI/chatterbox-turbo-ONNX**: + +- **Architecture**: 4-component ONNX model pipeline + - `speech_encoder.onnx` - Encodes reference voice audio + - `embed_tokens.onnx` - Token embedding layer + - `language_model.onnx` - Autoregressive language model (24 layers, 16 KV heads, 64 head dim) + - `conditional_decoder.onnx` - Decodes speech tokens to audio waveform + +- **Sample Rate**: 24,000 Hz output +- **Voice Cloning**: Required (no default voice support) +- **Special Tokens**: + - START_SPEECH_TOKEN: 6561 + - STOP_SPEECH_TOKEN: 6562 + - SILENCE_TOKEN: 4299 + +### 1.2 Current API Surface + +**Core TTS Functions:** +```rust +pub fn generate_tts(&mut self, _text: &str) -> Result, TtsError> + // Returns VoiceRequired error - voice cloning is mandatory + +pub fn generate_tts_with_voice(&mut self, text: &str, sample_audio_path: &Path) -> Result, TtsError> + // Voice cloning from file path + +pub fn generate_tts_with_samples(&mut self, text: &str, samples: &[f32], sample_rate: u32) -> Result, TtsError> + // Voice cloning from raw samples +``` + +**Audio Processing:** +- Input audio resampled to 24kHz +- Reference voice encoded into: + - `audio_features` - Acoustic features + - `prompt_tokens` - Token representation + - `speaker_embeddings` - Speaker identity + - `speaker_features` - Voice characteristics + +### 1.3 Current Limitations + +1. **No Streaming Support**: Current implementation generates complete audio before returning +2. **No Default Voice**: Requires voice reference audio for every call +3. **No HTTP Endpoint**: TTS is only available as a Rust library, not exposed via REST API +4. **Single Language**: Optimized for English, unclear multilingual support +5. **High Latency**: Full autoregressive generation before any audio output + +### 1.4 Related Components + +**Audio Processing (`makima/src/audio.rs`):** +- Uses Symphonia for audio decoding (MP3, WAV, FLAC, OGG, etc.) +- Resampling via sinc interpolation +- Stereo to mono mixdown +- Target: 16kHz mono for STT + +**Listen Endpoint (`makima/src/server/handlers/listen.rs`):** +- WebSocket-based streaming STT +- Uses Parakeet for transcription +- Sortformer for speaker diarization +- Already has real-time audio streaming infrastructure + +--- + +## 2. Qwen3-TTS-12Hz-0.6B-Base Model Analysis + +### 2.1 Model Capabilities + +| Feature | Specification | +|---------|---------------| +| **Model Size** | 0.6B parameters (lightweight variant) | +| **Voice Cloning** | 3-second reference audio only | +| **Streaming** | Dual-track hybrid architecture | +| **Minimum Latency** | 97ms end-to-end | +| **Languages** | 10 (Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian) | +| **Cross-lingual Cloning** | Japanese voice to English speech supported | +| **Speaker Similarity** | 0.95 (near human-level) | +| **Output Sample Rate** | Up to 48kHz (standard 24kHz) | + +### 2.2 Voice Cloning Requirements + +**Reference Audio:** +- **Minimum Duration**: 3 seconds +- **Recommended Duration**: 5-15 seconds +- **Format**: WAV preferred; also supports URL, base64, numpy array +- **Quality**: Clean, noise-free audio essential +- **Transcript**: Providing `ref_text` significantly improves quality + +**Cross-Lingual Usage (Japanese to English):** +```python +ref_audio = "makima_japanese.wav" # Japanese reference +ref_text = "日本語のテキスト" # Japanese transcription + +wavs, sr = model.generate_voice_clone( + text="This is English text", # English output + language="English", + ref_audio=ref_audio, + ref_text=ref_text, +) +``` + +### 2.3 Technical Requirements + +**Python Dependencies:** +```bash +pip install -U qwen-tts +pip install -U flash-attn --no-build-isolation # For optimal performance +``` + +**Hardware:** +- CUDA-compatible GPU required +- FlashAttention 2 for optimal memory usage +- Float16/bfloat16 precision support +- For <96GB RAM: `MAX_JOBS=4` for flash-attn installation + +**Model Loading:** +```python +from qwen_tts import Qwen3TTSModel +import torch + +model = Qwen3TTSModel.from_pretrained( + "Qwen/Qwen3-TTS-12Hz-0.6B-Base", + device_map="cuda:0", + dtype=torch.bfloat16, + attn_implementation="flash_attention_2", +) +``` + +### 2.4 Streaming Architecture + +**Dual-Track Hybrid Design:** +- Single model supports both streaming and non-streaming +- Audio output begins after minimal text input +- 97ms minimum latency achieved through: + - Proprietary Qwen3-TTS-Tokenizer-12Hz (efficient acoustic compression) + - Discrete multi-codebook LM (eliminates LM+DiT bottleneck) + - Lightweight non-DiT vocoder + +**Reusable Voice Clone Prompt (Critical for Performance):** +```python +# Pre-compute once +prompt_items = model.create_voice_clone_prompt( + ref_audio=ref_audio, + ref_text=ref_text, + x_vector_only_mode=False +) + +# Reuse for multiple generations +wavs, sr = model.generate_voice_clone( + text=["Line 1", "Line 2"], + language=["English", "English"], + voice_clone_prompt=prompt_items, # Cached prompt +) +``` + +--- + +## 3. Makima Voice Audio Sources + +### 3.1 Character Information + +- **Character**: Makima from Chainsaw Man anime +- **Japanese Voice Actress**: Tomori Kusunoki (楠木ともり) +- **English Voice Actress**: Suzie Yeung + +### 3.2 Potential Audio Sources + +| Source | Type | Notes | +|--------|------|-------| +| **Voicy Network Soundboard** | Official clips | MP3 download available, 20+ sound effects | +| **101Soundboards** | Fan-curated clips | Various character sounds | +| **Anime Episodes** | Source material | Highest quality, requires extraction | +| **Nikke: Goddess of Victory** | Game voicelines | Same voice actress (Tomori Kusunoki) | +| **Ko-fi (erusha)** | WAV files | x5 character impression audio files | + +### 3.3 Recommended Approach + +1. **Primary Source**: Extract 5-15 seconds of clean dialogue from Chainsaw Man anime (Japanese audio track) +2. **Selection Criteria**: + - Clear, isolated dialogue (no background music/effects) + - Natural speaking tone (not shouting/whispering) + - Variety of phonemes for better cloning +3. **Transcription**: Provide accurate Japanese transcription for `ref_text` +4. **Processing**: Convert to WAV format, ensure clean audio quality + +### 3.4 Legal Considerations + +- Voice cloning of real voice actors for commercial use may have legal implications +- Synthetic voice generation based on copyrighted characters may require licenses +- Consider using for internal/personal use only, or creating disclaimer + +--- + +## 4. Feasibility Assessment + +### 4.1 Live/Streaming TTS Feasibility: HIGHLY FEASIBLE + +**Evidence:** +- Qwen3-TTS achieves 97ms latency (well under 200ms real-time threshold) +- Existing WebSocket infrastructure in makima (`/api/v1/listen`) can be adapted +- Streaming architecture designed for interactive scenarios + +**Implementation Approach:** +1. Create new WebSocket endpoint `/api/v1/speak` mirroring listen endpoint +2. Pre-compute voice clone prompt on connection start +3. Stream audio chunks as they're generated +4. Use chunked audio encoding (similar to listen's binary message handling) + +### 4.2 Voice Cloning with Japanese Voice: FULLY SUPPORTED + +**Evidence:** +- Qwen3-TTS explicitly supports cross-lingual voice cloning +- Japanese is one of 10 supported languages +- 0.95 speaker similarity maintained across languages + +**Implementation Approach:** +1. Pre-process Makima voice clips (5-15 seconds Japanese audio) +2. Include Japanese transcription +3. Generate English speech while preserving voice characteristics + +### 4.3 Integration Challenges + +| Challenge | Difficulty | Mitigation | +|-----------|-----------|------------| +| **Python to Rust Integration** | Medium | Use Python subprocess or microservice | +| **GPU Memory** | Low | 0.6B model is lightweight | +| **Latency Target** | Low | 97ms base latency is excellent | +| **Audio Format Conversion** | Low | Existing symphonia infrastructure | +| **Default Voice Setup** | Low | One-time voice prompt caching | + +### 4.4 Architecture Options + +**Option A: Python Microservice** +``` +[Makima Rust Server] --HTTP/WebSocket--> [Python TTS Service] + | + [Qwen3-TTS Model] +``` +Pros: Clean separation, easy Python integration +Cons: Network overhead, deployment complexity + +**Option B: PyO3 Rust Bindings** +``` +[Makima Rust Server] --FFI--> [pyo3 Python Bindings] --> [Qwen3-TTS] +``` +Pros: Single process, lower latency +Cons: Complex build, Python GIL issues + +**Option C: ONNX Export (Like Current Chatterbox)** +``` +[Makima Rust Server] --ort--> [Qwen3-TTS ONNX Models] +``` +Pros: Pure Rust, consistent with existing architecture +Cons: May not have ONNX export available for Qwen3-TTS + +**Recommended: Option A (Python Microservice)** +- Fastest time to implementation +- Aligns with Qwen3-TTS's native Python API +- Can use WebSocket for streaming audio chunks +- Easy to deploy alongside existing makima server + +--- + +## 5. Preliminary Technical Approach + +### 5.1 Phase 1: Python TTS Microservice + +```python +# tts_service.py +from fastapi import FastAPI, WebSocket +from qwen_tts import Qwen3TTSModel +import torch +import base64 + +app = FastAPI() +model = None +voice_prompt = None + +@app.on_event("startup") +async def load_model(): + global model, voice_prompt + model = Qwen3TTSModel.from_pretrained( + "Qwen/Qwen3-TTS-12Hz-0.6B-Base", + device_map="cuda:0", + dtype=torch.bfloat16, + ) + # Pre-load Makima voice + voice_prompt = model.create_voice_clone_prompt( + ref_audio="makima_voice.wav", + ref_text="日本語の台詞...", + ) + +@app.websocket("/ws/speak") +async def speak(websocket: WebSocket): + await websocket.accept() + while True: + text = await websocket.receive_text() + wavs, sr = model.generate_voice_clone( + text=text, + language="English", + voice_clone_prompt=voice_prompt, + ) + # Stream audio chunks + audio_bytes = wavs[0].tobytes() + await websocket.send_bytes(audio_bytes) +``` + +### 5.2 Phase 2: Rust Integration + +```rust +// makima/src/server/handlers/speak.rs +pub async fn websocket_handler( + ws: WebSocketUpgrade, + State(state): State, +) -> Response { + ws.on_upgrade(|socket| handle_speak_socket(socket, state)) +} + +async fn handle_speak_socket(socket: WebSocket, state: SharedState) { + // Connect to Python TTS service + let tts_ws = tokio_tungstenite::connect_async("ws://localhost:8001/ws/speak").await?; + + // Forward text to TTS, stream audio back to client + // ... +} +``` + +### 5.3 API Design + +**WebSocket Endpoint: `/api/v1/speak`** + +**Client to Server Messages:** +```json +{ + "type": "start", + "sample_rate": 24000, + "encoding": "pcm16" +} + +{ + "type": "speak", + "text": "Hello, I am Makima." +} + +{ + "type": "stop" +} +``` + +**Server to Client Messages:** +```json +{ + "type": "ready", + "session_id": "uuid" +} + +{ + "type": "audio_chunk", + "data": "", + "is_final": false +} + +{ + "type": "complete" +} +``` + +--- + +## 6. Next Steps + +### Immediate Actions +1. [ ] Obtain Makima voice clips (5-15 seconds clean Japanese audio) +2. [ ] Create Japanese transcription of voice clips +3. [ ] Test Qwen3-TTS voice cloning with Makima samples +4. [ ] Benchmark latency on target hardware + +### Development Phases +1. **Phase 1**: Python TTS microservice proof-of-concept +2. **Phase 2**: WebSocket streaming integration +3. **Phase 3**: Rust proxy endpoint in makima +4. **Phase 4**: Listen page integration for bidirectional speech + +### Hardware Requirements +- CUDA-compatible GPU (minimum) +- Recommended: 8GB+ VRAM for 0.6B model with FlashAttention 2 +- Python 3.12+ environment + +--- + +## References + +- [Qwen3-TTS-12Hz-0.6B-Base on HuggingFace](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base) +- [Qwen3-TTS GitHub Repository](https://github.com/QwenLM/Qwen3-TTS) +- [Behind The Voice Actors - Makima](https://www.behindthevoiceactors.com/tv-shows/Chainsaw-Man/Makima/) +- [Voicy Network Chainsaw Man Soundboard](https://www.voicy.network/official-soundboards/anime/chainsaw-man) -- cgit v1.2.3