diff options
Diffstat (limited to 'docs/research')
| -rw-r--r-- | docs/research/TTS_RESEARCH_NOTES.md | 405 | ||||
| -rw-r--r-- | docs/research/rust-native-tts-research.md | 297 | ||||
| -rw-r--r-- | docs/research/tts-qwen3-research.md | 548 |
3 files changed, 1250 insertions, 0 deletions
diff --git a/docs/research/TTS_RESEARCH_NOTES.md b/docs/research/TTS_RESEARCH_NOTES.md new file mode 100644 index 0000000..72ac8c6 --- /dev/null +++ b/docs/research/TTS_RESEARCH_NOTES.md @@ -0,0 +1,405 @@ +# TTS Replacement Research Notes + +## Executive Summary + +This document summarizes research on replacing the existing TTS endpoint in makima with Qwen3-TTS-12Hz-0.6B-Base, with the goal of supporting voice cloning using Makima's Japanese voice speaking English, and achieving near-live/streaming TTS capabilities. + +--- + +## 1. Current TTS Implementation Analysis + +### 1.1 Current Model: Chatterbox-Turbo + +The existing TTS implementation in `makima/src/tts.rs` uses **ResembleAI/chatterbox-turbo-ONNX**: + +- **Architecture**: 4-component ONNX model pipeline + - `speech_encoder.onnx` - Encodes reference voice audio + - `embed_tokens.onnx` - Token embedding layer + - `language_model.onnx` - Autoregressive language model (24 layers, 16 KV heads, 64 head dim) + - `conditional_decoder.onnx` - Decodes speech tokens to audio waveform + +- **Sample Rate**: 24,000 Hz output +- **Voice Cloning**: Required (no default voice support) +- **Special Tokens**: + - START_SPEECH_TOKEN: 6561 + - STOP_SPEECH_TOKEN: 6562 + - SILENCE_TOKEN: 4299 + +### 1.2 Current API Surface + +**Core TTS Functions:** +```rust +pub fn generate_tts(&mut self, _text: &str) -> Result<Vec<f32>, TtsError> + // Returns VoiceRequired error - voice cloning is mandatory + +pub fn generate_tts_with_voice(&mut self, text: &str, sample_audio_path: &Path) -> Result<Vec<f32>, TtsError> + // Voice cloning from file path + +pub fn generate_tts_with_samples(&mut self, text: &str, samples: &[f32], sample_rate: u32) -> Result<Vec<f32>, TtsError> + // Voice cloning from raw samples +``` + +**Audio Processing:** +- Input audio resampled to 24kHz +- Reference voice encoded into: + - `audio_features` - Acoustic features + - `prompt_tokens` - Token representation + - `speaker_embeddings` - Speaker identity + - `speaker_features` - Voice characteristics + +### 1.3 Current Limitations + +1. **No Streaming Support**: Current implementation generates complete audio before returning +2. **No Default Voice**: Requires voice reference audio for every call +3. **No HTTP Endpoint**: TTS is only available as a Rust library, not exposed via REST API +4. **Single Language**: Optimized for English, unclear multilingual support +5. **High Latency**: Full autoregressive generation before any audio output + +### 1.4 Related Components + +**Audio Processing (`makima/src/audio.rs`):** +- Uses Symphonia for audio decoding (MP3, WAV, FLAC, OGG, etc.) +- Resampling via sinc interpolation +- Stereo to mono mixdown +- Target: 16kHz mono for STT + +**Listen Endpoint (`makima/src/server/handlers/listen.rs`):** +- WebSocket-based streaming STT +- Uses Parakeet for transcription +- Sortformer for speaker diarization +- Already has real-time audio streaming infrastructure + +--- + +## 2. Qwen3-TTS-12Hz-0.6B-Base Model Analysis + +### 2.1 Model Capabilities + +| Feature | Specification | +|---------|---------------| +| **Model Size** | 0.6B parameters (lightweight variant) | +| **Voice Cloning** | 3-second reference audio only | +| **Streaming** | Dual-track hybrid architecture | +| **Minimum Latency** | 97ms end-to-end | +| **Languages** | 10 (Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian) | +| **Cross-lingual Cloning** | Japanese voice to English speech supported | +| **Speaker Similarity** | 0.95 (near human-level) | +| **Output Sample Rate** | Up to 48kHz (standard 24kHz) | + +### 2.2 Voice Cloning Requirements + +**Reference Audio:** +- **Minimum Duration**: 3 seconds +- **Recommended Duration**: 5-15 seconds +- **Format**: WAV preferred; also supports URL, base64, numpy array +- **Quality**: Clean, noise-free audio essential +- **Transcript**: Providing `ref_text` significantly improves quality + +**Cross-Lingual Usage (Japanese to English):** +```python +ref_audio = "makima_japanese.wav" # Japanese reference +ref_text = "日本語のテキスト" # Japanese transcription + +wavs, sr = model.generate_voice_clone( + text="This is English text", # English output + language="English", + ref_audio=ref_audio, + ref_text=ref_text, +) +``` + +### 2.3 Technical Requirements + +**Python Dependencies:** +```bash +pip install -U qwen-tts +pip install -U flash-attn --no-build-isolation # For optimal performance +``` + +**Hardware:** +- CUDA-compatible GPU required +- FlashAttention 2 for optimal memory usage +- Float16/bfloat16 precision support +- For <96GB RAM: `MAX_JOBS=4` for flash-attn installation + +**Model Loading:** +```python +from qwen_tts import Qwen3TTSModel +import torch + +model = Qwen3TTSModel.from_pretrained( + "Qwen/Qwen3-TTS-12Hz-0.6B-Base", + device_map="cuda:0", + dtype=torch.bfloat16, + attn_implementation="flash_attention_2", +) +``` + +### 2.4 Streaming Architecture + +**Dual-Track Hybrid Design:** +- Single model supports both streaming and non-streaming +- Audio output begins after minimal text input +- 97ms minimum latency achieved through: + - Proprietary Qwen3-TTS-Tokenizer-12Hz (efficient acoustic compression) + - Discrete multi-codebook LM (eliminates LM+DiT bottleneck) + - Lightweight non-DiT vocoder + +**Reusable Voice Clone Prompt (Critical for Performance):** +```python +# Pre-compute once +prompt_items = model.create_voice_clone_prompt( + ref_audio=ref_audio, + ref_text=ref_text, + x_vector_only_mode=False +) + +# Reuse for multiple generations +wavs, sr = model.generate_voice_clone( + text=["Line 1", "Line 2"], + language=["English", "English"], + voice_clone_prompt=prompt_items, # Cached prompt +) +``` + +--- + +## 3. Makima Voice Audio Sources + +### 3.1 Character Information + +- **Character**: Makima from Chainsaw Man anime +- **Japanese Voice Actress**: Tomori Kusunoki (楠木ともり) +- **English Voice Actress**: Suzie Yeung + +### 3.2 Potential Audio Sources + +| Source | Type | Notes | +|--------|------|-------| +| **Voicy Network Soundboard** | Official clips | MP3 download available, 20+ sound effects | +| **101Soundboards** | Fan-curated clips | Various character sounds | +| **Anime Episodes** | Source material | Highest quality, requires extraction | +| **Nikke: Goddess of Victory** | Game voicelines | Same voice actress (Tomori Kusunoki) | +| **Ko-fi (erusha)** | WAV files | x5 character impression audio files | + +### 3.3 Recommended Approach + +1. **Primary Source**: Extract 5-15 seconds of clean dialogue from Chainsaw Man anime (Japanese audio track) +2. **Selection Criteria**: + - Clear, isolated dialogue (no background music/effects) + - Natural speaking tone (not shouting/whispering) + - Variety of phonemes for better cloning +3. **Transcription**: Provide accurate Japanese transcription for `ref_text` +4. **Processing**: Convert to WAV format, ensure clean audio quality + +### 3.4 Legal Considerations + +- Voice cloning of real voice actors for commercial use may have legal implications +- Synthetic voice generation based on copyrighted characters may require licenses +- Consider using for internal/personal use only, or creating disclaimer + +--- + +## 4. Feasibility Assessment + +### 4.1 Live/Streaming TTS Feasibility: HIGHLY FEASIBLE + +**Evidence:** +- Qwen3-TTS achieves 97ms latency (well under 200ms real-time threshold) +- Existing WebSocket infrastructure in makima (`/api/v1/listen`) can be adapted +- Streaming architecture designed for interactive scenarios + +**Implementation Approach:** +1. Create new WebSocket endpoint `/api/v1/speak` mirroring listen endpoint +2. Pre-compute voice clone prompt on connection start +3. Stream audio chunks as they're generated +4. Use chunked audio encoding (similar to listen's binary message handling) + +### 4.2 Voice Cloning with Japanese Voice: FULLY SUPPORTED + +**Evidence:** +- Qwen3-TTS explicitly supports cross-lingual voice cloning +- Japanese is one of 10 supported languages +- 0.95 speaker similarity maintained across languages + +**Implementation Approach:** +1. Pre-process Makima voice clips (5-15 seconds Japanese audio) +2. Include Japanese transcription +3. Generate English speech while preserving voice characteristics + +### 4.3 Integration Challenges + +| Challenge | Difficulty | Mitigation | +|-----------|-----------|------------| +| **Python to Rust Integration** | Medium | Use Python subprocess or microservice | +| **GPU Memory** | Low | 0.6B model is lightweight | +| **Latency Target** | Low | 97ms base latency is excellent | +| **Audio Format Conversion** | Low | Existing symphonia infrastructure | +| **Default Voice Setup** | Low | One-time voice prompt caching | + +### 4.4 Architecture Options + +**Option A: Python Microservice** +``` +[Makima Rust Server] --HTTP/WebSocket--> [Python TTS Service] + | + [Qwen3-TTS Model] +``` +Pros: Clean separation, easy Python integration +Cons: Network overhead, deployment complexity + +**Option B: PyO3 Rust Bindings** +``` +[Makima Rust Server] --FFI--> [pyo3 Python Bindings] --> [Qwen3-TTS] +``` +Pros: Single process, lower latency +Cons: Complex build, Python GIL issues + +**Option C: ONNX Export (Like Current Chatterbox)** +``` +[Makima Rust Server] --ort--> [Qwen3-TTS ONNX Models] +``` +Pros: Pure Rust, consistent with existing architecture +Cons: May not have ONNX export available for Qwen3-TTS + +**Recommended: Option A (Python Microservice)** +- Fastest time to implementation +- Aligns with Qwen3-TTS's native Python API +- Can use WebSocket for streaming audio chunks +- Easy to deploy alongside existing makima server + +--- + +## 5. Preliminary Technical Approach + +### 5.1 Phase 1: Python TTS Microservice + +```python +# tts_service.py +from fastapi import FastAPI, WebSocket +from qwen_tts import Qwen3TTSModel +import torch +import base64 + +app = FastAPI() +model = None +voice_prompt = None + +@app.on_event("startup") +async def load_model(): + global model, voice_prompt + model = Qwen3TTSModel.from_pretrained( + "Qwen/Qwen3-TTS-12Hz-0.6B-Base", + device_map="cuda:0", + dtype=torch.bfloat16, + ) + # Pre-load Makima voice + voice_prompt = model.create_voice_clone_prompt( + ref_audio="makima_voice.wav", + ref_text="日本語の台詞...", + ) + +@app.websocket("/ws/speak") +async def speak(websocket: WebSocket): + await websocket.accept() + while True: + text = await websocket.receive_text() + wavs, sr = model.generate_voice_clone( + text=text, + language="English", + voice_clone_prompt=voice_prompt, + ) + # Stream audio chunks + audio_bytes = wavs[0].tobytes() + await websocket.send_bytes(audio_bytes) +``` + +### 5.2 Phase 2: Rust Integration + +```rust +// makima/src/server/handlers/speak.rs +pub async fn websocket_handler( + ws: WebSocketUpgrade, + State(state): State<SharedState>, +) -> Response { + ws.on_upgrade(|socket| handle_speak_socket(socket, state)) +} + +async fn handle_speak_socket(socket: WebSocket, state: SharedState) { + // Connect to Python TTS service + let tts_ws = tokio_tungstenite::connect_async("ws://localhost:8001/ws/speak").await?; + + // Forward text to TTS, stream audio back to client + // ... +} +``` + +### 5.3 API Design + +**WebSocket Endpoint: `/api/v1/speak`** + +**Client to Server Messages:** +```json +{ + "type": "start", + "sample_rate": 24000, + "encoding": "pcm16" +} + +{ + "type": "speak", + "text": "Hello, I am Makima." +} + +{ + "type": "stop" +} +``` + +**Server to Client Messages:** +```json +{ + "type": "ready", + "session_id": "uuid" +} + +{ + "type": "audio_chunk", + "data": "<base64-encoded-audio>", + "is_final": false +} + +{ + "type": "complete" +} +``` + +--- + +## 6. Next Steps + +### Immediate Actions +1. [ ] Obtain Makima voice clips (5-15 seconds clean Japanese audio) +2. [ ] Create Japanese transcription of voice clips +3. [ ] Test Qwen3-TTS voice cloning with Makima samples +4. [ ] Benchmark latency on target hardware + +### Development Phases +1. **Phase 1**: Python TTS microservice proof-of-concept +2. **Phase 2**: WebSocket streaming integration +3. **Phase 3**: Rust proxy endpoint in makima +4. **Phase 4**: Listen page integration for bidirectional speech + +### Hardware Requirements +- CUDA-compatible GPU (minimum) +- Recommended: 8GB+ VRAM for 0.6B model with FlashAttention 2 +- Python 3.12+ environment + +--- + +## References + +- [Qwen3-TTS-12Hz-0.6B-Base on HuggingFace](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base) +- [Qwen3-TTS GitHub Repository](https://github.com/QwenLM/Qwen3-TTS) +- [Behind The Voice Actors - Makima](https://www.behindthevoiceactors.com/tv-shows/Chainsaw-Man/Makima/) +- [Voicy Network Chainsaw Man Soundboard](https://www.voicy.network/official-soundboards/anime/chainsaw-man) diff --git a/docs/research/rust-native-tts-research.md b/docs/research/rust-native-tts-research.md new file mode 100644 index 0000000..5bc75f7 --- /dev/null +++ b/docs/research/rust-native-tts-research.md @@ -0,0 +1,297 @@ +# Rust-Native Qwen3-TTS Integration Research + +## Executive Summary + +This document researches integrating **Qwen3-TTS-12Hz-0.6B-Base** directly into the makima Rust codebase, replacing the current Chatterbox TTS implementation. The goal is a **pure Rust** solution — no Python, no separate microservice. + +**Bottom line:** A Rust-native integration is feasible but requires significant implementation work. The most viable path is using **candle** (HuggingFace's Rust ML framework) to implement the model architecture natively, loading safetensors weights directly. The existing ONNX-based approach used for Chatterbox TTS is **not viable** for Qwen3-TTS due to architecture incompatibilities with ONNX exporters. + +--- + +## 1. Current TTS Implementation Analysis + +The existing Chatterbox TTS in `makima/src/tts.rs` uses: + +- **ONNX Runtime** via the `ort` crate (v2.0.0-rc.10) +- **Four ONNX model files**: `speech_encoder.onnx`, `embed_tokens.onnx`, `language_model.onnx`, `conditional_decoder.onnx` +- **tokenizers** crate for text tokenization +- **ndarray** for tensor manipulation +- **hf-hub** for model downloading +- **Pipeline**: encode voice → tokenize text → autoregressive token generation with KV cache → decode tokens to waveform +- **Architecture constants**: 24 layers, 16 KV heads, 64 head dim, 24kHz sample rate + +The pattern is well-established: download ONNX models from HuggingFace, load sessions, run inference with manual KV cache management. + +### STT Pattern (listen.rs) + +The Listen/STT handler in `makima/src/server/handlers/listen.rs` demonstrates the broader ML pattern: +- WebSocket-based streaming +- Lazy model loading via `SharedState::get_ml_models()` +- Models held behind `tokio::sync::Mutex` for async access +- `parakeet-rs` local crate for STT, `sortformer` for diarization +- All models are Rust-native with ONNX backends + +--- + +## 2. Qwen3-TTS-12Hz-0.6B-Base Architecture + +### Model Overview + +| Property | Value | +|----------|-------| +| **Parameters** | 0.6B | +| **Architecture** | `Qwen3TTSForConditionalGeneration` | +| **Output Sample Rate** | 24,000 Hz | +| **Token Frame Rate** | 12.5 Hz (~80ms per token) | +| **Model Format** | SafeTensors (1.83 GB main + 682 MB tokenizer) | +| **Total Size** | ~2.52 GB | +| **Precision** | bfloat16/float16 | + +### Components + +The model has **three distinct components**: + +#### A. Main Language Model (Talker) — 1.83 GB safetensors +- Hidden size: 1024 +- Layers: 28 +- Attention heads: 16 (8 KV heads) +- Intermediate size: 3072 +- Head dimension: 128 +- Text vocab size: 151,936 +- Max position embeddings: 32,768 +- Autoregressive transformer predicting speech token sequences from text + +#### B. Code Predictor (Multi-Token Prediction) — embedded in main model +- Hidden size: 1024 +- Layers: 5 +- Attention heads: 16 +- Number of code groups: 16 +- Codebook vocab size: 2048 +- Predicts residual codebooks (16 layers) after the main LM predicts the zeroth codebook + +#### C. Speech Tokenizer (Qwen3-TTS-Tokenizer-12Hz) — 682 MB safetensors +- Separate model in `speech_tokenizer/` directory +- GAN-based codec: encoder + decoder +- 16-layer multi-codebook RVQ (Residual Vector Quantization) +- First codebook: semantic (WavLM-guided) +- Remaining 15: acoustic details +- **Decoder**: lightweight causal ConvNet (no DiT/diffusion needed) +- Encodes reference audio → discrete codes, decodes codes → waveform + +### Inference Pipeline + +``` +Text Input + Reference Audio + ↓ +[Speech Tokenizer Encoder] → reference audio codes + speaker embedding + ↓ +[Text Tokenizer] → text token IDs + ↓ +[Language Model] → autoregressive generation of zeroth codebook tokens + ↓ +[Code Predictor / MTP] → predict remaining 15 codebook layers + ↓ +[Speech Tokenizer Decoder / Causal ConvNet] → waveform output (24kHz) +``` + +--- + +## 3. ONNX Export Feasibility — NOT VIABLE + +### Status: No ONNX support exists + +- **No official ONNX export** from Qwen team +- **No community ONNX conversion** for Qwen3-TTS +- The Qwen3 architecture is **not supported** by HuggingFace Optimum's ONNX exporter +- Users attempting export get: `ValueError: Trying to export a qwen3 model, that is a custom or unsupported architecture, but no custom onnx configuration was passed` +- Even for base Qwen3 LLMs (non-TTS), ONNX export has significant issues with MoE routing, hybrid attention, and novel architecture components + +### Why ONNX Won't Work for Qwen3-TTS + +1. **Custom architecture** — `Qwen3TTSForConditionalGeneration` is not a standard transformer; it combines LM + code predictor + speech tokenizer +2. **Multi-codebook MTP module** — the code predictor generates 16 codebook layers, a non-standard operation +3. **Causal ConvNet decoder** — the speech tokenizer's decoder is a custom GAN-trained ConvNet, not a standard vocoder +4. **Dynamic control flow** — dual-track streaming architecture with conditional branching +5. **No Optimum support** — would require writing a custom ONNX config from scratch for each sub-component + +**Verdict: The ONNX path (matching our Chatterbox approach) is a dead end for Qwen3-TTS.** + +--- + +## 4. Rust-Native Inference Options + +### Option A: Candle (HuggingFace) — RECOMMENDED + +[candle](https://github.com/huggingface/candle) is HuggingFace's minimalist Rust ML framework. + +**Why candle is the best fit:** + +| Factor | Assessment | +|--------|------------| +| **Qwen model support** | ✅ Has `qwen2` module in candle-transformers; Qwen3 variants supported | +| **SafeTensors loading** | ✅ Native first-class support (safetensors is a Rust crate) | +| **GPU support** | ✅ CUDA backend, Metal (macOS), CPU with MKL | +| **Tokenizer support** | ✅ Uses the same `tokenizers` crate makima already depends on | +| **Audio models** | ✅ Supports EnCodec, Whisper, MetaVoice, Parler-TTS | +| **KV cache** | ✅ Well-established patterns in existing model implementations | +| **Community** | ✅ Active; Crane project already lists Qwen3-TTS as "highest priority" | +| **Binary size** | ✅ Compiles to single binary, no Python dependency | + +**What needs to be implemented:** + +1. **Qwen3-TTS transformer layers** — extend existing `qwen2` model code for the 28-layer LM with TTS-specific modifications (speaker encoder concatenation, code predictor output heads) +2. **Code Predictor (MTP)** — 5-layer module that generates 16 codebook predictions from the LM hidden states +3. **Speech Tokenizer Encoder** — ConvNet encoder that converts reference audio to discrete multi-codebook tokens + speaker embeddings +4. **Speech Tokenizer Decoder** — causal ConvNet that reconstructs waveforms from discrete codes +5. **Multi-codebook handling** — manage 16 parallel codebook sequences + +**Estimated effort:** Medium-High. The LM backbone can reuse existing Qwen2/3 code. The speech tokenizer (encoder + decoder) is the most novel component. + +**Key crate dependencies to add:** +```toml +candle-core = "0.8" +candle-nn = "0.8" +candle-transformers = "0.8" +# Keep existing: tokenizers, hf-hub, ndarray (for compatibility) +``` + +### Option B: Crane (Candle-based TTS Engine) + +[Crane](https://github.com/lucasjinreal/Crane) is a pure Rust LLM inference engine built on candle, specifically designed for multi-modal models including TTS. + +**Key facts:** +- Already supports Spark-TTS (codec-based TTS with similar architecture) +- **Qwen3-TTS is listed as "Highest Priority" on their roadmap** +- Handles multi-module architectures (codec + LLM pipelines) +- Supports Qwen2.5, Moonshine ASR +- Claims 50x faster than PyTorch on Apple Silicon + +**Strategy:** Monitor Crane's Qwen3-TTS implementation. If they ship it, we could either: +- Use Crane as a dependency directly +- Port their implementation into makima's codebase +- Contribute to Crane and depend on it + +**Risk:** Crane is a relatively new project; depending on it adds supply chain risk. + +### Option C: qwen3-rs (Educational Reference) + +[qwen3-rs](https://github.com/reinterpretcat/qwen3-rs) is an educational project implementing Qwen3 inference from scratch in Rust. + +**Useful for:** Reference implementation of Qwen3 transformer layers, tokenization, KV cache, and safetensors loading — all without heavy ML framework dependencies. However, it only implements the base LLM, not the TTS-specific components. + +### Option D: Direct ort (ONNX Runtime) with Custom Export — FALLBACK + +If we could manually export each sub-component to ONNX: + +1. Export the 28-layer LM backbone (similar complexity to Chatterbox) +2. Export the code predictor separately +3. Export the speech tokenizer encoder/decoder + +This would match our existing Chatterbox pattern but requires Python scripting for the one-time export, and the Qwen3 architecture is explicitly unsupported by standard exporters. **Not recommended unless ONNX support materializes upstream.** + +### Option E: PyTorch C++ (libtorch) via FFI — NOT RECOMMENDED + +Using libtorch via Rust FFI bindings (`tch-rs` crate). This would: +- Add a ~2GB libtorch dependency +- Require complex build setup +- Introduce C++ dependency management +- Defeat the purpose of a pure Rust solution + +--- + +## 5. Recommended Approach + +### Phase 1: Candle-Based Implementation + +**Architecture:** + +``` +makima/src/tts/ +├── mod.rs // TTS trait + factory (select Chatterbox vs Qwen3) +├── chatterbox.rs // Existing ONNX-based Chatterbox (moved from tts.rs) +├── qwen3/ +│ ├── mod.rs // Qwen3TTS public API +│ ├── model.rs // Qwen3 LM transformer (28 layers) +│ ├── code_predictor.rs // MTP module (5 layers, 16 codebooks) +│ ├── speech_tokenizer.rs // Encoder + Decoder (causal ConvNet) +│ ├── config.rs // Model config from config.json +│ └── generate.rs // Autoregressive generation loop with KV cache +``` + +**Key implementation details:** + +1. **Load safetensors directly** — candle's `safetensors` support reads the 1.83GB main model and 682MB speech tokenizer +2. **Reuse Qwen2 attention** — candle-transformers already has `qwen2::Model` with RoPE, GQA, and KV cache +3. **Implement ConvNet codec** — the speech tokenizer's encoder/decoder is a causal 1D ConvNet; candle has `Conv1d` layers +4. **Multi-codebook RVQ** — implement the 16-codebook residual vector quantization lookup +5. **Speaker embedding** — extract from reference audio via the speech tokenizer encoder +6. **Streaming support** — the 12Hz model's causal architecture enables token-by-token waveform generation + +### Phase 2: Voice Assets + +The model supports voice cloning with reference audio. For the default Makima voice: +- Need 5-15 second Japanese-accented English audio clip +- Reference audio + transcript fed to speech tokenizer encoder +- Speaker embedding cached for reuse + +### Phase 3: Integration with Listen Page + +Following the pattern in `listen.rs`: +- TTS model loaded lazily via `SharedState` +- Protected behind `tokio::sync::Mutex` (or `RwLock` for concurrent reads) +- WebSocket endpoint for streaming TTS (emit audio chunks as tokens are generated) +- Bidirectional: STT (listen) → process → TTS (speak) loop + +--- + +## 6. Comparison Matrix + +| Criteria | ONNX (current pattern) | Candle | Crane | libtorch | +|----------|----------------------|--------|-------|----------| +| Pure Rust | ✅ (ort crate) | ✅ | ✅ | ❌ (C++ FFI) | +| Qwen3-TTS support | ❌ No export | ⚠️ Needs impl | ⚠️ Planned | ✅ (full PyTorch) | +| Single binary | ✅ | ✅ | ✅ | ❌ | +| GPU acceleration | ✅ | ✅ | ✅ | ✅ | +| SafeTensors loading | ❌ (needs ONNX) | ✅ | ✅ | ✅ | +| Streaming TTS | ✅ | ✅ | ✅ | ✅ | +| Maintenance burden | Low | Medium | Low (if adopted) | High | +| Implementation effort | N/A (blocked) | Medium-High | Low (if available) | Medium | +| Dependency size | ~50MB | ~5MB | ~5MB | ~2GB | + +--- + +## 7. Risk Assessment + +| Risk | Likelihood | Impact | Mitigation | +|------|-----------|--------|------------| +| Candle implementation takes longer than expected | Medium | Medium | Reference Crane's Spark-TTS implementation; use qwen3-rs as LM reference | +| Speech tokenizer ConvNet is complex to port | Medium | High | Study the PyTorch source in qwen-tts package; ConvNet layers are simpler than transformers | +| Model quality differs from reference PyTorch | Low | High | Validate with reference audio samples; ensure bfloat16 precision | +| Crane ships Qwen3-TTS before we finish | Medium | Positive | Adopt their implementation | +| GPU memory issues on target hardware | Low | Medium | 0.6B model is small (~2.5GB); fits in 4GB VRAM with float16 | + +--- + +## 8. Next Steps + +1. **Immediate:** Add `candle-core`, `candle-nn`, `candle-transformers` to Cargo.toml +2. **Week 1:** Implement Qwen3 LM backbone in candle (extend existing qwen2 model) +3. **Week 2:** Implement speech tokenizer encoder/decoder (ConvNet + RVQ) +4. **Week 2:** Implement code predictor (MTP module) +5. **Week 3:** Integration testing with reference audio; validate output quality +6. **Week 3:** Wire into makima server as TTS endpoint +7. **Ongoing:** Monitor Crane project for Qwen3-TTS implementation + +--- + +## Sources + +- [Qwen3-TTS-12Hz-0.6B-Base on HuggingFace](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base) +- [Qwen3-TTS Technical Report (arXiv)](https://arxiv.org/html/2601.15621v1) +- [Qwen3-TTS GitHub Repository](https://github.com/QwenLM/Qwen3-TTS) +- [Candle — HuggingFace Rust ML Framework](https://github.com/huggingface/candle) +- [Crane — Rust LLM Inference Engine](https://github.com/lucasjinreal/Crane) +- [qwen3-rs — Educational Qwen3 Rust Implementation](https://github.com/reinterpretcat/qwen3-rs) +- [candle-transformers Qwen2 model](https://docs.rs/candle-transformers/latest/candle_transformers/models/qwen2/index.html) +- [Qwen3-TTS-Tokenizer-12Hz on HuggingFace](https://huggingface.co/Qwen/Qwen3-TTS-Tokenizer-12Hz) +- [ONNX export issues for Qwen3](https://huggingface.co/onnx-community/Qwen3-1.7B-ONNX/discussions/1) diff --git a/docs/research/tts-qwen3-research.md b/docs/research/tts-qwen3-research.md new file mode 100644 index 0000000..a961b4f --- /dev/null +++ b/docs/research/tts-qwen3-research.md @@ -0,0 +1,548 @@ +# TTS Research: Qwen3-TTS-12Hz-0.6B-Base Integration + +## Executive Summary + +This document evaluates replacing the current Chatterbox TTS implementation with Qwen3-TTS-12Hz-0.6B-Base for the makima system. The goal is to enable near-real-time voice synthesis with voice cloning capabilities, defaulting to Makima's Japanese voice (Tomori Kusunoki) speaking English. + +**Key Findings:** +- Qwen3-TTS offers superior streaming capabilities (~97ms latency) compared to the current batch-only Chatterbox implementation +- Voice cloning requires only 3 seconds of reference audio +- No official ONNX export exists; Python/PyTorch inference required +- The 0.6B model is optimized for resource-constrained environments + +--- + +## 1. Current TTS Implementation Analysis + +### 1.1 Architecture Overview + +The current implementation uses **Chatterbox-Turbo-ONNX** from ResembleAI: + +``` +Location: makima/src/tts.rs +Model ID: ResembleAI/chatterbox-turbo-ONNX +Sample Rate: 24,000 Hz +``` + +**Components:** +| Component | File | Purpose | +|-----------|------|---------| +| `speech_encoder.onnx` | ~XX MB | Encodes reference audio to speaker embeddings | +| `embed_tokens.onnx` | ~XX MB | Token embedding layer | +| `language_model.onnx` | ~XX MB | Autoregressive text-to-speech token generation | +| `conditional_decoder.onnx` | ~XX MB | Converts speech tokens to waveform | +| `tokenizer.json` | ~KB | Text tokenization | + +### 1.2 Current API Surface + +```rust +pub struct ChatterboxTTS { + speech_encoder: Session, + embed_tokens: Session, + language_model: Session, + conditional_decoder: Session, + tokenizer: Tokenizer, +} + +impl ChatterboxTTS { + // Load from pretrained models + pub fn from_pretrained(model_dir: Option<&str>) -> Result<Self, TtsError>; + + // Generate speech (requires voice reference) + pub fn generate_tts(&mut self, _text: &str) -> Result<Vec<f32>, TtsError>; + + // Voice cloning from file path + pub fn generate_tts_with_voice( + &mut self, + text: &str, + sample_audio_path: &Path, + ) -> Result<Vec<f32>, TtsError>; + + // Voice cloning from raw samples + pub fn generate_tts_with_samples( + &mut self, + text: &str, + samples: &[f32], + sample_rate: u32, + ) -> Result<Vec<f32>, TtsError>; +} +``` + +### 1.3 Current Capabilities + +| Feature | Supported | Notes | +|---------|-----------|-------| +| Voice Cloning | **Yes** | Required for all synthesis | +| Streaming | **No** | Batch generation only | +| Languages | Limited | English-focused | +| ONNX Runtime | **Yes** | CPU inference via `ort` crate | +| GPU Acceleration | Partial | ONNX supports CUDA EP | +| Real-time Factor | Unknown | Not benchmarked | + +### 1.4 Integration Points + +The TTS module is currently: +- Exposed as `pub mod tts` in `lib.rs` +- Used in `main.rs` for testing +- **Not integrated with the web server** (no `/api/v1/tts` endpoint) + +The audio processing infrastructure is shared with the Listen (STT) module: +- `audio.rs` provides format conversion utilities +- `symphonia` for decoding various audio formats +- Resampling to target sample rates (16kHz for STT, 24kHz for TTS) + +--- + +## 2. Qwen3-TTS-12Hz-0.6B-Base Analysis + +### 2.1 Model Overview + +**Source:** [Hugging Face - Qwen/Qwen3-TTS-12Hz-0.6B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base) + +| Specification | Value | +|---------------|-------| +| Parameters | 0.6B | +| Release Date | January 22, 2026 | +| Architecture | Dual-Track hybrid streaming LM | +| Tokenizer | Qwen3-TTS-Tokenizer-12Hz | +| Frame Rate | 12.5 Hz | +| Output Sample Rate | 24 kHz | +| Languages | 10 (Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian) | + +### 2.2 Key Features + +| Feature | Status | Details | +|---------|--------|---------| +| **Voice Cloning** | Yes | 3-second minimum reference audio | +| **Streaming** | Yes | 97ms end-to-end latency | +| **Real-time** | Yes | First audio packet after single character | +| **Multilingual** | Yes | 10 languages supported | +| **Instruction Control** | No | Base model limitation | + +### 2.3 Streaming Architecture + +The Dual-Track architecture enables: +1. **Streaming text input** - Processes text incrementally +2. **Streaming audio output** - Emits audio packets as generated +3. **Multi-Token Prediction (MTP)** - Enables immediate speech decoding from first codec frame + +**Latency Benchmarks:** +- First token latency: ~97ms (end-to-end) +- Optimized TTFT: ~170ms on RTX 5090 (community fork) +- Initial implementations: ~800ms TTFT (before optimization) + +### 2.4 Voice Cloning Requirements + +| Requirement | Specification | +|-------------|---------------| +| Reference Audio Length | **3 seconds minimum** | +| Audio Format | WAV, MP3, or common formats | +| Input Methods | File path, URL, base64, numpy array | +| Reference Text | **Required** (transcript of reference audio) | +| X-Vector Only Mode | Optional (speaker embedding only, lower quality) | + +### 2.5 Python API + +```python +from qwen_tts import Qwen3TTSModel + +# Load model +model = Qwen3TTSModel.from_pretrained( + "Qwen/Qwen3-TTS-12Hz-0.6B-Base", + device_map="cuda:0", + dtype=torch.bfloat16, + attn_implementation="flash_attention_2", +) + +# Voice cloning +wavs, sr = model.generate_voice_clone( + text="Hello, this is a test.", + language="English", + ref_audio="reference.wav", + ref_text="Original speaker text from reference", +) + +# Reusable prompt (efficient for multiple generations) +prompt = model.create_voice_clone_prompt( + ref_audio="reference.wav", + ref_text="Reference transcript", +) + +wavs, sr = model.generate_voice_clone( + text="New text", + language="English", + voice_clone_prompt=prompt, +) +``` + +### 2.6 Dependencies + +``` +pip install -U qwen-tts +pip install -U flash-attn --no-build-isolation # Optional, recommended +``` + +**Requirements:** +- Python 3.12 recommended +- CUDA-capable GPU (for optimal performance) +- FlashAttention 2 compatible hardware +- PyTorch with bfloat16 support + +--- + +## 3. Feasibility Assessment + +### 3.1 Streaming/Live TTS Feasibility + +**Assessment: FEASIBLE with caveats** + +| Factor | Current State | Path Forward | +|--------|---------------|--------------| +| Streaming API | Experimental (community fork) | Use [dffdeeq/Qwen3-TTS-streaming](https://github.com/dffdeeq/Qwen3-TTS-streaming) or wait for official support | +| Latency Target | 97ms advertised | Achievable with optimization | +| First Token | ~170ms optimized | Acceptable for conversational use | +| Audio Stability | First 1-2s may have timbre issues | Known limitation, may need buffering | + +**Streaming Implementation Status:** +- Official repository: Streaming documented but **not released** +- Community fork: Working implementation with ~170ms TTFT +- vLLM-Omni: Offline inference only (online serving planned) + +### 3.2 Voice Cloning for Makima + +**Assessment: FULLY FEASIBLE** + +Requirements for Makima voice cloning: +1. **3+ seconds of clean audio** - Tomori Kusunoki (Japanese VA) speaking +2. **Transcript of the audio** - Required for full quality +3. **Audio format** - WAV/MP3 acceptable + +**Audio Sources:** +- Chainsaw Man anime clips +- Voice actress promotional material +- Behind The Voice Actors database + +**Considerations:** +- Japanese VA speaking English may work with explicit `language="English"` setting +- May need English-speaking Makima clips (Suzie Yeung, English dub VA) as fallback +- X-vector mode available if transcript is difficult to obtain + +### 3.3 Integration Complexity + +| Component | Complexity | Notes | +|-----------|------------|-------| +| Model Loading | Medium | Python subprocess or PyO3 bridge required | +| Streaming API | High | WebSocket integration needed | +| Voice Caching | Low | `create_voice_clone_prompt()` supports this | +| Audio Format | Low | Both use 24kHz output | +| ONNX Migration | N/A | **No ONNX export available** | + +### 3.4 ONNX vs Python Inference + +**Current approach (Chatterbox):** Rust + ONNX Runtime +- Pros: Native Rust, low latency, CPU-friendly +- Cons: Limited model ecosystem, no streaming + +**Required approach (Qwen3-TTS):** Python + PyTorch +- Pros: Full model access, streaming support, GPU acceleration +- Cons: Python subprocess overhead, dependency management + +**Integration Options:** + +1. **Python Subprocess/Service** + - Run `qwen-tts` as separate Python service + - Communicate via HTTP/WebSocket + - Cleanest separation, easiest to implement + +2. **PyO3 Bridge** + - Embed Python in Rust binary + - Higher complexity, tighter integration + - May have GIL contention issues + +3. **Custom ONNX Export** (Future) + - Not currently available + - Would require community effort + - No timeline from Qwen team + +**Recommendation:** Python service with WebSocket streaming + +--- + +## 4. Audio Clip Requirements + +### 4.1 For Voice Cloning Setup + +| Requirement | Specification | +|-------------|---------------| +| Minimum Duration | 3 seconds | +| Recommended Duration | 5-10 seconds | +| Format | WAV (preferred), MP3 | +| Sample Rate | Any (will be resampled) | +| Content | Clear speech, minimal background noise | +| Transcript | Required for full quality | + +### 4.2 Makima Voice Sources + +**Priority 1: Japanese VA (Tomori Kusunoki) speaking Japanese** +- Source: Chainsaw Man anime +- Challenge: Need clear dialogue without music/SFX +- Fallback: May not transfer well to English output + +**Priority 2: English VA (Suzie Yeung)** +- Source: Chainsaw Man English dub +- Advantage: Native English speaker for English output +- Trade-off: Different vocal characteristics from Japanese VA + +**Recommended Approach:** +1. Extract 5-10 second clips of both VAs +2. Test voice cloning quality with each +3. Select based on English speech naturalness +4. Store reference audio + transcript in `models/voices/makima/` + +### 4.3 Transcript Requirements + +For optimal voice cloning: +``` +ref_audio: "models/voices/makima/makima-reference.wav" +ref_text: "The exact words spoken in the reference audio" +``` + +X-vector fallback (lower quality, no transcript needed): +```python +prompt = model.create_voice_clone_prompt( + ref_audio="reference.wav", + x_vector_only_mode=True, # No transcript required +) +``` + +--- + +## 5. Preliminary Technical Approach + +### 5.1 Architecture Overview + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Makima Server (Rust) │ +├─────────────────────────────────────────────────────────────┤ +│ ┌─────────────┐ ┌─────────────┐ ┌──────────────────────┐│ +│ │ Listen (STT)│ │ TTS Proxy │ │ Chat/Contract APIs ││ +│ │ /api/v1/ │ │ /api/v1/tts │ │ /api/v1/... ││ +│ │ listen │ │ │ │ ││ +│ └──────┬──────┘ └──────┬──────┘ └──────────────────────┘│ +│ │ │ │ +│ │ ┌──────▼──────┐ │ +│ │ │ WebSocket │ │ +│ │ │ Bridge │ │ +│ │ └──────┬──────┘ │ +└─────────┼────────────────┼──────────────────────────────────┘ + │ │ + │ ┌──────▼──────┐ + │ │ Python TTS │ + │ │ Service │ + │ │ (Qwen3-TTS) │ + │ └─────────────┘ + │ + ┌──────▼──────┐ + │ ML Models │ + │ (Parakeet, │ + │ Sortformer) │ + └─────────────┘ +``` + +### 5.2 Python TTS Service + +**Proposed Architecture:** + +```python +# tts_service.py +import asyncio +from fastapi import FastAPI, WebSocket +from qwen_tts import Qwen3TTSModel + +app = FastAPI() +model = None +voice_prompts = {} + +@app.on_event("startup") +async def load_model(): + global model + model = Qwen3TTSModel.from_pretrained( + "Qwen/Qwen3-TTS-12Hz-0.6B-Base", + device_map="cuda:0", + dtype=torch.bfloat16, + attn_implementation="flash_attention_2", + ) + + # Pre-load Makima voice prompt + voice_prompts["makima"] = model.create_voice_clone_prompt( + ref_audio="models/voices/makima/reference.wav", + ref_text="[Makima reference transcript]", + ) + +@app.websocket("/ws/tts") +async def tts_stream(websocket: WebSocket): + await websocket.accept() + while True: + data = await websocket.receive_json() + text = data["text"] + voice = data.get("voice", "makima") + language = data.get("language", "English") + + # Generate with streaming (when available) + prompt = voice_prompts.get(voice) + wavs, sr = model.generate_voice_clone( + text=text, + language=language, + voice_clone_prompt=prompt, + ) + + # Send audio chunks + await websocket.send_bytes(wavs[0].tobytes()) + +@app.post("/api/tts") +async def tts_batch(request: TTSRequest): + # Batch fallback for non-streaming use cases + ... +``` + +### 5.3 Rust Integration + +**New Endpoint: `/api/v1/tts`** + +```rust +// server/handlers/tts.rs +pub async fn tts_websocket_handler( + ws: WebSocketUpgrade, + State(state): State<SharedState>, +) -> Response { + ws.on_upgrade(|socket| handle_tts_socket(socket, state)) +} + +async fn handle_tts_socket(socket: WebSocket, state: SharedState) { + // Proxy WebSocket to Python TTS service + let tts_client = state.tts_client.lock().await; + + let (mut sender, mut receiver) = socket.split(); + + while let Some(msg) = receiver.next().await { + match msg { + Ok(Message::Text(text)) => { + // Forward to Python service + let response = tts_client.generate(text).await; + + // Stream audio back + for chunk in response.audio_chunks { + sender.send(Message::Binary(chunk)).await.ok(); + } + } + _ => {} + } + } +} +``` + +### 5.4 Voice Prompt Caching + +```rust +// Pre-computed voice prompts stored in state +pub struct TtsConfig { + pub default_voice: String, + pub voices: HashMap<String, VoicePrompt>, +} + +pub struct VoicePrompt { + pub name: String, + pub ref_audio_path: PathBuf, + pub ref_text: String, + pub language: String, + // Cached prompt from Python service + pub cached_prompt_id: Option<String>, +} +``` + +### 5.5 Message Protocol + +**Client -> Server:** +```json +{ + "type": "synthesize", + "text": "Hello, I am Makima.", + "voice": "makima", + "language": "English", + "stream": true +} +``` + +**Server -> Client:** +```json +// Audio chunk +{"type": "audio", "data": "<base64 PCM>", "sample_rate": 24000, "final": false} + +// Completion +{"type": "complete", "duration_ms": 1234} + +// Error +{"type": "error", "code": "SYNTHESIS_ERROR", "message": "..."} +``` + +--- + +## 6. Implementation Phases + +### Phase 1: Research & Setup (Current) +- [x] Analyze current TTS implementation +- [x] Research Qwen3-TTS capabilities +- [x] Document feasibility and approach +- [ ] Acquire Makima voice reference clips +- [ ] Test voice cloning quality + +### Phase 2: Python Service +- [ ] Create Python TTS service with FastAPI +- [ ] Implement batch TTS endpoint +- [ ] Implement WebSocket streaming (when available) +- [ ] Add voice prompt management +- [ ] GPU optimization with FlashAttention 2 + +### Phase 3: Rust Integration +- [ ] Add TTS proxy endpoints to makima server +- [ ] WebSocket bridge implementation +- [ ] Voice configuration management +- [ ] Error handling and fallbacks + +### Phase 4: Production Ready +- [ ] Health checks for Python service +- [ ] Voice prompt caching optimization +- [ ] Latency benchmarking +- [ ] Integration with Listen page + +--- + +## 7. Open Questions + +1. **Streaming API Availability**: When will official streaming support be released? + - Fallback: Use community fork or batch with chunked responses + +2. **Voice Quality**: How well does Japanese VA voice clone to English speech? + - Action: Test with both Japanese and English VA samples + +3. **GPU Requirements**: What's the minimum VRAM for 0.6B model? + - Estimate: ~2-4GB with bf16 quantization + +4. **Latency Target**: What's acceptable for "close to live" TTS? + - Proposal: <500ms first audio, <100ms subsequent chunks + +5. **Transcript Acquisition**: How to obtain accurate transcripts for voice clips? + - Options: Manual transcription, Whisper ASR, community resources + +--- + +## 8. References + +- [Qwen3-TTS-12Hz-0.6B-Base (Hugging Face)](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base) +- [Qwen3-TTS GitHub Repository](https://github.com/QwenLM/Qwen3-TTS) +- [Qwen3-TTS Technical Report (arXiv)](https://arxiv.org/abs/2601.15621) +- [Streaming Inference Issue #77](https://github.com/QwenLM/Qwen3-TTS/issues/77) +- [Community Streaming Fork](https://github.com/dffdeeq/Qwen3-TTS-streaming) +- [Makima Voice Actors](https://www.behindthevoiceactors.com/characters/Chainsaw-Man/Makima/) +- [Chatterbox-Turbo-ONNX (Current Model)](https://huggingface.co/ResembleAI/chatterbox-turbo-ONNX) |