path: root/docs/research/tts-qwen3-research.md

# TTS Research: Qwen3-TTS-12Hz-0.6B-Base Integration

## Executive Summary

This document evaluates replacing the current Chatterbox TTS implementation with Qwen3-TTS-12Hz-0.6B-Base for the makima system. The goal is to enable near-real-time voice synthesis with voice cloning capabilities, defaulting to Makima's Japanese voice (Tomori Kusunoki) speaking English.

**Key Findings:**
- Qwen3-TTS offers superior streaming capabilities (~97ms latency) compared to the current batch-only Chatterbox implementation
- Voice cloning requires only 3 seconds of reference audio
- No official ONNX export exists; Python/PyTorch inference required
- The 0.6B model is optimized for resource-constrained environments

---

## 1. Current TTS Implementation Analysis

### 1.1 Architecture Overview

The current implementation uses **Chatterbox-Turbo-ONNX** from ResembleAI:

```
Location: makima/src/tts.rs
Model ID: ResembleAI/chatterbox-turbo-ONNX
Sample Rate: 24,000 Hz
```

**Components:**
| Component | File | Purpose |
|-----------|------|---------|
| `speech_encoder.onnx` | ~XX MB | Encodes reference audio to speaker embeddings |
| `embed_tokens.onnx` | ~XX MB | Token embedding layer |
| `language_model.onnx` | ~XX MB | Autoregressive text-to-speech token generation |
| `conditional_decoder.onnx` | ~XX MB | Converts speech tokens to waveform |
| `tokenizer.json` | ~KB | Text tokenization |

### 1.2 Current API Surface

```rust
pub struct ChatterboxTTS {
    speech_encoder: Session,
    embed_tokens: Session,
    language_model: Session,
    conditional_decoder: Session,
    tokenizer: Tokenizer,
}

impl ChatterboxTTS {
    // Load from pretrained models
    pub fn from_pretrained(model_dir: Option<&str>) -> Result<Self, TtsError>;

    // Generate speech (requires voice reference)
    pub fn generate_tts(&mut self, _text: &str) -> Result<Vec<f32>, TtsError>;

    // Voice cloning from file path
    pub fn generate_tts_with_voice(
        &mut self,
        text: &str,
        sample_audio_path: &Path,
    ) -> Result<Vec<f32>, TtsError>;

    // Voice cloning from raw samples
    pub fn generate_tts_with_samples(
        &mut self,
        text: &str,
        samples: &[f32],
        sample_rate: u32,
    ) -> Result<Vec<f32>, TtsError>;
}
```

### 1.3 Current Capabilities

| Feature | Supported | Notes |
|---------|-----------|-------|
| Voice Cloning | **Yes** | Required for all synthesis |
| Streaming | **No** | Batch generation only |
| Languages | Limited | English-focused |
| ONNX Runtime | **Yes** | CPU inference via `ort` crate |
| GPU Acceleration | Partial | ONNX supports CUDA EP |
| Real-time Factor | Unknown | Not benchmarked |

### 1.4 Integration Points

The TTS module is currently:
- Exposed as `pub mod tts` in `lib.rs`
- Used in `main.rs` for testing
- **Not integrated with the web server** (no `/api/v1/tts` endpoint)

The audio processing infrastructure is shared with the Listen (STT) module:
- `audio.rs` provides format conversion utilities
- `symphonia` for decoding various audio formats
- Resampling to target sample rates (16kHz for STT, 24kHz for TTS)

---

## 2. Qwen3-TTS-12Hz-0.6B-Base Analysis

### 2.1 Model Overview

**Source:** [Hugging Face - Qwen/Qwen3-TTS-12Hz-0.6B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base)

| Specification | Value |
|---------------|-------|
| Parameters | 0.6B |
| Release Date | January 22, 2026 |
| Architecture | Dual-Track hybrid streaming LM |
| Tokenizer | Qwen3-TTS-Tokenizer-12Hz |
| Frame Rate | 12.5 Hz |
| Output Sample Rate | 24 kHz |
| Languages | 10 (Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian) |

### 2.2 Key Features

| Feature | Status | Details |
|---------|--------|---------|
| **Voice Cloning** | Yes | 3-second minimum reference audio |
| **Streaming** | Yes | 97ms end-to-end latency |
| **Real-time** | Yes | First audio packet after single character |
| **Multilingual** | Yes | 10 languages supported |
| **Instruction Control** | No | Base model limitation |

### 2.3 Streaming Architecture

The Dual-Track architecture enables:
1. **Streaming text input** - Processes text incrementally
2. **Streaming audio output** - Emits audio packets as generated
3. **Multi-Token Prediction (MTP)** - Enables immediate speech decoding from first codec frame

**Latency Benchmarks:**
- First token latency: ~97ms (end-to-end)
- Optimized TTFT: ~170ms on RTX 5090 (community fork)
- Initial implementations: ~800ms TTFT (before optimization)

### 2.4 Voice Cloning Requirements

| Requirement | Specification |
|-------------|---------------|
| Reference Audio Length | **3 seconds minimum** |
| Audio Format | WAV, MP3, or common formats |
| Input Methods | File path, URL, base64, numpy array |
| Reference Text | **Required** (transcript of reference audio) |
| X-Vector Only Mode | Optional (speaker embedding only, lower quality) |

### 2.5 Python API

```python
from qwen_tts import Qwen3TTSModel

# Load model
model = Qwen3TTSModel.from_pretrained(
    "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
    device_map="cuda:0",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)

# Voice cloning
wavs, sr = model.generate_voice_clone(
    text="Hello, this is a test.",
    language="English",
    ref_audio="reference.wav",
    ref_text="Original speaker text from reference",
)

# Reusable prompt (efficient for multiple generations)
prompt = model.create_voice_clone_prompt(
    ref_audio="reference.wav",
    ref_text="Reference transcript",
)

wavs, sr = model.generate_voice_clone(
    text="New text",
    language="English",
    voice_clone_prompt=prompt,
)
```

### 2.6 Dependencies

```
pip install -U qwen-tts
pip install -U flash-attn --no-build-isolation  # Optional, recommended
```

**Requirements:**
- Python 3.12 recommended
- CUDA-capable GPU (for optimal performance)
- FlashAttention 2 compatible hardware
- PyTorch with bfloat16 support

---

## 3. Feasibility Assessment

### 3.1 Streaming/Live TTS Feasibility

**Assessment: FEASIBLE with caveats**

| Factor | Current State | Path Forward |
|--------|---------------|--------------|
| Streaming API | Experimental (community fork) | Use [dffdeeq/Qwen3-TTS-streaming](https://github.com/dffdeeq/Qwen3-TTS-streaming) or wait for official support |
| Latency Target | 97ms advertised | Achievable with optimization |
| First Token | ~170ms optimized | Acceptable for conversational use |
| Audio Stability | First 1-2s may have timbre issues | Known limitation, may need buffering |

**Streaming Implementation Status:**
- Official repository: Streaming documented but **not released**
- Community fork: Working implementation with ~170ms TTFT
- vLLM-Omni: Offline inference only (online serving planned)

### 3.2 Voice Cloning for Makima

**Assessment: FULLY FEASIBLE**

Requirements for Makima voice cloning:
1. **3+ seconds of clean audio** - Tomori Kusunoki (Japanese VA) speaking
2. **Transcript of the audio** - Required for full quality
3. **Audio format** - WAV/MP3 acceptable

**Audio Sources:**
- Chainsaw Man anime clips
- Voice actress promotional material
- Behind The Voice Actors database

**Considerations:**
- Japanese VA speaking English may work with explicit `language="English"` setting
- May need English-speaking Makima clips (Suzie Yeung, English dub VA) as fallback
- X-vector mode available if transcript is difficult to obtain

### 3.3 Integration Complexity

| Component | Complexity | Notes |
|-----------|------------|-------|
| Model Loading | Medium | Python subprocess or PyO3 bridge required |
| Streaming API | High | WebSocket integration needed |
| Voice Caching | Low | `create_voice_clone_prompt()` supports this |
| Audio Format | Low | Both use 24kHz output |
| ONNX Migration | N/A | **No ONNX export available** |

### 3.4 ONNX vs Python Inference

**Current approach (Chatterbox):** Rust + ONNX Runtime
- Pros: Native Rust, low latency, CPU-friendly
- Cons: Limited model ecosystem, no streaming

**Required approach (Qwen3-TTS):** Python + PyTorch
- Pros: Full model access, streaming support, GPU acceleration
- Cons: Python subprocess overhead, dependency management

**Integration Options:**

1. **Python Subprocess/Service**
   - Run `qwen-tts` as separate Python service
   - Communicate via HTTP/WebSocket
   - Cleanest separation, easiest to implement

2. **PyO3 Bridge**
   - Embed Python in Rust binary
   - Higher complexity, tighter integration
   - May have GIL contention issues

3. **Custom ONNX Export** (Future)
   - Not currently available
   - Would require community effort
   - No timeline from Qwen team

**Recommendation:** Python service with WebSocket streaming

---

## 4. Audio Clip Requirements

### 4.1 For Voice Cloning Setup

| Requirement | Specification |
|-------------|---------------|
| Minimum Duration | 3 seconds |
| Recommended Duration | 5-10 seconds |
| Format | WAV (preferred), MP3 |
| Sample Rate | Any (will be resampled) |
| Content | Clear speech, minimal background noise |
| Transcript | Required for full quality |

### 4.2 Makima Voice Sources

**Priority 1: Japanese VA (Tomori Kusunoki) speaking Japanese**
- Source: Chainsaw Man anime
- Challenge: Need clear dialogue without music/SFX
- Fallback: May not transfer well to English output

**Priority 2: English VA (Suzie Yeung)**
- Source: Chainsaw Man English dub
- Advantage: Native English speaker for English output
- Trade-off: Different vocal characteristics from Japanese VA

**Recommended Approach:**
1. Extract 5-10 second clips of both VAs
2. Test voice cloning quality with each
3. Select based on English speech naturalness
4. Store reference audio + transcript in `models/voices/makima/`

### 4.3 Transcript Requirements

For optimal voice cloning:
```
ref_audio: "models/voices/makima/makima-reference.wav"
ref_text: "The exact words spoken in the reference audio"
```

X-vector fallback (lower quality, no transcript needed):
```python
prompt = model.create_voice_clone_prompt(
    ref_audio="reference.wav",
    x_vector_only_mode=True,  # No transcript required
)
```

---

## 5. Preliminary Technical Approach

### 5.1 Architecture Overview

```
┌─────────────────────────────────────────────────────────────┐
│                    Makima Server (Rust)                     │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌──────────────────────┐│
│  │ Listen (STT)│  │ TTS Proxy   │  │ Chat/Contract APIs   ││
│  │ /api/v1/    │  │ /api/v1/tts │  │ /api/v1/...          ││
│  │ listen      │  │             │  │                      ││
│  └──────┬──────┘  └──────┬──────┘  └──────────────────────┘│
│         │                │                                  │
│         │         ┌──────▼──────┐                          │
│         │         │  WebSocket  │                          │
│         │         │   Bridge    │                          │
│         │         └──────┬──────┘                          │
└─────────┼────────────────┼──────────────────────────────────┘
          │                │
          │         ┌──────▼──────┐
          │         │ Python TTS  │
          │         │   Service   │
          │         │ (Qwen3-TTS) │
          │         └─────────────┘
          │
   ┌──────▼──────┐
   │  ML Models  │
   │ (Parakeet,  │
   │ Sortformer) │
   └─────────────┘
```

### 5.2 Python TTS Service

**Proposed Architecture:**

```python
# tts_service.py
import asyncio
from fastapi import FastAPI, WebSocket
from qwen_tts import Qwen3TTSModel

app = FastAPI()
model = None
voice_prompts = {}

@app.on_event("startup")
async def load_model():
    global model
    model = Qwen3TTSModel.from_pretrained(
        "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
        device_map="cuda:0",
        dtype=torch.bfloat16,
        attn_implementation="flash_attention_2",
    )

    # Pre-load Makima voice prompt
    voice_prompts["makima"] = model.create_voice_clone_prompt(
        ref_audio="models/voices/makima/reference.wav",
        ref_text="[Makima reference transcript]",
    )

@app.websocket("/ws/tts")
async def tts_stream(websocket: WebSocket):
    await websocket.accept()
    while True:
        data = await websocket.receive_json()
        text = data["text"]
        voice = data.get("voice", "makima")
        language = data.get("language", "English")

        # Generate with streaming (when available)
        prompt = voice_prompts.get(voice)
        wavs, sr = model.generate_voice_clone(
            text=text,
            language=language,
            voice_clone_prompt=prompt,
        )

        # Send audio chunks
        await websocket.send_bytes(wavs[0].tobytes())

@app.post("/api/tts")
async def tts_batch(request: TTSRequest):
    # Batch fallback for non-streaming use cases
    ...
```

### 5.3 Rust Integration

**New Endpoint: `/api/v1/tts`**

```rust
// server/handlers/tts.rs
pub async fn tts_websocket_handler(
    ws: WebSocketUpgrade,
    State(state): State<SharedState>,
) -> Response {
    ws.on_upgrade(|socket| handle_tts_socket(socket, state))
}

async fn handle_tts_socket(socket: WebSocket, state: SharedState) {
    // Proxy WebSocket to Python TTS service
    let tts_client = state.tts_client.lock().await;

    let (mut sender, mut receiver) = socket.split();

    while let Some(msg) = receiver.next().await {
        match msg {
            Ok(Message::Text(text)) => {
                // Forward to Python service
                let response = tts_client.generate(text).await;

                // Stream audio back
                for chunk in response.audio_chunks {
                    sender.send(Message::Binary(chunk)).await.ok();
                }
            }
            _ => {}
        }
    }
}
```

### 5.4 Voice Prompt Caching

```rust
// Pre-computed voice prompts stored in state
pub struct TtsConfig {
    pub default_voice: String,
    pub voices: HashMap<String, VoicePrompt>,
}

pub struct VoicePrompt {
    pub name: String,
    pub ref_audio_path: PathBuf,
    pub ref_text: String,
    pub language: String,
    // Cached prompt from Python service
    pub cached_prompt_id: Option<String>,
}
```

### 5.5 Message Protocol

**Client -> Server:**
```json
{
  "type": "synthesize",
  "text": "Hello, I am Makima.",
  "voice": "makima",
  "language": "English",
  "stream": true
}
```

**Server -> Client:**
```json
// Audio chunk
{"type": "audio", "data": "<base64 PCM>", "sample_rate": 24000, "final": false}

// Completion
{"type": "complete", "duration_ms": 1234}

// Error
{"type": "error", "code": "SYNTHESIS_ERROR", "message": "..."}
```

---

## 6. Implementation Phases

### Phase 1: Research & Setup (Current)
- [x] Analyze current TTS implementation
- [x] Research Qwen3-TTS capabilities
- [x] Document feasibility and approach
- [ ] Acquire Makima voice reference clips
- [ ] Test voice cloning quality

### Phase 2: Python Service
- [ ] Create Python TTS service with FastAPI
- [ ] Implement batch TTS endpoint
- [ ] Implement WebSocket streaming (when available)
- [ ] Add voice prompt management
- [ ] GPU optimization with FlashAttention 2

### Phase 3: Rust Integration
- [ ] Add TTS proxy endpoints to makima server
- [ ] WebSocket bridge implementation
- [ ] Voice configuration management
- [ ] Error handling and fallbacks

### Phase 4: Production Ready
- [ ] Health checks for Python service
- [ ] Voice prompt caching optimization
- [ ] Latency benchmarking
- [ ] Integration with Listen page

---

## 7. Open Questions

1. **Streaming API Availability**: When will official streaming support be released?
   - Fallback: Use community fork or batch with chunked responses

2. **Voice Quality**: How well does Japanese VA voice clone to English speech?
   - Action: Test with both Japanese and English VA samples

3. **GPU Requirements**: What's the minimum VRAM for 0.6B model?
   - Estimate: ~2-4GB with bf16 quantization

4. **Latency Target**: What's acceptable for "close to live" TTS?
   - Proposal: <500ms first audio, <100ms subsequent chunks

5. **Transcript Acquisition**: How to obtain accurate transcripts for voice clips?
   - Options: Manual transcription, Whisper ASR, community resources

---

## 8. References

- [Qwen3-TTS-12Hz-0.6B-Base (Hugging Face)](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base)
- [Qwen3-TTS GitHub Repository](https://github.com/QwenLM/Qwen3-TTS)
- [Qwen3-TTS Technical Report (arXiv)](https://arxiv.org/abs/2601.15621)
- [Streaming Inference Issue #77](https://github.com/QwenLM/Qwen3-TTS/issues/77)
- [Community Streaming Fork](https://github.com/dffdeeq/Qwen3-TTS-streaming)
- [Makima Voice Actors](https://www.behindthevoiceactors.com/characters/Chainsaw-Man/Makima/)
- [Chatterbox-Turbo-ONNX (Current Model)](https://huggingface.co/ResembleAI/chatterbox-turbo-ONNX)