Add Qwen3-TTS streaming endpoint for voice synthesis (#40)

* Task completion checkpoint

* Task completion checkpoint

* Task completion checkpoint

* Add Qwen3-TTS research document for live TTS replacement

Research findings for replacing Chatterbox TTS with Qwen3-TTS-12Hz-0.6B-Base:

- Current TTS: Chatterbox-Turbo-ONNX with batch-only generation, no streaming
- Qwen3-TTS: 97ms end-to-end latency, streaming support, 3-second voice cloning
- Voice cloning: Requires 3s reference audio + transcript (Makima voice planned)
- Integration: Python service with WebSocket bridge (no ONNX export available)
- Languages: 10 supported including English and Japanese

Document includes:
- Current architecture analysis (makima/src/tts.rs)
- Qwen3-TTS capabilities and requirements
- Feasibility assessment for live/streaming TTS
- Audio clip requirements for voice cloning
- Preliminary technical approach with architecture diagrams

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 03:11:15 UTC

* Add Qwen3-TTS research documentation

Comprehensive research on replacing Chatterbox TTS with Qwen3-TTS-12Hz-0.6B-Base:

- Current TTS implementation analysis (Chatterbox-Turbo-ONNX in makima/src/tts.rs)
- Qwen3-TTS capabilities: 97ms streaming latency, voice cloning with 3s reference
- Cross-lingual support: Japanese voice (Makima/Tomori Kusunoki) speaking English
- Python microservice architecture recommendation (FastAPI + WebSocket)
- Implementation phases and technical approach
- Hardware requirements and dependencies

Key findings:
- Live/streaming TTS is highly feasible with 97ms latency
- Voice cloning fully supported with 0.95 speaker similarity
- Recommended: Python microservice with WebSocket streaming

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add comprehensive Qwen3-TTS integration specification

This specification document defines the complete integration of
Qwen3-TTS-12Hz-0.6B-Base as a replacement for the existing Chatterbox-Turbo
TTS implementation. The document covers:

## Functional Requirements
- WebSocket endpoint /api/v1/speak for streaming TTS
- Voice cloning with default Makima voice (Japanese VA speaking English)
- Support for custom voice references
- Detailed client-to-server and server-to-client message protocols
- Integration with Listen page for bidirectional speech

## Non-Functional Requirements
- Latency targets: < 200ms first audio byte
- Audio quality: 24kHz, mono, PCM16/PCM32f
- Hardware requirements: CUDA GPU with 4-8GB VRAM
- Scalability: 10 concurrent sessions per GPU

## Architecture Specification
- Python TTS microservice with FastAPI/WebSocket
- Rust proxy endpoint in makima server
- Voice prompt caching mechanism (LRU cache)
- Error handling and recovery strategies

## API Contract
- Complete WebSocket message format definitions (TypeScript)
- Error codes and responses (TTS_UNAVAILABLE, SYNTHESIS_ERROR, etc.)
- Session state machine and lifecycle management

## Voice Asset Requirements
- Makima voice clip specifications (5-10s WAV, transcript required)
- Storage location: models/voices/makima/
- Metadata format for voice management

## Testing Strategy
- Unit tests for Python TTS service and Rust proxy
- Integration tests for WebSocket flow
- Latency benchmarks with performance targets
- Test data fixtures for various text lengths

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add Qwen3-TTS implementation plan

Comprehensive implementation plan for replacing Chatterbox-TTS with
Qwen3-TTS streaming TTS service, including:

- Task breakdown with estimated hours for each phase
- Phase 1: Python TTS microservice (FastAPI, WebSocket)
- Phase 2: Rust proxy integration (speak.rs, tts_client.rs)
- Detailed file changes and new module structure
- Testing plan with unit, integration, and latency benchmarks
- Risk assessment with mitigation strategies
- Success criteria for each phase

Based on specification in docs/specs/qwen3-tts-spec.md

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add author and research references to TTS implementation plan

Add links to research documentation and author attribution.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 03:25:06 UTC

* Add Python TTS service project structure (Phase 1.1-1.3)

Create the initial makima-tts Python service directory structure with:
- pyproject.toml with FastAPI, Qwen-TTS, and torch dependencies
- config.py with pydantic-settings TTSConfig class
- models.py with Pydantic message models (Start, Speak, Stop, Ready, etc.)

This implements tasks P1.1, P1.2, and P1.3 from the Qwen3-TTS implementation plan.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add TTS engine and voice manager for Qwen3-TTS (Phase 1.4-1.5)

Implement core TTS functionality:
- tts_engine.py: Qwen3-TTS wrapper with streaming audio chunk generation
- voice_manager.py: Voice prompt caching with LRU eviction and TTL support

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 03:30:06 UTC

* Add TTS proxy client and message types (Phase 2.1, 2.2, 2.4)

- Add tts_client.rs with TtsConfig, TtsCircuitBreaker, TtsError,
  TtsProxyClient, and TtsConnection structs for WebSocket proxying
- Add TTS message types to messages.rs (TtsAudioEncoding, TtsPriority,
  TtsStartMessage, TtsSpeakMessage, TtsStopMessage, TtsClientMessage,
  TtsReadyMessage, TtsAudioChunkMessage, TtsCompleteMessage,
  TtsErrorMessage, TtsStoppedMessage, TtsServerMessage)
- Export tts_client module from server mod.rs
- tokio-tungstenite already present in Cargo.toml

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add TTS WebSocket handler and route (Phase 2.3, 2.5, 2.6)

- Create speak.rs WebSocket handler that proxies to Python TTS service
- Add TtsState fields (tts_client, tts_config) to AppState
- Add with_tts() builder and is_tts_healthy() methods to AppState
- Register /api/v1/speak route in the router
- Add speak module export in handlers/mod.rs

The handler forwards WebSocket messages bidirectionally between
the client and the Python TTS microservice with proper error handling.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add Makima voice profile assets for TTS voice cloning

Creates the voice assets directory structure with:
- manifest.json containing voice configuration (voice_id, speaker,
  language, reference audio path, and Japanese transcript placeholder)
- README.md with instructions for obtaining voice reference audio

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add Rust-native Qwen3-TTS integration research document

Research findings for integrating Qwen3-TTS-12Hz-0.6B-Base directly into
the makima Rust codebase without Python. Key conclusions:
- ONNX export is not viable (unsupported architecture)
- Candle (HF Rust ML framework) is the recommended approach
- Model weights available in safetensors format (2.52GB total)
- Three components needed: LM backbone, code predictor, speech tokenizer
- Crane project has Qwen3-TTS as highest priority (potential upstream)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [WIP] Heartbeat checkpoint - 2026-01-27 11:21:43 UTC

* [WIP] Heartbeat checkpoint - 2026-01-27 11:24:19 UTC

* [WIP] Heartbeat checkpoint - 2026-01-27 11:26:43 UTC

* feat: implement Rust-native Qwen3-TTS using candle framework

Replace monolithic tts.rs with modular tts/ directory structure:

- tts/mod.rs: TtsEngine trait, TtsEngineFactory, shared types (AudioChunk,
  TtsError), and utility functions (save_wav, resample, argmax)
- tts/chatterbox.rs: existing ONNX-based ChatterboxTTS adapted to implement
  TtsEngine trait with Mutex-wrapped sessions for Send+Sync
- tts/qwen3/mod.rs: Qwen3Tts entry point with HuggingFace model loading
- tts/qwen3/config.rs: Qwen3TtsConfig parsing from HF config.json
- tts/qwen3/model.rs: 28-layer Qwen3 transformer with RoPE, GQA (16 heads,
  8 KV heads), SiLU MLP, RMS norm, and KV cache
- tts/qwen3/code_predictor.rs: 5-layer MTP module predicting 16 codebooks
- tts/qwen3/speech_tokenizer.rs: ConvNet encoder/decoder with 16-layer RVQ
- tts/qwen3/generate.rs: autoregressive generation loop with streaming support

Add candle-core, candle-nn, candle-transformers, safetensors to Cargo.toml.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: integrate TTS engine into speak WebSocket handler

- Update speak.rs handler to use TTS engine directly from SharedState
  instead of returning a stub "not implemented" error
- Add TtsEngine (OnceCell lazy-loaded) to AppState in state.rs with
  get_tts_engine() method for lazy initialization on first connection
- Implement full WebSocket protocol: client sends JSON speak/cancel/stop
  messages, server streams binary PCM audio chunks and audio_end signals
- Create voices/makima/manifest.json for Makima voice profile configuration
- All files compile successfully with zero errors

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add /speak TTS page with WebSocket audio playback

Add a new /speak frontend page for text-to-speech via WebSocket.
The page accepts text input and streams synthesized PCM audio through
the Web Audio API. Includes model loading indicator, cancel support,
and connection status. Also adds a loading bar to the listen page
ControlPanel during WebSocket connection.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

1 files changed, 281 insertions, 0 deletions

diff --git a/makima/src/tts/mod.rs b/makima/src/tts/mod.rs
new file mode 100644
index 0000000..2cd0412
--- /dev/null
+++ b/makima/src/tts/mod.rs
@@ -0,0 +1,281 @@
+//! TTS engine abstraction and implementations.
+//!
+//! Provides a trait-based TTS engine interface with two backends:
+//! - **Chatterbox**: ONNX-based TTS (legacy)
+//! - **Qwen3**: Pure Rust candle-based Qwen3-TTS-12Hz-0.6B
+
+use std::path::Path;
+
+pub mod chatterbox;
+pub mod qwen3;
+
+// Re-export primary types
+pub use chatterbox::ChatterboxTTS;
+pub use qwen3::Qwen3Tts;
+
+/// Audio output sample rate (both engines output 24kHz).
+pub const SAMPLE_RATE: u32 = 24_000;
+
+/// A chunk of generated audio for streaming output.
+#[derive(Debug, Clone)]
+pub struct AudioChunk {
+    /// PCM f32 samples in [-1.0, 1.0].
+    pub samples: Vec<f32>,
+    /// Sample rate (always 24000 for both engines).
+    pub sample_rate: u32,
+    /// Whether this is the final chunk in the stream.
+    pub is_final: bool,
+}
+
+impl AudioChunk {
+    /// Convert to 16-bit PCM bytes (little-endian) for WebSocket streaming.
+    pub fn to_pcm16_bytes(&self) -> Vec<u8> {
+        let mut buf = Vec::with_capacity(self.samples.len() * 2);
+        for &s in &self.samples {
+            let clamped = s.clamp(-1.0, 1.0);
+            let int_sample = (clamped * 32767.0) as i16;
+            buf.extend_from_slice(&int_sample.to_le_bytes());
+        }
+        buf
+    }
+}
+
+/// Errors that can occur during TTS operations.
+#[derive(Debug)]
+pub enum TtsError {
+    ModelLoad(String),
+    Inference(String),
+    Tokenizer(String),
+    Audio(crate::audio::AudioError),
+    Io(std::io::Error),
+    VoiceRequired,
+    Config(String),
+    Candle(String),
+}
+
+impl std::fmt::Display for TtsError {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            TtsError::ModelLoad(msg) => write!(f, "model load error: {msg}"),
+            TtsError::Inference(msg) => write!(f, "inference error: {msg}"),
+            TtsError::Tokenizer(msg) => write!(f, "tokenizer error: {msg}"),
+            TtsError::Audio(err) => write!(f, "audio error: {err}"),
+            TtsError::Io(err) => write!(f, "io error: {err}"),
+            TtsError::VoiceRequired => {
+                write!(f, "voice reference audio is required")
+            }
+            TtsError::Config(msg) => write!(f, "config error: {msg}"),
+            TtsError::Candle(msg) => write!(f, "candle error: {msg}"),
+        }
+    }
+}
+
+impl std::error::Error for TtsError {}
+
+impl From<crate::audio::AudioError> for TtsError {
+    fn from(value: crate::audio::AudioError) -> Self {
+        TtsError::Audio(value)
+    }
+}
+
+impl From<std::io::Error> for TtsError {
+    fn from(value: std::io::Error) -> Self {
+        TtsError::Io(value)
+    }
+}
+
+impl From<ort::Error> for TtsError {
+    fn from(value: ort::Error) -> Self {
+        TtsError::ModelLoad(value.to_string())
+    }
+}
+
+impl From<candle_core::Error> for TtsError {
+    fn from(value: candle_core::Error) -> Self {
+        TtsError::Candle(value.to_string())
+    }
+}
+
+/// Which TTS backend to use.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum TtsBackend {
+    /// ONNX-based Chatterbox TTS (legacy).
+    Chatterbox,
+    /// Candle-based Qwen3-TTS (preferred).
+    Qwen3,
+}
+
+/// TTS engine trait — implemented by both Chatterbox and Qwen3.
+#[async_trait::async_trait]
+pub trait TtsEngine: Send + Sync {
+    /// Generate complete audio from text with a voice reference.
+    async fn generate(
+        &self,
+        text: &str,
+        reference_audio: Option<&[f32]>,
+        reference_sample_rate: Option<u32>,
+    ) -> Result<Vec<AudioChunk>, TtsError>;
+
+    /// Check if the engine is loaded and ready.
+    fn is_ready(&self) -> bool;
+
+    /// Get the engine's output sample rate.
+    fn sample_rate(&self) -> u32 {
+        SAMPLE_RATE
+    }
+}
+
+/// Factory for creating TTS engines.
+pub struct TtsEngineFactory;
+
+impl TtsEngineFactory {
+    /// Create a TTS engine of the specified backend type.
+    pub fn create(backend: TtsBackend, model_dir: Option<&str>) -> Result<Box<dyn TtsEngine>, TtsError> {
+        match backend {
+            TtsBackend::Chatterbox => {
+                let engine = ChatterboxTTS::from_pretrained(model_dir)?;
+                Ok(Box::new(engine))
+            }
+            TtsBackend::Qwen3 => {
+                let device = candle_core::Device::Cpu; // Default to CPU; GPU selection happens at higher level
+                let engine = Qwen3Tts::from_pretrained(model_dir, &device)?;
+                Ok(Box::new(engine))
+            }
+        }
+    }
+}
+
+/// Save audio samples to a WAV file.
+pub fn save_wav(samples: &[f32], path: &Path) -> Result<(), TtsError> {
+    let mut file = std::fs::File::create(path)?;
+    write_wav(&mut file, samples, SAMPLE_RATE)?;
+    Ok(())
+}
+
+fn write_wav<W: std::io::Write>(
+    writer: &mut W,
+    samples: &[f32],
+    sample_rate: u32,
+) -> Result<(), std::io::Error> {
+    let num_samples = samples.len() as u32;
+    let num_channels: u16 = 1;
+    let bits_per_sample: u16 = 16;
+    let byte_rate = sample_rate * num_channels as u32 * bits_per_sample as u32 / 8;
+    let block_align = num_channels * bits_per_sample / 8;
+    let data_size = num_samples * num_channels as u32 * bits_per_sample as u32 / 8;
+    let file_size = 36 + data_size;
+
+    writer.write_all(b"RIFF")?;
+    writer.write_all(&file_size.to_le_bytes())?;
+    writer.write_all(b"WAVE")?;
+
+    writer.write_all(b"fmt ")?;
+    writer.write_all(&16u32.to_le_bytes())?;
+    writer.write_all(&1u16.to_le_bytes())?;
+    writer.write_all(&num_channels.to_le_bytes())?;
+    writer.write_all(&sample_rate.to_le_bytes())?;
+    writer.write_all(&byte_rate.to_le_bytes())?;
+    writer.write_all(&block_align.to_le_bytes())?;
+    writer.write_all(&bits_per_sample.to_le_bytes())?;
+
+    writer.write_all(b"data")?;
+    writer.write_all(&data_size.to_le_bytes())?;
+
+    for &sample in samples {
+        let clamped = sample.clamp(-1.0, 1.0);
+        let int_sample = (clamped * 32767.0) as i16;
+        writer.write_all(&int_sample.to_le_bytes())?;
+    }
+
+    Ok(())
+}
+
+/// Resample audio to 24kHz using simple linear interpolation.
+pub fn resample_to_24k(samples: &[f32], input_rate: u32) -> Vec<f32> {
+    if input_rate == SAMPLE_RATE {
+        return samples.to_vec();
+    }
+    if samples.is_empty() {
+        return Vec::new();
+    }
+
+    let ratio = input_rate as f64 / SAMPLE_RATE as f64;
+    let output_len = ((samples.len() as f64) / ratio).ceil() as usize;
+
+    let mut output = Vec::with_capacity(output_len);
+    for i in 0..output_len {
+        let src_idx = (i as f64 * ratio) as usize;
+        let sample = samples.get(src_idx).copied().unwrap_or(0.0);
+        output.push(sample);
+    }
+
+    output
+}
+
+/// Apply repetition penalty to logits based on previously generated tokens.
+pub fn apply_repetition_penalty(logits: &mut [f32], generated: &[i64], penalty: f32) {
+    for &token in generated {
+        if (token as usize) < logits.len() {
+            let score = logits[token as usize];
+            logits[token as usize] = if score < 0.0 {
+                score * penalty
+            } else {
+                score / penalty
+            };
+        }
+    }
+}
+
+/// Return the index of the maximum value in logits.
+pub fn argmax(logits: &[f32]) -> i64 {
+    logits
+        .iter()
+        .enumerate()
+        .max_by(|(_, a), (_, b)| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal))
+        .map(|(idx, _)| idx as i64)
+        .unwrap_or(0)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_argmax() {
+        let logits = vec![0.1, 0.5, 0.3, 0.8, 0.2];
+        assert_eq!(argmax(&logits), 3);
+    }
+
+    #[test]
+    fn test_resample_same_rate() {
+        let samples = vec![0.1, 0.2, 0.3];
+        let resampled = resample_to_24k(&samples, SAMPLE_RATE);
+        assert_eq!(resampled, samples);
+    }
+
+    #[test]
+    fn test_repetition_penalty() {
+        let mut logits = vec![1.0, 2.0, 3.0, 4.0];
+        let generated = vec![1, 3];
+        apply_repetition_penalty(&mut logits, &generated, 1.2);
+        assert!((logits[1] - 2.0 / 1.2).abs() < 1e-6);
+        assert!((logits[3] - 4.0 / 1.2).abs() < 1e-6);
+    }
+
+    #[test]
+    fn test_audio_chunk_to_pcm16() {
+        let chunk = AudioChunk {
+            samples: vec![0.0, 1.0, -1.0],
+            sample_rate: 24_000,
+            is_final: true,
+        };
+        let bytes = chunk.to_pcm16_bytes();
+        assert_eq!(bytes.len(), 6);
+        // 0.0 -> 0i16
+        assert_eq!(i16::from_le_bytes([bytes[0], bytes[1]]), 0);
+        // 1.0 -> 32767i16
+        assert_eq!(i16::from_le_bytes([bytes[2], bytes[3]]), 32767);
+        // -1.0 -> -32767i16
+        assert_eq!(i16::from_le_bytes([bytes[4], bytes[5]]), -32767);
+    }
+}