summaryrefslogtreecommitdiff
path: root/makima/frontend/src
diff options
context:
space:
mode:
authorsoryu <soryu@soryu.co>2026-01-28 02:54:17 +0000
committerGitHub <noreply@github.com>2026-01-28 02:54:17 +0000
commiteabd1304cce0e053cd32ec910d2f0ea429e8af14 (patch)
treefca3b08810a1dc0c0c610a8189a466cc23d5c547 /makima/frontend/src
parentc618174e60e4632d36d7352d83399508c72b2f42 (diff)
downloadsoryu-eabd1304cce0e053cd32ec910d2f0ea429e8af14.tar.gz
soryu-eabd1304cce0e053cd32ec910d2f0ea429e8af14.zip
Add Qwen3-TTS streaming endpoint for voice synthesis (#40)
* Task completion checkpoint * Task completion checkpoint * Task completion checkpoint * Add Qwen3-TTS research document for live TTS replacement Research findings for replacing Chatterbox TTS with Qwen3-TTS-12Hz-0.6B-Base: - Current TTS: Chatterbox-Turbo-ONNX with batch-only generation, no streaming - Qwen3-TTS: 97ms end-to-end latency, streaming support, 3-second voice cloning - Voice cloning: Requires 3s reference audio + transcript (Makima voice planned) - Integration: Python service with WebSocket bridge (no ONNX export available) - Languages: 10 supported including English and Japanese Document includes: - Current architecture analysis (makima/src/tts.rs) - Qwen3-TTS capabilities and requirements - Feasibility assessment for live/streaming TTS - Audio clip requirements for voice cloning - Preliminary technical approach with architecture diagrams Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * [WIP] Heartbeat checkpoint - 2026-01-27 03:11:15 UTC * Add Qwen3-TTS research documentation Comprehensive research on replacing Chatterbox TTS with Qwen3-TTS-12Hz-0.6B-Base: - Current TTS implementation analysis (Chatterbox-Turbo-ONNX in makima/src/tts.rs) - Qwen3-TTS capabilities: 97ms streaming latency, voice cloning with 3s reference - Cross-lingual support: Japanese voice (Makima/Tomori Kusunoki) speaking English - Python microservice architecture recommendation (FastAPI + WebSocket) - Implementation phases and technical approach - Hardware requirements and dependencies Key findings: - Live/streaming TTS is highly feasible with 97ms latency - Voice cloning fully supported with 0.95 speaker similarity - Recommended: Python microservice with WebSocket streaming Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add comprehensive Qwen3-TTS integration specification This specification document defines the complete integration of Qwen3-TTS-12Hz-0.6B-Base as a replacement for the existing Chatterbox-Turbo TTS implementation. The document covers: ## Functional Requirements - WebSocket endpoint /api/v1/speak for streaming TTS - Voice cloning with default Makima voice (Japanese VA speaking English) - Support for custom voice references - Detailed client-to-server and server-to-client message protocols - Integration with Listen page for bidirectional speech ## Non-Functional Requirements - Latency targets: < 200ms first audio byte - Audio quality: 24kHz, mono, PCM16/PCM32f - Hardware requirements: CUDA GPU with 4-8GB VRAM - Scalability: 10 concurrent sessions per GPU ## Architecture Specification - Python TTS microservice with FastAPI/WebSocket - Rust proxy endpoint in makima server - Voice prompt caching mechanism (LRU cache) - Error handling and recovery strategies ## API Contract - Complete WebSocket message format definitions (TypeScript) - Error codes and responses (TTS_UNAVAILABLE, SYNTHESIS_ERROR, etc.) - Session state machine and lifecycle management ## Voice Asset Requirements - Makima voice clip specifications (5-10s WAV, transcript required) - Storage location: models/voices/makima/ - Metadata format for voice management ## Testing Strategy - Unit tests for Python TTS service and Rust proxy - Integration tests for WebSocket flow - Latency benchmarks with performance targets - Test data fixtures for various text lengths Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add Qwen3-TTS implementation plan Comprehensive implementation plan for replacing Chatterbox-TTS with Qwen3-TTS streaming TTS service, including: - Task breakdown with estimated hours for each phase - Phase 1: Python TTS microservice (FastAPI, WebSocket) - Phase 2: Rust proxy integration (speak.rs, tts_client.rs) - Detailed file changes and new module structure - Testing plan with unit, integration, and latency benchmarks - Risk assessment with mitigation strategies - Success criteria for each phase Based on specification in docs/specs/qwen3-tts-spec.md Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add author and research references to TTS implementation plan Add links to research documentation and author attribution. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * [WIP] Heartbeat checkpoint - 2026-01-27 03:25:06 UTC * Add Python TTS service project structure (Phase 1.1-1.3) Create the initial makima-tts Python service directory structure with: - pyproject.toml with FastAPI, Qwen-TTS, and torch dependencies - config.py with pydantic-settings TTSConfig class - models.py with Pydantic message models (Start, Speak, Stop, Ready, etc.) This implements tasks P1.1, P1.2, and P1.3 from the Qwen3-TTS implementation plan. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add TTS engine and voice manager for Qwen3-TTS (Phase 1.4-1.5) Implement core TTS functionality: - tts_engine.py: Qwen3-TTS wrapper with streaming audio chunk generation - voice_manager.py: Voice prompt caching with LRU eviction and TTL support Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * [WIP] Heartbeat checkpoint - 2026-01-27 03:30:06 UTC * Add TTS proxy client and message types (Phase 2.1, 2.2, 2.4) - Add tts_client.rs with TtsConfig, TtsCircuitBreaker, TtsError, TtsProxyClient, and TtsConnection structs for WebSocket proxying - Add TTS message types to messages.rs (TtsAudioEncoding, TtsPriority, TtsStartMessage, TtsSpeakMessage, TtsStopMessage, TtsClientMessage, TtsReadyMessage, TtsAudioChunkMessage, TtsCompleteMessage, TtsErrorMessage, TtsStoppedMessage, TtsServerMessage) - Export tts_client module from server mod.rs - tokio-tungstenite already present in Cargo.toml Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add TTS WebSocket handler and route (Phase 2.3, 2.5, 2.6) - Create speak.rs WebSocket handler that proxies to Python TTS service - Add TtsState fields (tts_client, tts_config) to AppState - Add with_tts() builder and is_tts_healthy() methods to AppState - Register /api/v1/speak route in the router - Add speak module export in handlers/mod.rs The handler forwards WebSocket messages bidirectionally between the client and the Python TTS microservice with proper error handling. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add Makima voice profile assets for TTS voice cloning Creates the voice assets directory structure with: - manifest.json containing voice configuration (voice_id, speaker, language, reference audio path, and Japanese transcript placeholder) - README.md with instructions for obtaining voice reference audio Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add Rust-native Qwen3-TTS integration research document Research findings for integrating Qwen3-TTS-12Hz-0.6B-Base directly into the makima Rust codebase without Python. Key conclusions: - ONNX export is not viable (unsupported architecture) - Candle (HF Rust ML framework) is the recommended approach - Model weights available in safetensors format (2.52GB total) - Three components needed: LM backbone, code predictor, speech tokenizer - Crane project has Qwen3-TTS as highest priority (potential upstream) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * [WIP] Heartbeat checkpoint - 2026-01-27 11:21:43 UTC * [WIP] Heartbeat checkpoint - 2026-01-27 11:24:19 UTC * [WIP] Heartbeat checkpoint - 2026-01-27 11:26:43 UTC * feat: implement Rust-native Qwen3-TTS using candle framework Replace monolithic tts.rs with modular tts/ directory structure: - tts/mod.rs: TtsEngine trait, TtsEngineFactory, shared types (AudioChunk, TtsError), and utility functions (save_wav, resample, argmax) - tts/chatterbox.rs: existing ONNX-based ChatterboxTTS adapted to implement TtsEngine trait with Mutex-wrapped sessions for Send+Sync - tts/qwen3/mod.rs: Qwen3Tts entry point with HuggingFace model loading - tts/qwen3/config.rs: Qwen3TtsConfig parsing from HF config.json - tts/qwen3/model.rs: 28-layer Qwen3 transformer with RoPE, GQA (16 heads, 8 KV heads), SiLU MLP, RMS norm, and KV cache - tts/qwen3/code_predictor.rs: 5-layer MTP module predicting 16 codebooks - tts/qwen3/speech_tokenizer.rs: ConvNet encoder/decoder with 16-layer RVQ - tts/qwen3/generate.rs: autoregressive generation loop with streaming support Add candle-core, candle-nn, candle-transformers, safetensors to Cargo.toml. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: integrate TTS engine into speak WebSocket handler - Update speak.rs handler to use TTS engine directly from SharedState instead of returning a stub "not implemented" error - Add TtsEngine (OnceCell lazy-loaded) to AppState in state.rs with get_tts_engine() method for lazy initialization on first connection - Implement full WebSocket protocol: client sends JSON speak/cancel/stop messages, server streams binary PCM audio chunks and audio_end signals - Create voices/makima/manifest.json for Makima voice profile configuration - All files compile successfully with zero errors Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: add /speak TTS page with WebSocket audio playback Add a new /speak frontend page for text-to-speech via WebSocket. The page accepts text input and streams synthesized PCM audio through the Web Audio API. Includes model loading indicator, cancel support, and connection status. Also adds a loading bar to the listen page ControlPanel during WebSocket connection. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Diffstat (limited to 'makima/frontend/src')
-rw-r--r--makima/frontend/src/components/listen/ControlPanel.tsx36
-rw-r--r--makima/frontend/src/hooks/useSpeakWebSocket.ts329
-rw-r--r--makima/frontend/src/index.css6
-rw-r--r--makima/frontend/src/lib/api.ts1
-rw-r--r--makima/frontend/src/main.tsx9
-rw-r--r--makima/frontend/src/routes/listen.tsx1
-rw-r--r--makima/frontend/src/routes/speak.tsx159
7 files changed, 534 insertions, 7 deletions
diff --git a/makima/frontend/src/components/listen/ControlPanel.tsx b/makima/frontend/src/components/listen/ControlPanel.tsx
index f0e5702..f482ec4 100644
--- a/makima/frontend/src/components/listen/ControlPanel.tsx
+++ b/makima/frontend/src/components/listen/ControlPanel.tsx
@@ -1,6 +1,7 @@
import { useState } from "react";
import { Logo } from "../Logo";
import type { MicrophoneStatus } from "../../hooks/useMicrophone";
+import type { ConnectionStatus } from "../../hooks/useWebSocket";
import { ContractPickerModal } from "./ContractPickerModal";
export interface ContractOption {
@@ -22,6 +23,8 @@ interface ControlPanelProps {
selectedContractId: string | null;
onContractChange: (contractId: string | null) => void;
contractsLoading?: boolean;
+ // Connection status for loading state
+ connectionStatus?: ConnectionStatus;
}
function getStatusText(isListening: boolean, micStatus: MicrophoneStatus): string {
@@ -54,6 +57,7 @@ export function ControlPanel({
selectedContractId,
onContractChange,
contractsLoading,
+ connectionStatus,
}: ControlPanelProps) {
const [isModalOpen, setIsModalOpen] = useState(false);
const statusText = getStatusText(isListening, micStatus);
@@ -121,18 +125,36 @@ export function ControlPanel({
{/* Connection status */}
<div
- className={`inline-flex items-center gap-1.5 px-2 py-1 border ${
+ className={`inline-flex flex-col gap-1 px-2 py-1 border ${
isConnected
? "border-[#3f6fb3] text-[#75aafc]"
+ : connectionStatus === "connecting"
+ ? "border-[#3f6fb3] text-[#9bc3ff]"
: "border-[rgba(117,170,252,0.25)] text-[#9bc3ff]"
}`}
>
- <span
- className={`w-1.5 h-1.5 rounded-full ${
- isConnected ? "bg-[#75aafc]" : "bg-[#3f6fb3]"
- }`}
- />
- {isConnected ? "CONNECTED" : "DISCONNECTED"}
+ <div className="inline-flex items-center gap-1.5">
+ <span
+ className={`w-1.5 h-1.5 rounded-full ${
+ isConnected ? "bg-[#75aafc]" : "bg-[#3f6fb3]"
+ }`}
+ />
+ {isConnected
+ ? "CONNECTED"
+ : connectionStatus === "connecting"
+ ? "LOADING MODELS..."
+ : "DISCONNECTED"}
+ </div>
+ {connectionStatus === "connecting" && (
+ <div className="w-full h-1.5 bg-[#0f1c2f] overflow-hidden">
+ <div
+ className="h-full w-1/3 bg-[#75aafc]"
+ style={{
+ animation: "loading-slide 1.5s ease-in-out infinite",
+ }}
+ />
+ </div>
+ )}
</div>
</div>
diff --git a/makima/frontend/src/hooks/useSpeakWebSocket.ts b/makima/frontend/src/hooks/useSpeakWebSocket.ts
new file mode 100644
index 0000000..3ef8851
--- /dev/null
+++ b/makima/frontend/src/hooks/useSpeakWebSocket.ts
@@ -0,0 +1,329 @@
+import { useState, useCallback, useRef, useEffect } from "react";
+import { SPEAK_ENDPOINT } from "../lib/api";
+
+export type SpeakStatus =
+ | "disconnected"
+ | "connecting"
+ | "connected"
+ | "loading_model"
+ | "speaking"
+ | "error";
+
+export interface SpeakWebSocketState {
+ status: SpeakStatus;
+ error: string | null;
+}
+
+export function useSpeakWebSocket() {
+ const [state, setState] = useState<SpeakWebSocketState>({
+ status: "disconnected",
+ error: null,
+ });
+
+ const wsRef = useRef<WebSocket | null>(null);
+ const audioContextRef = useRef<AudioContext | null>(null);
+ const audioQueueRef = useRef<Float32Array[]>([]);
+ const isPlayingRef = useRef(false);
+ const modelLoadingTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
+ const nextPlayTimeRef = useRef(0);
+
+ // Clean up on unmount
+ useEffect(() => {
+ return () => {
+ if (wsRef.current) {
+ wsRef.current.close();
+ wsRef.current = null;
+ }
+ if (audioContextRef.current) {
+ audioContextRef.current.close();
+ audioContextRef.current = null;
+ }
+ if (modelLoadingTimerRef.current) {
+ clearTimeout(modelLoadingTimerRef.current);
+ modelLoadingTimerRef.current = null;
+ }
+ };
+ }, []);
+
+ const getAudioContext = useCallback((): AudioContext => {
+ if (!audioContextRef.current || audioContextRef.current.state === "closed") {
+ audioContextRef.current = new AudioContext({ sampleRate: 24000 });
+ }
+ return audioContextRef.current;
+ }, []);
+
+ const playAudioQueue = useCallback(() => {
+ if (isPlayingRef.current) return;
+ isPlayingRef.current = true;
+
+ const ctx = getAudioContext();
+
+ function scheduleNext() {
+ const chunk = audioQueueRef.current.shift();
+ if (!chunk) {
+ isPlayingRef.current = false;
+ return;
+ }
+
+ const buffer = ctx.createBuffer(1, chunk.length, 24000);
+ buffer.copyToChannel(chunk, 0);
+
+ const source = ctx.createBufferSource();
+ source.buffer = buffer;
+ source.connect(ctx.destination);
+
+ // Schedule playback at the right time to avoid gaps
+ const now = ctx.currentTime;
+ const startTime = Math.max(now, nextPlayTimeRef.current);
+ source.start(startTime);
+ nextPlayTimeRef.current = startTime + buffer.duration;
+
+ source.onended = () => {
+ if (audioQueueRef.current.length > 0) {
+ scheduleNext();
+ } else {
+ isPlayingRef.current = false;
+ }
+ };
+ }
+
+ scheduleNext();
+ }, [getAudioContext]);
+
+ const connect = useCallback((): Promise<boolean> => {
+ return new Promise((resolve) => {
+ if (wsRef.current?.readyState === WebSocket.OPEN) {
+ resolve(true);
+ return;
+ }
+
+ if (wsRef.current) {
+ wsRef.current.close();
+ wsRef.current = null;
+ }
+
+ setState({ status: "connecting", error: null });
+
+ try {
+ const ws = new WebSocket(SPEAK_ENDPOINT);
+ ws.binaryType = "arraybuffer";
+ wsRef.current = ws;
+
+ ws.onopen = () => {
+ setState({ status: "connected", error: null });
+ resolve(true);
+ };
+
+ ws.onmessage = (event) => {
+ // Binary data = PCM audio chunk
+ if (event.data instanceof ArrayBuffer) {
+ // Clear model loading timer on first audio data
+ if (modelLoadingTimerRef.current) {
+ clearTimeout(modelLoadingTimerRef.current);
+ modelLoadingTimerRef.current = null;
+ }
+
+ // Update status to speaking if not already
+ setState((s) => {
+ if (s.status === "loading_model" || s.status === "connected") {
+ return { ...s, status: "speaking" };
+ }
+ return s;
+ });
+
+ // Convert PCM16 LE to Float32
+ const pcm16 = new Int16Array(event.data);
+ const float32 = new Float32Array(pcm16.length);
+ for (let i = 0; i < pcm16.length; i++) {
+ float32[i] = pcm16[i] / 32768;
+ }
+
+ audioQueueRef.current.push(float32);
+ playAudioQueue();
+ return;
+ }
+
+ // Text data = JSON message
+ try {
+ const message = JSON.parse(event.data);
+
+ switch (message.type) {
+ case "audio_end":
+ // Clear model loading timer
+ if (modelLoadingTimerRef.current) {
+ clearTimeout(modelLoadingTimerRef.current);
+ modelLoadingTimerRef.current = null;
+ }
+ // Wait for audio queue to drain, then go back to connected
+ // Use a short delay to let buffered audio finish
+ {
+ const checkDone = () => {
+ if (audioQueueRef.current.length === 0 && !isPlayingRef.current) {
+ setState((s) => {
+ if (s.status === "speaking" || s.status === "loading_model") {
+ return { ...s, status: "connected" };
+ }
+ return s;
+ });
+ } else {
+ setTimeout(checkDone, 100);
+ }
+ };
+ checkDone();
+ }
+ break;
+
+ case "error":
+ if (modelLoadingTimerRef.current) {
+ clearTimeout(modelLoadingTimerRef.current);
+ modelLoadingTimerRef.current = null;
+ }
+ setState({
+ status: "error",
+ error: message.message || `Error: ${message.code}`,
+ });
+ break;
+ }
+ } catch {
+ console.error("Failed to parse speak WebSocket message:", event.data);
+ }
+ };
+
+ ws.onerror = () => {
+ setState({
+ status: "error",
+ error: "Failed to connect to speak server",
+ });
+ resolve(false);
+ };
+
+ ws.onclose = (event) => {
+ if (modelLoadingTimerRef.current) {
+ clearTimeout(modelLoadingTimerRef.current);
+ modelLoadingTimerRef.current = null;
+ }
+
+ let errorMessage: string | null = null;
+ if (event.code === 1006) {
+ errorMessage = "Connection failed - server may be unavailable";
+ } else if (event.code !== 1000 && event.code !== 1001) {
+ errorMessage = `Connection closed unexpectedly (code: ${event.code})`;
+ }
+
+ setState((s) => ({
+ status: "disconnected",
+ error: errorMessage || s.error,
+ }));
+ wsRef.current = null;
+ };
+ } catch (err) {
+ const message =
+ err instanceof Error ? err.message : "Failed to create WebSocket connection";
+ setState({ status: "error", error: message });
+ resolve(false);
+ }
+ });
+ }, [playAudioQueue]);
+
+ const speak = useCallback(
+ async (text: string) => {
+ if (!text.trim()) return;
+
+ // Connect if not connected
+ if (!wsRef.current || wsRef.current.readyState !== WebSocket.OPEN) {
+ const connected = await connect();
+ if (!connected) return;
+ }
+
+ // Reset audio state
+ audioQueueRef.current = [];
+ isPlayingRef.current = false;
+ nextPlayTimeRef.current = 0;
+
+ // Resume audio context if suspended (browser autoplay policy)
+ const ctx = getAudioContext();
+ if (ctx.state === "suspended") {
+ await ctx.resume();
+ }
+
+ // Start loading timer - if no audio arrives in 2 seconds, show loading state
+ modelLoadingTimerRef.current = setTimeout(() => {
+ setState((s) => {
+ if (s.status === "connected" || s.status === "connecting") {
+ return { ...s, status: "loading_model" };
+ }
+ return s;
+ });
+ modelLoadingTimerRef.current = null;
+ }, 2000);
+
+ // Send speak request
+ wsRef.current?.send(
+ JSON.stringify({ type: "speak", text })
+ );
+
+ setState((s) => ({ ...s, error: null }));
+ },
+ [connect, getAudioContext]
+ );
+
+ const cancel = useCallback(() => {
+ // Clear audio queue
+ audioQueueRef.current = [];
+ isPlayingRef.current = false;
+ nextPlayTimeRef.current = 0;
+
+ // Clear model loading timer
+ if (modelLoadingTimerRef.current) {
+ clearTimeout(modelLoadingTimerRef.current);
+ modelLoadingTimerRef.current = null;
+ }
+
+ // Send cancel message
+ if (wsRef.current?.readyState === WebSocket.OPEN) {
+ wsRef.current.send(JSON.stringify({ type: "cancel" }));
+ }
+
+ setState((s) => ({
+ ...s,
+ status: wsRef.current?.readyState === WebSocket.OPEN ? "connected" : "disconnected",
+ }));
+ }, []);
+
+ const disconnect = useCallback(() => {
+ // Clear audio queue
+ audioQueueRef.current = [];
+ isPlayingRef.current = false;
+ nextPlayTimeRef.current = 0;
+
+ if (modelLoadingTimerRef.current) {
+ clearTimeout(modelLoadingTimerRef.current);
+ modelLoadingTimerRef.current = null;
+ }
+
+ if (wsRef.current) {
+ // Send stop message before closing
+ if (wsRef.current.readyState === WebSocket.OPEN) {
+ wsRef.current.send(JSON.stringify({ type: "stop" }));
+ }
+ wsRef.current.close(1000, "User disconnected");
+ wsRef.current = null;
+ }
+
+ setState({ status: "disconnected", error: null });
+ }, []);
+
+ return {
+ ...state,
+ isConnected:
+ state.status === "connected" ||
+ state.status === "speaking" ||
+ state.status === "loading_model",
+ isSpeaking: state.status === "speaking",
+ isModelLoading: state.status === "loading_model",
+ speak,
+ cancel,
+ connect,
+ disconnect,
+ };
+}
diff --git a/makima/frontend/src/index.css b/makima/frontend/src/index.css
index 5c08006..f29873b 100644
--- a/makima/frontend/src/index.css
+++ b/makima/frontend/src/index.css
@@ -64,6 +64,12 @@ body {
background: rgba(117, 170, 252, 0.35);
}
+/* Loading bar animation for indeterminate progress */
+@keyframes loading-slide {
+ 0% { transform: translateX(-100%); }
+ 100% { transform: translateX(300%); }
+}
+
/* Grid overlay */
.grid-overlay {
position: fixed;
diff --git a/makima/frontend/src/lib/api.ts b/makima/frontend/src/lib/api.ts
index 4390b20..ca04ce7 100644
--- a/makima/frontend/src/lib/api.ts
+++ b/makima/frontend/src/lib/api.ts
@@ -99,6 +99,7 @@ async function authFetch(url: string, options: RequestInit = {}): Promise<Respon
});
}
export const LISTEN_ENDPOINT = `${WS_BASE}/api/v1/listen`;
+export const SPEAK_ENDPOINT = `${WS_BASE}/api/v1/speak`;
export const FILE_SUBSCRIBE_ENDPOINT = `${WS_BASE}/api/v1/files/subscribe`;
export const TASK_SUBSCRIBE_ENDPOINT = `${WS_BASE}/api/v1/mesh/tasks/subscribe`;
diff --git a/makima/frontend/src/main.tsx b/makima/frontend/src/main.tsx
index 383b732..ef1ba5c 100644
--- a/makima/frontend/src/main.tsx
+++ b/makima/frontend/src/main.tsx
@@ -19,6 +19,7 @@ import LoginPage from "./routes/login";
import SettingsPage from "./routes/settings";
import ContractFilePage from "./routes/contract-file";
import TemplatesPage from "./routes/templates";
+import SpeakPage from "./routes/speak";
createRoot(document.getElementById("root")!).render(
<StrictMode>
@@ -135,6 +136,14 @@ createRoot(document.getElementById("root")!).render(
</ProtectedRoute>
}
/>
+ <Route
+ path="/speak"
+ element={
+ <ProtectedRoute>
+ <SpeakPage />
+ </ProtectedRoute>
+ }
+ />
</Routes>
</BrowserRouter>
</SupervisorQuestionsProvider>
diff --git a/makima/frontend/src/routes/listen.tsx b/makima/frontend/src/routes/listen.tsx
index 55cf7e6..8af538e 100644
--- a/makima/frontend/src/routes/listen.tsx
+++ b/makima/frontend/src/routes/listen.tsx
@@ -207,6 +207,7 @@ export default function ListenPage() {
selectedContractId={selectedContractId}
onContractChange={setSelectedContractId}
contractsLoading={contractsLoading}
+ connectionStatus={ws.status}
/>
</div>
</main>
diff --git a/makima/frontend/src/routes/speak.tsx b/makima/frontend/src/routes/speak.tsx
new file mode 100644
index 0000000..c4692ff
--- /dev/null
+++ b/makima/frontend/src/routes/speak.tsx
@@ -0,0 +1,159 @@
+import { useState, useCallback } from "react";
+import { Masthead } from "../components/Masthead";
+import { useSpeakWebSocket } from "../hooks/useSpeakWebSocket";
+
+export default function SpeakPage() {
+ const [text, setText] = useState("");
+ const tts = useSpeakWebSocket();
+
+ const handleSpeak = useCallback(() => {
+ if (!text.trim()) return;
+ tts.speak(text);
+ }, [text, tts]);
+
+ const handleCancel = useCallback(() => {
+ tts.cancel();
+ }, [tts]);
+
+ const handleKeyDown = useCallback(
+ (e: React.KeyboardEvent<HTMLTextAreaElement>) => {
+ // Ctrl/Cmd + Enter to speak
+ if ((e.ctrlKey || e.metaKey) && e.key === "Enter") {
+ e.preventDefault();
+ handleSpeak();
+ }
+ },
+ [handleSpeak]
+ );
+
+ const statusLabel = (() => {
+ switch (tts.status) {
+ case "disconnected":
+ return "DISCONNECTED";
+ case "connecting":
+ return "CONNECTING...";
+ case "connected":
+ return "CONNECTED";
+ case "loading_model":
+ return "LOADING TTS MODEL...";
+ case "speaking":
+ return "SPEAKING";
+ case "error":
+ return "ERROR";
+ default:
+ return "IDLE";
+ }
+ })();
+
+ const statusColor = (() => {
+ switch (tts.status) {
+ case "connected":
+ case "speaking":
+ return "border-[#3f6fb3] text-[#75aafc]";
+ case "error":
+ return "border-red-400/50 text-red-400";
+ default:
+ return "border-[rgba(117,170,252,0.25)] text-[#9bc3ff]";
+ }
+ })();
+
+ const dotColor = (() => {
+ switch (tts.status) {
+ case "connected":
+ case "speaking":
+ return "bg-[#75aafc]";
+ case "error":
+ return "bg-red-400";
+ default:
+ return "bg-[#3f6fb3]";
+ }
+ })();
+
+ return (
+ <div className="relative z-10 h-screen flex flex-col overflow-hidden">
+ <Masthead showTicker={false} showNav />
+
+ <main className="flex-1 flex flex-col items-center justify-center p-4 md:p-8 gap-6 min-h-0 overflow-auto">
+ {/* Text input area */}
+ <div className="w-full max-w-2xl">
+ <textarea
+ value={text}
+ onChange={(e) => setText(e.target.value)}
+ onKeyDown={handleKeyDown}
+ placeholder="Enter text to speak..."
+ disabled={tts.isSpeaking || tts.isModelLoading}
+ className="w-full h-48 p-4 font-mono text-sm text-[#dbe7ff] bg-[#0d1b2d] border border-[#0f3c78] focus:border-[#3f6fb3] focus:outline-none placeholder-[#3f6fb3] resize-none transition-colors disabled:opacity-50"
+ />
+ <div className="mt-1 text-right font-mono text-xs text-[#3f6fb3]">
+ Ctrl+Enter to speak
+ </div>
+ </div>
+
+ {/* Controls row */}
+ <div className="w-full max-w-2xl flex items-center gap-4">
+ {/* Speak / Cancel button */}
+ {tts.isSpeaking || tts.isModelLoading ? (
+ <button
+ onClick={handleCancel}
+ className="px-6 py-2 font-mono text-sm text-red-400 bg-[#0d1b2d] border border-red-400/50 hover:border-red-400 transition-colors uppercase tracking-wide"
+ >
+ Cancel
+ </button>
+ ) : (
+ <button
+ onClick={handleSpeak}
+ disabled={!text.trim()}
+ className="px-6 py-2 font-mono text-sm text-[#dbe7ff] bg-[#0d1b2d] border border-[#0f3c78] hover:border-[#3f6fb3] transition-colors uppercase tracking-wide disabled:opacity-50 disabled:cursor-not-allowed"
+ >
+ Speak
+ </button>
+ )}
+
+ {/* Status indicator */}
+ <div
+ className={`inline-flex items-center gap-1.5 px-2 py-1 border font-mono text-xs tracking-wide uppercase ${statusColor}`}
+ >
+ <span className={`w-1.5 h-1.5 rounded-full ${dotColor}`} />
+ {statusLabel}
+ </div>
+ </div>
+
+ {/* Loading bar (indeterminate) */}
+ {tts.isModelLoading && (
+ <div className="w-full max-w-2xl">
+ <div className="w-full h-1.5 bg-[#0f1c2f] overflow-hidden">
+ <div
+ className="h-full w-1/3 bg-[#75aafc]"
+ style={{
+ animation: "loading-slide 1.5s ease-in-out infinite",
+ }}
+ />
+ </div>
+ <div className="mt-2 font-mono text-xs text-[#9bc3ff] text-center tracking-wide uppercase">
+ Loading TTS model... This may take a moment on first use.
+ </div>
+ </div>
+ )}
+
+ {/* Speaking animation bar */}
+ {tts.isSpeaking && (
+ <div className="w-full max-w-2xl">
+ <div className="w-full h-1.5 bg-[#0f1c2f] overflow-hidden">
+ <div
+ className="h-full w-full bg-[#75aafc] animate-pulse"
+ />
+ </div>
+ </div>
+ )}
+
+ {/* Error display */}
+ {tts.error && (
+ <div className="w-full max-w-2xl font-mono text-xs text-red-400 text-center px-4 py-2 border border-red-400/50 bg-red-400/10">
+ {tts.error}
+ </div>
+ )}
+ </main>
+
+ </div>
+ );
+}