diff options
Diffstat (limited to 'makima-architecture.md')
| -rw-r--r-- | makima-architecture.md | 503 |
1 files changed, 503 insertions, 0 deletions
diff --git a/makima-architecture.md b/makima-architecture.md new file mode 100644 index 0000000..69ab06f --- /dev/null +++ b/makima-architecture.md @@ -0,0 +1,503 @@ +# Makima Architecture Analysis + +## Executive Summary + +Makima is a distributed task orchestration system for managing AI coding agents (primarily Claude Code instances). It follows a client-server architecture with daemons running on local machines that execute tasks, while a central server coordinates work through contracts. The system already implements several patterns similar to ralph, including completion gates, autonomous loop mode, and circuit breakers. + +--- + +## 1. Current Architecture Overview + +### 1.1 High-Level Architecture + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ MAKIMA SERVER (Rust) │ +│ ┌─────────────┐ ┌──────────────┐ ┌─────────────┐ ┌──────────────────┐ │ +│ │ REST API │ │ WebSocket │ │ PostgreSQL │ │ LLM Tools │ │ +│ │ Handlers │ │ Hub │ │ DB │ │ (Chat, Analysis) │ │ +│ └─────────────┘ └──────────────┘ └─────────────┘ └──────────────────┘ │ +└─────────────────────────────────────────────────────────────────────────────┘ + │ + ┌───────────────┼───────────────┐ + ▼ ▼ ▼ + ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ + │ DAEMON 1 │ │ DAEMON 2 │ │ DAEMON N │ + │ (Worker) │ │ (Worker) │ │ (Worker) │ + ├──────────────┤ ├──────────────┤ ├──────────────┤ + │ Task Manager │ │ Task Manager │ │ Task Manager │ + │ Worktree │ │ Worktree │ │ Worktree │ + │ Manager │ │ Manager │ │ Manager │ + │ Process │ │ Process │ │ Process │ + │ Manager │ │ Manager │ │ Manager │ + └──────────────┘ └──────────────┘ └──────────────┘ + │ │ │ + ▼ ▼ ▼ + ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ + │ Claude Code │ │ Claude Code │ │ Claude Code │ + │ Instances │ │ Instances │ │ Instances │ + └──────────────┘ └──────────────┘ └──────────────┘ +``` + +### 1.2 Core Components + +#### Server-Side (`makima/src/server/`) + +| Component | Location | Responsibility | +|-----------|----------|----------------| +| **REST API** | `handlers/*.rs` | HTTP endpoints for contracts, tasks, files, mesh operations | +| **WebSocket Hub** | `handlers/mesh_daemon.rs`, `mesh_ws.rs` | Real-time communication with daemons | +| **Database** | `../db/` | PostgreSQL via sqlx for persistent state | +| **Authentication** | `auth.rs` | API key and JWT authentication | +| **LLM Integration** | `../llm/` | Claude/Groq clients, tool execution | + +#### Daemon-Side (`makima/src/daemon/`) + +| Component | Location | Responsibility | +|-----------|----------|----------------| +| **Task Manager** | `task/manager.rs` | Task lifecycle, concurrency control | +| **Task State** | `task/state.rs` | State machine for task progression | +| **Completion Gate** | `task/completion_gate.rs` | Autonomous loop termination logic | +| **Process Manager** | `process/` | Spawns and manages Claude Code subprocesses | +| **Worktree Manager** | `worktree/` | Git worktree isolation for tasks | +| **WebSocket Client** | `ws/` | Bidirectional communication with server | +| **Local DB** | `db/local.rs` | SQLite for crash recovery | +| **TUI** | `tui/` | Interactive terminal interface | + +--- + +## 2. Key Components and Responsibilities + +### 2.1 Contract System + +Contracts are the top-level organizational unit representing a body of work: + +```rust +// From db/models.rs +pub struct Contract { + pub id: Uuid, + pub name: String, + pub contract_type: String, // "simple" or "specification" + pub phase: String, // research, specify, plan, execute, review + pub status: String, // active, completed, archived + pub supervisor_task_id: Option<Uuid>, + pub autonomous_loop: bool, // Enable auto-restart on incomplete + pub phase_guard: bool, // Require user approval for phase transitions +} +``` + +**Contract Types:** +- `simple`: Plan → Execute workflow +- `specification`: Research → Specify → Plan → Execute → Review + +### 2.2 Task Orchestration + +Tasks represent individual units of work executed by Claude Code: + +```rust +// Simplified from db/models.rs +pub struct Task { + pub id: Uuid, + pub contract_id: Option<Uuid>, + pub parent_task_id: Option<Uuid>, + pub is_supervisor: bool, + pub status: String, // pending, running, paused, blocked, done, failed + pub plan: String, + pub daemon_id: Option<Uuid>, + pub continue_from_task_id: Option<Uuid>, // For task continuation chains + pub conversation_state: Option<serde_json::Value>, // For resumption +} +``` + +**Task Hierarchy:** +1. **Supervisor Tasks**: Long-running orchestrators that spawn worker tasks +2. **Worker Tasks**: Execute specific implementation work +3. **Subtask Chains**: Tasks can continue from other tasks' worktrees + +### 2.3 Task State Machine + +``` + ┌────────────────┐ + │ Initializing │ + └───────┬────────┘ + │ + ┌───────▼────────┐ + │ Starting │ + └───────┬────────┘ + │ + ┌───────────────▼───────────────┐ + │ Running │ + └───────┬───────┬───────┬───────┘ + │ │ │ + ┌───────────▼─┐ ┌───▼───┐ ┌─▼────────┐ + │ Paused │ │Blocked│ │Completed │ + └─────────────┘ └───────┘ └──────────┘ + │ │ + ▼ ▼ + ┌──────────────────────┐ + │ Failed/Interrupted │ + └──────────────────────┘ +``` + +### 2.4 Worktree Isolation + +Each task gets its own git worktree, providing: +- Complete isolation from other tasks +- Ability to merge changes via git +- Checkpointing via git commits + +```rust +// From daemon/worktree/mod.rs +pub struct WorktreeInfo { + pub path: PathBuf, + pub branch: String, + pub task_id: Uuid, +} +``` + +--- + +## 3. Existing Context Management Mechanisms + +### 3.1 Completion Gate (Ralph-Inspired) + +The `CompletionGate` system allows tasks to signal completion status: + +```rust +// From daemon/task/completion_gate.rs +pub struct CompletionGate { + pub ready: bool, + pub reason: Option<String>, + pub progress: Option<String>, + pub blockers: Option<Vec<String>>, +} +``` + +**Format in Claude output:** +```xml +<COMPLETION_GATE> +ready: true +reason: "All tests pass" +progress: "Implemented feature X" +</COMPLETION_GATE> +``` + +### 3.2 Circuit Breaker + +Prevents infinite loops in autonomous mode: + +```rust +pub struct CircuitBreaker { + pub runs_without_changes: u32, // Trips after 3 runs with no changes + pub same_error_count: u32, // Trips after 5 identical errors + pub iteration_count: u32, // Trips after 10 iterations + pub is_open: bool, + pub open_reason: Option<String>, +} +``` + +### 3.3 Autonomous Loop Mode + +When `autonomous_loop: true` on a contract: +1. Task runs to completion +2. Output is parsed for `COMPLETION_GATE` +3. If `ready: false`, task is restarted with `--continue` +4. Circuit breaker monitors for stuck states + +### 3.4 Supervisor State Persistence + +```rust +pub struct SupervisorState { + pub conversation_history: serde_json::Value, + pub pending_task_ids: Vec<Uuid>, + pub phase: String, + pub last_activity: DateTime<Utc>, +} +``` + +### 3.5 Conversation Snapshots + +```rust +pub struct ConversationSnapshot { + pub task_id: Uuid, + pub checkpoint_id: Option<Uuid>, + pub snapshot_type: String, // 'auto', 'manual', 'checkpoint' + pub conversation_state: serde_json::Value, +} +``` + +### 3.6 Phase Guidance System + +```rust +// From llm/phase_guidance.rs +pub struct PhaseDeliverables { + pub phase: String, + pub recommended_files: Vec<RecommendedFile>, + pub requires_repository: bool, + pub requires_tasks: bool, + pub guidance: String, +} +``` + +--- + +## 4. Extension Points for Ralph-Inspired Features + +### 4.1 Task Manager Hook Points + +Location: `daemon/task/manager.rs` + +| Hook Point | Current State | Extension Opportunity | +|------------|---------------|----------------------| +| `spawn_task()` | Creates worktree, spawns process | Add pre-flight checks, memory injection | +| `handle_output()` | Streams to server | Enhanced context extraction | +| `on_completion()` | Cleanup, status update | Post-task analysis, learning | +| `restart_with_continue()` | Autonomous loop restart | Context summarization | + +### 4.2 Process Manager Hook Points + +Location: `daemon/process/claude.rs` + +| Hook Point | Current State | Extension Opportunity | +|------------|---------------|----------------------| +| `build_command()` | Constructs CLI args | Dynamic prompt injection | +| `inject_system_prompt()` | Static prompts | Context-aware prompting | +| `parse_output()` | JSON message parsing | Structured output extraction | + +### 4.3 Server API Extension Points + +Location: `server/handlers/` + +| Endpoint Category | Files | Extension Opportunity | +|-------------------|-------|----------------------| +| Contract Daemon API | `contract_daemon.rs` | Enhanced progress tracking | +| Supervisor API | `mesh_supervisor.rs` | Smarter task scheduling | +| History API | `history.rs` | Learning from past sessions | + +### 4.4 Configuration Extension Points + +Location: `daemon/config.rs` + +```rust +pub struct ProcessConfig { + pub claude_command: String, + pub claude_args: Vec<String>, + pub env_vars: HashMap<String, String>, + // Extension: Add ralph-style configs + // pub context_window_size: usize, + // pub memory_extraction_enabled: bool, + // pub learning_mode: LearningMode, +} +``` + +--- + +## 5. Current Limitations That Ralph Patterns Could Address + +### 5.1 Context Window Management + +**Current State:** +- No explicit context window tracking +- Conversation history stored but not summarized +- `--continue` flag relies on Claude's session state + +**Ralph Pattern Opportunities:** +- Token counting per message +- Automatic context summarization when approaching limits +- Smart context pruning strategies + +### 5.2 Memory and Learning + +**Current State:** +- Output stored in `task_events` table +- Checkpoints stored with diffs +- No cross-task learning + +**Ralph Pattern Opportunities:** +- Extract patterns from successful completions +- Build knowledge base from task outputs +- Learn from failures to improve future prompts + +### 5.3 Progress Tracking + +**Current State:** +- `progress_summary` field on tasks +- `COMPLETION_GATE` for completion signaling +- Phase checklist for deliverables + +**Ralph Pattern Opportunities:** +- Structured progress metrics extraction +- Confidence scoring +- Automatic milestone detection + +### 5.4 Error Recovery + +**Current State:** +- Circuit breaker stops on repeated errors +- Daemon failover with retry count +- Manual intervention required for stuck tasks + +**Ralph Pattern Opportunities:** +- Intelligent error classification +- Automatic recovery strategies +- Error pattern learning + +### 5.5 Task Planning + +**Current State:** +- Human-written plans +- LLM-generated task breakdowns (via `task_output.rs`) +- Static orchestrator prompts + +**Ralph Pattern Opportunities:** +- Dynamic plan refinement +- Dependency inference +- Resource estimation + +--- + +## 6. Data Flow Diagrams + +### 6.1 Task Execution Flow + +``` +┌─────────┐ SpawnTask ┌──────────┐ +│ Server │ ───────────────► │ Daemon │ +└─────────┘ └────┬─────┘ + │ + ┌─────────▼─────────┐ + │ Task Manager │ + │ - Create worktree│ + │ - Setup env vars │ + └─────────┬─────────┘ + │ + ┌─────────▼─────────┐ + │ Process Manager │ + │ - Spawn Claude │ + │ - Inject prompt │ + └─────────┬─────────┘ + │ + ┌─────────▼─────────┐ + │ Claude Code │ + │ - Execute task │ + │ - Stream output │ + └─────────┬─────────┘ + │ + ┌───────────────────┼───────────────────┐ + ▼ ▼ ▼ + ┌────────────┐ ┌──────────────┐ ┌─────────────┐ + │ TaskOutput │ │ Checkpoints │ │ Completion │ + │ Events │ │ (Git) │ │ Gate │ + └────────────┘ └──────────────┘ └─────────────┘ +``` + +### 6.2 Autonomous Loop Flow + +``` +┌─────────────────────────────────────────────────────────┐ +│ AUTONOMOUS LOOP │ +└─────────────────────────────────────────────────────────┘ + │ + ┌───────────▼───────────┐ + │ Execute Task │ + └───────────┬───────────┘ + │ + ┌───────────▼───────────┐ + │ Parse COMPLETION_GATE │ + └───────────┬───────────┘ + │ + ┌───────────────┼───────────────┐ + │ │ │ + ▼ ▼ ▼ + ready: true ready: false No Gate Found + │ │ │ + ▼ ▼ ▼ + Complete ┌───────────┐ ┌───────────┐ + │ Check │ │ Circuit │ + │ Circuit │ │ Breaker │ + │ Breaker │ │ Trip? │ + └─────┬─────┘ └─────┬─────┘ + │ │ + ┌──────┴──────┐ ┌──────┴──────┐ + ▼ ▼ ▼ ▼ + Continue Stop Continue + with Loop with Loop +``` + +--- + +## 7. Key Files Reference + +### Daemon Core +- `src/daemon/mod.rs` - Module exports +- `src/daemon/task/manager.rs` - Task lifecycle (~1200 lines) +- `src/daemon/task/state.rs` - State machine +- `src/daemon/task/completion_gate.rs` - Ralph-style completion +- `src/daemon/process/claude.rs` - Claude subprocess management +- `src/daemon/worktree/manager.rs` - Git worktree isolation +- `src/daemon/ws/protocol.rs` - Server-daemon protocol + +### Server Core +- `src/server/mod.rs` - Route configuration +- `src/server/handlers/mesh_supervisor.rs` - Supervisor operations +- `src/server/handlers/contract_daemon.rs` - Contract CLI interface +- `src/server/handlers/history.rs` - Conversation history + +### LLM Integration +- `src/llm/phase_guidance.rs` - Phase deliverables +- `src/llm/contract_tools.rs` - Contract interaction tools +- `src/llm/task_output.rs` - Output analysis + +### CLI +- `src/bin/makima.rs` - CLI entry point +- `src/daemon/cli/mod.rs` - CLI commands + +--- + +## 8. Recommendations for Ralph Integration + +### Priority 1: Enhanced Context Management +- Add token counting to message parsing +- Implement context summarization triggers +- Create memory extraction pipeline + +### Priority 2: Structured Progress Tracking +- Extend `COMPLETION_GATE` format +- Add confidence scoring +- Implement milestone detection + +### Priority 3: Learning System +- Create patterns database +- Implement cross-task knowledge sharing +- Add success/failure analysis + +### Priority 4: Improved Error Recovery +- Classify error types +- Create recovery playbooks +- Implement automatic retry strategies + +--- + +## Appendix: Related Configuration + +### Daemon Configuration (`makima-daemon.toml`) +```toml +[server] +url = "wss://api.makima.jp" +api_key = "..." + +[process] +max_concurrent_tasks = 4 +claude_command = "claude" +heartbeat_commit_interval_secs = 300 + +[worktree] +base_dir = "~/.makima/worktrees" +repos_dir = "~/.makima/repos" +``` + +### Environment Variables +- `MAKIMA_API_KEY` - API authentication +- `MAKIMA_DAEMON_SERVER_URL` - Server WebSocket URL +- `MAKIMA_TASK_ID` - Set in task environment +- `MAKIMA_CONTRACT_ID` - Set in task environment |