Quickstart & Usage Guide

Prerequisites

Python 3.10+
An API key for at least one LLM provider (OpenAI, Anthropic, or OpenRouter)

Installation

git clone https://github.com/ImKeTT/VisualClaw.git
cd VisualClaw
pip install -e .

For optional features:

pip install -e ".[embedding]"   # embedding-based skill retrieval
pip install -e ".[rl]"          # RL training (Tinker/MinT)
pip install -e ".[scheduler]"   # idle-window scheduler + psutil
pip install -e ".[live]"        # Gemini Live + audio
pip install -e ".[video]"       # video pipeline (OpenCV)

Minimal Setup

1. Create `config.yaml` in your project root

gateway:
  port: 30100
  enabled_modules: [proxy, skill, memory]

proxy:
  providers:
    openai:
      api_base: "https://api.openai.com/v1"
      api_key: ""        # or set MM_PROXY_PROVIDERS_OPENAI_API_KEY

skill:
  skills_dir: visualclaw/skills_seed/seed_universal_mc

Or use environment variables only — no config file needed for basic use:

cp .env.example .env
# Set MM_PROXY_PROVIDERS_OPENAI_API_KEY or OPENAI_API_KEY in .env.
visualclaw start

2. Start the gateway

visualclaw start
# Listening on http://0.0.0.0:30100
# Modules loaded: proxy, skill, memory

# Background mode:
visualclaw start --daemon
visualclaw status
visualclaw stop

Connecting Your Agent Framework

VisualClaw supports two transport modes:

proxy: VisualClaw is the LLM endpoint. Agent traffic is sent to the gateway, which forwards to the upstream model after running the full pipeline.
http+plugin: The agent keeps using its own model/provider configuration. VisualClaw only exposes HTTP services such as /v1/skill/inject and /v1/memory/collect, while a framework plugin handles hook translation.

Proxy Mode

Set the API base URL to the gateway:

# For Anthropic-compatible agents (Claude Code, OpenClaw):
export ANTHROPIC_BASE_URL=http://localhost:30100

# For OpenAI-compatible agents:
export OPENAI_BASE_URL=http://localhost:30100/v1

The gateway transparently forwards to your configured upstream LLM while running the full pre/post pipeline.

HTTP + Plugin Mode

Start the gateway without proxy in enabled_modules, then let the framework plugin call VisualClaw's HTTP endpoints directly.

gateway:
  port: 30100
  enabled_modules: [skill, memory]

Install the plugin for your framework and point it at the gateway through VISUALCLAW_GATEWAY. See plugins/ directory for adapters.

Claude Code (~/.claude/settings.json):

{
  "hooks": {
    "UserPromptSubmit": [
      {"type": "command", "command": "bash /path/to/plugins/claude_code/hooks_multi.sh"}
    ]
  }
}

OpenClaw:

{
  "plugins": {
    "load": {
      "paths": ["/path/to/VisualClaw/plugins/openclaw"]
    },
    "entries": {
      "visualclaw": {
        "enabled": true
      }
    },
    "allow": ["visualclaw"]
  }
}

Then export:

export VISUALCLAW_GATEWAY=http://127.0.0.1:30100

Skills

Using Bundled Seed Skills

VisualClaw ships benchmark-oriented seed banks under visualclaw/skills_seed/. For VisualClawArena-style multiple-choice video tasks, use:

skill:
  skills_dir: visualclaw/skills_seed/seed_universal_mc

Adding Custom Skills

For your own deployment, keep writable skills in a local skills/ directory:

mkdir -p skills/my-skill
cat > skills/my-skill/SKILL.md << 'EOF'
---
name: my-skill
description: Use when the user asks about X to do Y.
category: coding
---

## My Skill

1. First step...
2. Second step...

**Anti-pattern:** Don't do Z.
EOF

Reload without restarting:

curl -X POST http://localhost:30100/v1/skill/reload

Viewing Skill Evolution History

visualclaw skills log --n 10
visualclaw skills log --full   # show full skill content

RL Training (madmax mode)

Enable RL to collect training data and fine-tune your model:

gateway:
  enabled_modules: [proxy, skill, memory, rl, scheduler]

rl:
  enabled: true
  batch_size: 4
  records_dir: records
  prm:
    enabled: true
    model: gpt-4o
    majority_votes: 3

scheduler:
  enabled: true
  mode: sleep_window      # train during 23:00-07:00
  sleep_start: "23:00"
  sleep_end: "07:00"

The scheduler monitors idle windows and automatically triggers POST /v1/rl/train.

Video Pipeline

Enable video for live scene understanding with smart glasses:

gateway:
  enabled_modules: [proxy, multimodal_video, skill, memory]

multimodal:
  video:
    enabled: true
    vlm_model: gemini-2.0-flash

live:
  enabled: true
  glasses_port: 8765

Connect the companion app (iOS/Android) to ws://your-device:8765. The gateway processes frames in the background and injects scene context into LLM prompts.

Governance

Constitution rules are loaded from constitution.yaml in your project root. If absent, the bundled visualclaw/config/constitution_default.yaml is used.

Customize for your deployment:

rules:
  - id: cost-cap-per-session
    description: "Block when session exceeds $10"
    severity: error
    enforcement: hook

  - id: skill-content-no-injection
    description: "Block evolved skills with injection patterns"
    severity: error
    enforcement: hook
    triggers:
      - tool: skill_evolution
        pattern: "(ignore previous|jailbreak)"

Config Migration (v2 → v3)

If you have an existing VisualClaw v2 flat config:

visualclaw config migrate --dry-run   # preview changes
visualclaw config migrate             # apply (backs up original to .yaml.v2.bak)

Health & Diagnostics

# Check gateway health
curl http://localhost:30100/v1/health
# → {"status":"ok","modules":["proxy","skill","memory","rl"],"uptime_sec":42}

# Check RL status
curl http://localhost:30100/v1/rl/status
# → {"active_model":"","queue_size":0,"success_rate":1.0,...}

# Check scheduler
visualclaw scheduler status

# List sessions
curl http://localhost:30100/v1/sessions  # via health endpoint modules list

FilesExpand file tree

quickstart.md

Latest commit

History