Kaelo is a Rust MCP server that gives AI agents (OpenCode, Claude Code, Cursor, etc.) reliable access to web content, even when sites try to block them.
Unlike existing tools that brute-force every request with the same fallback chain, Kaelo learns which strategy works for which domain and gets faster over time.
First visit to reddit.com: 8.1s (probing strategies)
Second visit to reddit.com: 1.5s (uses known-good strategy)
100th visit to reddit.com: 1.5s (same, every time)
Why Kaelo?
| Problem | How Kaelo solves it |
|---|---|
| Sites block AI agents | Multi-strategy fetching: HTTP → TLS impersonation → headless browser |
| Every request re-learns from scratch | Route cache remembers what works per domain |
| Agents waste tokens on bloated content | Token-aware extraction: budgets, dedup, targeted extraction |
| Existing tools are AGPL or paid | MIT licensed, local-first, zero API keys for fetch |
Run kaelo setup to auto-detect which clients are installed on your system.
MCP Tools
| Tool | Description | Parameters |
|---|---|---|
| web_fetch | Fetch a URL and return clean Markdown | url, strategy, token_budget, focus, no_cache, session |
| web_search | Search the web, optionally fetch top results | query, max_results, fetch_content |
| fetch_urls | Batch fetch up to 10 URLs in one call | urls, strategy, token_budget |
| ping | Health check, returns pong | — |
Tool details
web_fetch — the primary tool. Fetches a URL and returns extracted Markdown content. Supports CSS selector focusing (focus), token budget limits, browser sessions for cookie-aware navigation, and cache bypass.
web_search — searches DuckDuckGo (or a self-hosted SearXNG instance) and returns results. Set fetch_content: true to automatically fetch the top result's full content in a single call.
fetch_urls — parallel batch fetch. Pass up to 10 URLs and get combined Markdown output separated by ---. Each URL can have its own token budget.
Fetch Strategies
Kaelo auto-selects the best strategy per domain using its route cache. You can also force a specific strategy via the strategy parameter.
| Strategy | Backend | When to use |
|---|---|---|
| HttpSimple | reqwest | Static sites, APIs, most pages. The default for unknown domains. |
| TlsChrome | wreq (TLS impersonation) | Cloudflare-protected sites, bot-detection pages. |
| TlsMobile | wreq (mobile TLS) | Sites that block desktop bots but allow mobile traffic. |
| Headless | chromiumoxide | JS-heavy SPAs (React, Next.js, Vue, Blazor). Requires Chromium. |
| PublicApi | native HTTP | Reddit, Hacker News, YouTube. Uses native APIs for structured data. |
Leave strategy empty to let Kaelo auto-detect. It probes on the first visit and caches the result for subsequent requests.
Configuration
Config file
~/.config/kaelo/config.toml (created automatically on first run):
KAELO_CACHE_ENABLED=true # Master cache switch
KAELO_CACHE_MAX_SIZE=52428800 # 50 MB hard limit
KAELO_CACHE_MAX_ENTRY=102400 # 100 KB max per cached entry
KAELO_CACHE_TTL=3600 # 1 hour default TTL
KAELO_CACHE_COMPRESSION=gzip # Compress cached content
KAELO_DB_PATH=~/.config/kaelo/kaelo.db # SQLite database path
KAELO_SEARXNG_URL=http://... # SearXNG instance URL
KAELO_SEARCH_BACKEND=searxng # "duckduckgo" (default) or "searxng"
KAELO_DEFAULT_STRATEGY=HttpSimple # Override auto-detection for all requests
KAELO_LOG_LEVEL=debug # log level
Per-domain auth
For sites that require authentication, set environment variables with the domain:
# Bearer token for a private GitHub repo
KAELO_AUTH_GITHUB_COM="Bearer: ghp_xxxxxxxxxxxx"
# Basic auth for a staging site
KAELO_AUTH_STAGING_EXAMPLE_COM="Basic: user:pass"
# Custom header
KAELO_AUTH_API_EXAMPLE_COM="Header: X-API-Key: xxx"
CLI commands
kaelo serve # Start MCP server (stdio transport)
kaelo fetch <url> # Fetch a URL and print extracted Markdown
kaelo prove-it # Run self-test to verify installation
kaelo prove-it --challenge # Test against a Cloudflare-protected URL
kaelo cache status # Show cache size, entries, top domains
kaelo cache clear # Clear everything
kaelo cache clear --domain X # Clear specific domain
kaelo cache clear --older-than 24h # Clear old entries
kaelo cache export <path> # Export route cache strategies to JSON
kaelo cache import <path> # Import route cache strategies from JSON
kaelo cache import-pack <path> # Import community route pack
kaelo cache show-auth # Show stored per-domain auth (tokens redacted)
kaelo cache forget-auth <domain> # Remove stored auth for a domain
Architecture
Kaelo is a single Rust crate with a modular structure:
The route cache sits at the center. On the first request to a domain, Kaelo probes strategies (fastest first) and caches the winner. On every subsequent request, it goes straight to the known-good strategy, cutting latency from ~8s to ~1.5s.
