✨ New in v0.9.1: Patch release with 12 bug fixes across Docker, browser, and core. Adds preserve_classes/preserve_tags whitelist for PruningContentFilter, fixes Windows browser crash, Docker auth gate UI, HTTP timeout unit mismatch, and more. Release notes →
✨ Recent v0.9.0: Major secure-by-default release of the Docker API server. Auth is on by default, the server binds loopback unless given a token, and the request body is now an untrusted trust boundary. Release notes →
I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters.
In 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now it’s the most-starred crawler on GitHub.
I made it open source for availability, anyone can use it without a gate. Now I’m building the platform for affordability, anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing.
Fast in practice, async browser pool, caching, minimal hops
Full control, sessions, proxies, cookies, user scripts, hooks
Adaptive intelligence, learns site patterns, explores only what matters
Deploy anywhere, zero keys, CLI and Docker, cloud friendly
🚀 Quick Start
Install Crawl4AI:
# Install the package
pip install -U crawl4ai
# For pre release versions
pip install crawl4ai --pre
# Run post-installation setup
crawl4ai-setup
# Verify your installation
crawl4ai-doctor
If you encounter any browser-related issues, you can install them manually:
python -m playwright install --with-deps chromium
Run a simple web crawl with Python:
import asyncio
from crawl4ai import *
async def main():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="https://www.nbcnews.com/business",
)
print(result.markdown)
if __name__ == "__main__":
asyncio.run(main())
Or use the new command-line interface:
# Basic crawl with markdown output
crwl https://www.nbcnews.com/business -o markdown
# Deep crawl with BFS strategy, max 10 pages
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10
# Use LLM extraction with a specific question
crwl https://www.example.com/products -q "Extract all product prices"
💖 Support Crawl4AI
🎉 Sponsorship Program Now Open! After powering 51K+ developers and 1 year of growth, Crawl4AI is launching dedicated support for startups and enterprises. Be among the first 50 Founding Sponsors for permanent recognition in our Hall of Fame.
Crawl4AI is the #1 trending open-source web crawler on GitHub. Your support keeps it independent, innovative, and free for the community — while giving you direct access to premium benefits.
🤝 Sponsorship Tiers
🌱 Believer ($5/mo) — Join the movement for data democratization
🚀 Builder ($50/mo) — Priority support & early access to features
💼 Growing Team ($500/mo) — Bi-weekly syncs & optimization help
🏢 Data Infrastructure Partner ($2000/mo) — Full partnership with dedicated support Custom arrangements available - see SPONSORS.md for details & contact
Why sponsor?
No rate-limited APIs. No lock-in. Build and own your data pipeline with direct guidance from the creator of Crawl4AI.
🌌 Cosine Similarity: Find relevant content chunks based on user queries for semantic extraction.
🔎 CSS-Based Extraction: Fast schema-based data extraction using XPath and CSS selectors.
🔧 Schema Definition: Define custom schemas for extracting structured JSON from repetitive patterns.
🖥️ Managed Browser: Use user-owned browsers with full control, avoiding bot detection.
🔄 Remote Browser Control: Connect to Chrome Developer Tools Protocol for remote, large-scale data extraction.
👤 Browser Profiler: Create and manage persistent profiles with saved authentication states, cookies, and settings.
🔒 Session Management: Preserve browser states and reuse them for multi-step crawling.
🧩 Proxy Support: Seamlessly connect to proxies with authentication for secure access.
⚙️ Full Browser Control: Modify headers, cookies, user agents, and more for tailored crawling setups.
🌍 Multi-Browser Support: Compatible with Chromium, Firefox, and WebKit.
📐 Dynamic Viewport Adjustment: Automatically adjusts the browser viewport to match page content, ensuring complete rendering and capturing of all elements.
🖼️ Media Support: Extract images, audio, videos, and responsive image formats like srcset and picture.
🚀 Dynamic Crawling: Execute JS and wait for async or sync for dynamic content extraction.
📸 Screenshots: Capture page screenshots during crawling for debugging or analysis.
📂 Raw Data Crawling: Directly process raw HTML (raw:) or local files (file://).
🔗 Comprehensive Link Extraction: Extracts internal, external links, and embedded iframe content.
🛠️ Customizable Hooks: Define hooks at every step to customize crawling behavior (supports both string and function-based APIs).
💾 Caching: Cache data for improved speed and to avoid redundant fetches.
📄 Metadata Extraction: Retrieve structured metadata from web pages.
📡 IFrame Content Extraction: Seamless extraction from embedded iframe content.
🕵️ Lazy Load Handling: Waits for images to fully load, ensuring no content is missed due to lazy loading.
🔄 Full-Page Scanning: Simulates scrolling to load and capture all dynamic content, perfect for infinite scroll pages.
🐳 Dockerized Setup: Optimized Docker image with FastAPI server for easy deployment.
🔑 Secure Authentication: Built-in JWT token authentication for API security.
🔄 API Gateway: One-click deployment with secure token authentication for API-based workflows.
🌐 Scalable Architecture: Designed for mass-scale production and optimized server performance.
☁️ Cloud Deployment: Ready-to-deploy configurations for major cloud platforms.
🕶️ Stealth Mode: Avoid bot detection by mimicking real users.
🏷️ Tag-Based Content Extraction: Refine crawling based on custom tags, headers, or metadata.
🔗 Link Analysis: Extract and analyze all links for detailed data exploration.
🛡️ Error Handling: Robust error management for seamless execution.
🔐 CORS & Static Serving: Supports filesystem-based caching and cross-origin requests.
📖 Clear Documentation: Simplified and updated guides for onboarding and advanced usage.
🙌 Community Recognition: Acknowledges contributors and pull requests for transparency.
Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package or use Docker.
Choose the installation option that best fits your needs:
Basic Installation
For basic web crawling and scraping tasks:
pip install crawl4ai
crawl4ai-setup # Setup the browser
By default, this will install the asynchronous version of Crawl4AI, using Playwright for web crawling.
👉 Note: When you install Crawl4AI, the crawl4ai-setup should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods:
Through the command line:
playwright install
If the above doesn't work, try this more specific command:
python -m playwright install chromium
This second method has proven to be more reliable in some cases.
Installation with Synchronous Version
The sync version is deprecated and will be removed in future versions. If you need the synchronous version using Selenium:
pip install crawl4ai[sync]
Development Installation
For contributors who plan to modify the source code:
git clone https://github.com/unclecode/crawl4ai.git
cd crawl4ai
pip install -e . # Basic installation in editable mode
Install optional features:
pip install -e ".[torch]" # With PyTorch features
pip install -e ".[transformer]" # With Transformer features
pip install -e ".[cosine]" # With cosine similarity features
pip install -e ".[sync]" # With synchronous crawling (Selenium)
pip install -e ".[all]" # Install all optional features
🚀 Now Available! Our completely redesigned Docker implementation is here! This new solution makes deployment more efficient and seamless than ever.
New Docker Features
The new Docker implementation includes:
Real-time Monitoring Dashboard with live system metrics and browser pool visibility
Browser pooling with page pre-warming for faster response times
Interactive playground to test and generate request code
MCP integration for direct connection to AI tools like Claude Code
Comprehensive API endpoints including HTML extraction, screenshots, PDF generation, and JavaScript execution
Multi-architecture support with automatic detection (AMD64/ARM64)
Optimized resources with improved memory management
Getting Started
# Pull and run the latest release
docker pull unclecode/crawl4ai:latest
docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:latest
# Visit the monitoring dashboard at http://localhost:11235/dashboard
# Or the playground at http://localhost:11235/playground
Quick Test
Run a quick test (works for both Docker options):
import requests
# Submit a crawl job
response = requests.post(
"http://localhost:11235/crawl",
json={"urls": ["https://example.com"], "priority": 10}
)
if response.status_code == 200:
print("Crawl job submitted successfully.")
if "results" in response.json():
results = response.json()["results"]
print("Crawl job completed. Results:")
for result in results:
print(result)
else:
task_id = response.json()["task_id"]
print(f"Crawl job submitted. Task ID:: {task_id}")
result = requests.get(f"http://localhost:11235/task/{task_id}")
For more examples, see our Docker Examples. For advanced configuration, monitoring features, and production deployment, see our Self-Hosting Guide.
🔬 Advanced Usage Examples 🔬
You can check the project structure in the directory docs/examples. Over there, you can find a variety of examples; here, some popular examples are shared.
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai.content_filter_strategy import PruningContentFilter, BM25ContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
async def main():
browser_config = BrowserConfig(
headless=True,
verbose=True,
)
run_config = CrawlerRunConfig(
cache_mode=CacheMode.ENABLED,
markdown_generator=DefaultMarkdownGenerator(
content_filter=PruningContentFilter(threshold=0.48, threshold_type="fixed", min_word_threshold=0)
),
# markdown_generator=DefaultMarkdownGenerator(
# content_filter=BM25ContentFilter(user_query="WHEN_WE_FOCUS_BASED_ON_A_USER_QUERY", bm25_threshold=1.0)
# ),
)
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(
url="https://docs.micronaut.io/4.9.9/guide/",
config=run_config
)
print(len(result.markdown.raw_markdown))
print(len(result.markdown.fit_markdown))
if __name__ == "__main__":
asyncio.run(main())
import os
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig
from crawl4ai import LLMExtractionStrategy
from pydantic import BaseModel, Field
class OpenAIModelFee(BaseModel):
model_name: str = Field(..., description="Name of the OpenAI model.")
input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
output_fee: str = Field(..., description="Fee for output token for the OpenAI model.")
async def main():
browser_config = BrowserConfig(verbose=True)
run_config = CrawlerRunConfig(
word_count_threshold=1,
extraction_strategy=LLMExtractionStrategy(
# Here you can use any provider that Litellm library supports, for instance: ollama/qwen2
# provider="ollama/qwen2", api_token="no-token",
llm_config = LLMConfig(provider="openai/gpt-4o", api_token=os.getenv('OPENAI_API_KEY')),
schema=OpenAIModelFee.schema(),
extraction_type="schema",
instruction="""From the crawled content, extract all mentioned model names along with their fees for input and output tokens.
Do not miss any models in the entire content. One extracted model JSON format should look like this:
{"model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens"}."""
),
cache_mode=CacheMode.BYPASS,
)
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(
url='https://openai.com/api/pricing/',
config=run_config
)
print(result.extracted_content)
if __name__ == "__main__":
asyncio.run(main())
import os, sys
from pathlib import Path
import asyncio, time
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
async def test_news_crawl():
# Create a persistent user data directory
user_data_dir = os.path.join(Path.home(), ".crawl4ai", "browser_profile")
os.makedirs(user_data_dir, exist_ok=True)
browser_config = BrowserConfig(
verbose=True,
headless=True,
user_data_dir=user_data_dir,
use_persistent_context=True,
)
run_config = CrawlerRunConfig(
cache_mode=CacheMode.BYPASS
)
async with AsyncWebCrawler(config=browser_config) as crawler:
url = "ADDRESS_OF_A_CHALLENGING_WEBSITE"
result = await crawler.arun(
url,
config=run_config,
magic=True,
)
print(f"Successfully crawled {url}")
print(f"Content length: {len(result.markdown)}")
✨ Recent Updates
A patch release with 12 bug fixes and one new feature. The new preserve_classes / preserve_tags parameters for PruningContentFilter let you whitelist CSS classes or HTML tags that should never be pruned — useful for protecting short metadata elements like author names and timestamps.
A major, secure-by-default release of the Docker API server. The out-of-the-box deployment is hardened with defense in depth: authentication is on by default, the server binds loopback unless you give it a token, and the network request body is treated as an untrusted trust boundary.
A security-hardening release. Fixes critical Docker API vulnerabilities (AST sandbox escape RCE, hook sandbox RCE, hardcoded JWT secret, SSRF on webhook and crawl endpoints, arbitrary file write, monitor auth bypass, stored XSS, and unauthenticated JS execution), adds the DomainMapper feature, and ships a batch of scraping, deep-crawl, and LLM fixes. If you self-host the Docker API, upgrade immediately.
Replaced litellm dependency with unclecode-litellm due to a PyPI supply chain compromise affecting the original package. If you're on v0.8.5 or earlier, upgrade immediately.
pip install -U crawl4ai
Our biggest release since v0.8.0. Anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, and over 60 bug fixes.
🛡️ Anti-Bot Detection & Proxy Escalation:
3-tier detection: known vendors, generic block indicators, structural integrity checks
Automatic retry with proxy chain and fallback fetch function
from crawl4ai import CrawlerRunConfig
from crawl4ai.async_configs import ProxyConfig
config = CrawlerRunConfig(
proxy_config=[ProxyConfig.DIRECT, ProxyConfig(server="http://my-proxy:8080")],
max_retries=2,
fallback_fetch_function=my_web_unlocker,
)
🌑 Shadow DOM Flattening:
Extract content hidden inside shadow DOM components
This release introduces crash recovery for deep crawls, a new prefetch mode for fast URL discovery, and critical security fixes for Docker deployments.
🔄 Deep Crawl Crash Recovery:
on_state_change callback fires after each URL for real-time state persistence
resume_state parameter to continue from a saved checkpoint
JSON-serializable state for Redis/database storage
Works with BFS, DFS, and Best-First strategies
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy
strategy = BFSDeepCrawlStrategy(
max_depth=3,
resume_state=saved_state, # Continue from checkpoint
on_state_change=save_to_redis, # Called after each URL
)
⚡ Prefetch Mode for Fast URL Discovery:
prefetch=True skips markdown, extraction, and media processing
5-10x faster than full processing
Perfect for two-phase crawling: discover first, process selectively
config = CrawlerRunConfig(prefetch=True)
result = await crawler.arun("https://example.com", config=config)
# Returns HTML and links only - no markdown generation
🔒 Security Fixes (Docker API):
Hooks disabled by default (CRAWL4AI_HOOKS_ENABLED=false)
file:// URLs blocked on API endpoints to prevent LFI
This release focuses on stability with 11 bug fixes addressing issues reported by the community. No new features, but significant improvements to reliability.
🐳 Docker API Fixes:
Fixed ContentRelevanceFilter deserialization in deep crawl requests (#1642)
Fixed ProxyConfig JSON serialization in BrowserConfig.to_dict() (#1629)
Fixed .cache folder permissions in Docker image (#1638)
🤖 LLM Extraction Improvements:
Configurable rate limiter backoff with new LLMConfig parameters (#1269):
from crawl4ai import LLMConfig
config = LLMConfig(
provider="openai/gpt-4o-mini",
backoff_base_delay=5, # Wait 5s on first retry
backoff_max_attempts=5, # Try up to 5 times
backoff_exponential_factor=3 # Multiply delay by 3 each attempt
)
HTML input format support for LLMExtractionStrategy (#1178):
from crawl4ai import LLMExtractionStrategy
strategy = LLMExtractionStrategy(
llm_config=config,
instruction="Extract table data",
input_format="html" # Now supports: "html", "markdown", "fit_markdown"
)
Fixed raw HTML URL variable - extraction strategies now receive "Raw HTML" instead of HTML blob (#1116)
🔗 URL Handling:
Fixed relative URL resolution after JavaScript redirects (#1268)
Fixed import statement formatting in extracted code (#1181)
📦 Dependency Updates:
Replaced deprecated PyPDF2 with pypdf (#1412)
Pydantic v2 ConfigDict compatibility - no more deprecation warnings (#678)
🧠 AdaptiveCrawler:
Fixed query expansion to actually use LLM instead of hardcoded mock data (#1621)
from crawl4ai import AsyncWebCrawler, BrowserConfig
browser_config = BrowserConfig(
browser_type="undetected", # Use undetected Chrome
headless=True, # Can run headless with stealth
extra_args=[
"--disable-blink-features=AutomationControlled",
"--disable-web-security"
]
)
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun("https://protected-site.com")
# Successfully bypass Cloudflare, Akamai, and custom bot detection
🎨 Multi-URL Configuration: Different strategies for different URL patterns in one batch:
from crawl4ai import CrawlerRunConfig, MatchMode, CacheMode
configs = [
# Documentation sites - aggressive caching
CrawlerRunConfig(
url_matcher=["*docs*", "*documentation*"],
cache_mode=CacheMode.WRITE_ONLY,
markdown_generator_options={"include_links": True}
),
# News/blog sites - fresh content
CrawlerRunConfig(
url_matcher=lambda url: 'blog' in url or 'news' in url,
cache_mode=CacheMode.BYPASS
),
# Fallback for everything else
CrawlerRunConfig()
]
results = await crawler.arun_many(urls, config=configs)
# Each URL gets the perfect configuration automatically
🧠 Memory Monitoring: Track and optimize memory usage during crawling:
📊 Enhanced Table Extraction: Direct DataFrame conversion from web tables:
result = await crawler.arun("https://site-with-tables.com")
# New way - direct table access
if result.tables:
import pandas as pd
for table in result.tables:
df = pd.DataFrame(table['data'])
print(f"Table: {df.shape[0]} rows × {df.shape[1]} columns")
💰 GitHub Sponsors: 4-tier sponsorship system for project sustainability
🐳 Docker LLM Flexibility: Configure providers via environment variables
🧠 Adaptive Crawling: Your crawler now learns and adapts to website patterns automatically:
config = AdaptiveConfig(
confidence_threshold=0.7, # Min confidence to stop crawling
max_depth=5, # Maximum crawl depth
max_pages=20, # Maximum number of pages to crawl
strategy="statistical"
)
async with AsyncWebCrawler() as crawler:
adaptive_crawler = AdaptiveCrawler(crawler, config)
state = await adaptive_crawler.digest(
start_url="https://news.example.com",
query="latest news content"
)
# Crawler learns patterns and improves extraction over time
Crawl4AI follows standard Python version numbering conventions (PEP 440) to help users understand the stability and features of each release.
Our version numbers follow this pattern: MAJOR.MINOR.PATCH (e.g., 0.4.3)
Pre-release Versions
We use different suffixes to indicate development stages:
dev (0.4.3dev1): Development versions, unstable
a (0.4.3a1): Alpha releases, experimental features
b (0.4.3b1): Beta releases, feature complete but needs testing
rc (0.4.3): Release candidates, potential final version
Installation
Regular installation (stable version):
pip install -U crawl4ai
Install pre-release versions:
pip install crawl4ai --pre
Install specific version:
pip install crawl4ai==0.4.3b1
Why Pre-releases?
We use pre-releases to:
Test new features in real-world scenarios
Gather feedback before final releases
Ensure stability for production users
Allow early adopters to try new features
For production environments, we recommend using the stable version. For testing new features, you can opt-in to pre-releases using the --pre flag.
📖 Documentation & Roadmap
🚨 Documentation Update Alert: We're undertaking a major documentation overhaul next week to reflect recent updates and improvements. Stay tuned for a more comprehensive and up-to-date guide!
For current documentation, including installation instructions, advanced features, and API reference, visit our Documentation Website.
To check our development plans and upcoming features, visit our Roadmap.
[x] 0. Graph Crawler: Smart website traversal using graph search algorithms for comprehensive nested page extraction
[x] 1. Question-Based Crawler: Natural language driven web discovery and content extraction
[x] 2. Knowledge-Optimal Crawler: Smart crawling that maximizes knowledge while minimizing data extraction
[x] 3. Agentic Crawler: Autonomous system for complex multi-step crawling operations
[x] 4. Automated Schema Generator: Convert natural language to extraction schemas
[x] 5. Domain-Specific Scrapers: Pre-configured extractors for common platforms (academic, e-commerce)
[x] 6. Web Embedding Index: Semantic search infrastructure for crawled content
[x] 7. Interactive Playground: Web UI for testing, comparing strategies with AI assistance
[x] 8. Performance Monitor: Real-time insights into crawler operations
Our mission is to unlock the value of personal and enterprise data by transforming digital footprints into structured, tradeable assets. Crawl4AI empowers individuals and organizations with open-source tools to extract and structure data, fostering a shared data economy.
We envision a future where AI is powered by real human knowledge, ensuring data creators directly benefit from their contributions. By democratizing data and enabling ethical sharing, we are laying the foundation for authentic AI advancement.
Data Capitalization: Transform digital footprints into measurable, valuable assets.
Authentic AI Data: Provide AI systems with real human insights.
Shared Economy: Create a fair data marketplace that benefits data creators.
Open-Source Tools: Community-driven platforms for transparent data extraction.
Digital Asset Structuring: Tools to organize and value digital knowledge.
Ethical Data Marketplace: A secure, fair platform for exchanging structured data.
These companies provide core infrastructure and technology that power Crawl4AI’s capabilities — from web access and proxy networks to AI tooling and data pipelines.
| Company | About |
|------|------|
| <a href="https://www.joinmassive.com/" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="docs/assets/sponsors/massive_light.svg"><source media="(prefers-color-scheme: light)" srcset="docs/assets/sponsors/massive.svg"><img alt="Massive" src="docs/assets/sponsors/massive.svg" height="40"/></picture></a> | Massive is a web access API backed by millions of volunteer devices in 195+ countries. AI agents, models, and data pipelines use it to reach any site on the internet, reliably, in real time, and at scale. |
🏢 Enterprise Sponsors
Our enterprise sponsors support Crawl4AI and help scale it to power production-grade data pipelines.
| Company | About | Sponsorship Tier |
|------|------|----------------------------|
| <a href="https://kipo.ai" target="_blank"><img src="https://docs.crawl4ai.com/uploads/sponsors/20251013045751_2d54f57f117c651e.png" alt="DataSync" height="40"/></a> | Helps engineers and buyers find, compare, and source electronic & industrial parts in seconds, with specs, pricing, lead times & alternatives.| 🥇 Gold |
| <a href="https://www.kidocode.com/" target="_blank"><img src="https://docs.crawl4ai.com/uploads/sponsors/20251013045045_bb8dace3f0440d65.svg" alt="Kidocode" height="40"/></a> | Kidocode is a hybrid technology and entrepreneurship school for kids aged 5–18, offering both online and on-campus education. | 🥇 Gold |
| <a href="https://www.alephnull.sg/" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="docs/assets/sponsors/aleph_null_light.svg"><source media="(prefers-color-scheme: light)" srcset="docs/assets/sponsors/aleph_null.svg"><img alt="Aleph null" src="docs/assets/sponsors/aleph_null.svg" height="40"/></picture></a> | Singapore-based Aleph Null is Asia’s leading edtech hub, dedicated to student-centric, AI-driven education—empowering learners with the tools to thrive in a fast-changing world. | 🥇 Gold |
💼 Become a Strategic Partner or Sponsor
Interested in partnering with Crawl4AI?
Whether you’re a proxy provider, AI infrastructure company, cloud platform, or an organization looking to support the Crawl4AI ecosystem, we’d love to hear from you.
📩 Contact: hello@crawl4ai.com
🧑🤝 Individual Sponsors
A heartfelt thanks to our individual supporters! Every contribution helps us keep our opensource mission alive and thriving!
