<a href="https://crawl4ai.com/?ref=readme-banner">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/cloud-launch-banner-dark.svg">
<img alt="Crawl4AI Cloud is live. Soft launch: free credits every month, no card. Get your key." src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/cloud-launch-banner-light.svg" width="960">
</picture>
</a>
</div>
Crawl4AI turns any website into clean, LLM-ready Markdown for RAG, AI agents and data pipelines. Run the open-source web crawler and scraper yourself, free forever, or use it hosted with one key: /scrape a page to Markdown, it to JSON, the web, for Google's results page as data, a question with sources; MCP for your agent. Free: 10 a minute, 2 at a time, 1,000 credits every month. Plans: 120 a minute, 10 at a time, from $10 a month.
<details>
<summary>🤓 <strong>My Personal Story</strong></summary>
</details>
<details>
<summary>Why developers pick Crawl4AI</summary>
</details>
<details>
<summary>📝 <strong>Markdown generation</strong></summary>
</details>
<details>
<summary>📊 <strong>Structured data extraction</strong></summary>
</details>
<details>
<summary>🌐 <strong>Browser control</strong></summary>
</details>
<details>
<summary>🔎 <strong>Crawling and scraping</strong></summary>
</details>
<details>
<summary>🐳 <strong>Self-hosting (Docker)</strong></summary>
</details>
<details>
<summary>☁️ <strong>What the cloud adds</strong></summary>
</details>
<details>
<summary>🐍 <strong>pip</strong></summary>
</details>
<details>
<summary>🐳 <strong>Docker server</strong></summary>
</details>
<details>
<summary>⌨️ <strong>Command line (`crwl`)</strong></summary>
</details>
<details>
<summary>📝 <strong>Clean and fit Markdown</strong></summary>
</details>
<details>
<summary>🖥️ <strong>A JavaScript page and structured data, without an LLM</strong></summary>
</details>
<details>
<summary>📚 <strong>Structured data with an LLM</strong></summary>
</details>
<details>
<summary>🤖 <strong>Your own browser with a saved profile</strong></summary>
</details>
<details>
<summary>📈 <strong>1. Badge Attribution (Recommended)</strong></summary>
Add one of these badges to your README, documentation, or website:
</details>
<details>
<summary>📖 <strong>2. Text Attribution</strong></summary>
Add this line to your documentation:
```
This project uses Crawl4AI (https://github.com/unclecode/crawl4ai) for web data extraction.
```
</details>
<p align="left">
<a href="https://github.com/hafezparast"><img src="https://avatars.githubusercontent.com/u/14273305?s=60&v=4" style="border-radius:50%;" width="64px;"/></a>
<a href="https://github.com/ntohidi"><img src="https://avatars.githubusercontent.com/u/17140097?s=60&v=4" style="border-radius:50%;"width="64px;"/></a>
<a href="https://github.com/Sjoeborg"><img src="https://avatars.githubusercontent.com/u/17451310?s=60&v=4" style="border-radius:50%;"width="64px;"/></a>
<a href="https://github.com/romek-rozen"><img src="https://avatars.githubusercontent.com/u/30595969?s=60&v=4" style="border-radius:50%;"width="64px;"/></a>
<a href="https://github.com/Kourosh-Kiyani"><img src="https://avatars.githubusercontent.com/u/34105600?s=60&v=4" style="border-radius:50%;"width="64px;"/></a>
<a href="https://github.com/Etherdrake"><img src="https://avatars.githubusercontent.com/u/67021215?s=60&v=4" style="border-radius:50%;"width="64px;"/></a>
<a href="https://github.com/shaman247"><img src="https://avatars.githubusercontent.com/u/211010067?s=60&v=4" style="border-radius:50%;"width="64px;"/></a>
<a href="https://github.com/work-flow-manager"><img src="https://avatars.githubusercontent.com/u/217665461?s=60&v=4" style="border-radius:50%;"width="64px;"/></a>
</p>
<p align="center">
<a href="https://github.com/unclecode/crawl4ai/network/members"><img src="https://img.shields.io/github/forks/unclecode/crawl4ai?style=social" alt="GitHub Forks"/></a>
<a href="https://pypi.org/project/crawl4ai/"><img src="https://img.shields.io/pypi/pyversions/crawl4ai" alt="Python Version"/></a>
<a href="https://github.com/sponsors/unclecode"><img src="https://img.shields.io/github/sponsors/unclecode?style=flat&logo=GitHub-Sponsors&label=Sponsors&color=pink" alt="GitHub Sponsors"/></a>
<a href="https://x.com/crawl4ai"><img src="https://img.shields.io/badge/Follow%20on%20X-000000?style=flat&logo=x&logoColor=white" alt="Follow on X"/></a>
<a href="https://www.linkedin.com/company/crawl4ai"><img src="https://img.shields.io/badge/LinkedIn-0077B5?style=flat&logo=linkedin&logoColor=white" alt="Follow on LinkedIn"/></a>
</p>
<a href="https://www.star-history.com/?type=date&repos=unclecode%2Fcrawl4ai">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=unclecode/crawl4ai&type=date&theme=dark&legend=top-left&sealed_token=KuajrA7ScH8VT4KagC7nm1xbazTVaNs6rdok4At2dV6tDl91YR_dxmHhmsffjhFiWdLYlzdACxZ-cWLwp8tZHCYxSDMjITf3Vnu4mPns7YdLetyQBPHMQ2f_KakXdbvbVP6PofI82GNqGCVEXtPnZWHC8WM6CzZe6s6cJb6_ga_kn-jh-BeHdyuRRdVR" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=unclecode/crawl4ai&type=date&legend=top-left&sealed_token=KuajrA7ScH8VT4KagC7nm1xbazTVaNs6rdok4At2dV6tDl91YR_dxmHhmsffjhFiWdLYlzdACxZ-cWLwp8tZHCYxSDMjITf3Vnu4mPns7YdLetyQBPHMQ2f_KakXdbvbVP6PofI82GNqGCVEXtPnZWHC8WM6CzZe6s6cJb6_ga_kn-jh-BeHdyuRRdVR" />
<img alt="Star History Chart" src="https://api.star-history.com/chart?repos=unclecode/crawl4ai&type=date&legend=top-left&sealed_token=KuajrA7ScH8VT4KagC7nm1xbazTVaNs6rdok4At2dV6tDl91YR_dxmHhmsffjhFiWdLYlzdACxZ-cWLwp8tZHCYxSDMjITf3Vnu4mPns7YdLetyQBPHMQ2f_KakXdbvbVP6PofI82GNqGCVEXtPnZWHC8WM6CzZe6s6cJb6_ga_kn-jh-BeHdyuRRdVR" />
</picture>
</a>
/extract
/search
/serp
/answer
Two ways to use Crawl4AI
🐍 Run it yourself: open source, forever
pip install -U crawl4ai
crawl4ai-setup # installs the browser, once
import asyncio
from crawl4ai import AsyncWebCrawler
async def main():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(url="https://news.ycombinator.com")
print(result.markdown)
asyncio.run(main())
| | 🐍 Library | 🐳 Your own server | ☁️ Crawl4AI Cloud |
|---|---|---|---|
| Runs the browsers | you, in your Python process | you, in Docker on your machine | we do |
| JS-heavy pages and bot walls | your settings, your proxies | your settings, your proxies | handled for you, automatically |
| Web search | – | – | /search, /serp and /answer |
| Price | free, forever | free (your hosting) | 1,000 free credits every month; plans from $10 a month |
I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters.
In 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now it’s the most-starred crawler on GitHub.
I made it open source for availability, anyone can use it without a gate. Now I’m building the platform for affordability, anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing.
The dashboard is at http://localhost:11235/dashboard, the playground at http://localhost:11235/playground. LLM keys, MCP and every setting: Self-hosting guide.
# A page as Markdown
crwl https://news.ycombinator.com -o markdown
# Deep crawl, breadth first, at most 10 pages
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10
# Ask a question about a page (needs an LLM key: crwl config)
crwl https://www.example.com/products -q "Extract all product prices"
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai.content_filter_strategy import PruningContentFilterLXML
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
async def main():
run_config = CrawlerRunConfig(
cache_mode=CacheMode.BYPASS,
markdown_generator=DefaultMarkdownGenerator(
content_filter=PruningContentFilterLXML(threshold=0.48, threshold_type="fixed", min_word_threshold=0)
),
)
async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
result = await crawler.arun(url="https://en.wikipedia.org/wiki/Web_crawler", config=run_config)
print(len(result.markdown.raw_markdown), "characters of raw Markdown")
print(len(result.markdown.fit_markdown), "characters after the filter")
asyncio.run(main())
import asyncio, json
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, JsonCssExtractionStrategy
schema = {
"name": "Quotes",
"baseSelector": "div.quote",
"fields": [
{"name": "text", "selector": "span.text", "type": "text"},
{"name": "author", "selector": "small.author", "type": "text"},
{"name": "tags", "selector": "a.tag", "type": "list", "fields": [{"name": "tag", "type": "text"}]},
],
}
async def main():
run_config = CrawlerRunConfig(
extraction_strategy=JsonCssExtractionStrategy(schema),
scan_full_page=True, # scroll to the end, so the page loads every quote
scroll_delay=0.5,
cache_mode=CacheMode.BYPASS,
)
async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
result = await crawler.arun(url="https://quotes.toscrape.com/scroll", config=run_config)
quotes = json.loads(result.extracted_content)
print(f"Extracted {len(quotes)} quotes")
print(json.dumps(quotes[0], indent=2))
asyncio.run(main())
import os, asyncio
from pydantic import BaseModel, Field
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode, LLMConfig, LLMExtractionStrategy
class ModelFee(BaseModel):
model_name: str = Field(..., description="Name of the model.")
input_fee: str = Field(..., description="Fee for input tokens.")
output_fee: str = Field(..., description="Fee for output tokens.")
async def main():
run_config = CrawlerRunConfig(
cache_mode=CacheMode.BYPASS,
extraction_strategy=LLMExtractionStrategy(
# any provider LiteLLM supports, e.g. "ollama/llama3.3" with api_token="no-token"
llm_config=LLMConfig(provider="openai/gpt-4o-mini", api_token=os.getenv("OPENAI_API_KEY")),
schema=ModelFee.model_json_schema(),
extraction_type="schema",
instruction="Extract every model name with its input and output token fee.",
),
)
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(url="https://openai.com/api/pricing/", config=run_config)
print(result.extracted_content)
asyncio.run(main())
import os, asyncio
from pathlib import Path
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
async def main():
user_data_dir = os.path.join(Path.home(), ".crawl4ai", "browser_profile")
os.makedirs(user_data_dir, exist_ok=True)
browser_config = BrowserConfig(headless=True, user_data_dir=user_data_dir, use_persistent_context=True)
run_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS, magic=True)
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(url="ADDRESS_OF_A_CHALLENGING_WEBSITE", config=run_config)
print(result.success, len(result.markdown))
asyncio.run(main())
Our mission is to unlock the value of personal and enterprise data by turning digital footprints into structured, useful assets. Crawl4AI gives individuals and organizations open-source tools to extract and structure data, and a fair way to benefit from it. Full mission statement →
💖 Support Crawl4AI
⭐ Star the repo: it helps more people find it.
☁️ Use the cloud: crawl4ai.com. It funds the library.
🏢 Companies: the sponsor tiers and benefits are in SPONSORS.md.
🌟 Current Sponsors
🤝 Strategic Partners
These companies provide core infrastructure and technology that power Crawl4AI’s capabilities — from web access and proxy networks to AI tooling and data pipelines.
| Company | About |
|------|------|
| <a href="https://www.joinmassive.com/" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="docs/assets/sponsors/massive_light.svg"><source media="(prefers-color-scheme: light)" srcset="docs/assets/sponsors/massive.svg"><img alt="Massive" src="docs/assets/sponsors/massive.svg" height="40"/></picture></a> | Massive is a web access API backed by millions of volunteer devices in 195+ countries. AI agents, models, and data pipelines use it to reach any site on the internet, reliably, in real time, and at scale. |
🏢 Enterprise Sponsors
Our enterprise sponsors support Crawl4AI and help scale it to power production-grade data pipelines.
| Company | About | Sponsorship Tier |
|------|------|----------------------------|
| <a href="https://kipo.ai" target="_blank"><img src="https://docs.crawl4ai.com/uploads/sponsors/20251013045751_2d54f57f117c651e.png" alt="DataSync" height="40"/></a> | Helps engineers and buyers find, compare, and source electronic & industrial parts in seconds, with specs, pricing, lead times & alternatives.| 🥇 Gold |
| <a href="https://www.kidocode.com/" target="_blank"><img src="https://docs.crawl4ai.com/uploads/sponsors/20251013045045_bb8dace3f0440d65.svg" alt="Kidocode" height="40"/></a> | Kidocode is a hybrid technology and entrepreneurship school for kids aged 5–18, offering both online and on-campus education. | 🥇 Gold |
| <a href="https://www.alephnull.sg/" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="docs/assets/sponsors/aleph_null_light.svg"><source media="(prefers-color-scheme: light)" srcset="docs/assets/sponsors/aleph_null.svg"><img alt="Aleph null" src="docs/assets/sponsors/aleph_null.svg" height="40"/></picture></a> | Singapore-based Aleph Null is Asia’s leading edtech hub, dedicated to student-centric, AI-driven education—empowering learners with the tools to thrive in a fast-changing world. | 🥇 Gold |
💼 Become a Strategic Partner or Sponsor
Interested in partnering with Crawl4AI?
Whether you’re a proxy provider, AI infrastructure company, cloud platform, or an organization looking to support the Crawl4AI ecosystem, we’d love to hear from you.
📩 Contact: hello@crawl4ai.com
🧑🤝 Individual Sponsors
A heartfelt thanks to our individual supporters! Every contribution helps us keep our opensource mission alive and thriving!
Building crawlers or AI agents for a living? DM me on X. I want to work with people like you, and we are hiring.
From a company? We have an enterprise offer and we tailor it to your business. SOC 2 Type I is done, Type II is in progress. Write to hello@crawl4ai.com.
Happy crawling! 🕸️🚀
Star History
Ecosystem Role
Standard MoltPulse indexed agent.
Embed Badge
Show off your Pulse Score in your GitHub README to build trust and rank higher.
