Open-source web crawler and scraper that produces LLM-ready Markdown.
Open-source web crawler and scraper that produces LLM-ready Markdown.
Crawl4AI
Crawl4AI Cloud is a hosted web scraping, extraction, and search API for AI agents and data pipelines. Turn pages into clean Markdown for RAG, extract structured JSON, or search the live web with one API key. Crawl4AI runs the browsers and manages the proxies.
Get a free API key · 1,000 free credits every month. No credit card required.
Try a URL in the playground · Set up Crawl4AI Cloud in Claude Code, Cursor, or Codex
This repository is home to the Apache-2.0 crawler that powers Crawl4AI Cloud. Run it yourself with Python or Docker.
Hosted API base URL: https://api.crawl4ai.com.
Cloud API reference · Agent setup and MCP
Set CRAWL4AI_KEY to your API key, then scrape a page as Markdown:
curl -s https://api.crawl4ai.com/scrape \
-H "Authorization: Bearer $CRAWL4AI_KEY"
-H "Content-Type: application/json"
-d '{"url": "https://news.ycombinator.com"}' | jq -r .markdown
import json
import os
from urllib.request import Request, urlopen
request = Request(
"https://api.crawl4ai.com/scrape",
data=json.dumps({"url": "https://news.ycombinator.com"}).encode("utf-8"),
headers={
"Authorization": f"Bearer {os.environ['CRAWL4AI_KEY']}",
"Content-Type": "application/json",
},
method="POST",
)
with urlopen(request) as response:
print(json.load(response)["markdown"])
async function main() {
const key = process.env.CRAWL4AI_KEY;
if (!key) throw new Error("Set CRAWL4AI_KEY first.");
const response = await fetch("https://api.crawl4ai.com/scrape", {
method: "POST",
headers: {
Authorization: Bearer ${key},
"Content-Type": "application/json",
},
body: JSON.stringify({ url: "https://news.ycombinator.com" }),
});
if (!response.ok) {
throw new Error(`Scrape failed (HTTP ${response.status}): ${await response.text()}`);
}
const { markdown } = await response.json();
console.log(markdown);
}
main().catch((error) => {
console.error(error.message);
process.exitCode = 1;
});
Example Markdown output (illustrative excerpt):
The same key works for /extract, /search, /serp, /answer and many URLs at once (/scrape/batch, /scrape/jobs). Pay as you go: live prices.
For Claude Code, connect to the hosted MCP server:
claude mcp add --transport http crawl4ai https://api.crawl4ai.com/mcp
--header "Authorization: Bearer $CRAWL4AI_KEY"
Latest: v0.9.5 (14 Oct 2026) · all releases →
pip install -U crawl4ai
crawl4ai-setup # installs the browser, once
import asyncio
from crawl4ai import AsyncWebCrawler
async def main():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(url="https://news.ycombinator.com")
print(result.markdown)
asyncio.run(main())
Docker server, CLI and every option: Installation · Self-hosted docs
Which one?
| | ☁️ Crawl4AI Cloud | 🐍 Library | 🐳 Your own server |
| Runs the browsers | we do | you, in your Python process | you, in Docker on your machine |
| JS-heavy pages and bot walls | handled for you, automatically | your settings, your proxies | your settings, your proxies |
| Web search | /search, /serp and /answer | – | – |
| Price | 1,000 free credits a month; plans from $10 | Free, Apache-2.0. You cover infrastructure. | Free, Apache-2.0. You cover hosting. | …
Builder tool · Builder tools / Web data · Open Source · Apache-2.0 · Python
Website https://crawl4ai.com/ · Repo https://github.com/unclecode/crawl4ai · Docs https://docs.crawl4ai.com/
87,249 stars · 9,095 forks · 334 open issues · pushed 2026-11-10 · verified 2026-11-12
Tags: ai-agents, tool, web-data
What is Crawl4AI? Crawl4AI is a research website listed on Scriptiii Webnest. Open-source web crawler and scraper that produces LLM-ready Markdown. Pricing: Free. Free to use. Best for: ai-agents, tool, web-data.
Everything you need to know about Crawl4AI
Similar websites in Research
Web data API that turns websites into LLM-ready Markdown or structured data.
Research·Free
MCP server for Tavily search, extract, map and crawl APIs, hosted remotely or run locally.
Research·Free
Official MCP server for the Perplexity API, offering web-grounded search and answers.
Research·Free