toolcloudflareweb-scrapingcrawlingapimarkdowncontent-extractionUpdated 2026-08-11

Cloudflare /crawl Endpoint

One API call crawls an entire site and returns clean content in HTML, Markdown, or plaintext - no scripts, no browser management, no scraping infrastructure.

View original tweet

What It Is

Cloudflare's /crawl endpoint is a managed crawling API built on top of their Browser Rendering product. A single POST request crawls an entire domain, following links and extracting content, then returns structured output in the format you specify. It handles JavaScript rendering, pagination, and link traversal internally - the caller gets clean content without managing headless browsers, proxy rotation, or rate limiting.

The current endpoint is an asynchronous crawl job with HTML, Markdown, or JSON output. render: true uses a headless browser for JavaScript-dependent pages; render: false uses a faster static fetch on Workers. Scope, depth, limit, include/exclude patterns, headers, cookies, and authentication must be explicit in the run receipt. Markdown is useful for ingestion, but it is not proof that every page, script-rendered state, or protected route was captured. Source: Cloudflare Browser Rendering /crawl docs, reviewed 2026-08-12

The reviewed local artifact shows the concrete API shape: a POST to https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl with bearer token auth and a JSON body containing the target url. That makes this a Cloudflare-account API route, not a general anonymous crawl primitive. Source: local X image artifact, 2026-06-30

Robots and operator controls: Cloudflare emphasized /crawl respects robots.txt, self-identifies as a bot, cannot bypass Cloudflare bot detection or CAPTCHAs, and honors AI Crawl Control, Content Signals, and Pay-Per-Crawl. The current API also exposes crawl-purpose controls. Preserve the requested purpose, robots/content-signal outcome, scope, credentials used, job ID, completion status, and page/error counts with every source receipt. Source: X/@CloudflareDev thread, 2026-03-11; Cloudflare docs, reviewed 2026-08-12

Relevance to Kevin's Stack

This fills a specific gap in the Agent Operations Skills toolkit. Currently, the wiki's ingestion pipeline uses Jina Reader for single-page extraction and Firecrawl for multi-page crawls in cloud automations. Cloudflare /crawl provides a middle option: multi-page crawling without Firecrawl's per-page pricing, backed by Cloudflare's existing infrastructure (Workers + Browser Rendering). For source-compile automation runs that need to ingest documentation sites or multi-page references, this is a lower-cost alternative to Firecrawl.

It also complements Defuddle - Web Content Extraction (local single-page extraction) and Autobrowse (Browserbase Skills) (browser-based site learning). The hierarchy for multi-page crawling becomes: Firecrawl MCP for cloud automations → Cloudflare /crawl for bulk site ingestion → Jina Reader for single pages → defuddle as local fallback.

Do not use /crawl to bypass site controls. The source thread explicitly says it respects robots.txt, cannot bypass Cloudflare bot detection or CAPTCHAs, and self-identifies as a bot. For Kevin's wiki/source-compile pipeline, this is strongest for documentation sites and owned/allowed sources where Cloudflare account auth, API scopes, and crawl costs are acceptable.

Signal

  • 19,837 likes, 25,545 bookmarks on X - extremely high engagement, indicating strong developer demand for managed crawling APIs
  • Announcement from @CloudflareDev (official account), signaling production-grade reliability

Timeline

Related Pages

Agent Operations SkillsAutobrowse (Browserbase Skills)Defuddle - Web Content Extraction

3 pages link here

Camofox BrowserToolsWeb Scraping Stealth (TLS Fingerprinting & Anti-Bot Evasion)ConceptsX Bookmarks: Dev Tools (Mar–Jun 2026)Tools