Skip to main content
Back

AnyCrawl Hits 3.4k Stars Challenging Firecrawl With Self-Hosted Open-Source Scraping

AnyCrawl is an MIT-licensed Node.js crawler that converts sites into markdown, runs LLM extraction, and scrapes SERPs with multi-threaded workers.

Repo, Full Access
  • AnyCrawl is an MIT-licensed Node.js crawler that outputs LLM-ready markdown and structured JSON.
  • Four endpoints: scrape, site crawl, batch scrape, and SERP search across Google, Bing, and Baidu.
  • Three engines available: cheerio for static HTML, plus Playwright and Puppeteer for JS-heavy pages.
  • Built-in LLM extraction via JSON schema, compatible with any OpenAI-style provider or Atlas Cloud.
  • Self-hostable via Docker Compose with API key generation, proxy support, and multi-threaded workers.
  • Hosted plans start free at 1,500 credits monthly, scaling to 100,000 credits on Business.

A new open-source project called AnyCrawl has been climbing GitHub as a direct alternative to hosted scraping APIs like Firecrawl. The Node.js and TypeScript toolkit turns arbitrary web pages into clean markdown, runs structured JSON extraction with an LLM, and pulls organized results from search engines like Google, Bing, and Baidu. The repo sits at around 3.4k stars under an MIT license, so you can self-host the whole thing or use the hosted API at anycrawl.dev.

Most crawling libraries were built before retrieval-augmented generation was a common workload, and their output still looks like raw HTML soup. AnyCrawl ships markdown conversion, JSON schema extraction, and SERP parsing as first-class endpoints, all behind the same REST surface.

Four verbs, one REST surface

The API is organized around four endpoints:

  • Scrape a single URL and get back markdown, HTML, or structured JSON.
  • Crawl an entire site with depth limits, path filters, and scope strategies (same-domain, same-hostname, same-origin, or all).
  • Batch scrape a known list of URLs as one async job, then poll for paginated results.