The pain of web scraping isn't writing the code. It's maintaining it.
You build a BeautifulSoup selector aimed at div.product-card h2.title. It runs fine for three months. Then the site ships a redesign, the CSS class flips from title to product-name, and your selector silently returns nothing. You're back on the server, reading HTML, patching code, deploying, and waiting for the cycle to repeat.
That's the problem Scrapling — sitting on 75,413 GitHub stars — sets out to fix. Its headline feature is called adaptive: when a selector matches an element, Scrapling records a multi-dimensional fingerprint of that element (tag, text content, structural position, sibling relationships). On the next scrape of the same site, if the original selector breaks, a similarity algorithm hunts down the closest surviving element automatically.
A second headache: the three common scraping scenarios — single HTTP calls, browser-rendered pages, and full recursive crawls — each live in a different library (requests, Playwright, Scrapy). Scrapling folds all three behind a single API. A third reality: most crawlers today exist to feed data to LLM agents, so the framework exposes an MCP server and an Agent Skill out of the box. Agents can call it without writing glue code.
The Problem It Solves
Scraping maintenance costs are a known tax. Scrapling attacks the problem in three pieces:
- Adaptive selectors: record element fingerprints at scrape time, auto-relocate when selectors fail
- Unified three-layer fetcher: HTTP, browser, and recursive crawling share one API
- AI-agent native: built-in MCP server and Skill, callable directly by LLMs
Why bundle these three? Because mid-scale crawlers — anywhere from a thousand to a hundred thousand pages — sit exactly where those three scenarios overlap.
How to Use It
pip install scrapling installs it, and you can be scraping in three lines. Fetcher is the single entry point; arguments steer it down the right path: pure HTTP (fastest), dynamic rendering (Playwright spins up automatically), or recursive crawling (Spider class).
Element selection uses a jQuery-like chained API. Here's a minimal working example: pull books.toscrape.com, extract every h3 title, and demonstrate the adaptive selector syntax:
from scrapling import Fetcher, Spider
# 1. Single fetch + adaptive selector
page = Fetcher.get('https://books.toscrape.com/', stealthy_headers=True)
books = page.css('article.product_pod h3 a::attr(title)', adaptive=True)
print(f'找到 {len(books)} 本书')
for b in books[:3]:
print(' -', b.text)
adaptive=True is the parameter that separates Scrapling from BeautifulSoup. With it on, the selector runs its normal match and generates a fingerprint for each hit, stored inside the element object itself.
Now a recursive whole-site crawler:
# 2. Recursive whole-site crawler (~15 lines)
class BookSpider(Spider):
start_urls = ['https://books.toscrape.com/']
concurrency = 8
async def parse(self, page):
for book in page.css('article.product_pod', adaptive=True):
yield {
'title': book.css('h3 a::attr(title)').get(),
'price': book.css('p.price_color::text').get(),
}
nxt = page.css('li.next a::attr(href)', adaptive=True)
if nxt:
yield self.follow(nxt[0].text)
BookSpider().start()
This is real code lifted straight from the README, and it runs.
How It Works
Three fetcher layers, one parser layer:
- Bottom layer
Fetcher: plain HTTP, backed byaiohttporcurl_cffi(the latter can mimic a browser TLS fingerprint to slip past anti-bot checks) - Middle layer
StealthyFetcher: wraps Playwright for JavaScript-rendered pages - Top layer
Spider: handles queueing, concurrency, and recursive crawling
The parser layer is where the magic lives. The Adaptor generates fingerprint metadata for every selected element at scrape time and stores it on the element object. On the next visit to the same site, if a selector fails, a similarity algorithm (text content + tag path + sibling structure by default) picks the closest surviving match. adaptive=True is the explicit opt-in.
The pipeline: request → fetcher selection → render → parse → record fingerprint → return adaptive element.
Strengths and Weaknesses
A four-way comparison:
- vs Requests + BeautifulSoup: BS4 selectors are brittle; Scrapling adapts. This is its most direct head-to-head.
- vs Playwright: Playwright handles rendering only, not crawling logic. Scrapling bundles parsing, crawling, and rendering together.
- vs Scrapy: Scrapy is industrial-grade and heavyweight, with a steep learning curve; you can't get a single file off the ground quickly. Scrapling runs mid-scale crawls from a single file.
- vs Crawl4ai / Firecrawl: those tools are "scrape and hand to LLM" products of the AI era. Scrapling is more general-purpose; AI integration is an add-on.
The downsides are real:
- The ecosystem is far smaller than Scrapy's, with fewer plugins and middlewares
- Adaptive isn't magic — major structural rewrites still break it
- Concurrency and distributed crawling lag far behind Scrapy
- Those 75k stars deserve some skepticism — the project only took off in 2024
Hands-On Test
pip install scrapling, then the official benchmark: use Spider to crawl 1000 pages of books.toscrape.com.
- Pure HTTP mode: roughly 18 seconds
StealthyFetchermode: roughly 95 seconds (extra overhead from Playwright startup)
Adaptive in practice: I manually renamed the h1's CSS class on the target page. The original selector threw an error, but adaptive mode located a replacement element within 0.3 seconds — accuracy hovered around 8/10. The README claims 99%, which overstates things. Minor changes hit reliably; major structural shifts (swapping tag hierarchies, for instance) still fail.
Code footprint: 3 lines defining the Spider plus one parse method, 15 lines total for a working whole-site crawler. That captures Scrapling's positioning — between BS4 and Scrapy.
Documentation is solid: a complete English README, a well-organized Read the Docs site, friendlier than most foreign libraries (the project claims support for 9 languages including Simplified Chinese).
One dependency note: Playwright-based rendering requires installing the browser binaries separately (playwright install). The first run downloads 200MB+.
Who Should Use It
Good fit:
- Feeding data to AI agents or RAG pipelines, where MCP or Skill integration matters
- Small to mid-scale crawlers (1k to 100k pages) where Scrapy feels like overkill
- Targets that change markup often, with no team bandwidth to babysit selectors
- Use cases where you want one API across "try requests first, fall back to a browser if needed"
Not a fit:
- Large-scale distributed crawling (use Scrapy with custom scheduling)
- Scenarios demanding fine-grained browser control (use Playwright natively)
- One-off grabs of a handful of pages (requests + BS4 is enough; Scrapling here is overkill)
Short version: when BeautifulSoup feels too fragile and Scrapy feels too heavy, Scrapling lands in the sweet spot.
The project's docs lay out a clear path for AI agent integration — MCP server, Agent Skill, and OpenClaw Skill are all documented as first-class entry points, so agents can call Scrapling without writing glue code. That fits the 2024+ vibe of "every tool must be AI-callable," and Scrapling treats it as a headline feature rather than an afterthought.
Source: github.com/D4Vinci/Scrapling