Scrapling

Open-source Python web scraping framework

Scrapling is an open-source web scraping framework for Python, created and maintained by Karim Shoair. It is released under the BSD-3-Clause licence and handles everything from a single request to a full-scale crawl.

Features

According to the project's documentation, Scrapling's parser can relocate elements when a site's layout changes, so scrapers keep working after page updates. Its fetchers are described as able to bypass anti-bot systems, including Cloudflare Turnstile. A spider framework supports concurrent crawls, multiple sessions, pause and resume, and proxy rotation. The project also includes a command-line interface, an MCP server and an agent skill.

History

The first release, version 0.1, was published on the Python Package Index on 13 October 2024.

Availability

Scrapling is installed with pip install scrapling. Source code is on GitHub, documentation is hosted on Read the Docs, and Docker images are published on Docker Hub and the GitHub Container Registry. The project is funded partly through GitHub Sponsors, and its README lists proxy and data-service companies as sponsors. It has a community Discord server and an X account.