🔥 The Web Data API for AI - Power AI agents with clean web data

Scrapy, a fast high-level web crawling & scraping framework for Python.

The all-in-one AI productivity accelerator. On device and privacy first with no annoying setup or configuration.

:arrow_double_down: Dumb downloader that scrapes the web

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring—all for free or enjoy our SaaS plan!

Python scraper based on AI

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

An AI-powered research assistant that performs iterative, deep research on any topic by combining search engines, web scraping, and large language models. The goal of this repo is to provide the simplest implementation of a deep research agent - e.g. an agent that can refine its research direction overtime and deep dive into a topic.

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥

Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection

APIs for browser automation, testing, and bypassing bot-detection.

Clone any website with one command using AI coding agents

Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.

Lighter web automation with Python

List of libraries, tools and APIs for web scraping and data processing.

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

A Chrome DevTools Protocol driver for web automation and scraping.

Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.

🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.

Declarative web scraping

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.

Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

Self-hosted webscraper.

Code samples from the book Web Scraping with Python http://shop.oreilly.com/product/0636920034391.do

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework