Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome :star:

A simple, easy to use framework for adding randomized, anonymous IP addresses and user-agents to web scrapers, crawlers, and penetration testing solutions

Python scraper that grabs user agent strings and populates them into sqlite/JSON files

This tool automatically retrieves the user agent from each browser.

Yellow Pages scraper webapp made with Flask, requests, and beautifulsoup. Outputs results to users as csvs. Uses random proxy and user-agent for each request.

Selenium scraper of goodrx.com (for IE, Chrome, and mobile Safari) with and useragentstring.com user-agents

AGENTIC AI BOILERPLATE — starter kit buat bikin AI agent pakai Agno AGI + Google Gemini + Qdrant (RAG), dengan web scraper “bersih”, Google Search, dan user memory; siap buat cari, baca, dan jawab dari sumber tepercaya ⚡🤖

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome :star:

Built an agentic linkedIn lead scraper that performs query based google search, fetches profiles and scrapes necessary information like user name, position, about, email and the profile url

🤖 AI-Powered LinkedIn Scraper | Extract Emails & Profiles Ethically with Python + Selenium ✅ Bypasses LinkedIn restrictions with human-like behavior ✅ Enriches data using OpenAI API (job titles, company info) ✅ Exports to CSV/Google Sheets for CRM integration ✅ Anti-detection techniques (random delays, user-agent rotation)

Python-based web scraper using Selenium with user agent rotation, error handling, and structured JSON storage. Supports batch URL processing from CSV files.

The list of User-Agent and IPv4 addresses to block unwanted crawlers, bad robots, suspicious spiders, junk web-scrapers, malicious spammers, and unauthorized access including DDoS attack.

Practical examples of how header spoofing works. Shows you the tricks scrapers use like changing user agents and mimicking human behavior. Code works in browser console or Python. Built this while learning about bot detection.

An asynchronous web scraper built with Python that collects and analyzes bike price data from Amazon, featuring proxy rotation, user-agent spoofing, and robust error handling. Built with aiohttp for high-performance concurrent requests and pandas for data processing, with comprehensive logging and anti-detection measures.

This project is a web scraper using Puppeteer, Cheerio, and Axios. It scrapes product details (generic name, strength, brand, price) from medex.com.bd and updates them to an API. It includes proxy rotation, stealth browsing, randomized user agents, human-like scrolling, and auto-restart on Cloudflare blocks.

This tool automatically retrieves the user agent from each browser.

A Python tool to generate and save random user-agent strings for web scraping and automation purposes. Useful for projects like YouTube automation, scraping websites, or bypassing anti-bot measures. The user-agent list is saved in a text file for later use.

easy python module to get a list of user agents

Web scraper with dynamic proxy and user agent.

User-facing platform integrating scrapers, data lake, MCP server, and Agentic AI into a unified advisory system.

fork of https://github.com/ruippeixotog/scala-scraper to set user agent and referrer in connection header

A browser for master web scrapers. It saves cookies, has a decent user agent, sends referer urls, and let's you (the scraper) easily submit forms on the page. It *doesn't* run the page's javascript yet, however.

🕵️‍♂️ Welcome to AI Web Scraper Agent, a powerful and user-friendly app to scrape websites using natural language via OpenAI or Llama 3.2 (Ollama).

A simple Python web scraper to crawl a specific domain (like documentation sites), extract text content, and save it to a file. Configurable crawl depth, delay, and user agent.

A Python web scraper for extracting real estate listings from Mubawab.ma using BeautifulSoup. Supports multiple cities and languages (French/English) with user-agent rotation and flexible CSS selectors to handle dynamic page structures.

A smart, agent-powered job recommendation system built using Model Context Protocol (MCP), Apify job scrapers, and OpenAI. This project fetches real-time job listings from LinkedIn and Naukri, processes user queries, and returns personalized job recommendations with LLM-powered reasoning. ✨ Ideal for learning MCP tooling, agentic workflows, API .

AI Real Estate Agent is an Apify Actor that searches for real estate listings on Zillow based on user queries. It extracts ZIP codes using an LLM (GPT-3.5-Turbo) and fetches property listings using the Zillow Scraper.

ShadowScrape is an advanced asynchronous Python web scraper with a sleek GUI. It features rate limiting, ban detection, user-agent rotation, optional image scraping, and export to JSON or CSV. Built with Playwright and CustomTkinter for cross-platform ease and performance.

Dynamic JS Website Parser with Selenium. Selenium-based scraper for dynamic JavaScript-driven websites. Handles user-agent rotation and other anti-bot measures to reliably extract required product attributes. Scraped data is processed and saved in Excel, JSON, and CSV formats for easy use.

A robust, multi-process web scraper with a Tkinter GUI. It uses Scrapy and Trafilatura for intelligent data extraction from a URL queue. Features configurable PDF/ZIP packaging, request delays, user-agent rotation, exclusion lists, and granular logging control. The application is packaged for Windows using PyInstaller.