This is a crawler for Sina Weiqun website(WAP) information, including given Weiqun's posts, replies, users and their follow relation. Written in Python 2.7.1, store data in SQLite3. Relation-crawling part customized on Github Project sina_reptile.

10000+ awesome gpts store and gpts crawler

a parser + crawler for .DS_Store files exposed publically

Scrapes a large portion of the app store and outputs information for each app.

✨ Article Crawler is a package used to crawl articles with Markdown format from a specific webpage and store them locally in HTML / Markdown formats.

Pimcore Search Bundle. Data Provider: Search via Crawler, ObjectList. Index Provider: Store via Lucene, Elasticsearch or OpenSearch

A repository to store Deep Web (onion domain) crawler, scraper, and NLP tools for Tor network.

A crawler to fetch and store housing rent data from various flat and flatmates groups on Facebook

A Web Crawler that crawle some job looking sites, analyze them, store data

AppStore and Google Play crawler written in F#

PHP Console Crawler to Download Products from a Store on MercadoLivre.com.br

Python scalable Play Store crawler

Crunchy a Crunchbase crawler. Used MongoDB For storing data and used kafka for data streaming. rabbitmq for queuing purpose

A demo of a serverless crawler built on AWS Lambda (scheduled tasks) and store results in S3

A simple, scalable scrapper for apps data being listed on Apple's AppStore

A simple crawler on app information from App Store

Crawler of the "Apple Store" podcasts directory

A web crawler, built to be easily dropped into PHP applications, which behaves just like a regular web browser, interpreting location redirects and storing cookies automatically.

This is where we will store our crawler specific code.

A simple script based on CasperJS which craws and purchases FREE games from the PlayStation Store(R) to your account.

Web crawler capable of scraping the reviews and ratings data of any app on Google Play Store then saves in CSV file format for further data analysis.

It is an HTTP crawler which looks for domains in <a> tags and sotres them into a sqlite database next to their IP address. it works recursively among the stored domains.

A web scraping project, where a web crawler receives information from an online store and sorts the data using Pandas

Squirrel is a web crawler designed to collect all pages from Iranian websites, enabling you to download and store web page content in a structured format.

crawl windows application from windows store on windows 8

A simple youtube crawler application that uses the youtube api v3 to crawl videos info and store them in a sqlite database

HBase-Writer is an extension to the Heritrix open source crawler written by the Internet Archive (http://crawler.archive.org/) that enables it to store crawled content directly into HBase tables.

Use crawler to retrieve financial information of TWSE and use Flask to set up restful API to stored and get data into the database

A distributed on demand web crawler, using Redis for job and url store, Hbase for page store.

Mac App Store (MAS) Metadata Crawler