supercrawler

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

crawlerdistributed-crawlerrobotsitemapweb-crawler

Apache License 2.0

Created on Jul 17, 2016

Updated on Jan 9, 2026

Stars

382

Forks

63

Watchers

382

Open Issues

23

Repository Health Score

❤️

46/100

Poor

Overall repository health assessment

Score Breakdown

Activity

Slow updates - updated within 3 months

10/30

33%

Issues Analytics

28

Total Issues

All time

7

Open

25% of total

21

Closed

Recent Commits

Version 2.0.0

Brendon Boshell•6 years ago

202819cView on GitHub

Update README.md

Brendon Boshell•6 years ago

2216cd4View on GitHub

Merge branch 'useragent'

Brendon Boshell•6 years ago

8f1cae3View on GitHub

Update README.md

Brendon Boshell•6 years ago

9841aafView on GitHub

Merge branch 'filter-sitemaps-2'

Brendon Boshell•6 years ago

be8ddb3View on GitHub

Bump sequelize from 5.15.0 to 5.15.1

dependabot[bot]•6 years ago

5da46ccView on GitHub

Update README

Brendon Boshell•6 years ago

cdb1cd2View on GitHub

Allow userAgent option to be a function

Herman J. Radtke III•6 years ago

8ce53bbView on GitHub

Apply urlFilter to sitemap urls within sitemap

Herman J. Radtke III•6 years ago

059e55eView on GitHub

Add error message to 'crawledurl' event

Herman J. Radtke III•6 years ago

bfc8ff7View on GitHub

Version 1.7.2

Brendon Boshell•6 years ago

6f85107View on GitHub

Update DbUrlList to use symbol operators

Herman J. Radtke III•6 years ago

deda1a0View on GitHub

Version 1.7.1

Brendon Boshell•6 years ago

4f5260aView on GitHub

Rollback sinon (so tests pass)

Brendon Boshell•6 years ago

4b3d174View on GitHub

dependencies update

MrRefactoring•6 years ago

98c1e95View on GitHub

View all commits

Community

382 stars, 63 forks

1/30

3%

Documentation

Has description, wiki, license

20/20

100%

Maintenance

6.0% issue ratio

15/20

75%

Health score is calculated based on activity, community engagement, documentation quality, and maintenance practices

Languages

JavaScript

100.0%

Dependencies15 total

10

Production

5

Development

Production Dependencies (10)

bluebird^3.5.4

cheerio^1.0.0-rc.3

ioredis^4.9.3

lodash^4.17.11

Top Contributors

1

brendonboshell

User

75

commits

2

cbess

User

14

commits

3

hjr3

User

4

commits

4

taina0407

User

2

commits

5

MrRefactoring

User

1

commits

6

simoncpu

User

1

commits

7

dependabot[bot]

Bot

1

commits

Languages

JavaScript

100.0%

Issues Analytics

28

Total Issues

All time

7

Open

25% of total

21

Closed

Dependencies15 total

10

Production

5

Development

Production Dependencies (10)

bluebird^3.5.4

cheerio^1.0.0-rc.3

ioredis^4.9.3

lodash^4.17.11

mime-types

^2.1.24

node-cache^4.2.0

request^2.88.0

robots-parser^2.1.1

sequelize^5.8.5

sqlite3^4.0.7

Development Dependencies (5)

chai^4.2.0

mocha^6.1.4

nyc^14.1.0

proxyquire^2.1.0

sinon^1.17.7

supercrawlerv2.0.0

75% of total

3mo

Avg Close Time

Avg response

Issues Activity: Last 6 months

Hottest Issues

1

#36 How to know when crawl finished?

7

closed

2

#8 Is there a way to get the ID of the inserted link on the addHandler("text/html", function(context)). My first time in git posting a issue, maybe this is not a issue :)

7

closed

3

#13 How to avoid to be see as a robot?

5

closed

4

#10 Is the htmlLinkParser `hostnames` option actually implemented?

5

closed

5

#7 Script doesn't exit when using RedisUrlList

4

open

Recent Commits

Version 2.0.0

Brendon Boshell•6 years ago

202819cView on GitHub

Update README.md

Brendon Boshell•6 years ago

2216cd4View on GitHub

Merge branch 'useragent'

Brendon Boshell•6 years ago

8f1cae3View on GitHub

Update README.md

Brendon Boshell•6 years ago

9841aafView on GitHub

Merge branch 'filter-sitemaps-2'

Brendon Boshell•6 years ago

be8ddb3View on GitHub

Bump sequelize from 5.15.0 to 5.15.1

dependabot[bot]•6 years ago

5da46ccView on GitHub

Update README

Brendon Boshell•6 years ago

cdb1cd2View on GitHub

Allow userAgent option to be a function

Herman J. Radtke III•6 years ago

8ce53bbView on GitHub

Apply urlFilter to sitemap urls within sitemap

Herman J. Radtke III•6 years ago

059e55eView on GitHub

Add error message to 'crawledurl' event

Herman J. Radtke III•6 years ago

bfc8ff7View on GitHub

Version 1.7.2

Brendon Boshell•6 years ago

6f85107View on GitHub

Update DbUrlList to use symbol operators

Herman J. Radtke III•6 years ago

deda1a0View on GitHub

Version 1.7.1

Brendon Boshell•6 years ago

4f5260aView on GitHub

Rollback sinon (so tests pass)

Brendon Boshell•6 years ago

4b3d174View on GitHub

dependencies update

MrRefactoring•6 years ago

98c1e95View on GitHub

View all commits

Top Contributors

1

brendonboshell

User

75

commits

2

cbess

User

14

commits

3

hjr3

User

4

commits

4

taina0407

User

2

commits

5

MrRefactoring

User

1

commits

6

simoncpu

User

1

commits

7

dependabot[bot]

Bot

1

commits

GitHub Explorer

supercrawler

Score Breakdown

Production Dependencies (10)

Production Dependencies (10)

Development Dependencies (5)

Issues Activity: Last 6 months

Hottest Issues