Cross-platform persistent and distributed web crawler :link:
Stars
113
Forks
9
Watchers
Open Issues
0
Overall repository health assessment
No package.json found
This might not be a Node.js project
User
71
commits
Update README.md
f20aff4
Add tor option, fix tests
fec52cd
Add tor option
af7e13f
Added trash limit flag
074db3e
Quit if too much trash
40d790b
Store doing urls, so they aren't lost
4ddc312
also count trash links
f174f0a
ab1c08e
Merge branch 'master' of github.com:schollz/crawler
44d55c6
Default port for tests has changed
39cd3fe
0d81433
Run spellcheck
d185f97
Update out-of-date README
73894f8
Update README with goreleaser info
30b87c6
Update goreleaser
5c69c7f