fix: increased empty attempt number to prevent workers leaving early

perf: limit concurrent playwright pages with semaphore to reduce memory usage

feat: per-domain rate limiting respecting robots crawl-delay

fix: prevent concurrent robots.txt fetches with per-domain locking

refactor: introduce producer/asyncio.Queue to eliminate deadlocks on concurrent inserts

perf: batch insert links to reduce db roundtrips

perf: batch insert links to reduce db roundtrips

perf: increase connection pool size and add httpx limits for high concurrency

chore: removed unused get_crawl_delay function

fix: updated javascript rendered page heuristics

fix: removed excel export option when datasets get to large

Add MIT License to the project

feat: add finished_at field when crawl job ends

fix: resolve illegal character exception duting export

docs: update README