Create requirements.txt

Merge pull request #8 from toshohirasawa/master

unexpected 408 error while getting stats of downloaded files

Update README.md

Merge branch 'master' of ssh://github.com/igorbrigadir/DownloadConceptualCaptions

fix duplicate file names

add training head request report

Update download_data.py

use shelve to save progress and results df

add report

move mimetype check, pretend to be googlebot with headers, follow redirects properly

Merge pull request #2 from elliottd/master

add function to check with HEAD request

change multiprocess part, make sure results come back in order

Merge branch 'master' of https://github.com/elliottd/DownloadConceptualCaptions