NOTE: This is a question I found on StackOverflow which I’ve archived here, because the answer is so effing phenomenal.
If you are not into long explanations, see [Paolo Bergantino’s answer][2].
NOTE: This is a question I found on StackOverflow which I’ve archived here, because the answer is so effing phenomenal.
If you are not into long explanations, see [Paolo Bergantino’s answer][2].
Do you have a WARC file of a website all downloaded and ready to be added to the Internet Archive? Great! You can do that with the Internet Archive's web-based uploader, but it's not ideal and it can't handle really big uploads. Here's how you can upload your WARC files to the IA from the command line, and without worrying about a size restriction.
First, you need to get your Access Key and Secret Key from the Internet Archive for the S3-like API. Here's where you can get that for your IA account: http://archive.org/account/s3.php Don't share those with other people!
Here's their documentation file about how to use it, if you need some extra help: http://archive.org/help/abouts3.txt
Next, you should copy the following files to a text file and edit them as needed:
export IA_S3_ACCESS_KEY="YOUR-ACCESS-KEY-FROM-THE-IA-GOES-HERE"Each of these commands will run an ad hoc http static server in your current (or specified) directory, available at http://localhost:8000. Use this power wisely.
$ python -m SimpleHTTPServer 8000| .-(ed@curry 05:24:32) ~ | |
| `-->wget --recursive --warc-file=c4lj.warc.gz http://journal.code4lib.org | |
| FINISHED --2011-12-02 05:17:11-- | |
| Total wall clock time: 19m 24s | |
| Downloaded: 1524 files, 99M in 4m 17s (395 KB/s) | |
| .-(ed@curry 05:45:22) ~ | |
| `-->ls -lh c4lj.warc.gz.warc.gz | |
| -rw-rw-r-- 1 ed ed 85M 2011-12-02 05:17 c4lj.warc.gz.warc.gz |