Original idea from Transfer files from an FTP server to S3 by "Hack N Cheese".
I moved roughly a terrabyte in less than an hour. Granted, I couldn't take advantage of lftp's --parallel=30 switch due to my ftp source limiting me to one connection at a time, but use-pget-n=N did seem to help out.
- Get a fast Ubuntu 14.4 EC2 box on Amazon for temporary usage (I went with
m1.xlarge) so data tranfers aren't limited by your local bandwidth at least. I also attached a fat 2TB EBS volume and symlinked it to/bigdisk, and made sure the EBS volume was deleted after I terminated this EC2 box. I hopelftp2.6.4 is available as a stable package by the next time I attempt this. - Build
lftp2.6.4+ (Not easy to compile, so read theINSTALLfile and plow through all your missing dependencies - you'll also need to re-run `sudo ./configure && su