Shellbye
/ install_scrapy_in_centos_6_5.sh
Last active
November 26, 2016 04:36
Install scrapy in CentOS 6.5 in virtualenv, and install mysql to store the data crawled. After the installation, the `spider` dir will contains all the downloaded files and the `spider_env` is the virtualenv dir
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| mkdir spider | |
| cd spider/ | |
| yum update -y | |
| # install pip on centos6 | |
| rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm | |
| yum install -y python-pip | |
| # install wget | |
| yum install -y wget |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # ref http://stackoverflow.com/questions/3297196/how-to-set-up-a-squid-proxy-with-basic-username-and-password-authentication | |
| # first install apache: | |
| # $ sudo apt-get install -y apache2-utils | |
| # add these five lines to /etc/squid3/squid.conf | |
| auth_param basic program /usr/lib/squid3/basic_ncsa_auth /etc/squid3/passwords | |
| auth_param basic realm proxy | |
| acl authenticated proxy_auth REQUIRED | |
| http_access allow authenticated | |
| http_port 3128 # or whatever you like |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/sh | |
| sudo apt-get update | |
| sudo apt-get install -y squid3 | |
| sudo cp /etc/squid3/squid.conf /etc/squid3/squid.conf.original | |
| sudo chmod a-w /etc/squid3/squid.conf.original | |
| echo "use vim /etc/squid3/squid.conf to change port(default 3128) and http_access allow" | |
| # change port: http_port 8888 | |
| # allow access: http_access allow all <==this should at top of all http_access | |
| # ref: https://help.ubuntu.com/lts/serverguide/squid.html |
NewerOlder