cloudera安装文档

cloudera安装文档.txt

1.各个节点安装必需的包
yum install -y openssh  openssh-clients vim libxslt libaio


2.配置ssh无密码认证（个人认为可以忽略的步骤）
（1）修改hosts.allow 将新的IP都加进去
（2）修改ssh的配置文件 允许root登录 ，再重启ssh服务
（3）在一台机器上ssh所有机器，获取所有的机器的公钥，再生成自己的ssh-key，无密码登录自己，把.ssh目录打包，传到每个机器上

3.修改hosts文件
把每个机器的IP和主机名做对应，注意：删掉之前重复的
vim  /etc/hosts

4.添加hadoop用户，并设置sudo权限
useradd hadoop
echo "hadoop" |passwd hadoop --stdin
echo "hadoop    ALL=(ALL) NOPASSWD: ALL"  >> /etc/sudoers

5.每个节点配置ntp服务
由于阿里云的机器，都已经配置好ntp的配置文件了，所以只需要开启ntp服务即可
service ntpd start
chkconfig ntpd on

6.每台机器安装java

7.每台机器改系统参数
echo "vm.swappiness = 0"  >> /etc/sysctl.conf
sysctl -p 
echo "echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag"  >> /etc/rc.local
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

8.每台机器都要建用户
useradd --system --home=/opt/cm-5.6.0/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment  "Cloudera SCM User" cloudera-scm

9.主节点安装mysql （其他节点不需要安装mysql）
cd /opt
wget http://cdn.mysql.com/Downloads/MySQL-5.6/MySQL-client-5.6.29-1.el6.x86_64.rpm
wget http://cdn.mysql.com/Downloads/MySQL-5.6/MySQL-server-5.6.29-1.el6.x86_64.rpm
wget http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.36.zip

yum remove mysql mysql-server mysql-libs compat-mysql51
yum -y remove mysql-libs-5.1.71*
rm -rf /var/lib/mysql
rm -f /etc/my.cnf

rpm -ivh  MySQL-server-5.6.29-1.el6.x86_64.rpm
rpm -ivh  MySQL-client-5.6.29-1.el6.x86_64.rpm 
chkconfig mysql on
service mysql start
service mysql status 
查看mysql的密码
cat /root/.mysql_secret 
重新设置mysql的密码
mysql>  set password for 'root'@'localhost'=password('hadoop');

mysql> create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;  
mysql>  create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;    
mysql> create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;    
mysql> create database Oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

mysql> grant all privileges on *.* to 'root'@'ALYSZ101-129' identified by 'hadoop' with grant option;     
这一步赋权的时候，一定要用主机名，不可以用IP   
mysql> flush privileges;


10.主节点安装cloudera manager server
文件必须解压到/opt目录下，不能解压到其它地方。 
cd  /opt
wget http://archive-primary.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.6.0_x86_64.tar.gz
tar xf cloudera-manager-el6-cm5.6.0_x86_64.tar.gz
unzip mysql-connector-java-5.1.36.zip
cp  mysql-connector-java-5.1.36/mysql-connector-java-5.1.36-bin.jar  /opt/cm-5.6.0/share/cmf/lib/
/opt/cm-5.6.0/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -phadoop --scm-host localhost scm scm scm

vim /opt/cm-5.6.0/etc/cloudera-scm-agent/config.ini
server_host=ALYSZ101-129
scp -P 60088 -r cm-5.6.0/  root@ALYSZ65-240:/opt
scp -P 60088 -r cm-5.6.0/  root@ALYSZ101-205:/opt

cd /opt/cloudera/parcel-repo/
wget http://archive-primary.cloudera.com/cdh5/parcels/5.6.0/CDH-5.6.0-1.cdh5.6.0.p0.45-el6.parcel
wget http://archive-primary.cloudera.com/cdh5/parcels/5.6.0/CDH-5.6.0-1.cdh5.6.0.p0.45-el6.parcel.sha1
wget http://archive-primary.cloudera.com/cdh5/parcels/5.6.0/manifest.json
mv CDH-5.6.0-1.cdh5.6.0.p0.45-el6.parcel.sha1 CDH-5.6.0-1.cdh5.6.0.p0.45-el6.parcel.sha

/opt/cm-5.6.0/etc/init.d/cloudera-scm-server start


11 其他的agent启动:
/opt/cm-5.6.0/etc/init.d/cloudera-scm-agent start
拷贝文件到目录，如果当时没有，就等目录创建了再拷贝
cp /opt/mysql-connector-java-5.1.36-bin.jar /opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hive/lib/
cp /opt/mysql-connector-java-5.1.36-bin.jar  /var/lib/oozie

最后访问：http://120.76.101.XXX:7180/ 
用户名：admin
密码：XXXXXXXXX

参考文档：
http://www.kancloud.cn/cxfeel/cdh543/43518