Skip to content

Instantly share code, notes, and snippets.

@novohool

novohool/使用CDH搭建Hadoop集群

Created May 21, 2018 00:34
Show Gist options
  • Save novohool/8bb83c54d5b1bf86bdcb72fef91ce4ea to your computer and use it in GitHub Desktop.
Save novohool/8bb83c54d5b1bf86bdcb72fef91ce4ea to your computer and use it in GitHub Desktop.
Download ZIP
hadoop 集群搭建
Raw
使用CDH搭建Hadoop集群
master 内存 9500MB 40G
slave-1 内存 7500MB 40G
slave-2 内存 7500MB 40G
安装包：
链接：https://pan.baidu.com/s/1bvgoTFSlvjHq2_TZd333dg 密码：enyl
## Master 主
```shell
安装CDH【https://segmentfault.com/a/1190000012540680】
安装包下载
http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.13.1/RPMS/x86_64/
http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.13.1_x86_64.tar.gz
http://archive.cloudera.com/cdh5/parcels/5.13.1/
最终安装目录这里查看/opt/cm-5.13.1/etc/cloudera-scm-agent/config.ini
初始化【所有机器】
1.更改hosts【所有机器的hosts一致】
2.更改时区-并加入定时任务-1小时同步一次
3.关闭selinux
4.关闭iptables
5.安装ssh lrzsz glibc2.14 gcc gcc-c++
6.ssh免密登陆
其中时间
--------------------------------------------------------------------------------
Master服务器:
yum install ntp -y
cat >>/etc/ntp.conf<<EOF
server time5.aliyun.com
server 0.cn.pool.ntp.org
server 0.asia.pool.ntp.org
server 3.asia.pool.ntp.org
restrict 0.cn.pool.ntp.org nomodify notrap noquery
restrict 0.asia.pool.ntp.org nomodify notrap noquery
restrict 3.asia.pool.ntp.org nomodify notrap noquery
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10
EOF
service ntpd start
chkconfig ntpd on
ntpq -p
Slave 客户端：
cat >>/etc/ntp.conf<<EOF
server master prefer
EOF
service ntpd start
chkconfig ntpd on
ntpdate -u master
clock -w
-------------------------------------------------------------------------------
迅雷下载离线包
https://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.13.1_x86_64.tar.gz
移动文件夹
mv cloudera cm-5.13.1 /opt/
设置环境变量:
export TARBALL_ROOT=/opt/cm-5.13.1/
迅雷下载CDH包
cd /opt/cloudera/parcel-repo/
地址：http://archive.cloudera.com/cdh5/parcels/5.13.1/
包名 CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcel CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcel.sha1 cloudera-manager-el6-cm5.13.1_x86_64.tar.gz
复制文件
cp CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcel.sha1 CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcel.sha
配置mysql-connector：
mkdir -p /usr/share/java
cd /usr/local/src && tar -zxvf mysql-connector-java-5.1.45.tar.gz
cp mysql-connector-java-5.1.45/mysql-connector-java-5.1.45-bin.jar /usr/share/java/mysql-connector-java.jar
cp mysql-connector-java-5.1.45/mysql-connector-java-5.1.45-bin.jar /opt/cm-5.13.1/share/cmf/lib/mysql-connector-java.jar
mkdir -p /opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/hive/lib/
cp mysql-connector-java-5.1.45/mysql-connector-java-5.1.45-bin.jar /opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/hive/lib/mysql-connector-java.jar
迅雷下载RPM包并安装【主要安装 manage 和 java】
地址：http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.13.1/RPMS/x86_64/
mkdir /opt/RPM
cd /opt/RPM
rm -rf /usr/lib64/cmf/agent
yum -y install postgresql-server-8.4.20-8.el6_9.x86_64
cd /usr/local/src;yum localinstall --nogpgcheck --skip-broken jdk-6u31-linux-amd64.rpm oracle*.rpm
yum localinstall cloudera-manager-daemons-5.13.1-1.cm5131.p0.2.el6.x86_64.rpm cloudera-manager-agent-5.13.1-1.cm5131.p0.2.el6.x86_64.rpm
复制agent到其他节点并安装
scp -r /opt/cm-5.13.1 slave-1:/opt
scp -r /opt/cloudera slave-1:/opt
scp -r /opt/RPM slave-1:/opt
所有节点配置agent
vim /opt/cm-5.13.1/etc/cloudera-scm-agent/config.ini
所有节点上创建用户
mkdir -p /opt/cloudera/parcel-cache
useradd --system --home=/opt/cm-5.13.1/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
创建数据库
{
yum -y install mysql-server
service mysqld start
chkconfig mysqld on
mysql -e "create database cmf;create database hive;create database amon;create database rman;create database monitor;create database oozie;create database hue"
mysqladmin -u root password 'passwd'
export MYSQL_PWD='passwd'
}
数据库root用户添加局域网访问
mysql -uroot -p'passwd' -e "grant all privileges on *.* to root@'master' identified by 'passwd' with grant option;flush privileges;"
slave需要改成相应的slave
配置cmf
/opt/cm-5.13.1/share/cmf/schema/scm_prepare_database.sh mysql -hmaster --scm-host master cmf root passwd
查看配置是否成功
cat /opt/cm-5.13.1/etc/cloudera-scm-server/db.properties
修改CMF_JAVA_OPTS参数2G改成4G
vi /opt/cm-5.13.1/etc/default/cloudera-scm-server
配置cmf路径 /opt/cm-5.13.1/sbin/cmf-server
export CMF_PATH="${CMF_SERVER_ROOT:-/opt/cm-5.13.1/share/cmf}"
/opt/cm-5.13.1/etc/
创建目录
mkdir /var/lib/cloudera-scm-server
启动
ln -s /opt/cm-5.13.1/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server
/opt/cm-5.13.1/etc/init.d/cloudera-scm-server start
/opt/cm-5.13.1/etc/init.d/cloudera-scm-agent start 【默认agent不支持离线安装包，可用rpm方式管理】
查看日志
tail -100 /opt/cm-5.13.1/log/cloudera-scm-server/cloudera-scm-server.log
修改
mysql -uroot -p"passwd" -e "show global variables like 'wait_timeout';"
vi /etc/my.cnf
wait_timeout = 31536000
service mysqld restart
安装完成访问
http://192.168.1.169:7180
默认账号密码为admin
查看数据库的账号密码
地址:http://192.168.1.169:7180/cmf/express-wizard/hosts
搜索主机 master,slave-1,slave-2
图形界面安装需要自定义CDH版本为http://archive.cloudera.com/cdh5/parcels/5.13.1/
安装CDH出错【可以在控制面板删除出错的主机】如下：
rm -rf /opt/cm-5.13.1/lib/cloudera-scm-agent/*
/opt/cm-5.13.1/etc/init.d/cloudera-scm-agent restart
还不行的话重置服务端
mysql -uroot -p"passwd" -e "drop database cmf;create database cmf;"
/opt/cm-5.13.1/share/cmf/schema/scm_prepare_database.sh mysql -hmaster --scm-host master cmf root passwd
/opt/cm-5.13.1/etc/init.d/cloudera-scm-server restart
内核设置
echo "10" > /proc/sys/vm/swappiness
vim /etc/sysctl.conf 添加 vm.swappiness = 10
sysctl -p
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
这两条写到/etc/rc.local
JobTracker 本地数据目录
TaskTracker 本地数据目录
mkdir -p /data/job /data/task
```
## 从
```shell
客户端改用rpm包安装【安装正常图形界面就只会显示CDH安装】
cd /usr/local/src;yum localinstall --nogpgcheck --skip-broken jdk-6u31-linux-amd64.rpm oracle*.rpm
yum localinstall cloudera-manager-daemons-5.13.1-1.cm5131.p0.2.el6.x86_64.rpm cloudera-manager-agent-5.13.1-1.cm5131.p0.2.el6.x86_64.rpm
修改配置文件
/etc/cloudera-scm-agent/config.ini
chkconfig cloudera-scm-agent on
安装故障处理
通过网页解除授权，然后再删除相应主机，即可。
若故障依旧进一步排除。
查看主机【第一次看不到】:http://192.168.1.169:7180/cmf/express-wizard/hosts
管理主机:http://192.168.1.169:7180/cmf/hardware/hosts
第一次需要重启客户端
｛
/opt/cm-5.13.1/etc/init.d/cloudera-scm-agent stop
umount /opt/cm-5.13.1/run/cloudera-scm-agent/process
umount /var/run/cloudera-scm-agent/process
ps aux|grep supervisor
rm -rf /tmp/.scm_prepare_node.lock
/etc/init.d/cloudera-scm-agent stop
killall python python2.6
rm -rf /var/lib/cloudera-scm-agent/*
/etc/init.d/cloudera-scm-agent start
｝
重启客户端后发现当前接管的服务器有三台【无需搜索主机 master,slave-1,slave-2】
地址:http://192.168.1.169:7180/cmf/express-wizard/hosts
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment