Cloudera Setup - VM way

[Setup for the Centos minimal version]

[Note: After setting up the Static network then use SSH to connect the VM for easy editing]

## Static network setup in Centos 7.X minimal
1. Check the default gateway, dns servers address of the host machine
> Default gateway: 192.168.188.1
> DNS servers: 8.8.8.8, 8.8.4.4
2. VM -> Network -> Adapter 1 = NAT, Adapter 2 = Bridged Adapter
3. Start the VM
4. Modify as below
4.1 $ vi /etc/sysconfig/network-scripts/ifcfg-enp0s3 [edit the set up for the ethernet]
    BOOTPROTO=static
    ONBOOT=yes
    IPADDR=192.168.188.88
    GATEWAY=192.168.188.1
    DNS1=8.8.8.8
    DNS2=8.8.4.4

4.2 $ vi /etc/sysconfig/network [configure default getaway]
    NETWORKING=yes
    HOSTNAME=master
    GATEWAY=192.168.188.1

4.3 $ vi /etc/resolv.conf [configure DNS server]
    nameserver 8.8.8.8
    nameserver 8.8.4.4
5. $ /etc/init.d/network restart [restart the network]
6. $ ip addr [check ip address]

# hack: after everything is setup while booting the network doesn't start. If everything fails do this as below
$ vi /etc/rc.d/rc.local
> /etc/init.d/network restart
$ chmod u+x /etc/rc.d/rc.local

Source: https://lintut.com/how-to-configure-static-ip-address-on-centos-7/

## Change the hostname
1. $ vi /etc/sysconfig/network
   HOSTNAME=master
2. $ hostnamectl set-hostname master [enable the hostname on the server]
3. $ /etc/init.d/network restart [restart the network]

Source: https://support.rackspace.com/how-to/centos-hostname-change/

[Setup for the Centos UI version]

01. Download THE Centos 7 --> http://isoredirect.centos.org/centos/7/isos/x86_64/CentOS-7-x86_64-DVD-1611.iso
02. Setup the VirtualMachine.
03. Setup the VirtualMachine network --> http://www.snapdba.com/2013/04/internet-and-ssh-access-to-virtualbox-vm-with-nat-and-host-only-adapters/#.WQmn1HV95nw
04. Install CentOS
05. Always do the configuration in the root user.
06. Configure the network first in CentOS. (For network setup refer to link in #3)
    06.1 Disable IPV6 from the both type of network
    06.2 Check the MAC address to see which one refers to which (don't blind guess)
    06.3 Check the network server from, File > Preferences > Network > Host-only Network > Edits and note the IP address
    06.4 For the manual ip address for "host-only network" setup it should be as same as the network address in #4.3 
07. Remove OPENJDK
    $ yum remove java*
    [No need to setup at this stage, but for future reference!
    	$ wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u162-b12/0da788060d494f5095bf8624735fa2f1/jdk-8u162-linux-i586.rpm
    	$ sudo rpm -ivh jdk-8u162-linux-i586.rpm
    	$ vim /etc/profile.d/java.sh
    	  set the proper java home
    ]
08. $ vi /etc/sysconfig/network
    NETWORKING=yes
    GATEWAY=0.0.0.0
    HOSTNAME=base.example.com
09. /etc/init.d/network restart
10. check the hostname
    $ hostname -f
    10.1 if the hostname didnt change the set the hostname as follows
         $ hostnamectl set-hostname host.domain
11. $ vi /etc/sysconfig/network-scripts/ifcfg-enp0s3/
    11.1 change to "ONBOOT=yes"
    11.2 add "SEARCH=example.com"
12. $ vi /etc/sysconfig/network-scripts/ifcfg-enp0s8/
    12.1 change "ONBOOT=no" to "ONBOOY=yes"
13. /etc/init.d/network restart
14. check for the network connectivity. (ping 8.8.8.8)
15. $ cat /etc/resolv.conf
    you should be able to see the domain name as follows :
    ...
    search example.com
    ...
16. $ vi /etc/selinux/config [All Node]
    15.1 change to "SELINUX=disabled"
17. $ vi /etc/yum/pluginconf.d/fastestmirror.conf
    16.1 change to "enabled=0"
18. Disable ipv6 [All Node]
    $ sysctl -w net.ipv6.conf.all.disable_ipv6=1
19. Disable and Stop firewall
    $ systemctl disable firewalld
    $ systemctl stop firewalld
    $ systemctl mask --now firewalld
20. $ /etc/init.d/network restart
21. Change the swappiness [All Node]
    $ cat /proc/sys/vm/swappiness
    $ echo "vm.swappiness=1" >> /etc/sysctl.conf
22. Transparent hugepage bug workaround: [All Node]
    $ vi /etc/rc.d/rc.local
      if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
         echo never > /sys/kernel/mm/transparent_hugepage/enabled
      fi
      if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
         echo never > /sys/kernel/mm/transparent_hugepage/defrag
      fi
    $ chmod u+x /etc/rc.d/rc.local
    $ cat /sys/kernel/mm/transparent_hugepage/defrag
      [always] madvise never
23. Install NTP [All Node]
    $ sudo yum -y install ntp
    $ ntpdate 0.rhel.pool.ntp.org
    $ systemctl enable ntpd
    $ systemctl start ntpd
    $ sudo chkconfig ntpd on
24. $ yum -y update
25. $ reboot
26. Installing mysql [Master Node] [Try this stuff: https://dev.mysql.com/doc/mysql-yum-repo-quick-guide/en/#repo-qg-yum-fresh-install]
    26.01 $ wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
          if asked for mysql version 5.5 then use as below:
          $ wget http://repo.mysql.com/mysql-community-release-el5-5.noarch.rpm
    26.02 $ sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
    26.03 $ yum -y update
    26.04 $ yum install -y mysql-server
    26.05 $ systemctl start mysqld
    26.06 $ /sbin/chkconfig mysqld on
    26.07 $ service mysqld start
    26.08 $ /usr/bin/mysql_secure_installation
            [...]
            Enter current password for root (enter for none):
            OK, successfully used password, moving on...
            [...]
            Set root password? [Y/n] y
            New password:
            Re-enter new password:
            Remove anonymous users? [Y/n] Y
            [...]
            Disallow root login remotely? [Y/n] N
            [...]
            Remove test database and access to it [Y/n] Y
            [...]
            Reload privilege tables now? [Y/n] Y
            All done!

    26.09  Installing the MySQL JDBC Driver [All Node]
           26.09.01 $ wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.42.tar.gz
           26.09.02 $ tar zxvf mysql-connector-java-5.1.42.tar.gz
           26.09.03 $ mkdir -p /usr/share/java/
           26.09.04 $ cp mysql-connector-java-5.1.42/mysql-connector-java-5.1.42-bin.jar /usr/share/java/mysql-connector-java.jar
           26.09.05 Change the name of the mysql-connector-java-5.1.42-bin.jar to mysql-connector-java.jar
                    $ mv /usr/share/java/mysql-connector-java-5.1.42-bin.jar /usr/share/java/mysql-connector-java.jar
    
    26.10  Creating Databases for Activity Monitor, Reports Manager, Hive Metastore Server, Hue Server, Sentry Server, Cloudera Navigator Audit Server, and Cloudera Navigator Metadata Server

           23.10.01 $ mysql -u root -p
                    Enter password:

           23.10.02 mysql> create database amon DEFAULT CHARACTER SET utf8;
                    mysql> grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password';

                    mysql> create database rman DEFAULT CHARACTER SET utf8;
                    mysql> grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman_password';

                    mysql> create database hive DEFAULT CHARACTER SET utf8;
                    mysql> grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive_password';

                    mysql> create database scm DEFAULT CHARACTER SET utf8;
                    mysql> grant all on scm.* TO 'scm'@'%' IDENTIFIED BY 'scm_password';
                    
                    mysql> create database hue DEFAULT CHARACTER SET utf8;
                    mysql> grant all on hue.* TO 'hue'@'%' IDENTIFIED BY 'hue_password';

                    mysql> create database oozie DEFAULT CHARACTER SET utf8;
                    mysql> grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie_password';

                    mysql> create database metastore DEFAULT CHARACTER SET utf8;
                    mysql> grant all on metastore.* TO 'metastore'@'%' IDENTIFIED BY 'metastore_password';

                    mysql> create database sentry DEFAULT CHARACTER SET utf8;
                    mysql> grant all on sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry_password';

                    mysql> create database nav DEFAULT CHARACTER SET utf8;
                    mysql> grant all on nav.* TO 'nav'@'%' IDENTIFIED BY 'nav_password';

                    mysql> create database navms DEFAULT CHARACTER SET utf8;
                    mysql> grant all on navms.* TO 'navms'@'%' IDENTIFIED BY 'navms_password';

27. Install oracle-j2sdk1.7 [All Node]
      27.01 $ wget https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo
      27.02 $ cp cloudera-manager.repo /etc/yum.repos.d/
      27.03 $ vi /etc/yum.repos.d/cloudera-manager.repo
            > baseurl=https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.10.1/
            $ yum -y update
      27.04 $ yum install -y oracle-j2sdk1.7
28. After reboot, in the VirtualBox menu, select -> "Devices", and then select -> "Insert Guest editions CD Image".
    This insert a DVD with the iso image of the guest additions in the DVD Player of the VM, mount the DVD with the following commands to access this DVD.
    28.01 $ mkdir /media/VBGuest
    28.02 $ mount -r /dev/cdrom /media/VBGuest

29. Setup SSH (To simplify the access between hosts, install and setup SSH keys and defined them as already authorized)
    29.01 $ yum -y install perl openssh-clients
    29.02 $ ssh-keygen (type enter, enter, enter)
    29.03 $ cd ~/.ssh
    29.04 $ cp id_rsa.pub authorized_keys
    29.05 Modify the ssh configuration file. This will prevent the question when connecting with SSH to the host.
          $ vi /etc/ssh/ssh_config
            Find "StrictHostKeyChecking".
            Uncomment and change to "StrictHostKeyChecking no"
30. $ init 0
31. In VirtualBox, clone the base server.
    For the first node (master), change the memory settings to 10GB of memory.
    Most of the roles will be installed on this node, and therefore it is important that it have sufficient memory available. Rest other (hadoop2, hadoop3, hadoop4) clones, change to 4GB memory.
32. Now start all the nodes.
33. Configure the network as below for all the hadoop nodes
    33.01 Update the IPAddress of each node. Settings -> Network ->  Select enp0s8 -> Settings -> IPV4.
          33.01.01 Change the ipaddress to
                   192.168.188.80 (master), 192.168.188.91 (slave1), 192.168.188.92 (slave2), 192.168.188.93 (slave3)
    33.02 Add the hostname for all the nodes in all the nodes "hosts" file [Note about the domain name of the machine from the Host machine]
          $ vi /etc/hosts [machine-ip-address hostname.domainname alias-for-the-machine]
          192.168.188.80 master.fritz.box master
          192.168.188.91 slave1.fritz.box slave1
          192.168.188.92 slave2.fritz.box slave2
          192.168.188.93 slave3.fritz.box slave3
	  ## the output of the FQDN should match with the output of the hostname. ref: https://www.cloudera.com/documentation/enterprise/5-11-x/topics/cdh_ig_networknames_configure.html
	  $ hostname -f
	  $ host -v -t A `hostname`
    33.03 change hostname for all the nodes if set-in hostname in /etc/sysconfig/network didnt work
          $ hostnamectl set-hostname master
    33.04 restart the hostname change
          $ systemctl restart systemd-hostnamed
34. $ /etc/init.d/network restart
35. $ init 6
36. Configuring the Master/Slave Nodes. The master will have server, daemon, agent installed and running.
    Slave will have only daemon and agent installed and running.
    ## Install the Cloudera Manager [Master node]
    $ yum install -y cloudera-manager-daemons cloudera-manager-server cloudera-manager-agent
    $ vi /etc/default/cloudera-scm-agent
      export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
    $ vi /etc/cloudera-scm-agent/config.ini
      [General]
      # Hostname of the CM server.
      server_host=192.168.188.80 [Change the server_host to point to the Master node IP]
    ## Install the Cloudera Manager [Slave nodes]
    $ yum install -y cloudera-manager-daemons cloudera-manager-agent
    $ vi /etc/default/cloudera-scm-agent
      export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
    $ vi /etc/cloudera-scm-agent/config.ini
      [General]
      # Hostname of the CM server.
      server_host=192.168.188.80 [Change the server_host to point to the Master node IP]
    36.01 prepare database for cloudera manager
          # check the "/usr/share/java/mysql-connector-java.jar" is present
          # /usr/share/cmf/schema/scm_prepare_database.sh 'database-type' 'database_name' 'user' 'user_password'
	  $ /usr/share/cmf/schema/scm_prepare_database.sh mysql scm scm scm_password
    36.02 Start cloudera manager [Master node]
	  $ service cloudera-scm-server start
	  $ chkconfig cloudera-scm-server on
	  Start cloudera manager [Slave nodes]
	  $ service cloudera-scm-agent start
	  $ chkconfig cloudera-scm-agent on
    Wait for sometime to download the requried packages.
    Next use a web-browser to connect to the cloudera manager. address: "192.168.188.80:7180"    
37. Once started now start with the configurations and installation.    
38. While in the setup mode of the Cloudera distribution
    i) When the database setup comes, remove the port from the Database Host Name and keep only the hostname.
       example: hadoop1.example.com:7432 .. thus remove the :7432
   ii) When asked for the database password, put the respective database, username, and password for the services created 
       in the mysql database before.

------ Kerberos Setup -------

01. http://blog.puneethabm.in/configure-hadoop-security-with-cloudera-manager-using-kerberos/ [follow for the kerberos]
02. https://gist.github.com/ashrithr/4767927948eca70845db
  
------ Teragen Testing -------

$ su - hdfs
$ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce/hadoop-examples.jar teragen -D dfs.blocksize=16777216 65536000 /user/neymar/tgen640

------ Hive High Availabilty -------

https://www.cloudera.com/documentation/enterprise/5-5-x/topics/admin_ha_hivemetastore.html

1. Configure the HDFS Web Interface Role of Hue service(s) Hue to be an HTTPFS role instead of a NameNode. --> https://www.cloudera.com/documentation/enterprise/5-3-x/topics/cdh_hag_hdfs_ha_cdh_components_config.html#concept_rj1_hsq_bp
2. For each of the Hive service(s) Hive, stop the Hive service, back up the Hive Metastore Database to a persistent store, run the service command "Update Hive Metastore NameNodes", then restart the Hive services. --> https://www.cloudera.com/documentation/enterprise/5-3-x/topics/cdh_hag_hdfs_ha_cdh_components_config.html#topic_2_6_3

------ Mysql ------
connecting to mysql using ip address:
1. https://stackoverflow.com/questions/19101243/error-1130-hy000-host-is-not-allowed-to-connect-to-this-mysql-server#answer-19101356

Setup 4 node Cloudera cluster using Virtual Machine