Cloudera Setup - AWS way

0. Login with Root mode and always give the commands with root access
   $ sudo su

1. Pre-Installation Steps [In all the nodes]

1.1  Fix Swappiness:
	$ cat /proc/sys/vm/swappiness
	$ sysctl -w vm.swappiness=1
	$ cat /proc/sys/vm/swappiness
	$ vi /etc/sysctl.conf
      vm.swappiness=1 [add this line in the end]

1.2 Disable firewall and selinux:
	$ /etc/init.d/iptables stop
	$ chkconfig iptables off
	$ chkconfig ip6tables off
	$ vi /etc/selinux/config
 	  change to "SELINUX=disabled" or "SELINUX=permissive"

1.3 Transparent hugepage bug workaround:
    $ vi /etc/rc.d/rc.local
      if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
         echo never > /sys/kernel/mm/transparent_hugepage/enabled
      fi
      if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
         echo never > /sys/kernel/mm/transparent_hugepage/defrag
      fi
    $ chmod u+x /etc/rc.d/rc.local
    $ cat /sys/kernel/mm/transparent_hugepage/defrag
      always madvise [never]

1.4 Disable transparent hugepage support:
	$ vi /etc/grub.conf
      #add "transparent_hugepage=never" to the kernel 
	$ reboot
      #check that it worked
	  $ cat /sys/kernel/mm/transparent_hugepage/enabled
	    >> always madvise [never]

1.5 Set ulimits for common users:
	$ vi /etc/security/limits.d/90-nproc.conf
      #change the value 1024 on category (*) to 4096
      #logout and login
      #check to see that the limit has been set
	  $ ulimit -u (should respond with 4096)
	  $ ulimit -n (should respond with 1024)

1.6 Install ntpd:
	$ yum install -y ntp
 	$ chkconfig ntpd on
	$ service ntpd start
	$ service ntpd status

1.7 Install nscd (to cache dns entries): 
	$ yum install -y nscd
	$ service nscd start
	$ chkconfig nscd on

1.8 $ yum update
    $ reboot

2. Install mysql [Server in Master node, client in slave nodes and connector in all the nodes]

2.1 Download the mysql repo: [All the nodes]
    $ wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
    $ rpm -ivh mysql-community-release-el7-5.noarch.rpm
    
2.2 Default version of MySQL enabled in this repository is 5.7. However, depending upon the version asked, we need to change the repo file. [All nodes]
    # first check the /etc/yum.repos.d/mysql-community.repo repo file of mysql then for example, If we are asked to setup for mysql 5.5 then do this:
	$ cat /etc/yum.repos.d/mysql-community.repo 
	  [mysql-connectors-community]
	  name=MySQL Connectors Community
	  baseurl=http://repo.mysql.com/yum/mysql-connectors-community/el/6/$basearch/
	  enabled=1
	  gpgcheck=1
	  gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

	  [mysql-tools-community]
	  name=MySQL Tools Community
	  baseurl=http://repo.mysql.com/yum/mysql-tools-community/el/6/$basearch/
	  enabled=1
	  gpgcheck=1
	  gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

	  # Enable to use MySQL 5.5
	  [mysql55-community]
	  name=MySQL 5.5 Community Server
	  baseurl=http://repo.mysql.com/yum/mysql-5.5-community/el/6/$basearch/
	  enabled=1
	  gpgcheck=1
	  gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

	  # Enable to use MySQL 5.6
	  [mysql56-community]
	  name=MySQL 5.6 Community Server
	  baseurl=http://repo.mysql.com/yum/mysql-5.6-community/el/6/$basearch/
	  enabled=0
	  gpgcheck=1
	  gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

	  [mysql57-community]
	  name=MySQL 5.7 Community Server
	  baseurl=http://repo.mysql.com/yum/mysql-5.7-community/el/6/$basearch/
	  enabled=0
	  gpgcheck=1
	  gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

	  # check the content of the [mysql55-community]
	  # make "enabled=1" for this only
	  # for other contents in [mysql56-community] and [mysql57-community]
      # make "enabled=0"
    $ yum update
    $ yum install mysql-server [Master Node]
    $ yum install mysql        [Slave Nodes]

2.3 Start mysql service in all the nodes:
    $ service mysqld start
    $ chkconfig mysqld on

2.4 Installing the MySQL JDBC Driver [All Nodes]
    $ wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.42.tar.gz
    $ tar zxvf mysql-connector-java-5.1.42.tar.gz
    $ mkdir -p /usr/share/java/
    $ cp mysql-connector-java-5.1.42/mysql-connector-java-5.1.42-bin.jar /usr/share/java/mysql-connector-java.jar
    
    # Change the name of the mysql-connector-java-5.1.42-bin.jar to mysql-connector-java.jar
    $ mv /usr/share/java/mysql-connector-java-5.1.42-bin.jar /usr/share/java/mysql-connector-java.jar

2.5 Run MySQL secure installation: [Master Node]
    $ /usr/bin/mysql_secure_installation
      [...]
      Enter current password for root (enter for none): [just press enter]
      OK, successfully used password, moving on...
      [...]
      Set root password? [Y/n] y
      New password:
      Re-enter new password:
      Remove anonymous users? [Y/n] Y
      [...]
      Disallow root login remotely? [Y/n] N
      [...]
      Remove test database and access to it [Y/n] Y
      [...]
      Reload privilege tables now? [Y/n] Y
      All done!

2.6 Restart the MySQL service
    $ service mysqld restart

2.7 Creating Databases for Activity Monitor, Reports Manager, Hive Metastore Server, Hue Server, Sentry Server, Cloudera Navigator Audit Server, and Cloudera Navigator Metadata Server [Master Node]

    $ mysql -u root -p
    Enter password:

    mysql> create database amon DEFAULT CHARACTER SET utf8;
    mysql> grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password';

    mysql> create database rman DEFAULT CHARACTER SET utf8;
    mysql> grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman_password';

    mysql> create database hive DEFAULT CHARACTER SET utf8;
    mysql> grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive_password';

    mysql> create database scm DEFAULT CHARACTER SET utf8;
    mysql> grant all on scm.* TO 'scm'@'%' IDENTIFIED BY 'scm_password';
                    
    mysql> create database hue DEFAULT CHARACTER SET utf8;
    mysql> grant all on hue.* TO 'hue'@'%' IDENTIFIED BY 'hue_password';

    mysql> create database oozie DEFAULT CHARACTER SET utf8;
    mysql> grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie_password';

    mysql> create database metastore DEFAULT CHARACTER SET utf8;
    mysql> grant all on metastore.* TO 'metastore'@'%' IDENTIFIED BY 'metastore_password';

    mysql> create database sentry DEFAULT CHARACTER SET utf8;
    mysql> grant all on sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry_password';

    mysql> create database nav DEFAULT CHARACTER SET utf8;
    mysql> grant all on nav.* TO 'nav'@'%' IDENTIFIED BY 'nav_password';

    mysql> create database navms DEFAULT CHARACTER SET utf8;
    mysql> grant all on navms.* TO 'navms'@'%' IDENTIFIED BY 'navms_password';

    mysql> show databases;

3. Install Cloudera Manager [Master node]
   
3.1 Download CM from the repo
    $ wget https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo

    # If asked for a particular CM version then need to change the base url.
    # for example, if needed to download CM 5.8.3 then 
    $ vi cloudera-manager.repo

      [cloudera-manager]
      # Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64        
      name=Cloudera Manager
      baseurl=https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.8.3/                  [change this line url to point to 5.8.3 version]
      gpgkey =https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera
      gpgcheck = 1

    $ cp cloudera-manager.repo /etc/yum.repos.d/
    $ yum install -y oracle-j2sdk1.7
    $ yum install cloudera-manager-daemons cloudera-manager-server

3.2 prepare database for cloudera manager
	$ /usr/share/cmf/schema/scm_prepare_database.sh 'database-type' 'database_name' 'user' 'user_password'
    e.g.(sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql scm scm scm_password)
    $ service cloudera-scm-server start

    # Wait for sometime to download the requried packages.
    # Login on a browser using the public ip and the default port 7180
	# default credentials are username:admin, password, admin
    # choose cdh parcels, use pem file to login and don't select single use mode


EXTRAS::

Way to add new user and add to new groups

0. Login and change to root mode.
   $ sudo su
1. Adding the user with UID
   $ adduser -u 2010 neymar
   $ adduser -u 2016 ronaldo
2. Create group and add user
   $ groupadd merengues
   $ groupadd barca
   $ usermod -a -G merengues neymar
   $ usermod -a -G barca ronaldo
3. $ grep ronaldo /etc/passwd
   $ grep neymar /etc/passwd
4. $ grep barca /etc/group
   $ grep merengues /etc/group


TESTING:: [In Master Node]


First, prepare test data by running Teragen. Only do this once.

Teragen testing: [note about the byte size, number of maps]

$ su - hdfs
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen -Ddfs.blocksize=16M -Dmapred.map.tasks=6 50000000 /user/raffles/terasort/terasort-input

Run terasort on the data prepped by Teragen.  Run as many times as you'd like:

Terasort testing:

$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar terasort /user/raffles/terasort/terasort-input /user/raffles/terasort/terasort-output