Easiest HDFS cluster in the world with kubernetes.
Inspiration from kimoonkim/kubernetes-HDFS
kubectl create -f namenode.yaml
kubectl create -f datanode.yaml
Setup a port-forward to so you can see it is alive:
| ### KERNEL TUNING ### | |
| # Increase size of file handles and inode cache | |
| fs.file-max = 2097152 | |
| # Do less swapping | |
| vm.swappiness = 10 | |
| vm.dirty_ratio = 60 | |
| vm.dirty_background_ratio = 2 |
| #!/bin/bash | |
| # bash generate random alphanumeric string | |
| # | |
| # bash generate random 32 character alphanumeric string (upper and lowercase) and | |
| NEW_UUID=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1) | |
| # bash generate random 32 character alphanumeric string (lowercase only) | |
| cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1 |
| #!/bin/bash | |
| # bash generate random alphanumeric string | |
| # | |
| # bash generate random 32 character alphanumeric string (upper and lowercase) and | |
| NEW_UUID=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1) | |
| # bash generate random 32 character alphanumeric string (lowercase only) | |
| cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1 |
| #!/usr/bin/env bash | |
| # install docker | |
| # https://docs.docker.com/engine/installation/linux/ubuntulinux/ | |
| # install docker-compose | |
| # https://docs.docker.com/compose/install/ | |
| # install letsencrypt | |
| # https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-16-04 |
| https://dl.google.com/translate/offline/v3/r1/c.zip | |
| https://dl.google.com/translate/offline/v3/r1/profiles.txt |
| https://dl.google.com/translate/offline/v3/r1/c.zip | |
| https://dl.google.com/translate/offline/v3/r1/profiles.txt |
| http://storage.googleapis.com/books/ngrams/books/googlebooks-eng-all-1gram-20120701-1.gz | |
| http://storage.googleapis.com/books/ngrams/books/googlebooks-eng-all-1gram-20120701-2.gz | |
| http://storage.googleapis.com/books/ngrams/books/googlebooks-eng-all-1gram-20120701-0.gz | |
| http://storage.googleapis.com/books/ngrams/books/googlebooks-eng-all-1gram-20120701-3.gz | |
| http://storage.googleapis.com/books/ngrams/books/googlebooks-eng-all-1gram-20120701-4.gz | |
| http://storage.googleapis.com/books/ngrams/books/googlebooks-eng-all-1gram-20120701-5.gz | |
| http://storage.googleapis.com/books/ngrams/books/googlebooks-eng-all-1gram-20120701-6.gz | |
| http://storage.googleapis.com/books/ngrams/books/googlebooks-eng-all-1gram-20120701-7.gz | |
| http://storage.googleapis.com/books/ngrams/books/googlebooks-eng-all-1gram-20120701-8.gz | |
| http://storage.googleapis.com/books/ngrams/books/googlebooks-eng-all-1gram-20120701-9.gz |
Easiest HDFS cluster in the world with kubernetes.
Inspiration from kimoonkim/kubernetes-HDFS
kubectl create -f namenode.yaml
kubectl create -f datanode.yaml
Setup a port-forward to so you can see it is alive:
Set up Kubernetes on 3 Debian Jessie virtual machines: One master. Two nodes. Additionally do this without any "magic" so that what is required to be running to make everything work is plain and obvious.
We will be using flannel for the inter-machine networking layer. Mainly because it is useful and it seems to be pretty popular.