mgpradeepa

In the distributed world, while writing map-reduce codes, there are many situations where the input data seems to be non partitiionable. In which case all the data though would be picked up by multiple mappers, it gets mapped to the same key. Once all the mappers are run and if we end up in having one key with huge list of values, then we would be burdening the reducers. When I say burdening reducers it invovles burdening all the steps after mapper till the data enters the reducer nodes.

Lets take an example to deal more on this situation and see how this can be resolved.

Challenge: Finding of average of natural numbers.

Bird-view:

mgpradeepa

Spring Integration framework.

Mainly focusses on the Enterprise Integration Application Framework. Design patterns are obtained and utilized as a integration solution from those.

One of the component is Message

Message is a generic wrapper for a java object combined with the metadata used by the framework. Payload & Headers are the key components of a message. Payload:- Can be of anything and any type data. Headers:-
Holds commonly required information like id, time_stamp, correlation_id and return address.

mgpradeepa

Create user
Login as root
$ sudo su -
$ useradd -m <userName> -p <password>

Add to the group 
$ usermod -aG <someGroup> <userName>

For granting sudo or root access
visudo

mgpradeepa

OS: Centos
Requisite: JAVA 7+8

sudo vi /etc/profile
export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk
export JRE_HOME=/usr/lib/jvm/jre

Download Kafka from any of the given website: the one use here is kafka_2.11-0.10.1.1.tgz
Untar:  tar -xvf kafka_2.11-0.10.1.1.tgz
sudo mv kafka_2.11-0.10.1.1.tgz /opt

mgpradeepa

#!/bin/bash

#script extraido de: http://paulocassiano.wordpress.com/2008/08/29/deixando-o-gedit-com-a-cara-do-textmate/
#tip for better "resolution" here: http://blog.siverti.com.br/2008/05/22/fonte-monaco-no-ubuntugedit/

cd /usr/share/fonts/truetype/

#TODO: put validation if folder already exists
sudo mkdir ttf-monaco

mgpradeepa

Single responsibility to single method
No more than 10 lines of code in a method.

when ever you see extract it right away
have your code comments on the way

use shortcuts on your side
IntelliJ is the best ide to slide

move the responsibility to the classes of its own

mgpradeepa

The docker system in which docker is installed or running can run into dangling state. 

   To check whats the disk consumption due to Docker 
    docker system df 
    
    Primarily the issues comes up is with low disk space. Of course this is re-claimable.      
        
    Cause for Low disk space are 
    (a) docker images created during build could be in dangling state
                                      or would have been stored or persisted for a longer time. 

mgpradeepa

In WINDOWS machines use the following steps

Start putty gen tool in windows
Look at ACTIONS section and choose "Load" button
Browse through the folders to pull the ppk file fo which you need to create a pem file
Key passphrase is optional and cick YES to ignore
From the menu , choose Conversions  --> Drop down --> Export OpenSSH Key.
Just say 'Yes' for warning of not entering passphrase
Save the file as <xyz>.pem

mgpradeepa

Apache kafka

Platform for real-time distributed streaming which just behaves like any other MESSAGING system

Design goals

• Scalability
• High Volume
• Data Transformations
• Low Latency
• Fault Tolerant

mgpradeepa

** /boot folder gets filled up as and when upgrades are done. How to free it up **

Lets check by auto remove

sudo apt autoremove If the above does not remove old ones lets first check the current kernel thats being used

uname -r

Lists the current used kernel version. Lets remove the earlier versions of the kernel upgrades