Install Hadoop on Mac (M1, M2) with Apple Silicon

Install brew (Homebrew)
Install hadoop
```
brew install hadoop
```

cd into hadoop configuration files

cd /opt/homebrew/Cellar/hadoop/3.3.4/libexec/etc/hadoop

Update hadoop-env.sh conf
```
vim hadoop-env.sh
```
Find export JAVA_HOME line and uncomment it and put your $JAVA_HOME path in there you can also find it using /usr/libexec/java_home

Update core-site.xml conf

vim core-site.xml

Add configuration

<configuration>
 <property>
  <name>fs.defaultFS</name>
  <value>hdfs://localhost:9000</value>
 </property>
</configuration>

Update hdfs-site.xml

vim hdfs-site.xml

Add configuration

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

Update mapred-site.xml

vim mapred-site.xml

Add configuration

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.application.classpath</name>   
    <value>
      $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
    </value>
  </property>
</configuration>

Update yarn-site.xml

vim yarn-site.xml

Add configuration

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.env-whitelist</name>  
    <value>
      JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
    </value>
  </property>
</configuration>

Enable remote login from "Sharing" in system preferences and allow Administrators from "Only these users:" radio button

Run

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Run

stop-all.sh
hadoop namenode -format
start-all.sh

Run jps to verify all the installation

16641 NameNode
16882 SecondaryNameNode
17170 NodeManager
16741 DataNode
17279 Jps
17071 ResourceManager

Grab popcorns and work with Hadoop!

fahadsiddiqui/install-hadoop-mac-apple-silicon.md