Install Hadoop on Mac (M1, M2) with Apple Silicon

install-hadoop-mac-apple-silicon.md

• Install brew (Homebrew)

• Install hadoop

  brew install hadoop

• cd into hadoop configuration files

  cd /opt/homebrew/Cellar/hadoop/3.3.4/libexec/etc/hadoop
  

• Update hadoop-env.sh conf

  vim hadoop-env.sh
  

  Find export JAVA_HOME line and uncomment it and put your $JAVA_HOME path in there you can also find it using /usr/libexec/java_home

• Update core-site.xml conf

  vim core-site.xml

  Add configuration

  <configuration>
   <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
   </property>
  </configuration>

• Update hdfs-site.xml

  vim hdfs-site.xml

  Add configuration

  <configuration>
    <property>
      <name>dfs.replication</name>
      <value>1</value>
    </property>
  </configuration>

• Update mapred-site.xml

  vim mapred-site.xml

  Add configuration

  <configuration>
    <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
    </property>
    <property>
      <name>mapreduce.application.classpath</name>   
      <value>
        $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
      </value>
    </property>
  </configuration>

• Update yarn-site.xml

  vim yarn-site.xml

  Add configuration

  <configuration>
    <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
    </property>
    <property>
      <name>yarn.nodemanager.env-whitelist</name>  
      <value>
        JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
      </value>
    </property>
  </configuration>

• Enable remote login from "Sharing" in system preferences and allow Administrators from "Only these users:" radio button

• Run

  cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
  

• Run

  stop-all.sh
  hadoop namenode -format
  start-all.sh

• Run jps to verify all the installation

  16641 NameNode
  16882 SecondaryNameNode
  17170 NodeManager
  16741 DataNode
  17279 Jps
  17071 ResourceManager
  

• Grab popcorns and work with Hadoop!

I keep getting:

WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
ERROR: Invalid HADOOP_COMMON_HOME