Simple MapReduce Design

design_doc.md

Driver script image

require 'simple_map_reduce'

c = SimpleMapReduce::Driver::Config.new
job = SimpleMapReduce::Driver::Job.new(config: c)
job.map_task = map_task_object
job.reduce_task = map_task_object
job.input_s3_file_path = 's3://....'
job.job_tracker_url = 'http://hoge.tarou/'
job.start!

feature.md

JobTracker

• api server
• job management worker

Partitioner

JobWorker

• api server
• job execution worker

note.md

local minio config

SimpleMapReduce.s3_config = {
  access_key_id: 'MINIO_ACCESS_KEY',
  secret_access_key: 'MINIO_SECRET_KEY',
  endpoint: 'http://127.0.0.1:9000',
  region: 'us-east-1',
  force_path_style: true
}

Partitioner in Hadoop

• Hadoop uses Partitioner in order to decide which node should process a okey-value pair generated by map task
• The default implement of Partitioner is HashPartitioner
  • implementation(hadoop2.9.0)
• Of course, hash partioning may not result best optimized division. Because hash partitioning divides the list of hash into equal sized lists roughly by the key of hash, sometimes the amount of reduce input size is skewded among the datanodes.

User Definition Script Eval

• By using module_eval, defining class inside the specified module.

module UserDefinitionClass
end

UserDefinitionClass.module_eval("
    class WordCount
      def map
      end

      def reduce
      end
    end
  "
)