Skip to content

Instantly share code, notes, and snippets.

@costin
Created December 16, 2013 10:12
Show Gist options
  • Save costin/7984824 to your computer and use it in GitHub Desktop.
Save costin/7984824 to your computer and use it in GitHub Desktop.
Output from running Cascalog Experiment/Example with Elasticsearch Hadoop
;; lein run -m trial.input
04:27:11,920 INFO main util.HadoopUtil - resolving application jar from found main method on: clojure.main
04:27:11,922 INFO main planner.HadoopPlanner - using application jar: /root/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar
04:27:11,932 INFO main property.AppProps - using app.id: B395F55317CB44779C61FF6B9243266D
04:27:12,447 INFO flow write-to-ES util.Version - Concurrent, Inc - Cascading 2.5.1
04:27:12,451 INFO flow write-to-ES flow.Flow - [write-to-ES] starting
04:27:12,451 INFO flow write-to-ES flow.Flow - [write-to-ES] source: Lfs["TextDelimited[['id', 'name', 'url', 'picture']]"]["resources/artists.dat"]
04:27:12,451 INFO flow write-to-ES flow.Flow - [write-to-ES] sink: ESHadoopTap["ESHadoopScheme[['name', 'url', 'picture']]"]["radio/artists"]
04:27:12,451 INFO flow write-to-ES flow.Flow - [write-to-ES] parallel execution is enabled: false
04:27:12,451 INFO flow write-to-ES flow.Flow - [write-to-ES] starting jobs: 1
04:27:12,452 INFO flow write-to-ES flow.Flow - [write-to-ES] allocating threads: 1
04:27:12,455 INFO pool-1-thread-1 flow.FlowStep - [write-to-ES] starting step: (1/1) radio/artists
04:27:12,518 WARN pool-1-thread-1 util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
04:27:12,654 WARN pool-1-thread-1 mr.ESOutputFormat - Speculative execution enabled for reducer - consider disabling it to prevent data corruption
04:27:12,664 WARN pool-1-thread-1 snappy.LoadSnappy - Snappy native library is available
04:27:12,664 WARN pool-1-thread-1 snappy.LoadSnappy - Snappy native library not loaded
04:27:12,668 INFO pool-1-thread-1 mapred.FileInputFormat - Total input paths to process : 1
04:27:12,988 INFO pool-1-thread-1 flow.FlowStep - [write-to-ES] submitted hadoop job: job_local_0001
04:27:12,988 INFO pool-1-thread-1 flow.FlowStep - [write-to-ES] tracking url: http://localhost:8080/
04:27:13,058 INFO Thread-2 util.ProcessTree - setsid exited with exit code 0
04:27:13,066 INFO Thread-2 mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@320b34b1
04:27:13,072 INFO Thread-2 io.MultiInputSplit - current split input path: file:/tmp/clojure/elasticsearch-hadoop-trial/resources/artists.dat
04:27:13,087 INFO Thread-2 mapred.MapTask - numReduceTasks: 0
04:27:13,091 INFO Thread-2 hadoop.FlowMapper - cascading version: 2.5.1
04:27:13,091 INFO Thread-2 hadoop.FlowMapper - child jvm opts: -Xmx200m
04:27:13,122 INFO Thread-2 hadoop.FlowMapper - sourcing from: Lfs["TextDelimited[['id', 'name', 'url', 'picture']]"]["resources/artists.dat"]
04:27:13,123 INFO Thread-2 hadoop.FlowMapper - sinking to: ESHadoopTap["ESHadoopScheme[['name', 'url', 'picture']]"]["radio/artists"]
04:27:15,030 INFO Thread-2 mapred.Task - Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
04:27:15,032 INFO Thread-2 mapred.LocalJobRunner - file:/tmp/clojure/elasticsearch-hadoop-trial/resources/artists.dat:0+105583
04:27:15,033 INFO Thread-2 mapred.Task - Task 'attempt_local_0001_m_000000_0' done.
;; lein run -m trial.output
04:35:31,597 INFO main property.AppProps - using app.id: 6EC1A3BDE7AF49ACBAA8D3DFBB448D18
04:35:32,408 INFO flow read-from-ES util.Version - Concurrent, Inc - Cascading 2.5.1
04:35:32,414 INFO flow read-from-ES flow.Flow - [read-from-ES] starting
04:35:32,414 INFO flow read-from-ES flow.Flow - [read-from-ES] source: ESLocalTap["ESLocalScheme[[UNKNOWN]->[ALL]]"]["radio/artists"]
04:35:32,414 INFO flow read-from-ES flow.Flow - [read-from-ES] sink: StdOutTap["TextLine[['num', 'line']->[ALL]]"]["stdOut"]
04:35:32,415 INFO flow read-from-ES flow.Flow - [read-from-ES] parallel execution is enabled: true
04:35:32,415 INFO flow read-from-ES flow.Flow - [read-from-ES] starting jobs: 1
04:35:32,415 INFO flow read-from-ES flow.Flow - [read-from-ES] allocating threads: 1
04:35:32,444 INFO pool-1-thread-1 flow.FlowStep - [read-from-ES] starting step: local
Diary of Dreams http://www.last.fm/music/Diary+of+Dreams http://userserve-ak.last.fm/serve/252/3052066.jpg
Carpathian Forest http://www.last.fm/music/Carpathian+Forest http://userserve-ak.last.fm/serve/252/40222717.jpg
Moonspell http://www.last.fm/music/Moonspell http://userserve-ak.last.fm/serve/252/2181591.jpg
Grendel http://www.last.fm/music/Grendel http://userserve-ak.last.fm/serve/252/5872875.jpg
Artista sconosciuto http://www.last.fm/music/Artista+sconosciuto http://userserve-ak.last.fm/serve/252/17969009.png
Feindflug http://www.last.fm/music/Feindflug http://userserve-ak.last.fm/serve/252/17291471.jpg
...
;; lein with-profile cascalog run -m trial.cascalog.input
04:47:13,113 INFO main util.HadoopUtil - resolving application jar from found main method on: clojure.main
04:47:13,308 INFO main planner.HadoopPlanner - using application jar: /root/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar
04:47:13,349 INFO main property.AppProps - using app.id: 868A32BDDF834B8893809F2786B35ED2
04:47:18,373 INFO flow util.Version - Concurrent, Inc - Cascading 2.5.1
04:47:18,416 INFO flow flow.Flow - [] starting
04:47:18,422 INFO flow flow.Flow - [] source: MemorySourceTap["MemorySourceScheme[[UNKNOWN]->[ALL]]"]["/3f5cd5a6-c816-48a7-95d3-425ffeb7cc85"]
04:47:18,422 INFO flow flow.Flow - [] sink: ESHadoopTap["ESHadoopScheme[['?name', '?url', '?picture']]"]["radio/cascalog"]
04:47:18,423 INFO flow flow.Flow - [] parallel execution is enabled: false
04:47:18,423 INFO flow flow.Flow - [] starting jobs: 1
04:47:18,423 INFO flow flow.Flow - [] allocating threads: 1
04:47:18,424 INFO pool-1-thread-1 flow.FlowStep - [] starting step: (1/1) radio/cascalog
04:47:18,906 WARN pool-1-thread-1 util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
04:47:19,146 WARN pool-1-thread-1 mr.ESOutputFormat - Speculative execution enabled for reducer - consider disabling it to prevent data corruption
04:47:19,949 INFO pool-1-thread-1 flow.FlowStep - [] submitted hadoop job: job_local_0001
04:47:19,950 INFO pool-1-thread-1 flow.FlowStep - [] tracking url: http://localhost:8080/
04:47:20,143 INFO Thread-2 util.ProcessTree - setsid exited with exit code 0
04:47:20,205 INFO Thread-2 mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@2babbba1
04:47:20,235 INFO Thread-2 mapred.MapTask - numReduceTasks: 0
04:47:20,258 INFO Thread-2 hadoop.FlowMapper - cascading version: 2.5.1
04:47:20,258 INFO Thread-2 hadoop.FlowMapper - child jvm opts: -Xmx200m
04:47:20,509 INFO Thread-2 hadoop.FlowMapper - sourcing from: MemorySourceTap["MemorySourceScheme[[UNKNOWN]->[ALL]]"]["/3f5cd5a6-c816-48a7-95d3-425ffeb7cc85"]
04:47:20,509 INFO Thread-2 hadoop.FlowMapper - sinking to: ESHadoopTap["ESHadoopScheme[['?name', '?url', '?picture']]"]["radio/cascalog"]
04:47:21,260 INFO Thread-2 mapred.Task - Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
04:47:21,263 INFO Thread-2 mapred.LocalJobRunner -
04:47:21,264 INFO Thread-2 mapred.Task - Task 'attempt_local_0001_m_000000_0' done.
;; lein with-profile cascalog run -m trial.cascalog.input/delimited
04:51:11,413 INFO main util.HadoopUtil - resolving application jar from found main method on: clojure.main
04:51:11,414 INFO main planner.HadoopPlanner - using application jar: /root/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar
04:51:11,420 INFO main property.AppProps - using app.id: 85C1B67C31CE4D04ABA4E28337A787C1
04:51:11,854 INFO flow util.Version - Concurrent, Inc - Cascading 2.5.1
04:51:11,860 INFO flow flow.Flow - [] starting
04:51:11,860 INFO flow flow.Flow - [] source: Lfs["TextDelimited[['?id', '?name', '?url', '?picture']]"]["resources/artists.dat"]
04:51:11,860 INFO flow flow.Flow - [] sink: ESHadoopTap["ESHadoopScheme[['?name', '?url', '?picture']]"]["radio/cascalog"]
04:51:11,861 INFO flow flow.Flow - [] parallel execution is enabled: false
04:51:11,861 INFO flow flow.Flow - [] starting jobs: 1
04:51:11,861 INFO flow flow.Flow - [] allocating threads: 1
04:51:11,864 INFO pool-1-thread-1 flow.FlowStep - [] starting step: (1/1) radio/cascalog
04:51:11,902 WARN pool-1-thread-1 util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
04:51:11,996 WARN pool-1-thread-1 mr.ESOutputFormat - Speculative execution enabled for reducer - consider disabling it to prevent data corruption
04:51:12,009 WARN pool-1-thread-1 snappy.LoadSnappy - Snappy native library is available
04:51:12,009 WARN pool-1-thread-1 snappy.LoadSnappy - Snappy native library not loaded
04:51:12,024 INFO pool-1-thread-1 mapred.FileInputFormat - Total input paths to process : 1
04:51:12,417 INFO pool-1-thread-1 flow.FlowStep - [] submitted hadoop job: job_local_0001
04:51:12,417 INFO pool-1-thread-1 flow.FlowStep - [] tracking url: http://localhost:8080/
04:51:12,573 INFO Thread-2 util.ProcessTree - setsid exited with exit code 0
04:51:12,604 INFO Thread-2 mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@66f472ff
04:51:12,621 INFO Thread-2 io.MultiInputSplit - current split input path: file:/tmp/clojure/elasticsearch-hadoop-trial/resources/artists.dat
04:51:12,636 INFO Thread-2 mapred.MapTask - numReduceTasks: 0
04:51:12,650 INFO Thread-2 hadoop.FlowMapper - cascading version: 2.5.1
04:51:12,654 INFO Thread-2 hadoop.FlowMapper - child jvm opts: -Xmx200m
04:51:12,743 INFO Thread-2 hadoop.FlowMapper - sourcing from: Lfs["TextDelimited[['?id', '?name', '?url', '?picture']]"]["resources/artists.dat"]
04:51:12,743 INFO Thread-2 hadoop.FlowMapper - sinking to: ESHadoopTap["ESHadoopScheme[['?name', '?url', '?picture']]"]["radio/cascalog"]
04:51:14,027 INFO Thread-2 mapred.Task - Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
04:51:14,030 INFO Thread-2 mapred.LocalJobRunner - file:/tmp/clojure/elasticsearch-hadoop-trial/resources/artists.dat:0+105583
04:51:14,030 INFO Thread-2 mapred.Task - Task 'attempt_local_0001_m_000000_0' done.
;; lein with-profile cascalog run -m trial.cascalog.output
04:48:53,033 INFO main mr.ESInputFormat - Discovered mapping {cascalog=[?name=STRING, ?picture=STRING, ?url=STRING]} for [radio/cascalog]
04:48:53,035 INFO main mr.ESInputFormat - Created [5] shard-splits
04:48:53,218 INFO main util.HadoopUtil - resolving application jar from found main method on: clojure.main
04:48:53,219 INFO main planner.HadoopPlanner - using application jar: /root/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar
04:48:53,229 INFO main property.AppProps - using app.id: A00171D8771549BAB3D197986171D2BE
04:48:53,643 INFO flow util.Version - Concurrent, Inc - Cascading 2.5.1
04:48:53,651 INFO flow flow.Flow - [] starting
04:48:53,652 INFO flow flow.Flow - [] source: ESHadoopTap["ESHadoopScheme[[UNKNOWN]->[ALL]]"]["radio/cascalog"]
04:48:53,652 INFO flow flow.Flow - [] sink: StdoutTap["SequenceFile[[UNKNOWN]->['!__G__1965', '!__G__1966', '!__G__1967']]"]["/tmp/temp56157297470734119652513332312376"]
04:48:53,653 INFO flow flow.Flow - [] parallel execution is enabled: false
04:48:53,653 INFO flow flow.Flow - [] starting jobs: 1
04:48:53,653 INFO flow flow.Flow - [] allocating threads: 1
04:48:53,657 INFO pool-1-thread-1 flow.FlowStep - [] starting step: (1/1) ...7470734119652513332312376
04:48:53,714 WARN pool-1-thread-1 util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
04:48:53,815 INFO pool-1-thread-1 mr.ESInputFormat - Discovered mapping {cascalog=[?name=STRING, ?picture=STRING, ?url=STRING]} for [radio/cascalog]
04:48:53,815 INFO pool-1-thread-1 mr.ESInputFormat - Created [5] shard-splits
04:48:54,135 INFO pool-1-thread-1 flow.FlowStep - [] submitted hadoop job: job_local_0001
04:48:54,142 INFO pool-1-thread-1 flow.FlowStep - [] tracking url: http://localhost:8080/
04:48:54,276 INFO Thread-2 util.ProcessTree - setsid exited with exit code 0
04:48:54,300 INFO Thread-2 mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@730eb2f0
04:48:54,331 INFO Thread-2 mapred.MapTask - numReduceTasks: 0
04:48:54,378 INFO Thread-2 hadoop.FlowMapper - cascading version: 2.5.1
04:48:54,378 INFO Thread-2 hadoop.FlowMapper - child jvm opts: -Xmx200m
04:48:54,422 INFO Thread-2 hadoop.FlowMapper - sourcing from: ESHadoopTap["ESHadoopScheme[[UNKNOWN]->[ALL]]"]["radio/cascalog"]
04:48:54,422 INFO Thread-2 hadoop.FlowMapper - sinking to: StdoutTap["SequenceFile[[UNKNOWN]->['!__G__1965', '!__G__1966', '!__G__1967']]"]["/tmp/temp56157297470734119652513332312376"]
04:48:54,438 INFO Thread-2 mapred.Task - Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
04:48:54,441 INFO Thread-2 mapred.LocalJobRunner -
04:48:54,441 INFO Thread-2 mapred.Task - Task attempt_local_0001_m_000000_0 is allowed to commit now
04:48:54,443 INFO Thread-2 mapred.FileOutputCommitter - Saved output of task 'attempt_local_0001_m_000000_0' to file:/tmp/temp56157297470734119652513332312376
04:48:54,443 INFO Thread-2 mapred.LocalJobRunner - ShardInputSplit [node=[4MDSN8Q4QlawIAzcYic6sw/Snowfall|192.168.1.176:9200],shard=3]
04:48:54,444 INFO Thread-2 mapred.Task - Task 'attempt_local_0001_m_000000_0' done.
04:48:54,447 INFO Thread-2 mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@50c8c3b8
04:48:54,452 INFO Thread-2 mapred.MapTask - numReduceTasks: 0
04:48:54,454 INFO Thread-2 hadoop.FlowMapper - cascading version: 2.5.1
04:48:54,454 INFO Thread-2 hadoop.FlowMapper - child jvm opts: -Xmx200m
04:48:54,466 INFO Thread-2 hadoop.FlowMapper - sourcing from: ESHadoopTap["ESHadoopScheme[[UNKNOWN]->[ALL]]"]["radio/cascalog"]
04:48:54,467 INFO Thread-2 hadoop.FlowMapper - sinking to: StdoutTap["SequenceFile[[UNKNOWN]->['!__G__1965', '!__G__1966', '!__G__1967']]"]["/tmp/temp56157297470734119652513332312376"]
04:48:54,472 INFO Thread-2 mapred.Task - Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting
04:48:54,472 INFO Thread-2 mapred.LocalJobRunner -
04:48:54,472 INFO Thread-2 mapred.Task - Task attempt_local_0001_m_000001_0 is allowed to commit now
04:48:54,474 INFO Thread-2 mapred.FileOutputCommitter - Saved output of task 'attempt_local_0001_m_000001_0' to file:/tmp/temp56157297470734119652513332312376
04:48:54,474 INFO Thread-2 mapred.LocalJobRunner - ShardInputSplit [node=[4MDSN8Q4QlawIAzcYic6sw/Snowfall|192.168.1.176:9200],shard=4]
04:48:54,474 INFO Thread-2 mapred.Task - Task 'attempt_local_0001_m_000001_0' done.
04:48:54,476 INFO Thread-2 mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1b1fb141
04:48:54,478 INFO Thread-2 mapred.MapTask - numReduceTasks: 0
04:48:54,484 INFO Thread-2 hadoop.FlowMapper - cascading version: 2.5.1
04:48:54,484 INFO Thread-2 hadoop.FlowMapper - child jvm opts: -Xmx200m
04:48:54,495 INFO Thread-2 hadoop.FlowMapper - sourcing from: ESHadoopTap["ESHadoopScheme[[UNKNOWN]->[ALL]]"]["radio/cascalog"]
04:48:54,495 INFO Thread-2 hadoop.FlowMapper - sinking to: StdoutTap["SequenceFile[[UNKNOWN]->['!__G__1965', '!__G__1966', '!__G__1967']]"]["/tmp/temp56157297470734119652513332312376"]
04:48:54,510 INFO Thread-2 mapred.Task - Task:attempt_local_0001_m_000002_0 is done. And is in the process of commiting
04:48:54,510 INFO Thread-2 mapred.LocalJobRunner -
04:48:54,511 INFO Thread-2 mapred.Task - Task attempt_local_0001_m_000002_0 is allowed to commit now
04:48:54,512 INFO Thread-2 mapred.FileOutputCommitter - Saved output of task 'attempt_local_0001_m_000002_0' to file:/tmp/temp56157297470734119652513332312376
04:48:54,513 INFO Thread-2 mapred.LocalJobRunner - ShardInputSplit [node=[4MDSN8Q4QlawIAzcYic6sw/Snowfall|192.168.1.176:9200],shard=0]
04:48:54,514 INFO Thread-2 mapred.Task - Task 'attempt_local_0001_m_000002_0' done.
04:48:54,516 INFO Thread-2 mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7d8aea9e
04:48:54,525 INFO Thread-2 mapred.MapTask - numReduceTasks: 0
04:48:54,529 INFO Thread-2 hadoop.FlowMapper - cascading version: 2.5.1
04:48:54,531 INFO Thread-2 hadoop.FlowMapper - child jvm opts: -Xmx200m
04:48:54,539 INFO Thread-2 hadoop.FlowMapper - sourcing from: ESHadoopTap["ESHadoopScheme[[UNKNOWN]->[ALL]]"]["radio/cascalog"]
04:48:54,539 INFO Thread-2 hadoop.FlowMapper - sinking to: StdoutTap["SequenceFile[[UNKNOWN]->['!__G__1965', '!__G__1966', '!__G__1967']]"]["/tmp/temp56157297470734119652513332312376"]
04:48:54,546 INFO Thread-2 mapred.Task - Task:attempt_local_0001_m_000003_0 is done. And is in the process of commiting
04:48:54,547 INFO Thread-2 mapred.LocalJobRunner -
04:48:54,547 INFO Thread-2 mapred.Task - Task attempt_local_0001_m_000003_0 is allowed to commit now
04:48:54,548 INFO Thread-2 mapred.FileOutputCommitter - Saved output of task 'attempt_local_0001_m_000003_0' to file:/tmp/temp56157297470734119652513332312376
04:48:54,549 INFO Thread-2 mapred.LocalJobRunner - ShardInputSplit [node=[4MDSN8Q4QlawIAzcYic6sw/Snowfall|192.168.1.176:9200],shard=1]
04:48:54,549 INFO Thread-2 mapred.Task - Task 'attempt_local_0001_m_000003_0' done.
04:48:54,550 INFO Thread-2 mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@12330301
04:48:54,554 INFO Thread-2 mapred.MapTask - numReduceTasks: 0
04:48:54,559 INFO Thread-2 hadoop.FlowMapper - cascading version: 2.5.1
04:48:54,559 INFO Thread-2 hadoop.FlowMapper - child jvm opts: -Xmx200m
04:48:54,565 INFO Thread-2 hadoop.FlowMapper - sourcing from: ESHadoopTap["ESHadoopScheme[[UNKNOWN]->[ALL]]"]["radio/cascalog"]
04:48:54,565 INFO Thread-2 hadoop.FlowMapper - sinking to: StdoutTap["SequenceFile[[UNKNOWN]->['!__G__1965', '!__G__1966', '!__G__1967']]"]["/tmp/temp56157297470734119652513332312376"]
04:48:54,577 INFO Thread-2 mapred.Task - Task:attempt_local_0001_m_000004_0 is done. And is in the process of commiting
04:48:54,577 INFO Thread-2 mapred.LocalJobRunner -
04:48:54,579 INFO Thread-2 mapred.Task - Task attempt_local_0001_m_000004_0 is allowed to commit now
04:48:54,581 INFO Thread-2 mapred.FileOutputCommitter - Saved output of task 'attempt_local_0001_m_000004_0' to file:/tmp/temp56157297470734119652513332312376
04:48:54,581 INFO Thread-2 mapred.LocalJobRunner - ShardInputSplit [node=[4MDSN8Q4QlawIAzcYic6sw/Snowfall|192.168.1.176:9200],shard=2]
04:48:54,581 INFO Thread-2 mapred.Task - Task 'attempt_local_0001_m_000004_0' done.
04:48:54,693 INFO pool-1-thread-1 mapred.FileInputFormat - Total input paths to process : 5
RESULTS
-----------------------
jeroen http://github.com/jeroenvandijk https://secure.gravatar.com/avatar/3e626f6013805e96a95e999fc691b21f?s=420
-----------------------
04:48:54,719 INFO flow util.Hadoop18TapUtil - deleting temp path /tmp/temp56157297470734119652513332312376/_temporary
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment