Skip to content

Instantly share code, notes, and snippets.

@apple-corps
Created January 20, 2015 19:30
Show Gist options
  • Select an option

  • Save apple-corps/002f6629d5db24d28592 to your computer and use it in GitHub Desktop.

Select an option

Save apple-corps/002f6629d5db24d28592 to your computer and use it in GitHub Desktop.
hadoop standby namenode stuck getting block information
2015-01-20 12:20:43,444 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file http://us3sm2zk011r08.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50815793&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4, http://us3sm2zk010r07.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50815793&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4 of size 100746 edits # 569 loaded in 0 seconds
2015-01-20 12:20:43,964 INFO org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Loaded 569 edits starting from txid 50815792
2015-01-20 12:22:44,496 INFO org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering log roll on remote NameNode us3sm2nn011r08.comp.prod.local/10.51.28.141:8020
2015-01-20 12:22:44,619 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@4eba7f05 expecting start txid #50816362
2015-01-20 12:22:44,619 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file http://us3sm2zk012r09.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816362&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4, http://us3sm2zk010r07.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816362&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4
2015-01-20 12:22:44,619 INFO org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Fast-forwarding stream 'http://us3sm2zk012r09.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816362&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4, http://us3sm2zk010r07.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816362&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4' to transaction ID 50816362
2015-01-20 12:22:44,619 INFO org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Fast-forwarding stream 'http://us3sm2zk012r09.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816362&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4' to transaction ID 50816362
2015-01-20 12:22:44,627 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file http://us3sm2zk012r09.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816362&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4, http://us3sm2zk010r07.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816362&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4 of size 34 edits # 2 loaded in 0 seconds
2015-01-20 12:22:45,147 INFO org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Loaded 2 edits starting from txid 50816361
2015-01-20 12:24:45,681 INFO org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering log roll on remote NameNode us3sm2nn011r08.comp.prod.local/10.51.28.141:8020
2015-01-20 12:24:45,790 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@11c1576e expecting start txid #50816364
2015-01-20 12:24:45,790 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file http://us3sm2zk010r07.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816364&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4, http://us3sm2zk011r08.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816364&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4
2015-01-20 12:24:45,790 INFO org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Fast-forwarding stream 'http://us3sm2zk010r07.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816364&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4, http://us3sm2zk011r08.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816364&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4' to transaction ID 50816364
2015-01-20 12:24:45,790 INFO org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Fast-forwarding stream 'http://us3sm2zk010r07.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816364&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4' to transaction ID 50816364
2015-01-20 12:24:45,797 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file http://us3sm2zk010r07.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816364&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4, http://us3sm2zk011r08.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816364&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4 of size 34 edits # 2 loaded in 0 seconds
2015-01-20 12:24:46,925 INFO org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Loaded 2 edits starting from txid 50816363
2015-01-20 12:26:47,458 INFO org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering log roll on remote NameNode us3sm2nn011r08.comp.prod.local/10.51.28.141:8020
2015-01-20 12:26:47,550 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@60630531 expecting start txid #50816366
2015-01-20 12:26:47,550 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file http://us3sm2zk012r09.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816366&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4, http://us3sm2zk010r07.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816366&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4
2015-01-20 12:26:47,550 INFO org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Fast-forwarding stream 'http://us3sm2zk012r09.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816366&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4, http://us3sm2zk010r07.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816366&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4' to transaction ID 50816366
2015-01-20 12:26:47,550 INFO org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Fast-forwarding stream 'http://us3sm2zk012r09.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816366&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4' to transaction ID 50816366
2015-01-20 12:26:47,557 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file http://us3sm2zk012r09.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816366&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4, http://us3sm2zk010r07.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816366&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4 of size 34 edits # 2 loaded in 0 seconds
2015-01-20 12:26:47,950 INFO org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Loaded 2 edits starting from txid 50816365
2015-01-20 12:28:48,359 INFO org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering log roll on remote NameNode us3sm2nn011r08.comp.prod.local/10.51.28.141:8020
2015-01-20 12:28:48,438 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@317d5b4a expecting start txid #50816368
2015-01-20 12:28:48,439 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file http://us3sm2zk010r07.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816368&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4, http://us3sm2zk011r08.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816368&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4
2015-01-20 12:28:48,439 INFO org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Fast-forwarding stream 'http://us3sm2zk010r07.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816368&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4, http://us3sm2zk011r08.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816368&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4' to transaction ID 50816368
2015-01-20 12:28:48,439 INFO org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Fast-forwarding stream 'http://us3sm2zk010r07.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816368&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4' to transaction ID 50816368
2015-01-20 12:28:48,450 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file http://us3sm2zk010r07.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816368&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4, http://us3sm2zk011r08.comp.prod.local:8480/getJournal?jid=whprod&segmentTxId=50816368&storageInfo=-55%3A977585766%3A0%3ACID-1ccb02a5-bfd7-4808-a925-8e9804d40ec4 of size 34 edits # 2 loaded in 0 seconds
2015-01-20 12:28:48,848 INFO org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Loaded 2 edits starting from txid 50816367
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment