gistfile1.txt

1341938784.370396802 src/CoordinatorService.cc:885 in CoordinatorService::verifyServerFailure default NOTICE[2476:2]: Verified host failure: id 1 ("infrc:host=192.168.1.122,port=12249")
1341938784.370421087 src/CoordinatorService.cc:434 in CoordinatorService::hintServerDown default NOTICE[2476:2]: Server id 1 has crashed, notifying the cluster and starting recovery
1341938784.372789723 src/MasterRecoveryManager.cc:469 in MasterRecoveryManager::restartMasterRecovery default NOTICE[2476:2]: Scheduling recovery of master 1
1341938784.372906364 src/CoordinatorService.cc:427 in CoordinatorService::hintServerDown default NOTICE[2476:2]: Spurious crash report on unknown server id 1
1341938784.372925556 src/CoordinatorService.cc:427 in CoordinatorService::hintServerDown default NOTICE[2476:2]: Spurious crash report on unknown server id 1
1341938784.373121881 src/MasterRecoveryManager.cc:193 in MasterRecoveryManagerInternal::MaybeStartRecoveryTask::performTask default NOTICE[2476:3]: Starting recovery of server 1 (now 1 active recoveries)
1341938784.373401275 src/Recovery.cc:440 in Recovery::buildReplicaMap default NOTICE[2476:3]: Segment 0 of length 197628 bytes from backup infrc:host=192.168.1.145,port=12249 is the head of the log
1341938784.373444316 src/Recovery.cc:649 in Recovery::startRecoveryMasters default NOTICE[2476:3]: Starting recovery for 4 partitions
1341938784.373461236 src/Recovery.cc:550 in RecoveryInternal::MasterStartTask::send default NOTICE[2476:3]: Initiating recovery with recovery master infrc:host=192.168.1.127,port=12249, partition 0
1341938784.373486574 src/Recovery.cc:550 in RecoveryInternal::MasterStartTask::send default NOTICE[2476:3]: Initiating recovery with recovery master infrc:host=192.168.1.130,port=12249, partition 1
1341938784.373509580 src/Recovery.cc:550 in RecoveryInternal::MasterStartTask::send default NOTICE[2476:3]: Initiating recovery with recovery master infrc:host=192.168.1.129,port=12249, partition 2
1341938784.373531690 src/Recovery.cc:550 in RecoveryInternal::MasterStartTask::send default NOTICE[2476:3]: Initiating recovery with recovery master infrc:host=192.168.1.124,port=12249, partition 3
1341938784.374883187 src/Recovery.cc:582 in RecoveryInternal::MasterStartTask::wait default WARNING[2476:3]: recover failed on infrc:host=192.168.1.124,port=12249, trying next master; failure was: RPC response improperly formatted
1341938784.374899484 src/Recovery.cc:550 in RecoveryInternal::MasterStartTask::send default NOTICE[2476:3]: Initiating recovery with recovery master infrc:host=192.168.1.124,port=12249, partition 3
1341938784.374967070 src/Recovery.cc:582 in RecoveryInternal::MasterStartTask::wait default WARNING[2476:3]: recover failed on infrc:host=192.168.1.124,port=12249, trying next master; failure was: RPC response improperly formatted
1341938784.374982448 src/Recovery.cc:550 in RecoveryInternal::MasterStartTask::send default NOTICE[2476:3]: Initiating recovery with recovery master infrc:host=192.168.1.124,port=12249, partition 3
1341938784.375020549 src/ServiceManager.cc:348 in ServiceManager::workerMain default ERROR[2476:2]: worker: basic_string::_S_construct NULL not valid
1341938784.375020827 src/Recovery.cc:582 in RecoveryInternal::MasterStartTask::wait default WARNING[2476:3]: recover failed on infrc:host=192.168.1.130,port=12249, trying next master; failure was: RPC response improperly formatted
1341938784.375068103 src/Recovery.cc:550 in RecoveryInternal::MasterStartTask::send default NOTICE[2476:3]: Initiating recovery with recovery master infrc:host=192.168.1.130,port=12249, partition 1
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_S_construct NULL not valid
1341938784.375102925 src/Recovery.cc:582 in RecoveryInternal::MasterStartTask::wait default WARNING[2476:3]: recover failed on infrc:host=192.168.1.124,port=12249, trying next master; failure was: RPC response improperly formatted
Aborted (core dumped)
[christian@rc21 obj.master]$