Skip to content

Instantly share code, notes, and snippets.

@jdurgin

jdurgin/osd_wallclock_profile.txt

Created September 3, 2020 22:55
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save jdurgin/9628c529e009bd61e06f9dac8da2518e to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save jdurgin/9628c529e009bd61e06f9dac8da2518e to your computer and use it in GitHub Desktop.
Download ZIP
4k random write osd wallclock profile
Raw
osd_wallclock_profile.txt
Thread: 1 (ceph-osd) - 1000 samples
+ 100.00% main
+ 100.00% AsyncMessenger::wait
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 2 (log) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% ceph::logging::Log::entry
+ 99.90% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| + 99.90% pthread_cond_wait@@GLIBC_2.3.2
+ 0.10% std::unique_lock<std::mutex>::unlock
Thread: 3 (service) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% CephContextServiceThread::entry
+ 100.00% std::condition_variable::wait_for<unsigned long, std::ratio<1l, 1000000000l> >
+ 100.00% std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >
+ 100.00% std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >
+ 100.00% __gthread_cond_timedwait
+ 100.00% pthread_cond_timedwait@@GLIBC_2.3.2
Thread: 4 (admin_socket) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% execute_native_thread_routine
+ 100.00% AdminSocket::entry
+ 100.00% poll
+ 100.00% poll
Thread: 5 (msgr-worker-0) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% execute_native_thread_routine
+ 100.00% std::_Function_handler<void(), NetworkStack::add_thread(unsigned int)::<lambda()> >::_M_invoke(const std::_Any_data &)
+ 100.00% NetworkStack::<lambda()>::operator()
+ 100.00% EventCenter::process_events
+ 100.00% EpollDriver::event_wait
+ 100.00% epoll_wait
Thread: 6 (msgr-worker-1) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% execute_native_thread_routine
+ 100.00% std::_Function_handler<void(), NetworkStack::add_thread(unsigned int)::<lambda()> >::_M_invoke(const std::_Any_data &)
+ 100.00% NetworkStack::<lambda()>::operator()
+ 100.00% EventCenter::process_events
+ 44.20% AsyncConnection::process
| + 43.60% std::function<void (char*, long)>::operator()(char*, long) const
| | + 43.60% std::_Function_handler<void(char*, long int), ProtocolV1::read(CONTINUATION_RX_TYPE<ProtocolV1>&, int, char*)::<lambda(char*, int)> >::_M_invoke(const std::_Any_data &, char *&&, long &&)
| | + 43.60% ProtocolV1::<lambda(char*, int)>::operator()
| | + 22.00% ProtocolV1::handle_message_footer
| | | + 13.10% DispatchQueue::fast_dispatch
| | | | + 13.10% DispatchQueue::fast_dispatch
| | | | + 12.50% Messenger::ms_fast_dispatch
| | | | | + 12.40% Dispatcher::ms_fast_dispatch2
| | | | | | + 12.40% OSD::ms_fast_dispatch
| | | | | | + 8.30% OSD::enqueue_op
| | | | | | + 4.00% OpTracker::create_request<OpRequest, Message*>
| | | | | + 0.10% boost::intrusive_ptr<Message const>::intrusive_ptr<Message>
| | | | + 0.50% DispatchQueue::post_dispatch
| | | | + 0.10% DispatchQueue::pre_dispatch
| | | + 7.00% decode_message
| | | + 0.60% ceph::buffer::v14_2_0::list::clear
| | | + 0.20% ceph::time_detail::mono_clock::now
| | | + 0.20% PerfCounters::tinc
| | | + 0.20% Connection::is_blackhole
| | | + 0.10% std::mutex::unlock
| | | + 0.10% std::mutex::lock
| | | + 0.10% boost::intrusive_ptr<Message const>::~intrusive_ptr
| | | + 0.10% boost::intrusive_ptr<Connection>::intrusive_ptr
| | | + 0.10% ProtocolV1::<lambda(auto:102)>::operator()<CephContext*>
| | | + 0.10% Messenger::ms_fast_preprocess
| | + 14.30% ProtocolV1::read_message_data
| | | + 13.80% ProtocolV1::read
| | | | + 13.80% AsyncConnection::read(unsigned int, char*, std::function<void (char*, long)>)
| | | | + 13.80% AsyncConnection::read_until
| | | | + 12.50% AsyncConnection::read_bulk
| | | | | + 12.40% ConnectedSocket::read
| | | | | + 12.20% PosixConnectedSocketImpl::read
| | | | | + 12.10% read
| | | | | + 12.10% read
| | | | | + 0.60% __pthread_enable_asynccancel
| | | | + 1.30% memcpy
| | | + 0.20% ceph::buffer::v14_2_0::ptr::~ptr
| | | + 0.10% ceph::buffer::v14_2_0::list::iterator_impl<false>::get_current_ptr
| | | + 0.10% ProtocolV1::read_message_footer
| | + 2.80% ProtocolV1::handle_message_front
| | + 1.50% ProtocolV1::throttle_dispatch_queue
| | + 0.80% ProtocolV1::wait_message
| | + 0.80% ProtocolV1::throttle_bytes
| | + 0.50% ProtocolV1::handle_message_header
| | + 0.50% ProtocolV1::handle_message_data
| | + 0.30% ProtocolV1::handle_message
| + 0.60% AsyncConnection::read(unsigned int, char*, std::function<void (char*, long)>)
+ 28.60% AsyncConnection::handle_write
| + 28.60% ProtocolV1::write_event
| + 19.60% ProtocolV1::write_message
| | + 16.40% AsyncConnection::_try_send
| | | + 16.40% ConnectedSocket::send
| | | + 16.40% PosixConnectedSocketImpl::send
| | | + 16.40% PosixConnectedSocketImpl::do_sendmsg
| | | + 16.30% sendmsg
| | + 1.80% RefCountedObject::put
| | + 1.10% ceph::buffer::v14_2_0::list::append
| | + 0.20% std::__atomic_base<unsigned long>::operator++
| | + 0.10% ceph::buffer::v14_2_0::list::claim_append
| + 6.10% ProtocolV1::prepare_send_message
| + 1.10% ProtocolV1::_get_next_outgoing
| + 1.00% std::mutex::lock
| + 0.30% std::mutex::unlock
| + 0.30% PerfCounters::tinc
| + 0.20% ceph::time_detail::mono_clock::now
+ 25.50% EpollDriver::event_wait
| + 25.50% epoll_wait
+ 1.40% C_handle_notify::do_request
+ 0.10% std::deque<EventCallback*, std::allocator<EventCallback*> >::~deque
+ 0.10% std::deque<EventCallback*, std::allocator<EventCallback*> >::deque
Thread: 7 (msgr-worker-2) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% execute_native_thread_routine
+ 100.00% std::_Function_handler<void(), NetworkStack::add_thread(unsigned int)::<lambda()> >::_M_invoke(const std::_Any_data &)
+ 100.00% NetworkStack::<lambda()>::operator()
+ 100.00% EventCenter::process_events
+ 97.90% EpollDriver::event_wait
| + 97.90% epoll_wait
+ 2.10% AsyncConnection::handle_write
Thread: 8 (signal_handler) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% SignalHandler::entry
+ 100.00% poll
+ 100.00% poll
Thread: 9 (OpHistorySvc) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% OpHistoryServiceThread::entry
+ 99.60% usleep
| + 99.60% nanosleep
+ 0.20% std::pair<utime_t, boost::intrusive_ptr<TrackedOp> >::~pair
+ 0.20% OpHistory::_insert_delayed
Thread: 10 (ceph-osd) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% execute_native_thread_routine
+ 100.00% ceph::timer_detail::timer<ceph::time_detail::coarse_mono_clock>::timer_thread
+ 100.00% std::condition_variable::wait_until<ceph::time_detail::coarse_mono_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >
+ 100.00% std::condition_variable::__wait_until_impl<std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >
+ 100.00% __gthread_cond_timedwait
+ 100.00% pthread_cond_timedwait@@GLIBC_2.3.2
Thread: 11 (safe_timer) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% SafeTimerThread::entry
+ 100.00% SafeTimer::timer_thread
+ 100.00% std::condition_variable::wait_until<ceph::time_detail::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >
+ 100.00% std::condition_variable::__wait_until_impl<std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >
+ 100.00% __gthread_cond_timedwait
+ 100.00% pthread_cond_timedwait@@GLIBC_2.3.2
Thread: 12 (safe_timer) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% SafeTimerThread::entry
+ 100.00% SafeTimer::timer_thread
+ 95.00% std::condition_variable::wait_until<ceph::time_detail::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >
| + 95.00% std::condition_variable::__wait_until_impl<std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >
| + 95.00% __gthread_cond_timedwait
| + 95.00% pthread_cond_timedwait@@GLIBC_2.3.2
+ 5.00% Context::complete
Thread: 13 (safe_timer) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% SafeTimerThread::entry
+ 100.00% SafeTimer::timer_thread
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 14 (safe_timer) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% SafeTimerThread::entry
+ 100.00% SafeTimer::timer_thread
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 15 (fn_anonymous) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% Finisher::finisher_thread_entry
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 16 (bstore_aio) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% KernelDevice::AioCompletionThread::entry
+ 100.00% KernelDevice::_aio_thread
+ 94.80% aio_queue_t::get_next_completed
| + 94.70% __io_getevents_0_4
| + 0.10% io_getevents
+ 4.70% BlueStore::_txc_state_proc
Thread: 17 (bstore_discard) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% KernelDevice::DiscardThread::entry
+ 100.00% KernelDevice::_discard_thread
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 18 (rocksdb:low0) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% execute_native_thread_routine
+ 100.00% rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper
+ 100.00% rocksdb::ThreadPoolImpl::Impl::BGThread
+ 74.20% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| + 74.20% pthread_cond_wait@@GLIBC_2.3.2
+ 25.80% std::function<void ()>::operator()() const
+ 25.80% rocksdb::DBImpl::BGWorkCompaction
+ 25.60% rocksdb::DBImpl::BackgroundCallCompaction
| + 25.60% rocksdb::DBImpl::BackgroundCompaction
| + 25.60% rocksdb::CompactionJob::Run
| + 25.60% rocksdb::CompactionJob::ProcessKeyValueCompaction
| + 17.70% rocksdb::CompactionIterator::Next
| | + 10.00% rocksdb::MergingIterator::Next
| | | + 9.70% rocksdb::IteratorWrapperBase<rocksdb::Slice>::Next
| | | + 0.20% rocksdb::BinaryHeap<rocksdb::IteratorWrapperBase<rocksdb::Slice>*, rocksdb::MinIteratorComparator>::replace_top
| | + 7.60% rocksdb::CompactionIterator::NextFromInput
| + 5.40% rocksdb::BlockBasedTableBuilder::Add
| + 1.10% rocksdb::CompactionJob::FinishCompactionOutputFile
| + 0.30% rocksdb::VersionSet::MakeInputIterator
| + 0.30% rocksdb::MergingIterator::SeekToFirst
| + 0.30% rocksdb::FileMetaData::UpdateBoundaries
| + 0.20% rocksdb::CompactionIterator::PrepareOutput
| + 0.10% rocksdb::CompactionJob::SubcompactionState::ShouldStopBefore
+ 0.20% tcmalloc::ThreadCache::Scavenge()
Thread: 19 (rocksdb:low1) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% execute_native_thread_routine
+ 100.00% rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper
+ 100.00% rocksdb::ThreadPoolImpl::Impl::BGThread
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 20 (rocksdb:high0) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% execute_native_thread_routine
+ 100.00% rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper
+ 100.00% rocksdb::ThreadPoolImpl::Impl::BGThread
+ 63.80% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| + 63.80% pthread_cond_wait@@GLIBC_2.3.2
+ 36.20% std::function<void ()>::operator()() const
+ 36.20% rocksdb::DBImpl::BackgroundCallFlush
+ 36.20% rocksdb::DBImpl::BackgroundFlush
+ 36.20% rocksdb::DBImpl::FlushMemTablesToOutputFiles
+ 36.20% rocksdb::DBImpl::FlushMemTableToOutputFile
+ 36.20% rocksdb::FlushJob::Run
+ 36.20% rocksdb::FlushJob::WriteLevel0Table
+ 36.20% rocksdb::BuildTable
+ 17.30% rocksdb::BlockBasedTableBuilder::Add
| + 8.40% rocksdb::BlockBasedTableBuilder::Flush
| + 4.90% rocksdb::BlockBuilder::Add
| + 1.50% rocksdb::FullFilterBlockBuilder::Add
| + 0.90% rocksdb::ShortenedIndexBuilder::AddIndexEntry
| + 0.50% rocksdb::NotifyCollectTableCollectorsOnAdd
| + 0.30% rocksdb::BlockBasedTableBuilder::ok
| + 0.20% std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::assign
| + 0.10% rocksdb::FlushBlockBySizePolicy::Update
| + 0.10% rocksdb::ExtractUserKey
+ 15.20% rocksdb::CompactionIterator::Next
| + 14.20% rocksdb::CompactionIterator::NextFromInput
| | + 6.00% rocksdb::MergeHelper::MergeUntil
| | + 4.10% rocksdb::MemTableIterator::key
| | + 1.80% rocksdb::MemTableIterator::value
| | + 0.50% rocksdb::(anonymous namespace)::BytewiseComparatorImpl::Equal
| | + 0.40% rocksdb::IterKey::SetInternalKey
| | + 0.20% rocksdb::MemTableIterator::Valid
| | + 0.10% rocksdb::Slice::Slice
| | + 0.10% rocksdb::RangeDelAggregator::ShouldDelete
| | + 0.10% rocksdb::ParseInternalKey
| | + 0.10% rocksdb::MemTableIterator::Next
| | + 0.10% rocksdb::CompactionIterator::findEarliestVisibleSnapshot
| + 0.90% rocksdb::MemTableIterator::Next
+ 2.00% rocksdb::BlockBasedTableBuilder::Finish
+ 0.60% rocksdb::WritableFileWriter::Sync
+ 0.50% rocksdb::TableCache::NewIterator
+ 0.20% rocksdb::ThreadStatusUtil::SetThreadOperationProperty
+ 0.20% rocksdb::FileMetaData::UpdateBoundaries
Thread: 21 (bstore_aio) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% KernelDevice::AioCompletionThread::entry
+ 100.00% KernelDevice::_aio_thread
+ 100.00% aio_queue_t::get_next_completed
+ 100.00% __io_getevents_0_4
Thread: 22 (bstore_discard) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% KernelDevice::DiscardThread::entry
+ 100.00% KernelDevice::_discard_thread
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 23 (bstore_aio) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% KernelDevice::AioCompletionThread::entry
+ 100.00% KernelDevice::_aio_thread
+ 100.00% aio_queue_t::get_next_completed
+ 100.00% __io_getevents_0_4
Thread: 24 (bstore_discard) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% KernelDevice::DiscardThread::entry
+ 100.00% KernelDevice::_discard_thread
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 25 (ceph-osd) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% execute_native_thread_routine
+ 100.00% rocksdb::DeleteScheduler::BackgroundEmptyTrash
+ 100.00% rocksdb::InstrumentedCondVar::Wait
+ 100.00% rocksdb::InstrumentedCondVar::WaitInternal
+ 100.00% rocksdb::port::CondVar::Wait
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 26 (rocksdb:dump_st) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% execute_native_thread_routine
+ 100.00% std::thread::_State_impl<std::thread::_Invoker<std::tuple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}> > >::_M_run()
+ 100.00% std::thread::_Invoker<std::tuple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}> >::operator()()
+ 100.00% std::thread::_Invoker<std::tuple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>)
+ 100.00% std::__invoke<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}>(std::__invoke_result&&, (rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}&&)...)
+ 100.00% std::__invoke_impl<void, rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}>(std::__invoke_other, rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}&&)
+ 100.00% rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}::operator()() const
+ 100.00% rocksdb::RepeatableThread::thread
+ 100.00% rocksdb::RepeatableThread::wait
+ 100.00% rocksdb::InstrumentedCondVar::TimedWait
+ 100.00% rocksdb::InstrumentedCondVar::TimedWaitInternal
+ 100.00% rocksdb::port::CondVar::TimedWait
+ 100.00% pthread_cond_timedwait@@GLIBC_2.3.2
Thread: 27 (rocksdb:pst_st) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% execute_native_thread_routine
+ 100.00% std::thread::_State_impl<std::thread::_Invoker<std::tuple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}> > >::_M_run()
+ 100.00% std::thread::_Invoker<std::tuple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}> >::operator()()
+ 100.00% std::thread::_Invoker<std::tuple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>)
+ 100.00% std::__invoke<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}>(std::__invoke_result&&, (rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}&&)...)
+ 100.00% std::__invoke_impl<void, rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}>(std::__invoke_other, rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}&&)
+ 100.00% rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}::operator()() const
+ 100.00% rocksdb::RepeatableThread::thread
+ 100.00% rocksdb::RepeatableThread::wait
+ 100.00% rocksdb::InstrumentedCondVar::TimedWait
+ 100.00% rocksdb::InstrumentedCondVar::TimedWaitInternal
+ 100.00% rocksdb::port::CondVar::TimedWait
+ 100.00% pthread_cond_timedwait@@GLIBC_2.3.2
Thread: 28 (cfin) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% Finisher::finisher_thread_entry
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 29 (bstore_kv_sync) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% BlueStore::KVSyncThread::entry
+ 100.00% BlueStore::_kv_sync_thread
+ 66.00% BlueStore::_txc_apply_kv
| + 61.80% RocksDBStore::submit_transaction
| | + 61.40% RocksDBStore::submit_common
| | | + 61.00% rocksdb::DBImpl::Write
| | | + 60.90% rocksdb::DBImpl::WriteImpl
| | | + 43.60% rocksdb::WriteBatchInternal::InsertInto
| | | | + 43.40% rocksdb::WriteBatch::Iterate
| | | | | + 31.40% rocksdb::MemTableInserter::PutCF
| | | | | | + 31.30% rocksdb::MemTableInserter::PutCFImpl
| | | | | | + 30.70% rocksdb::MemTable::Add
| | | | | | | + 25.10% rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>::Insert<false>
| | | | | | | | + 21.30% rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>::RecomputeSpliceLevels
| | | | | | | | | + 20.10% rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>::FindSpliceForLevel<true>
| | | | | | | | | + 10.80% rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>::KeyIsAfterNode
| | | | | | | | | | + 10.60% rocksdb::MemTable::KeyComparator::operator()
| | | | | | | | | | + 8.00% rocksdb::InternalKeyComparator::CompareKeySeq
| | | | | | | | | | + 1.30% rocksdb::GetLengthPrefixedSlice
| | | | | | | | | + 0.80% rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>::Node::Next
| | | | | | | | + 1.60% rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>::KeyIsAfterNode
| | | | | | | | + 1.10% rocksdb::MemTable::KeyComparator::operator()
| | | | | | | | + 0.30% rocksdb::MemTableRep::KeyComparator::decode_key
| | | | | | | | + 0.20% rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>::Node::SetNext
| | | | | | | | + 0.10% rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>::Node::NoBarrier_SetNext
| | | | | | | + 2.40% __memmove_avx_unaligned_erms
| | | | | | | + 1.20% rocksdb::(anonymous namespace)::SkipListRep::Allocate
| | | | | | | + 1.10% rocksdb::MemTable::UpdateFlushState
| | | | | | | + 0.20% rocksdb::EncodeVarint32
| | | | | | | + 0.10% std::unique_ptr<rocksdb::MemTableRep, std::default_delete<rocksdb::MemTableRep> >::operator->
| | | | | | | + 0.10% std::__atomic_base<unsigned long>::operator unsigned long
| | | | | | + 0.20% rocksdb::MemTableInserter::get_post_process_info
| | | | | | + 0.20% rocksdb::MemTableInserter::SeekToColumnFamily
| | | | | | + 0.10% rocksdb::MemTableInserter::CheckMemtableFull
| | | | | | + 0.10% rocksdb::ColumnFamilyMemTablesImpl::GetMemTable
| | | | | + 9.50% rocksdb::MemTableInserter::MergeCF
| | | | | + 0.80% rocksdb::MemTableInserter::DeleteCF
| | | | | + 0.50% rocksdb::ReadRecordFromWriteBatch
| | | | | + 0.20% rocksdb::Status::operator=
| | | | | + 0.10% rocksdb::Slice::Slice
| | | | + 0.10% rocksdb::MemTableInserter::MemTableInserter
| | | + 12.50% rocksdb::DBImpl::WriteToWAL
| | | | + 12.30% rocksdb::DBImpl::WriteToWAL
| | | | | + 12.30% rocksdb::log::Writer::AddRecord
| | | | | + 11.90% rocksdb::log::Writer::EmitPhysicalRecord
| | | | | + 7.40% rocksdb::WritableFileWriter::Append
| | | | | + 2.50% rocksdb::crc32c::crc32c_3way
| | | | | + 1.70% rocksdb::WritableFileWriter::Flush
| | | | | + 0.10% rocksdb::Status::operator=
| | | | + 0.10% rocksdb::DBImpl::MergeBatch
| | | + 2.40% rocksdb::InstrumentedMutex::Lock
| | | + 0.40% rocksdb::WriteThread::EnterAsBatchGroupLeader
| | | + 0.40% rocksdb::InstrumentedMutex::Unlock
| | | + 0.20% rocksdb::WriteBatchInternal::Count
| | | + 0.20% rocksdb::DBImpl::WriteStatusCheck
| | | + 0.10% rocksdb::WriteThread::Writer::Writer
| | | + 0.10% rocksdb::WriteThread::Writer::ShouldWriteToMemtable
| | | + 0.10% rocksdb::WriteThread::Writer::FinalStatus
| | | + 0.10% rocksdb::WriteThread::JoinBatchGroup
| | | + 0.10% rocksdb::WriteThread::ExitAsBatchGroupLeader
| | | + 0.10% rocksdb::DBImpl::WriteContext::~WriteContext
| | | + 0.10% rocksdb::DBImpl::PreprocessWrite
| | + 0.30% ceph_clock_now
| | + 0.10% PerfCounters::inc
| + 0.60% std::shared_ptr<KeyValueDB::TransactionImpl>::shared_ptr
| + 0.50% ceph::time_detail::mono_clock::now
| + 0.20% std::_Rb_tree_const_iterator<boost::intrusive_ptr<BlueStore::Onode> >::operator++
+ 20.00% RocksDBStore::submit_transaction_sync
| + 19.90% RocksDBStore::submit_common
| | + 19.90% rocksdb::DBImpl::Write
| | + 19.90% rocksdb::DBImpl::WriteImpl
| | + 19.50% rocksdb::DBImpl::WriteToWAL
| | | + 19.40% rocksdb::WritableFileWriter::Sync
| | | | + 19.40% rocksdb::WritableFileWriter::SyncInternal
| | | | + 19.40% BlueRocksWritableFile::Sync
| | | | + 19.30% BlueFS::fsync
| | | | + 19.30% BlueFS::_fsync
| | | | + 18.00% BlueFS::_flush
| | | | | + 18.00% BlueFS::_flush_range
| | | | | + 17.60% KernelDevice::aio_write
| | | | | | + 17.60% KernelDevice::_sync_write
| | | | | | + 12.90% sync_file_range
| | | | | | + 4.60% pwritev64
| | | | | | + 0.10% ceph::buffer::v14_2_0::list::prepare_iov<std::vector<iovec, std::allocator<iovec> > >
| | | | | + 0.20% __memset_avx2_erms
| | | | | + 0.10% PerfCounters::inc
| | | | | + 0.10% IOContext::aio_wait
| | | | + 1.20% BlueFS::_flush_bdev_safely
| | | | + 0.10% BlueFS::_flush_and_sync_log
| | | + 0.10% rocksdb::DBImpl::WriteToWAL
| | + 0.30% rocksdb::InstrumentedMutex::Unlock
| | + 0.10% rocksdb::InstrumentedMutex::Lock
| + 0.10% PerfCounters::tinc
+ 10.60% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| + 10.60% pthread_cond_wait@@GLIBC_2.3.2
| + 0.20% __pthread_mutex_cond_lock
+ 1.00% std::condition_variable::notify_one()
+ 0.80% BlueStore::BlueStoreThrottle::log_state_latency
+ 0.40% KernelDevice::flush
+ 0.20% std::unique_lock<std::mutex>::unique_lock
+ 0.20% std::unique_lock<std::mutex>::lock
+ 0.20% std::__atomic_base<int>::operator--
+ 0.10% std::deque<BlueStore::TransContext*, std::allocator<BlueStore::TransContext*> >::~deque
+ 0.10% std::deque<BlueStore::TransContext*, std::allocator<BlueStore::TransContext*> >::size
+ 0.10% BlueStore::BlueStoreThrottle::release_kv_throttle
Thread: 30 (bstore_kv_final) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% BlueStore::KVFinalizeThread::entry
+ 100.00% BlueStore::_kv_finalize_thread
+ 57.60% BlueStore::_txc_state_proc
| + 42.10% BlueStore::_txc_finish
| | + 30.50% BlueStore::SharedBlob::finish_write
| | | + 27.50% BlueStore::BufferSpace::_finish_write
| | | | + 25.30% BlueStore::CacheShard::_trim
| | | | | + 25.20% TwoQBufferCacheShard::_trim_to
| | | | | + 11.90% ceph::buffer::v14_2_0::list::clear
| | | | | | + 11.60% ceph::buffer::v14_2_0::list::buffers_t::clear_and_dispose
| | | | | | + 11.20% ceph::buffer::v14_2_0::ptr_node::disposer::operator()
| | | | | | + 10.50% ceph::buffer::v14_2_0::ptr_node::~ptr_node
| | | | | | | + 10.50% ceph::buffer::v14_2_0::ptr::~ptr
| | | | | | | + 8.60% ceph::buffer::v14_2_0::raw_posix_aligned::~raw_posix_aligned
| | | | | | | + 1.30% ceph::buffer::v14_2_0::ptr::release
| | | | | | | + 0.20% tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned int)
| | | | | | | + 0.20% tc_free
| | | | | | | + 0.20% ceph::buffer::v14_2_0::raw_posix_aligned::operator delete
| | | | | | + 0.30% tc_free
| | | | | | + 0.10% tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned int)
| | | | | + 9.90% BlueStore::BufferSpace::_rm_buffer
| | | | | + 2.40% boost::intrusive::list_impl<boost::intrusive::mhtraits<BlueStore::Buffer, boost::intrusive::list_member_hook<void, void, void>, &BlueStore::Buffer::lru_item>, unsigned long, true, void>::erase
| | | | | + 0.50% boost::intrusive::reverse_iterator<boost::intrusive::list_iterator<boost::intrusive::mhtraits<BlueStore::Buffer, boost::intrusive::list_member_hook<void, void, void>, &BlueStore::Buffer::lru_item>, false> >::operator*
| | | | | + 0.20% boost::intrusive::list_impl<boost::intrusive::mhtraits<BlueStore::Buffer, boost::intrusive::list_member_hook<void, void, void>, &BlueStore::Buffer::lru_item>, unsigned long, true, void>::push_front
| | | | + 0.80% ceph::buffer::v14_2_0::list::reassign_to_mempool
| | | | + 0.50% BlueStore::Buffer::maybe_rebuild
| | | | + 0.30% TwoQBufferCacheShard::_add
| | | + 2.20% std::lock_guard<std::recursive_mutex>::lock_guard
| | | + 0.40% std::lock_guard<std::recursive_mutex>::~lock_guard
| | + 5.30% BlueStore::TransContext::~TransContext
| | + 2.90% BlueStore::_txc_release_alloc
| | + 0.70% BlueStore::BlueStoreThrottle::log_state_latency
| | + 0.40% std::_Rb_tree_const_iterator<boost::intrusive_ptr<BlueStore::SharedBlob> >::operator++
| | + 0.40% boost::intrusive_ptr<BlueStore::OpSequencer>::intrusive_ptr
| | + 0.30% BlueStore::BlueStoreThrottle::complete
| | + 0.20% std::set<boost::intrusive_ptr<BlueStore::SharedBlob>, std::less<boost::intrusive_ptr<BlueStore::SharedBlob> >, std::allocator<boost::intrusive_ptr<BlueStore::SharedBlob> > >::clear
| | + 0.20% std::lock_guard<std::mutex>::~lock_guard
| | + 0.20% std::condition_variable::notify_all()
| | + 0.10% std::set<boost::intrusive_ptr<BlueStore::SharedBlob>, std::less<boost::intrusive_ptr<BlueStore::SharedBlob> >, std::allocator<boost::intrusive_ptr<BlueStore::SharedBlob> > >::end
| | + 0.10% std::lock_guard<std::mutex>::lock_guard
| + 14.30% BlueStore::_txc_committed_kv
| | + 13.00% ContextQueue::queue
| | | + 8.10% std::condition_variable::notify_all()
| | | + 3.50% std::scoped_lock<std::mutex>::scoped_lock
| | | + 1.10% std::scoped_lock<std::mutex>::~scoped_lock
| | | + 0.20% std::__cxx11::list<Context*, std::allocator<Context*> >::clear
| | | + 0.10% std::__cxx11::list<Context*, std::allocator<Context*> >::insert<std::_List_iterator<Context*>, void>
| | + 0.50% BlueStore::BlueStoreThrottle::complete_kv
| | + 0.30% std::lock_guard<std::mutex>::~lock_guard
| | + 0.10% utime_t::operator double
| | + 0.10% std::lock_guard<std::mutex>::lock_guard
| | + 0.10% ceph_clock_now
| | + 0.10% BlueStore::log_latency_fn(char const*, int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > const&, double, std::function<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > const&)>) const
| + 0.90% BlueStore::BlueStoreThrottle::log_state_latency
| + 0.10% BlueStore::TransContext::get_state_name
+ 42.10% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| + 42.10% pthread_cond_wait@@GLIBC_2.3.2
| + 0.70% __pthread_mutex_cond_lock
+ 0.10% std::deque<BlueStore::TransContext*, std::allocator<BlueStore::TransContext*> >::pop_front
+ 0.10% BitmapAllocator::get_fragmentation
Thread: 31 (bstore_mempool) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% BlueStore::MempoolThread::entry
+ 98.00% std::condition_variable::wait_for<unsigned long, std::ratio<1l, 1000000000l> >
| + 98.00% std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >
| + 98.00% std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >
| + 98.00% __gthread_cond_timedwait
| + 98.00% pthread_cond_timedwait@@GLIBC_2.3.2
+ 2.00% PriorityCache::Manager::tune_memory
Thread: 32 (ms_dispatch) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% DispatchQueue::DispatchThread::entry
+ 100.00% DispatchQueue::entry
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 33 (ms_local) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% DispatchQueue::LocalDeliveryThread::entry
+ 100.00% DispatchQueue::run_local_delivery
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 34 (safe_timer) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% SafeTimerThread::entry
+ 100.00% SafeTimer::timer_thread
+ 100.00% std::condition_variable::wait_until<ceph::time_detail::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >
+ 100.00% std::condition_variable::__wait_until_impl<std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >
+ 100.00% __gthread_cond_timedwait
+ 100.00% pthread_cond_timedwait@@GLIBC_2.3.2
Thread: 35 (fn_anonymous) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% Finisher::finisher_thread_entry
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 36 (safe_timer) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% SafeTimerThread::entry
+ 100.00% SafeTimer::timer_thread
+ 89.30% std::condition_variable::wait_until<ceph::time_detail::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >
| + 89.30% std::condition_variable::__wait_until_impl<std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >
| + 89.30% __gthread_cond_timedwait
| + 89.30% pthread_cond_timedwait@@GLIBC_2.3.2
+ 10.70% Context::complete
+ 10.70% MgrClient::_send_stats
+ 9.70% MgrClient::_send_report
+ 1.00% MgrClient::_send_pgstats
Thread: 37 (ms_dispatch) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% DispatchQueue::DispatchThread::entry
+ 100.00% DispatchQueue::entry
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 38 (ms_local) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% DispatchQueue::LocalDeliveryThread::entry
+ 100.00% DispatchQueue::run_local_delivery
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 39 (ms_dispatch) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% DispatchQueue::DispatchThread::entry
+ 100.00% DispatchQueue::entry
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 40 (ms_local) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% DispatchQueue::LocalDeliveryThread::entry
+ 100.00% DispatchQueue::run_local_delivery
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 41 (ms_dispatch) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% DispatchQueue::DispatchThread::entry
+ 100.00% DispatchQueue::entry
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 42 (ms_local) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% DispatchQueue::LocalDeliveryThread::entry
+ 100.00% DispatchQueue::run_local_delivery
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 43 (ms_dispatch) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% DispatchQueue::DispatchThread::entry
+ 100.00% DispatchQueue::entry
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 44 (ms_local) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% DispatchQueue::LocalDeliveryThread::entry
+ 100.00% DispatchQueue::run_local_delivery
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 45 (ms_dispatch) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% DispatchQueue::DispatchThread::entry
+ 100.00% DispatchQueue::entry
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 46 (ms_local) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% DispatchQueue::LocalDeliveryThread::entry
+ 100.00% DispatchQueue::run_local_delivery
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 47 (ms_dispatch) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% DispatchQueue::DispatchThread::entry
+ 100.00% DispatchQueue::entry
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 48 (ms_local) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% DispatchQueue::LocalDeliveryThread::entry
+ 100.00% DispatchQueue::run_local_delivery
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 49 (fn_anonymous) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% Finisher::finisher_thread_entry
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 50 (finisher) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% Finisher::finisher_thread_entry
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 51 (safe_timer) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% SafeTimerThread::entry
+ 100.00% SafeTimer::timer_thread
+ 97.80% std::condition_variable::wait_until<ceph::time_detail::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >
| + 97.80% std::condition_variable::__wait_until_impl<std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >
| + 97.80% __gthread_cond_timedwait
| + 97.80% pthread_cond_timedwait@@GLIBC_2.3.2
+ 2.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 0.20% HandleWatchTimeout::complete
Thread: 52 (safe_timer) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% SafeTimerThread::entry
+ 100.00% SafeTimer::timer_thread
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 53 (ceph-osd) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% execute_native_thread_routine
+ 100.00% ceph::timer_detail::timer<ceph::time_detail::mono_clock>::timer_thread
+ 99.40% std::condition_variable::wait_until<ceph::time_detail::mono_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >
| + 99.40% std::condition_variable::__wait_until_impl<std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >
| + 99.40% __gthread_cond_timedwait
| + 99.40% pthread_cond_timedwait@@GLIBC_2.3.2
+ 0.50% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 0.10% std::unique_lock<std::mutex>::lock
Thread: 54 (osd_srv_agent) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% OSDService::AgentThread::entry
+ 100.00% OSDService::agent_entry
+ 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
Thread: 55 (tp_osd_tp) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% ShardedThreadPool::WorkThreadSharded::entry
+ 100.00% ShardedThreadPool::shardedthreadpool_worker
+ 99.90% OSD::ShardedOpWQ::_process
| + 50.70% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| | + 50.70% pthread_cond_wait@@GLIBC_2.3.2
| | + 0.60% __pthread_mutex_cond_lock
| + 38.10% OpQueueItem::run
| | + 38.10% PGOpItem::run
| | + 38.10% OSD::dequeue_op
| | + 38.00% PrimaryLogPG::do_request
| | | + 37.90% PrimaryLogPG::do_op
| | | | + 32.20% PrimaryLogPG::execute_ctx
| | | | | + 28.80% PrimaryLogPG::issue_repop
| | | | | | + 28.80% ReplicatedBackend::submit_transaction
| | | | | | + 22.20% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)
| | | | | | | + 22.20% BlueStore::queue_transactions
| | | | | | | + 14.90% BlueStore::_txc_add_transaction
| | | | | | | | + 12.10% BlueStore::_write
| | | | | | | | | + 12.00% BlueStore::_do_write
| | | | | | | | | + 5.70% BlueStore::ExtentMap::fault_range
| | | | | | | | | + 4.40% BlueStore::_do_alloc_write
| | | | | | | | | + 1.40% BlueStore::_do_write_data
| | | | | | | | | + 0.30% BlueStore::ExtentMap::compress_extent_map
| | | | | | | | | + 0.20% BlueStore::_wctx_finish
| | | | | | | | + 1.20% BlueStore::_omap_setkeys
| | | | | | | | + 0.60% BlueStore::_setattrs
| | | | | | | | + 0.30% BlueStore::Collection::get_onode
| | | | | | | | + 0.20% ceph::os::Transaction::iterator::decode_attrset_bl
| | | | | | | | + 0.20% ceph::os::Transaction::iterator::decode_attrset
| | | | | | | | + 0.10% BlueStore::_omap_rmkeys
| | | | | | | | + 0.10% BlueStore::_get_collection
| | | | | | | + 2.70% BlueStore::_txc_write_nodes
| | | | | | | + 1.90% BlueStore::_txc_state_proc
| | | | | | | + 1.50% BlueStore::BlueStoreThrottle::try_start_transaction
| | | | | | | + 0.60% BlueStore::_txc_create
| | | | | | | + 0.30% BlueStore::_txc_finalize_kv
| | | | | | | + 0.10% ceph::time_detail::mono_clock::now
| | | | | | | + 0.10% PerfCounters::inc
| | | | | | + 5.10% non-virtual thunk to PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)
| | | | | | + 0.50% generate_transaction
| | | | | | + 0.30% std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >::vector
| | | | | | + 0.30% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::~vector
| | | | | | + 0.20% std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >::~unique_ptr
| | | | | | + 0.10% std::set<pg_shard_t, std::less<pg_shard_t>, std::allocator<pg_shard_t> >::insert<std::_Rb_tree_const_iterator<pg_shard_t> >
| | | | | | + 0.10% ceph::os::Transaction::register_on_commit
| | | | | + 1.60% PrimaryLogPG::prepare_transaction
| | | | | + 1.50% PeeringState::update_trim_to
| | | | | + 0.10% tc_newarray
| | | | | + 0.10% PrimaryLogPG::new_repop
| | | | | + 0.10% PrimaryLogPG::OpContext::reset_obs
| | | | + 3.40% PrimaryLogPG::find_object_context
| | | | + 0.50% MOSDOp::finish_decode
| | | | + 0.40% PG::check_in_progress_op
| | | | + 0.30% PGLog::IndexedLog::get_request
| | | | + 0.20% PrimaryLogPG::OpContext::OpContext
| | | | + 0.20% Message::clear_payload
| | | | + 0.10% std::map<hobject_t, std::shared_ptr<ObjectContext>, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, std::shared_ptr<ObjectContext> > > >::find
| | | | + 0.10% ceph_clock_now
| | | | + 0.10% PrimaryLogPG::maybe_handle_cache
| | | | + 0.10% PrimaryLogPG::is_degraded_or_backfilling_object
| | | | + 0.10% PerfCounters::tinc
| | | | + 0.10% MOSDOp::get_snapid
| | | | + 0.10% MOSDOp::get_map_epoch
| | | + 0.10% PGBackend::handle_message
| | + 0.10% ceph_clock_now
| + 8.50% OSD::ShardedOpWQ::handle_oncommits
| + 1.60% PG::lock
| + 0.30% std::mutex::lock
| + 0.20% std::unique_lock<std::mutex>::unique_lock
| + 0.10% std::deque<OpQueueItem, std::allocator<OpQueueItem> >::push_back
| + 0.10% std::__cxx11::list<Context*, std::allocator<Context*> >::~list
| + 0.10% OpQueueItem::~OpQueueItem
| + 0.10% OpQueueItem::is_peering
| + 0.10% ContextQueue::move_to
+ 0.10% ceph::HeartbeatMap::reset_timeout
Thread: 56 (tp_osd_tp) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% ShardedThreadPool::WorkThreadSharded::entry
+ 100.00% ShardedThreadPool::shardedthreadpool_worker
+ 100.00% OSD::ShardedOpWQ::_process
+ 68.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| + 68.00% pthread_cond_wait@@GLIBC_2.3.2
| + 1.60% __pthread_mutex_cond_lock
+ 24.60% OpQueueItem::run
| + 24.50% PGOpItem::run
| | + 24.40% OSD::dequeue_op
| | | + 24.40% PrimaryLogPG::do_request
| | | + 24.30% PrimaryLogPG::do_op
| | | | + 20.80% PrimaryLogPG::execute_ctx
| | | | | + 17.40% PrimaryLogPG::issue_repop
| | | | | | + 17.30% ReplicatedBackend::submit_transaction
| | | | | | | + 13.20% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)
| | | | | | | | + 13.20% BlueStore::queue_transactions
| | | | | | | | + 9.00% BlueStore::_txc_add_transaction
| | | | | | | | + 1.40% BlueStore::_txc_write_nodes
| | | | | | | | + 1.40% BlueStore::BlueStoreThrottle::try_start_transaction
| | | | | | | | + 0.90% BlueStore::_txc_state_proc
| | | | | | | | + 0.30% BlueStore::_txc_create
| | | | | | | | + 0.20% BlueStore::_txc_finalize_kv
| | | | | | | + 3.30% non-virtual thunk to PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)
| | | | | | | + 0.30% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::~vector
| | | | | | | + 0.20% generate_transaction
| | | | | | | + 0.20% PrimaryLogPG::apply_stats
| | | | | | | + 0.10% std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >::~unique_ptr
| | | | | | + 0.10% PGLog::IndexedLog::add
| | | | | + 1.50% PrimaryLogPG::prepare_transaction
| | | | | + 1.40% PeeringState::update_trim_to
| | | | | + 0.30% PrimaryLogPG::new_repop
| | | | | + 0.10% pg_pool_t::is_pool_snaps_mode
| | | | | + 0.10% MOSDOpReply::MOSDOpReply
| | | | + 2.10% PrimaryLogPG::find_object_context
| | | | + 0.40% PG::check_in_progress_op
| | | | + 0.30% PrimaryLogPG::OpContext::OpContext
| | | | + 0.20% Message::clear_payload
| | | | + 0.10% PrimaryLogPG::maybe_await_blocked_head
| | | | + 0.10% PrimaryLogPG::is_unreadable_object
| | | | + 0.10% PG::op_has_sufficient_caps
| | | | + 0.10% MOSDOp::finish_decode
| | | + 0.10% PG::can_discard_request
| | + 0.10% __pthread_mutex_unlock_usercnt
| + 0.10% PGPeeringItem::run
+ 5.30% OSD::ShardedOpWQ::handle_oncommits
+ 1.20% PG::lock
+ 0.30% std::unique_lock<std::mutex>::unique_lock
+ 0.10% std::set<std::pair<spg_t, unsigned int>, std::less<std::pair<spg_t, unsigned int> >, std::allocator<std::pair<spg_t, unsigned int> > >::~set
+ 0.10% std::mutex::lock
+ 0.10% std::__cxx11::list<Context*, std::allocator<Context*> >::~list
+ 0.10% boost::intrusive_ptr<PG>::intrusive_ptr
+ 0.10% WeightedPriorityQueue<OpQueueItem, unsigned long>::dequeue
+ 0.10% OpQueueItem::is_peering
Thread: 57 (tp_osd_tp) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% ShardedThreadPool::WorkThreadSharded::entry
+ 100.00% ShardedThreadPool::shardedthreadpool_worker
+ 99.90% OSD::ShardedOpWQ::_process
| + 51.50% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| | + 51.50% pthread_cond_wait@@GLIBC_2.3.2
| | + 0.30% __pthread_mutex_cond_lock
| + 37.30% OpQueueItem::run
| | + 37.20% PGOpItem::run
| | | + 37.10% OSD::dequeue_op
| | | | + 36.90% PrimaryLogPG::do_request
| | | | | + 36.60% PrimaryLogPG::do_op
| | | | | | + 30.50% PrimaryLogPG::execute_ctx
| | | | | | | + 26.60% PrimaryLogPG::issue_repop
| | | | | | | | + 26.20% ReplicatedBackend::submit_transaction
| | | | | | | | | + 21.10% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)
| | | | | | | | | | + 21.00% BlueStore::queue_transactions
| | | | | | | | | | | + 14.20% BlueStore::_txc_add_transaction
| | | | | | | | | | | | + 11.60% BlueStore::_write
| | | | | | | | | | | | | + 11.60% BlueStore::_do_write
| | | | | | | | | | | | | + 5.00% BlueStore::ExtentMap::fault_range
| | | | | | | | | | | | | + 4.70% BlueStore::_do_alloc_write
| | | | | | | | | | | | | + 1.30% BlueStore::_do_write_data
| | | | | | | | | | | | | + 0.20% BlueStore::ExtentMap::compress_extent_map
| | | | | | | | | | | | | + 0.10% BlueStore::_wctx_finish
| | | | | | | | | | | | | + 0.10% BlueStore::WriteContext::~WriteContext
| | | | | | | | | | | | + 1.20% BlueStore::_omap_setkeys
| | | | | | | | | | | | + 0.50% BlueStore::Collection::get_onode
| | | | | | | | | | | | + 0.20% ceph::os::Transaction::iterator::decode_bl
| | | | | | | | | | | | + 0.20% BlueStore::_setattrs
| | | | | | | | | | | | + 0.10% ceph::os::Transaction::iterator::decode_attrset_bl
| | | | | | | | | | | | + 0.10% ceph::os::Transaction::iterator::decode_attrset
| | | | | | | | | | | | + 0.10% _dump_transaction<30>
| | | | | | | | | | | | + 0.10% BlueStore::_get_collection
| | | | | | | | | | | + 2.40% BlueStore::_txc_write_nodes
| | | | | | | | | | | + 2.10% BlueStore::_txc_state_proc
| | | | | | | | | | | + 1.60% BlueStore::BlueStoreThrottle::try_start_transaction
| | | | | | | | | | | + 0.40% BlueStore::_txc_finalize_kv
| | | | | | | | | | | + 0.10% ceph::time_detail::mono_clock::now
| | | | | | | | | | | + 0.10% BlueStore::log_latency
| | | | | | | | | | | + 0.10% BlueStore::_txc_create
| | | | | | | | | | + 0.10% TrackedOp::put
| | | | | | | | | + 3.60% non-virtual thunk to PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)
| | | | | | | | | + 0.30% generate_transaction
| | | | | | | | | + 0.20% std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >::vector
| | | | | | | | | + 0.20% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::push_back
| | | | | | | | | + 0.10% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::~vector
| | | | | | | | | + 0.10% std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >::~unique_ptr
| | | | | | | | | + 0.10% std::set<pg_shard_t, std::less<pg_shard_t>, std::allocator<pg_shard_t> >::insert<std::_Rb_tree_const_iterator<pg_shard_t> >
| | | | | | | | | + 0.10% std::set<hobject_t, std::less<hobject_t>, std::allocator<hobject_t> >::~set
| | | | | | | | | + 0.10% boost::intrusive_ptr<ReplicatedBackend::InProgressOp>::intrusive_ptr
| | | | | | | | | + 0.10% PrimaryLogPG::bless_context
| | | | | | | | | + 0.10% PrimaryLogPG::apply_stats
| | | | | | | | | + 0.10% PGBackend::get_osdmap
| | | | | | | | + 0.20% PGTransaction::add_obc
| | | | | | | | + 0.10% boost::intrusive_ptr<OpRequest>::~intrusive_ptr
| | | | | | | + 1.90% PeeringState::update_trim_to
| | | | | | | + 1.70% PrimaryLogPG::prepare_transaction
| | | | | | | + 0.10% std::shared_ptr<ObjectContext>::shared_ptr
| | | | | | | + 0.10% PrimaryLogPG::new_repop
| | | | | | | + 0.10% MOSDOpReply::MOSDOpReply
| | | | | | + 4.40% PrimaryLogPG::find_object_context
| | | | | | + 0.40% PGLog::IndexedLog::get_request
| | | | | | + 0.30% OpRequest::mark_started
| | | | | | + 0.30% Message::clear_payload
| | | | | | + 0.30% MOSDOp::finish_decode
| | | | | | + 0.10% boost::intrusive_ptr<Session>::~intrusive_ptr
| | | | | | + 0.10% PrimaryLogPG::check_laggy_requeue
| | | | | | + 0.10% PG::op_has_sufficient_caps
| | | | | | + 0.10% PG::check_in_progress_op
| | | | | + 0.10% PG::can_discard_request
| | | | + 0.20% ceph_clock_now
| | | + 0.10% pthread_mutex_unlock
| | + 0.10% PGPeeringItem::run
| + 7.50% OSD::ShardedOpWQ::handle_oncommits
| + 2.70% PG::lock
| + 0.30% WeightedPriorityQueue<OpQueueItem, unsigned long>::dequeue
| + 0.20% std::mutex::lock
| + 0.20% ContextQueue::move_to
| + 0.10% std::unique_lock<std::mutex>::unique_lock
| + 0.10% std::__cxx11::list<Context*, std::allocator<Context*> >::~list
+ 0.10% ceph::HeartbeatMap::reset_timeout
Thread: 58 (tp_osd_tp) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% ShardedThreadPool::WorkThreadSharded::entry
+ 100.00% ShardedThreadPool::shardedthreadpool_worker
+ 99.90% OSD::ShardedOpWQ::_process
| + 63.90% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| | + 63.90% pthread_cond_wait@@GLIBC_2.3.2
| | + 1.20% __pthread_mutex_cond_lock
| + 27.30% OpQueueItem::run
| | + 27.30% PGOpItem::run
| | + 27.30% OSD::dequeue_op
| | + 27.20% PrimaryLogPG::do_request
| | + 27.00% PrimaryLogPG::do_op
| | | + 23.70% PrimaryLogPG::execute_ctx
| | | | + 21.40% PrimaryLogPG::issue_repop
| | | | | + 21.10% ReplicatedBackend::submit_transaction
| | | | | | + 16.90% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)
| | | | | | | + 16.90% BlueStore::queue_transactions
| | | | | | | + 11.10% BlueStore::_txc_add_transaction
| | | | | | | | + 10.20% BlueStore::_write
| | | | | | | | | + 10.20% BlueStore::_do_write
| | | | | | | | | + 4.70% BlueStore::_do_alloc_write
| | | | | | | | | + 3.80% BlueStore::ExtentMap::fault_range
| | | | | | | | | + 1.40% BlueStore::_do_write_data
| | | | | | | | | + 0.10% BlueStore::_choose_write_options
| | | | | | | | | + 0.10% BlueStore::WriteContext::~WriteContext
| | | | | | | | | + 0.10% BlueStore::ExtentMap::dirty_range
| | | | | | | | + 0.40% BlueStore::_omap_setkeys
| | | | | | | | + 0.20% BlueStore::_setattrs
| | | | | | | | + 0.10% ceph::os::Transaction::iterator::decode_bl
| | | | | | | | + 0.10% ceph::os::Transaction::iterator::decode_attrset_bl
| | | | | | | | + 0.10% ceph::os::Transaction::iterator::decode_attrset
| | | | | | | + 1.70% BlueStore::_txc_write_nodes
| | | | | | | + 1.70% BlueStore::_txc_state_proc
| | | | | | | + 1.50% BlueStore::BlueStoreThrottle::try_start_transaction
| | | | | | | + 0.40% BlueStore::_txc_create
| | | | | | | + 0.30% BlueStore::_txc_finalize_kv
| | | | | | | + 0.10% ceph::time_detail::mono_clock::now
| | | | | | | + 0.10% BlueStore::log_latency
| | | | | | + 2.80% non-virtual thunk to PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)
| | | | | | + 0.50% generate_transaction
| | | | | | + 0.30% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::~vector
| | | | | | + 0.20% std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >::~vector
| | | | | | + 0.20% std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >::~unique_ptr
| | | | | | + 0.10% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::push_back
| | | | | | + 0.10% non-virtual thunk to PrimaryLogPG::op_applied(eversion_t const&)
| | | | | + 0.20% PGLog::IndexedLog::add
| | | | | + 0.10% tc_newarray
| | | | + 1.00% PeeringState::update_trim_to
| | | | + 0.70% PrimaryLogPG::prepare_transaction
| | | | + 0.20% MOSDOpReply::MOSDOpReply
| | | | + 0.10% PrimaryLogPG::new_repop
| | | | + 0.10% PrimaryLogPG::OpContext::reset_obs
| | | | + 0.10% PrimaryLogPG::OpContext::register_on_finish<PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)::<lambda()> >
| | | | + 0.10% PrimaryLogPG::OpContext::register_on_commit<PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)::<lambda()> >
| | | + 2.30% PrimaryLogPG::find_object_context
| | | + 0.30% MOSDOp::finish_decode
| | | + 0.10% boost::intrusive_ptr<OpRequest>::~intrusive_ptr
| | | + 0.10% PrimaryLogPG::is_degraded_or_backfilling_object
| | | + 0.10% PrimaryLogPG::OpContext::OpContext
| | | + 0.10% PerfCounters::tinc
| | | + 0.10% OpRequest::mark_started
| | | + 0.10% Message::clear_payload
| | + 0.10% PGBackend::handle_message
| + 6.30% OSD::ShardedOpWQ::handle_oncommits
| + 1.50% PG::lock
| + 0.30% std::unique_lock<std::mutex>::unique_lock
| + 0.20% std::mutex::lock
| + 0.20% OpQueueItem::maybe_get_op
| + 0.10% boost::intrusive_ptr<PG>::intrusive_ptr
| + 0.10% ContextQueue::move_to
+ 0.10% ceph::HeartbeatMap::reset_timeout
Thread: 59 (tp_osd_tp) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% ShardedThreadPool::WorkThreadSharded::entry
+ 100.00% ShardedThreadPool::shardedthreadpool_worker
+ 99.80% OSD::ShardedOpWQ::_process
| + 50.50% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| | + 50.50% pthread_cond_wait@@GLIBC_2.3.2
| | + 0.70% __pthread_mutex_cond_lock
| + 39.40% OpQueueItem::run
| | + 39.40% PGOpItem::run
| | + 39.40% OSD::dequeue_op
| | + 39.00% PrimaryLogPG::do_request
| | | + 38.80% PrimaryLogPG::do_op
| | | | + 32.30% PrimaryLogPG::execute_ctx
| | | | | + 29.30% PrimaryLogPG::issue_repop
| | | | | | + 29.30% ReplicatedBackend::submit_transaction
| | | | | | + 24.50% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)
| | | | | | | + 24.50% BlueStore::queue_transactions
| | | | | | | + 18.00% BlueStore::_txc_add_transaction
| | | | | | | | + 15.40% BlueStore::_write
| | | | | | | | | + 15.40% BlueStore::_do_write
| | | | | | | | | + 7.30% BlueStore::ExtentMap::fault_range
| | | | | | | | | + 6.40% BlueStore::_do_alloc_write
| | | | | | | | | + 1.10% BlueStore::_do_write_data
| | | | | | | | | + 0.20% BlueStore::_choose_write_options
| | | | | | | | | + 0.10% interval_set<unsigned long, std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > > >::union_of
| | | | | | | | | + 0.10% BlueStore::ExtentMap::dirty_range
| | | | | | | | | + 0.10% BlueStore::ExtentMap::compress_extent_map
| | | | | | | | + 0.90% BlueStore::_omap_setkeys
| | | | | | | | + 0.80% BlueStore::Collection::get_onode
| | | | | | | | + 0.30% BlueStore::_setattrs
| | | | | | | | + 0.20% ceph::os::Transaction::begin
| | | | | | | | + 0.10% std::vector<boost::intrusive_ptr<BlueStore::Collection>, std::allocator<boost::intrusive_ptr<BlueStore::Collection> > >::vector
| | | | | | | | + 0.10% ceph::os::Transaction::iterator::decode_attrset_bl
| | | | | | | | + 0.10% ceph::os::Transaction::iterator::decode_attrset
| | | | | | | + 2.50% BlueStore::_txc_write_nodes
| | | | | | | + 1.50% BlueStore::_txc_state_proc
| | | | | | | + 1.50% BlueStore::BlueStoreThrottle::try_start_transaction
| | | | | | | + 0.50% BlueStore::_txc_create
| | | | | | | + 0.40% BlueStore::_txc_finalize_kv
| | | | | | | + 0.10% ceph::time_detail::mono_clock::now
| | | | | | + 3.70% non-virtual thunk to PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)
| | | | | | + 0.40% generate_transaction
| | | | | | + 0.20% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::~vector
| | | | | | + 0.20% PrimaryLogPG::apply_stats
| | | | | | + 0.10% std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >::~vector
| | | | | | + 0.10% std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >::~unique_ptr
| | | | | | + 0.10% non-virtual thunk to PrimaryLogPG::op_applied(eversion_t const&)
| | | | | + 1.80% PrimaryLogPG::prepare_transaction
| | | | | + 0.60% PeeringState::update_trim_to
| | | | | + 0.20% PrimaryLogPG::new_repop
| | | | | + 0.10% std::shared_ptr<ObjectContext>::shared_ptr
| | | | | + 0.10% PrimaryLogPG::eval_repop
| | | | | + 0.10% PrimaryLogPG::complete_read_ctx
| | | | | + 0.10% PrimaryLogPG::OpContext::reset_obs
| | | | + 4.60% PrimaryLogPG::find_object_context
| | | | + 0.60% PGLog::IndexedLog::get_request
| | | | + 0.20% PG::op_has_sufficient_caps
| | | | + 0.20% PG::check_in_progress_op
| | | | + 0.10% pg_t::contains
| | | | + 0.10% PrimaryLogPG::maybe_await_blocked_head
| | | | + 0.10% PrimaryLogPG::get_rw_locks
| | | | + 0.10% PrimaryLogPG::check_laggy
| | | | + 0.10% PerfCounters::tinc
| | | | + 0.10% OSD::init_op_flags
| | | | + 0.10% Message::clear_payload
| | | | + 0.10% MOSDOp::get_snapid
| | | | + 0.10% MOSDOp::finish_decode
| | | + 0.10% std::unordered_map<entity_name_t, std::__cxx11::list<boost::intrusive_ptr<OpRequest>, std::allocator<boost::intrusive_ptr<OpRequest> > >, std::hash<entity_name_t>, std::equal_to<entity_name_t>, std::allocator<std::pair<entity_name_t const, std::__cxx11::list<boost::intrusive_ptr<OpRequest>, std::allocator<boost::intrusive_ptr<OpRequest> > > > > >::find
| | | + 0.10% PG::can_discard_request
| | + 0.20% OpRequest::mark_reached_pg
| | + 0.10% operator-
| | + 0.10% OSDService::maybe_share_map
| + 7.40% OSD::ShardedOpWQ::handle_oncommits
| + 1.40% PG::lock
| + 0.40% std::mutex::lock
| + 0.20% std::unique_lock<std::mutex>::unique_lock
| + 0.20% OpQueueItem::~OpQueueItem
| + 0.10% std::unordered_map<spg_t, std::unique_ptr<OSDShardPGSlot, std::default_delete<OSDShardPGSlot> >, std::hash<spg_t>, std::equal_to<spg_t>, std::allocator<std::pair<spg_t const, std::unique_ptr<OSDShardPGSlot, std::default_delete<OSDShardPGSlot> > > > >::find
| + 0.10% WeightedPriorityQueue<OpQueueItem, unsigned long>::dequeue
| + 0.10% ContextQueue::move_to
+ 0.10% std::atomic<bool>::operator bool
Thread: 60 (tp_osd_tp) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% ShardedThreadPool::WorkThreadSharded::entry
+ 100.00% ShardedThreadPool::shardedthreadpool_worker
+ 99.80% OSD::ShardedOpWQ::_process
| + 64.30% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| | + 64.30% pthread_cond_wait@@GLIBC_2.3.2
| | + 1.80% __pthread_mutex_cond_lock
| + 28.10% OpQueueItem::run
| | + 28.10% PGOpItem::run
| | + 28.10% OSD::dequeue_op
| | + 27.90% PrimaryLogPG::do_request
| | + 27.90% PrimaryLogPG::do_op
| | + 23.60% PrimaryLogPG::execute_ctx
| | | + 21.20% PrimaryLogPG::issue_repop
| | | | + 21.10% ReplicatedBackend::submit_transaction
| | | | | + 17.30% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)
| | | | | | + 17.20% BlueStore::queue_transactions
| | | | | | + 12.40% BlueStore::_txc_add_transaction
| | | | | | | + 10.50% BlueStore::_write
| | | | | | | | + 10.50% BlueStore::_do_write
| | | | | | | | + 4.70% BlueStore::ExtentMap::fault_range
| | | | | | | | + 4.00% BlueStore::_do_alloc_write
| | | | | | | | + 1.60% BlueStore::_do_write_data
| | | | | | | | + 0.10% interval_set<unsigned long, std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > > >::union_of
| | | | | | | | + 0.10% BlueStore::ExtentMap::compress_extent_map
| | | | | | | + 0.40% BlueStore::_omap_setkeys
| | | | | | | + 0.30% BlueStore::_setattrs
| | | | | | | + 0.30% BlueStore::Collection::get_onode
| | | | | | | + 0.10% std::vector<boost::intrusive_ptr<BlueStore::Onode>, std::allocator<boost::intrusive_ptr<BlueStore::Onode> > >::~vector
| | | | | | | + 0.10% std::unique_lock<std::shared_mutex>::~unique_lock
| | | | | | | + 0.10% std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v14_2_0::ptr, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v14_2_0::ptr> > >::~map
| | | | | | | + 0.10% ghobject_t::ghobject_t
| | | | | | | + 0.10% ceph::os::Transaction::begin
| | | | | | | + 0.10% ceph::buffer::v14_2_0::list::~list
| | | | | | | + 0.10% BlueStore::_omap_rmkeys
| | | | | | | + 0.10% BlueStore::_get_collection
| | | | | | + 1.70% BlueStore::_txc_state_proc
| | | | | | + 1.60% BlueStore::_txc_write_nodes
| | | | | | + 0.90% BlueStore::BlueStoreThrottle::try_start_transaction
| | | | | | + 0.20% BlueStore::_txc_finalize_kv
| | | | | | + 0.10% std::shared_ptr<KeyValueDB::TransactionImpl>::~shared_ptr
| | | | | | + 0.10% ceph::time_detail::mono_clock::now
| | | | | | + 0.10% BlueStore::_txc_calc_cost
| | | | | + 2.20% non-virtual thunk to PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)
| | | | | + 1.00% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::~vector
| | | | | + 0.30% generate_transaction
| | | | | + 0.10% std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >::~vector
| | | | | + 0.10% std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >::~unique_ptr
| | | | | + 0.10% PrimaryLogPG::op_applied
| | | | + 0.10% PGTransaction::add_obc
| | | + 1.50% PeeringState::update_trim_to
| | | + 0.60% PrimaryLogPG::prepare_transaction
| | | + 0.10% PrimaryLogPG::new_repop
| | | + 0.10% PrimaryLogPG::OpContext::reset_obs
| | | + 0.10% MOSDOpReply::MOSDOpReply
| | + 2.50% PrimaryLogPG::find_object_context
| | + 0.30% PrimaryLogPG::OpContext::OpContext
| | + 0.30% PGLog::IndexedLog::get_request
| | + 0.30% MOSDOp::finish_decode
| | + 0.20% PG::check_in_progress_op
| | + 0.10% hobject_t::get_head
| | + 0.10% PrimaryLogPG::maybe_await_blocked_head
| | + 0.10% OpRequest::mark_started
| | + 0.10% OpRequest::includes_pg_op
| | + 0.10% OSDService::_check_full
| | + 0.10% OSDMap::is_blacklisted
| | + 0.10% Message::clear_payload
| + 5.60% OSD::ShardedOpWQ::handle_oncommits
| + 1.10% PG::lock
| + 0.20% std::mutex::lock
| + 0.20% OpQueueItem::maybe_get_op
| + 0.10% std::deque<OpQueueItem, std::allocator<OpQueueItem> >::push_back
| + 0.10% WeightedPriorityQueue<OpQueueItem, unsigned long>::dequeue
| + 0.10% OpQueueItem::~OpQueueItem
+ 0.20% ceph::HeartbeatMap::reset_timeout
Thread: 61 (tp_osd_tp) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% ShardedThreadPool::WorkThreadSharded::entry
+ 100.00% ShardedThreadPool::shardedthreadpool_worker
+ 99.90% OSD::ShardedOpWQ::_process
| + 57.70% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| | + 57.70% pthread_cond_wait@@GLIBC_2.3.2
| | + 0.90% __pthread_mutex_cond_lock
| + 32.30% OpQueueItem::run
| | + 32.30% PGOpItem::run
| | + 32.10% OSD::dequeue_op
| | | + 31.90% PrimaryLogPG::do_request
| | | | + 31.80% PrimaryLogPG::do_op
| | | | | + 27.30% PrimaryLogPG::execute_ctx
| | | | | | + 23.40% PrimaryLogPG::issue_repop
| | | | | | | + 23.30% ReplicatedBackend::submit_transaction
| | | | | | | | + 18.90% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)
| | | | | | | | | + 18.80% BlueStore::queue_transactions
| | | | | | | | | | + 12.50% BlueStore::_txc_add_transaction
| | | | | | | | | | | + 10.20% BlueStore::_write
| | | | | | | | | | | | + 10.20% BlueStore::_do_write
| | | | | | | | | | | | + 5.10% BlueStore::_do_alloc_write
| | | | | | | | | | | | + 3.70% BlueStore::ExtentMap::fault_range
| | | | | | | | | | | | + 1.10% BlueStore::_do_write_data
| | | | | | | | | | | | + 0.10% BlueStore::_wctx_finish
| | | | | | | | | | | | + 0.10% BlueStore::WriteContext::~WriteContext
| | | | | | | | | | | + 0.80% BlueStore::_omap_setkeys
| | | | | | | | | | | + 0.40% BlueStore::Collection::get_onode
| | | | | | | | | | | + 0.20% ceph::os::Transaction::iterator::decode_attrset_bl
| | | | | | | | | | | + 0.20% ceph::os::Transaction::iterator::decode_attrset
| | | | | | | | | | | + 0.20% BlueStore::_setattrs
| | | | | | | | | | | + 0.20% BlueStore::_get_collection
| | | | | | | | | | | + 0.10% ceph::os::Transaction::iterator::~iterator
| | | | | | | | | | | + 0.10% ceph::buffer::v14_2_0::list::~list
| | | | | | | | | | | + 0.10% BlueStore::_omap_rmkeys
| | | | | | | | | | + 1.90% BlueStore::_txc_state_proc
| | | | | | | | | | + 1.90% BlueStore::BlueStoreThrottle::try_start_transaction
| | | | | | | | | | + 1.60% BlueStore::_txc_write_nodes
| | | | | | | | | | + 0.50% BlueStore::_txc_create
| | | | | | | | | | + 0.30% BlueStore::_txc_finalize_kv
| | | | | | | | | | + 0.10% ceph::time_detail::mono_clock::now
| | | | | | | | | + 0.10% TrackedOp::put
| | | | | | | | + 3.30% non-virtual thunk to PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)
| | | | | | | | + 0.40% generate_transaction
| | | | | | | | + 0.30% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::~vector
| | | | | | | | + 0.20% std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >::~unique_ptr
| | | | | | | | + 0.10% std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >::vector
| | | | | | | | + 0.10% boost::intrusive_ptr<ReplicatedBackend::InProgressOp>::intrusive_ptr
| | | | | | | + 0.10% PeeringState::pre_submit_op
| | | | | | + 2.30% PeeringState::update_trim_to
| | | | | | + 1.40% PrimaryLogPG::prepare_transaction
| | | | | | + 0.10% PrimaryLogPG::new_repop
| | | | | | + 0.10% PG::get_next_version
| | | | | + 2.70% PrimaryLogPG::find_object_context
| | | | | + 0.40% PrimaryLogPG::maybe_await_blocked_head
| | | | | + 0.30% PG::check_in_progress_op
| | | | | + 0.30% Message::clear_payload
| | | | | + 0.20% PG::op_has_sufficient_caps
| | | | | + 0.10% ceph_clock_now
| | | | | + 0.10% boost::intrusive_ptr<Session>::~intrusive_ptr
| | | | | + 0.10% PrimaryLogPG::OpContext::OpContext
| | | | | + 0.10% PGLog::IndexedLog::get_request
| | | | | + 0.10% OSDService::_check_full
| | | | | + 0.10% MOSDOp::finish_decode
| | | | + 0.10% pg_t::get_hobj_start
| | | + 0.10% operator-
| | | + 0.10% OpRequest::mark_reached_pg
| | + 0.20% __pthread_mutex_unlock_usercnt
| + 7.80% OSD::ShardedOpWQ::handle_oncommits
| + 1.30% PG::lock
| + 0.30% std::unique_lock<std::mutex>::unique_lock
| + 0.30% std::mutex::lock
| + 0.10% std::deque<OpQueueItem, std::allocator<OpQueueItem> >::push_back
| + 0.10% boost::intrusive_ptr<PG>::intrusive_ptr
+ 0.10% ceph::HeartbeatMap::reset_timeout
Thread: 62 (tp_osd_tp) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% ShardedThreadPool::WorkThreadSharded::entry
+ 100.00% ShardedThreadPool::shardedthreadpool_worker
+ 100.00% OSD::ShardedOpWQ::_process
+ 61.50% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| + 61.50% pthread_cond_wait@@GLIBC_2.3.2
| + 1.60% __pthread_mutex_cond_lock
+ 28.60% OpQueueItem::run
| + 28.50% PGOpItem::run
| | + 28.50% OSD::dequeue_op
| | + 28.40% PrimaryLogPG::do_request
| | | + 28.40% PrimaryLogPG::do_op
| | | + 23.80% PrimaryLogPG::execute_ctx
| | | | + 21.00% PrimaryLogPG::issue_repop
| | | | | + 20.90% ReplicatedBackend::submit_transaction
| | | | | | + 16.20% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)
| | | | | | | + 16.20% BlueStore::queue_transactions
| | | | | | | + 12.10% BlueStore::_txc_add_transaction
| | | | | | | | + 9.60% BlueStore::_write
| | | | | | | | + 0.90% BlueStore::_omap_setkeys
| | | | | | | | + 0.60% BlueStore::Collection::get_onode
| | | | | | | | + 0.20% std::vector<boost::intrusive_ptr<BlueStore::Onode>, std::allocator<boost::intrusive_ptr<BlueStore::Onode> > >::vector
| | | | | | | | + 0.20% ceph::os::Transaction::iterator::decode_attrset_bl
| | | | | | | | + 0.20% BlueStore::_setattrs
| | | | | | | | + 0.10% std::unique_lock<std::shared_mutex>::unique_lock
| | | | | | | | + 0.10% ceph::os::Transaction::iterator::decode_attrset
| | | | | | | | + 0.10% BlueStore::_omap_rmkeys
| | | | | | | + 1.60% BlueStore::_txc_write_nodes
| | | | | | | + 1.20% BlueStore::_txc_state_proc
| | | | | | | + 0.90% BlueStore::BlueStoreThrottle::try_start_transaction
| | | | | | | + 0.20% BlueStore::_txc_create
| | | | | | | + 0.10% PerfCounters::inc
| | | | | | | + 0.10% BlueStore::_txc_finalize_kv
| | | | | | + 3.40% non-virtual thunk to PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)
| | | | | | + 0.70% generate_transaction
| | | | | | + 0.20% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::~vector
| | | | | | + 0.20% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::push_back
| | | | | | + 0.10% std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >::vector
| | | | | | + 0.10% PrimaryLogPG::op_applied
| | | | | + 0.10% PGTransaction::add_obc
| | | | + 1.30% PeeringState::update_trim_to
| | | | + 0.90% PrimaryLogPG::prepare_transaction
| | | | + 0.30% MOSDOpReply::MOSDOpReply
| | | | + 0.10% PrimaryLogPG::new_repop
| | | | + 0.10% PrimaryLogPG::eval_repop
| | | + 2.80% PrimaryLogPG::find_object_context
| | | + 0.30% PGLog::IndexedLog::get_request
| | | + 0.30% Message::clear_payload
| | | + 0.30% MOSDOp::finish_decode
| | | + 0.20% PrimaryLogPG::maybe_await_blocked_head
| | | + 0.20% PrimaryLogPG::OpContext::OpContext
| | | + 0.10% PG::op_has_sufficient_caps
| | | + 0.10% OpRequest::mark_started
| | | + 0.10% OSDService::_check_full
| | + 0.10% OpRequest::mark_reached_pg
| + 0.10% PGPeeringItem::run
+ 6.40% OSD::ShardedOpWQ::handle_oncommits
+ 2.30% PG::lock
+ 0.60% std::unique_lock<std::mutex>::unique_lock
+ 0.20% OpQueueItem::~OpQueueItem
+ 0.10% std::vector<OSDShard*, std::allocator<OSDShard*> >::operator[]
+ 0.10% std::mutex::lock
+ 0.10% std::deque<OpQueueItem, std::allocator<OpQueueItem> >::push_back
+ 0.10% ContextQueue::move_to
Thread: 63 (tp_osd_tp) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% ShardedThreadPool::WorkThreadSharded::entry
+ 100.00% ShardedThreadPool::shardedthreadpool_worker
+ 100.00% OSD::ShardedOpWQ::_process
+ 58.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| + 58.00% pthread_cond_wait@@GLIBC_2.3.2
| + 0.70% __pthread_mutex_cond_lock
+ 39.30% OpQueueItem::run
| + 39.30% PGOpItem::run
| + 39.30% OSD::dequeue_op
| + 38.90% PrimaryLogPG::do_request
| | + 38.90% PrimaryLogPG::do_op
| | + 31.20% PrimaryLogPG::execute_ctx
| | | + 26.20% PrimaryLogPG::issue_repop
| | | | + 25.90% ReplicatedBackend::submit_transaction
| | | | | + 20.30% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)
| | | | | | + 20.30% BlueStore::queue_transactions
| | | | | | + 14.60% BlueStore::_txc_add_transaction
| | | | | | | + 11.20% BlueStore::_write
| | | | | | | | + 11.20% BlueStore::_do_write
| | | | | | | | + 5.70% BlueStore::ExtentMap::fault_range
| | | | | | | | + 3.80% BlueStore::_do_alloc_write
| | | | | | | | + 1.40% BlueStore::_do_write_data
| | | | | | | | + 0.10% BlueStore::_wctx_finish
| | | | | | | | + 0.10% BlueStore::ExtentMap::compress_extent_map
| | | | | | | + 1.20% BlueStore::_omap_setkeys
| | | | | | | + 0.70% BlueStore::Collection::get_onode
| | | | | | | + 0.40% ceph::os::Transaction::iterator::decode_attrset_bl
| | | | | | | + 0.20% ghobject_t::ghobject_t
| | | | | | | + 0.20% ceph::os::Transaction::iterator::decode_attrset
| | | | | | | + 0.20% ceph::os::Transaction::begin
| | | | | | | + 0.20% BlueStore::_setattrs
| | | | | | | + 0.10% std::unique_lock<std::shared_mutex>::~unique_lock
| | | | | | | + 0.10% ceph::buffer::v14_2_0::list::~list
| | | | | | | + 0.10% BlueStore::_omap_rmkeys
| | | | | | + 2.30% BlueStore::_txc_state_proc
| | | | | | + 1.50% BlueStore::_txc_write_nodes
| | | | | | + 1.20% BlueStore::BlueStoreThrottle::try_start_transaction
| | | | | | + 0.40% BlueStore::_txc_create
| | | | | | + 0.10% ceph::time_detail::mono_clock::now
| | | | | | + 0.10% BlueStore::log_latency
| | | | | | + 0.10% BlueStore::_txc_finalize_kv
| | | | | + 3.70% non-virtual thunk to PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)
| | | | | + 0.50% generate_transaction
| | | | | + 0.40% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::~vector
| | | | | + 0.30% std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >::vector
| | | | | + 0.20% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::push_back
| | | | | + 0.10% std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >::~unique_ptr
| | | | | + 0.10% ceph::make_ref<ReplicatedBackend::InProgressOp, unsigned long&, Context*&, boost::intrusive_ptr<OpRequest>&, eversion_t const&>
| | | | | + 0.10% boost::intrusive_ptr<OpRequest>::~intrusive_ptr
| | | | | + 0.10% PrimaryLogPG::op_applied
| | | | | + 0.10% PGBackend::get_osdmap
| | | | + 0.20% PGLog::IndexedLog::add
| | | + 2.60% PeeringState::update_trim_to
| | | + 1.70% PrimaryLogPG::prepare_transaction
| | | + 0.30% PrimaryLogPG::new_repop
| | | + 0.30% MOSDOpReply::MOSDOpReply
| | | + 0.10% PrimaryLogPG::eval_repop
| | + 4.20% PrimaryLogPG::find_object_context
| | + 0.60% MOSDOp::finish_decode
| | + 0.50% Message::clear_payload
| | + 0.40% PrimaryLogPG::maybe_await_blocked_head
| | + 0.40% PGLog::IndexedLog::get_request
| | + 0.30% PrimaryLogPG::maybe_handle_cache
| | + 0.30% PerfCounters::tinc
| | + 0.20% PrimaryLogPG::OpContext::OpContext
| | + 0.20% PG::check_in_progress_op
| | + 0.10% std::shared_ptr<ObjectContext>::~shared_ptr
| | + 0.10% PrimaryLogPG::is_unreadable_object
| | + 0.10% PrimaryLogPG::is_degraded_or_backfilling_object
| | + 0.10% PrimaryLogPG::<lambda(auto:173)>::operator()<CephContext*>
| | + 0.10% MOSDOp::get_reqid
| + 0.20% OpRequest::mark_reached_pg
| + 0.20% OSDService::maybe_share_map
+ 1.80% PG::lock
+ 0.40% WeightedPriorityQueue<OpQueueItem, unsigned long>::dequeue
+ 0.20% std::mutex::lock
+ 0.20% OpQueueItem::~OpQueueItem
+ 0.10% boost::intrusive_ptr<PG>::intrusive_ptr
Thread: 64 (tp_osd_tp) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% ShardedThreadPool::WorkThreadSharded::entry
+ 100.00% ShardedThreadPool::shardedthreadpool_worker
+ 100.00% OSD::ShardedOpWQ::_process
+ 70.60% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| + 70.60% pthread_cond_wait@@GLIBC_2.3.2
| + 1.90% __pthread_mutex_cond_lock
+ 26.60% OpQueueItem::run
| + 26.60% PGOpItem::run
| + 26.60% OSD::dequeue_op
| + 26.30% PrimaryLogPG::do_request
| | + 26.30% PrimaryLogPG::do_op
| | + 22.50% PrimaryLogPG::execute_ctx
| | | + 19.90% PrimaryLogPG::issue_repop
| | | | + 19.70% ReplicatedBackend::submit_transaction
| | | | | + 15.90% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)
| | | | | | + 15.90% BlueStore::queue_transactions
| | | | | | + 11.60% BlueStore::_txc_add_transaction
| | | | | | | + 9.00% BlueStore::_write
| | | | | | | + 0.80% BlueStore::_omap_setkeys
| | | | | | | + 0.70% BlueStore::Collection::get_onode
| | | | | | | + 0.30% ceph::os::Transaction::iterator::decode_attrset_bl
| | | | | | | + 0.20% BlueStore::_omap_rmkeys
| | | | | | | + 0.10% std::vector<boost::intrusive_ptr<BlueStore::Collection>, std::allocator<boost::intrusive_ptr<BlueStore::Collection> > >::~vector
| | | | | | | + 0.10% std::unique_lock<std::shared_mutex>::unique_lock
| | | | | | | + 0.10% ghobject_t::ghobject_t
| | | | | | | + 0.10% ceph::os::Transaction::iterator::decode_bl
| | | | | | | + 0.10% ceph::os::Transaction::iterator::decode_attrset
| | | | | | | + 0.10% BlueStore::_setattrs
| | | | | | + 2.00% BlueStore::_txc_write_nodes
| | | | | | + 1.10% BlueStore::BlueStoreThrottle::try_start_transaction
| | | | | | + 0.90% BlueStore::_txc_state_proc
| | | | | | + 0.20% BlueStore::_txc_finalize_kv
| | | | | | + 0.10% BlueStore::_txc_create
| | | | | + 3.00% non-virtual thunk to PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)
| | | | | + 0.30% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::~vector
| | | | | + 0.20% generate_transaction
| | | | | + 0.10% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::push_back
| | | | | + 0.10% std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >::~unique_ptr
| | | | | + 0.10% ceph::os::Transaction::Transaction
| | | | + 0.10% PGLog::IndexedLog::add
| | | + 1.50% PeeringState::update_trim_to
| | | + 0.40% PrimaryLogPG::prepare_transaction
| | | + 0.20% PrimaryLogPG::OpContext::register_on_commit<PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)::<lambda()> >
| | | + 0.20% MOSDOpReply::MOSDOpReply
| | | + 0.10% PrimaryLogPG::new_repop
| | | + 0.10% PrimaryLogPG::OpContext::reset_obs
| | | + 0.10% PrimaryLogPG::OpContext::register_on_finish<PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)::<lambda()> >
| | + 2.30% PrimaryLogPG::find_object_context
| | + 0.40% Message::clear_payload
| | + 0.30% MOSDOp::finish_decode
| | + 0.20% boost::intrusive_ptr<Session>::~intrusive_ptr
| | + 0.20% PrimaryLogPG::OpContext::OpContext
| | + 0.10% PrimaryLogPG::is_unreadable_object
| | + 0.10% PrimaryLogPG::check_laggy
| | + 0.10% PG::check_in_progress_op
| | + 0.10% OpRequest::mark_started
| + 0.10% OpRequest::mark_reached_pg
| + 0.10% OSDService::maybe_share_map
+ 2.00% PG::lock
+ 0.40% std::unique_lock<std::mutex>::unique_lock
+ 0.10% std::mutex::lock
+ 0.10% boost::intrusive_ptr<PG>::intrusive_ptr
+ 0.10% OpQueueItem::~OpQueueItem
+ 0.10% OpQueueItem::get_ordering_token
Thread: 65 (tp_osd_tp) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% ShardedThreadPool::WorkThreadSharded::entry
+ 100.00% ShardedThreadPool::shardedthreadpool_worker
+ 99.90% OSD::ShardedOpWQ::_process
| + 56.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| | + 56.00% pthread_cond_wait@@GLIBC_2.3.2
| | + 0.50% __pthread_mutex_cond_lock
| + 40.20% OpQueueItem::run
| | + 40.00% PGOpItem::run
| | | + 39.90% OSD::dequeue_op
| | | | + 39.80% PrimaryLogPG::do_request
| | | | | + 39.70% PrimaryLogPG::do_op
| | | | | | + 32.30% PrimaryLogPG::execute_ctx
| | | | | | | + 28.30% PrimaryLogPG::issue_repop
| | | | | | | | + 27.70% ReplicatedBackend::submit_transaction
| | | | | | | | | + 21.80% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)
| | | | | | | | | | + 21.80% BlueStore::queue_transactions
| | | | | | | | | | + 14.90% BlueStore::_txc_add_transaction
| | | | | | | | | | | + 11.80% BlueStore::_write
| | | | | | | | | | | | + 11.80% BlueStore::_do_write
| | | | | | | | | | | | + 5.00% BlueStore::_do_alloc_write
| | | | | | | | | | | | + 4.90% BlueStore::ExtentMap::fault_range
| | | | | | | | | | | | + 1.50% BlueStore::_do_write_data
| | | | | | | | | | | | + 0.30% BlueStore::ExtentMap::compress_extent_map
| | | | | | | | | | | | + 0.10% boost::intrusive_ptr<BlueStore::Onode>::~intrusive_ptr
| | | | | | | | | | | + 1.10% BlueStore::_omap_setkeys
| | | | | | | | | | | + 0.80% BlueStore::Collection::get_onode
| | | | | | | | | | | + 0.30% ceph::os::Transaction::iterator::decode_attrset
| | | | | | | | | | | + 0.30% BlueStore::_setattrs
| | | | | | | | | | | + 0.20% ceph::os::Transaction::iterator::decode_attrset_bl
| | | | | | | | | | | + 0.20% ceph::buffer::v14_2_0::list::~list
| | | | | | | | | | | + 0.10% ceph::os::Transaction::begin
| | | | | | | | | | | + 0.10% BlueStore::_omap_rmkeys
| | | | | | | | | | + 2.70% BlueStore::_txc_state_proc
| | | | | | | | | | + 1.80% BlueStore::_txc_write_nodes
| | | | | | | | | | + 1.70% BlueStore::BlueStoreThrottle::try_start_transaction
| | | | | | | | | | + 0.50% BlueStore::_txc_finalize_kv
| | | | | | | | | | + 0.20% BlueStore::_txc_create
| | | | | | | | | + 4.50% non-virtual thunk to PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)
| | | | | | | | | + 0.60% generate_transaction
| | | | | | | | | + 0.20% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::~vector
| | | | | | | | | + 0.20% std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >::~unique_ptr
| | | | | | | | | + 0.10% std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >::vector
| | | | | | | | | + 0.10% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::push_back
| | | | | | | | | + 0.10% operator!=
| | | | | | | | | + 0.10% ceph::make_ref<ReplicatedBackend::InProgressOp, unsigned long&, Context*&, boost::intrusive_ptr<OpRequest>&, eversion_t const&>
| | | | | | | | + 0.20% pthread_self@GLIBC_2.2.5
| | | | | | | | + 0.20% PGTransaction::add_obc
| | | | | | | | + 0.10% PGLog::IndexedLog::add
| | | | | | | | + 0.10% C_OSD_RepopCommit::C_OSD_RepopCommit
| | | | | | | + 2.20% PeeringState::update_trim_to
| | | | | | | + 1.30% PrimaryLogPG::prepare_transaction
| | | | | | | + 0.20% PrimaryLogPG::OpContext::register_on_success<PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)::<lambda()> >
| | | | | | | + 0.10% PrimaryLogPG::new_repop
| | | | | | | + 0.10% PrimaryLogPG::OpContext::register_on_finish<PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)::<lambda()> >
| | | | | | | + 0.10% PrimaryLogPG::OpContext::register_on_commit<PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)::<lambda()> >
| | | | | | + 4.40% PrimaryLogPG::find_object_context
| | | | | | + 1.00% Message::clear_payload
| | | | | | + 0.40% PGLog::IndexedLog::get_request
| | | | | | + 0.40% MOSDOp::finish_decode
| | | | | | + 0.20% PG::check_in_progress_op
| | | | | | + 0.10% tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))
| | | | | | + 0.10% operator-=
| | | | | | + 0.10% object_locator_t::~object_locator_t
| | | | | | + 0.10% object_locator_t::object_locator_t
| | | | | | + 0.10% boost::intrusive_ptr<OpRequest>::~intrusive_ptr
| | | | | | + 0.10% PrimaryLogPG::maybe_await_blocked_head
| | | | | | + 0.10% PrimaryLogPG::is_degraded_or_backfilling_object
| | | | | | + 0.10% OpRequest::mark_started
| | | | | | + 0.10% OSD::init_op_flags
| | | | | + 0.10% PG::can_discard_request
| | | | + 0.10% OpRequest::mark_reached_pg
| | | + 0.10% __pthread_mutex_unlock_usercnt
| | + 0.20% PGPeeringItem::run
| + 3.00% PG::lock
| + 0.20% std::mutex::lock
| + 0.20% boost::intrusive_ptr<PG>::intrusive_ptr
| + 0.10% std::unique_lock<std::mutex>::unique_lock
| + 0.10% OpRequest::get_reqid
| + 0.10% OpQueueItem::maybe_get_op
+ 0.10% ceph::HeartbeatMap::reset_timeout
Thread: 66 (tp_osd_tp) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% ShardedThreadPool::WorkThreadSharded::entry
+ 100.00% ShardedThreadPool::shardedthreadpool_worker
+ 100.00% OSD::ShardedOpWQ::_process
+ 70.40% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| + 70.40% pthread_cond_wait@@GLIBC_2.3.2
| + 1.00% __pthread_mutex_cond_lock
+ 26.90% OpQueueItem::run
| + 26.90% PGOpItem::run
| + 26.70% OSD::dequeue_op
| | + 26.50% PrimaryLogPG::do_request
| | | + 26.30% PrimaryLogPG::do_op
| | | | + 22.00% PrimaryLogPG::execute_ctx
| | | | | + 19.70% PrimaryLogPG::issue_repop
| | | | | | + 19.30% ReplicatedBackend::submit_transaction
| | | | | | | + 14.20% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)
| | | | | | | | + 14.20% BlueStore::queue_transactions
| | | | | | | | + 9.40% BlueStore::_txc_add_transaction
| | | | | | | | + 2.00% BlueStore::BlueStoreThrottle::try_start_transaction
| | | | | | | | + 1.50% BlueStore::_txc_write_nodes
| | | | | | | | + 1.20% BlueStore::_txc_state_proc
| | | | | | | | + 0.10% BlueStore::_txc_create
| | | | | | | + 3.90% non-virtual thunk to PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)
| | | | | | | + 0.30% generate_transaction
| | | | | | | + 0.20% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::~vector
| | | | | | | + 0.10% std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >::~vector
| | | | | | | + 0.10% std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >::vector
| | | | | | | + 0.10% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::push_back
| | | | | | | + 0.10% std::set<pg_shard_t, std::less<pg_shard_t>, std::allocator<pg_shard_t> >::insert<std::_Rb_tree_const_iterator<pg_shard_t> >
| | | | | | | + 0.10% ceph::make_ref<ReplicatedBackend::InProgressOp, unsigned long&, Context*&, boost::intrusive_ptr<OpRequest>&, eversion_t const&>
| | | | | | | + 0.10% PrimaryLogPG::op_applied
| | | | | | + 0.30% PGLog::IndexedLog::add
| | | | | | + 0.10% PGTransaction::add_obc
| | | | | + 1.50% PeeringState::update_trim_to
| | | | | + 0.60% PrimaryLogPG::prepare_transaction
| | | | | + 0.10% PrimaryLogPG::OpContext::register_on_finish<PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)::<lambda()> >
| | | | | + 0.10% MOSDOpReply::MOSDOpReply
| | | | + 2.90% PrimaryLogPG::find_object_context
| | | | + 0.50% Message::clear_payload
| | | | + 0.30% PG::check_in_progress_op
| | | | + 0.10% boost::intrusive_ptr<Session>::~intrusive_ptr
| | | | + 0.10% PrimaryLogPG::is_degraded_or_backfilling_object
| | | | + 0.10% PGLog::IndexedLog::get_request
| | | | + 0.10% OpRequest::mark_started
| | | | + 0.10% OSDMap::is_blacklisted
| | | | + 0.10% MOSDOp::finish_decode
| | | + 0.10% PG::can_discard_request
| | + 0.20% OpRequest::mark_reached_pg
| + 0.20% __pthread_mutex_unlock_usercnt
+ 1.70% PG::lock
+ 0.40% std::mutex::lock
+ 0.30% WeightedPriorityQueue<OpQueueItem, unsigned long>::dequeue
+ 0.20% std::unique_lock<std::mutex>::unique_lock
+ 0.10% OpQueueItem::get_ordering_token
Thread: 67 (tp_osd_tp) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% ShardedThreadPool::WorkThreadSharded::entry
+ 100.00% ShardedThreadPool::shardedthreadpool_worker
+ 100.00% OSD::ShardedOpWQ::_process
+ 54.80% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| + 54.80% pthread_cond_wait@@GLIBC_2.3.2
| + 1.00% __pthread_mutex_cond_lock
+ 42.50% OpQueueItem::run
| + 42.50% PGOpItem::run
| + 42.30% OSD::dequeue_op
| | + 42.20% PrimaryLogPG::do_request
| | | + 41.90% PrimaryLogPG::do_op
| | | | + 36.10% PrimaryLogPG::execute_ctx
| | | | | + 31.10% PrimaryLogPG::issue_repop
| | | | | | + 31.00% ReplicatedBackend::submit_transaction
| | | | | | + 24.30% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)
| | | | | | | + 24.30% BlueStore::queue_transactions
| | | | | | | + 18.10% BlueStore::_txc_add_transaction
| | | | | | | | + 15.40% BlueStore::_write
| | | | | | | | | + 15.40% BlueStore::_do_write
| | | | | | | | | + 7.20% BlueStore::ExtentMap::fault_range
| | | | | | | | | + 6.30% BlueStore::_do_alloc_write
| | | | | | | | | + 1.40% BlueStore::_do_write_data
| | | | | | | | | + 0.20% BlueStore::ExtentMap::compress_extent_map
| | | | | | | | | + 0.10% BlueStore::WriteContext::~WriteContext
| | | | | | | | | + 0.10% BlueStore::GarbageCollector::GarbageCollector
| | | | | | | | | + 0.10% BlueStore::ExtentMap::dirty_range
| | | | | | | | + 0.90% BlueStore::_omap_setkeys
| | | | | | | | + 0.80% BlueStore::Collection::get_onode
| | | | | | | | + 0.30% BlueStore::_setattrs
| | | | | | | | + 0.20% BlueStore::_omap_rmkeys
| | | | | | | | + 0.10% std::unique_lock<std::shared_mutex>::unique_lock
| | | | | | | | + 0.10% ghobject_t::~ghobject_t
| | | | | | | | + 0.10% ceph::os::Transaction::iterator::decode_attrset_bl
| | | | | | | | + 0.10% ceph::os::Transaction::begin
| | | | | | | | + 0.10% ceph::buffer::v14_2_0::list::~list
| | | | | | | + 2.00% BlueStore::_txc_write_nodes
| | | | | | | + 1.70% BlueStore::_txc_state_proc
| | | | | | | + 1.60% BlueStore::BlueStoreThrottle::try_start_transaction
| | | | | | | + 0.60% BlueStore::_txc_create
| | | | | | | + 0.20% BlueStore::_txc_finalize_kv
| | | | | | + 4.40% non-virtual thunk to PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)
| | | | | | + 1.10% generate_transaction
| | | | | | + 0.40% std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >::vector
| | | | | | + 0.40% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::~vector
| | | | | | + 0.10% std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >::~vector
| | | | | | + 0.10% std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >::~unique_ptr
| | | | | | + 0.10% PrimaryLogPG::get_acting_recovery_backfill_shards
| | | | | | + 0.10% PrimaryLogPG::apply_stats
| | | | | + 2.70% PeeringState::update_trim_to
| | | | | + 1.80% PrimaryLogPG::prepare_transaction
| | | | | + 0.30% PrimaryLogPG::new_repop
| | | | | + 0.10% PG::filter_snapc
| | | | | + 0.10% MOSDOpReply::MOSDOpReply
| | | | + 2.90% PrimaryLogPG::find_object_context
| | | | + 0.60% Message::clear_payload
| | | | + 0.40% PGLog::IndexedLog::get_request
| | | | + 0.40% MOSDOp::finish_decode
| | | | + 0.30% PG::check_in_progress_op
| | | | + 0.20% PrimaryLogPG::maybe_await_blocked_head
| | | | + 0.20% PrimaryLogPG::OpContext::OpContext
| | | | + 0.20% PerfCounters::tinc
| | | | + 0.10% boost::intrusive_ptr<Session>::~intrusive_ptr
| | | | + 0.10% PrimaryLogPG::maybe_handle_cache
| | | | + 0.10% PrimaryLogPG::check_laggy
| | | | + 0.10% PG::op_has_sufficient_caps
| | | | + 0.10% PG::get_osdmap
| | | | + 0.10% OpRequest::mark_started
| | | + 0.10% std::unordered_map<entity_name_t, std::__cxx11::list<boost::intrusive_ptr<OpRequest>, std::allocator<boost::intrusive_ptr<OpRequest> > >, std::hash<entity_name_t>, std::equal_to<entity_name_t>, std::allocator<std::pair<entity_name_t const, std::__cxx11::list<boost::intrusive_ptr<OpRequest>, std::allocator<boost::intrusive_ptr<OpRequest> > > > > >::find
| | | + 0.10% OpRequest::get_source
| | + 0.10% OpRequest::mark_reached_pg
| + 0.10% pthread_mutex_unlock
| + 0.10% __pthread_mutex_unlock_usercnt
+ 1.60% PG::lock
+ 0.60% std::mutex::lock
+ 0.50% WeightedPriorityQueue<OpQueueItem, unsigned long>::dequeue
Thread: 68 (tp_osd_tp) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% ShardedThreadPool::WorkThreadSharded::entry
+ 100.00% ShardedThreadPool::shardedthreadpool_worker
+ 99.90% OSD::ShardedOpWQ::_process
| + 70.20% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| | + 70.20% pthread_cond_wait@@GLIBC_2.3.2
| | + 1.60% __pthread_mutex_cond_lock
| + 27.00% OpQueueItem::run
| | + 26.90% PGOpItem::run
| | | + 26.90% OSD::dequeue_op
| | | + 26.70% PrimaryLogPG::do_request
| | | | + 26.70% PrimaryLogPG::do_op
| | | | + 23.00% PrimaryLogPG::execute_ctx
| | | | | + 20.00% PrimaryLogPG::issue_repop
| | | | | | + 19.90% ReplicatedBackend::submit_transaction
| | | | | | | + 14.70% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)
| | | | | | | | + 14.70% BlueStore::queue_transactions
| | | | | | | | + 9.50% BlueStore::_txc_add_transaction
| | | | | | | | + 1.70% BlueStore::BlueStoreThrottle::try_start_transaction
| | | | | | | | + 1.60% BlueStore::_txc_write_nodes
| | | | | | | | + 1.60% BlueStore::_txc_state_proc
| | | | | | | | + 0.10% ceph::time_detail::mono_clock::now
| | | | | | | | + 0.10% BlueStore::_txc_finalize_kv
| | | | | | | | + 0.10% BlueStore::_txc_create
| | | | | | | + 3.50% non-virtual thunk to PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)
| | | | | | | + 0.70% generate_transaction
| | | | | | | + 0.40% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::~vector
| | | | | | | + 0.20% std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >::vector
| | | | | | | + 0.10% std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >::~vector
| | | | | | | + 0.10% std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >::~unique_ptr
| | | | | | | + 0.10% ReplicatedBackend::issue_op
| | | | | | | + 0.10% PrimaryLogPG::apply_stats
| | | | | | + 0.10% PGLog::IndexedLog::add
| | | | | + 1.50% PeeringState::update_trim_to
| | | | | + 0.70% PrimaryLogPG::prepare_transaction
| | | | | + 0.30% MOSDOpReply::MOSDOpReply
| | | | | + 0.20% PrimaryLogPG::new_repop
| | | | | + 0.10% std::shared_ptr<ObjectContext>::~shared_ptr
| | | | | + 0.10% PrimaryLogPG::OpContext::reset_obs
| | | | | + 0.10% PG::get_next_version
| | | | + 2.40% PrimaryLogPG::find_object_context
| | | | + 0.40% Message::clear_payload
| | | | + 0.30% MOSDOp::finish_decode
| | | | + 0.10% PrimaryLogPG::maybe_await_blocked_head
| | | | + 0.10% PrimaryLogPG::check_laggy
| | | | + 0.10% PrimaryLogPG::OpContext::OpContext
| | | | + 0.10% PGLog::IndexedLog::get_request
| | | | + 0.10% PG::check_in_progress_op
| | | | + 0.10% ObjectContext::is_blocked
| | | + 0.20% OpRequest::mark_reached_pg
| | + 0.10% std::unique_ptr<OpQueueItem::OpQueueable, std::default_delete<OpQueueItem::OpQueueable> >::operator->
| + 2.00% PG::lock
| + 0.20% std::unique_lock<std::mutex>::unique_lock
| + 0.20% std::mutex::lock
| + 0.20% WeightedPriorityQueue<OpQueueItem, unsigned long>::dequeue
+ 0.10% ceph::HeartbeatMap::reset_timeout
Thread: 69 (tp_osd_tp) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% ShardedThreadPool::WorkThreadSharded::entry
+ 100.00% ShardedThreadPool::shardedthreadpool_worker
+ 100.00% OSD::ShardedOpWQ::_process
+ 65.30% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| + 65.30% pthread_cond_wait@@GLIBC_2.3.2
| + 1.10% __pthread_mutex_cond_lock
| + 0.10% __pthread_enable_asynccancel
+ 32.60% OpQueueItem::run
| + 32.60% PGOpItem::run
| + 32.50% OSD::dequeue_op
| + 32.20% PrimaryLogPG::do_request
| | + 32.10% PrimaryLogPG::do_op
| | + 26.40% PrimaryLogPG::execute_ctx
| | | + 21.70% PrimaryLogPG::issue_repop
| | | | + 21.60% ReplicatedBackend::submit_transaction
| | | | + 16.80% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)
| | | | | + 16.70% BlueStore::queue_transactions
| | | | | | + 10.70% BlueStore::_txc_add_transaction
| | | | | | | + 8.70% BlueStore::_write
| | | | | | | + 0.50% BlueStore::_omap_setkeys
| | | | | | | + 0.50% BlueStore::Collection::get_onode
| | | | | | | + 0.30% ceph::os::Transaction::iterator::decode_attrset
| | | | | | | + 0.20% ghobject_t::ghobject_t
| | | | | | | + 0.20% BlueStore::_setattrs
| | | | | | | + 0.20% BlueStore::_get_collection
| | | | | | | + 0.10% std::vector<boost::intrusive_ptr<BlueStore::Onode>, std::allocator<boost::intrusive_ptr<BlueStore::Onode> > >::~vector
| | | | | | + 2.00% BlueStore::_txc_state_proc
| | | | | | + 1.80% BlueStore::_txc_write_nodes
| | | | | | + 1.70% BlueStore::BlueStoreThrottle::try_start_transaction
| | | | | | + 0.30% BlueStore::_txc_finalize_kv
| | | | | | + 0.10% ceph::time_detail::mono_clock::now
| | | | | + 0.10% TrackedOp::put
| | | | + 3.10% non-virtual thunk to PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)
| | | | + 0.80% generate_transaction
| | | | + 0.20% std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >::vector
| | | | + 0.20% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::~vector
| | | | + 0.10% std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >::~vector
| | | | + 0.10% std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >::~unique_ptr
| | | | + 0.10% boost::intrusive_ptr<OpRequest>::~intrusive_ptr
| | | | + 0.10% ReplicatedBackend::issue_op
| | | | + 0.10% PGBackend::get_osdmap
| | | + 1.90% PeeringState::update_trim_to
| | | + 1.70% PrimaryLogPG::prepare_transaction
| | | + 0.30% MOSDOpReply::MOSDOpReply
| | | + 0.20% PrimaryLogPG::new_repop
| | | + 0.20% PrimaryLogPG::OpContext::reset_obs
| | | + 0.20% PG::filter_snapc
| | | + 0.10% tc_newarray
| | | + 0.10% PrimaryLogPG::OpContext::register_on_success<PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)::<lambda()> >
| | + 3.90% PrimaryLogPG::find_object_context
| | + 0.30% PrimaryLogPG::maybe_await_blocked_head
| | + 0.30% Message::clear_payload
| | + 0.20% PrimaryLogPG::OpContext::OpContext
| | + 0.20% PG::check_in_progress_op
| | + 0.20% MOSDOp::finish_decode
| | + 0.10% tc_newarray
| | + 0.10% PrimaryLogPG::check_laggy
| | + 0.10% PGLog::IndexedLog::get_request
| | + 0.10% PG::op_has_sufficient_caps
| | + 0.10% PG::get_osdmap
| | + 0.10% OSDService::_check_full
| + 0.10% operator-
| + 0.10% ceph_clock_now
| + 0.10% OpRequest::mark_reached_pg
+ 1.40% PG::lock
+ 0.20% std::mutex::lock
+ 0.20% WeightedPriorityQueue<OpQueueItem, unsigned long>::dequeue
+ 0.10% std::unique_lock<std::mutex>::unique_lock
+ 0.10% WeightedPriorityQueue<OpQueueItem, unsigned long>::empty
+ 0.10% OpQueueItem::~OpQueueItem
Thread: 70 (tp_osd_tp) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% ShardedThreadPool::WorkThreadSharded::entry
+ 100.00% ShardedThreadPool::shardedthreadpool_worker
+ 99.90% OSD::ShardedOpWQ::_process
| + 66.60% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| | + 66.60% pthread_cond_wait@@GLIBC_2.3.2
| | + 1.50% __pthread_mutex_cond_lock
| + 31.10% OpQueueItem::run
| | + 31.00% PGOpItem::run
| | | + 31.00% OSD::dequeue_op
| | | + 31.00% PrimaryLogPG::do_request
| | | + 30.70% PrimaryLogPG::do_op
| | | | + 24.20% PrimaryLogPG::execute_ctx
| | | | | + 21.90% PrimaryLogPG::issue_repop
| | | | | | + 21.60% ReplicatedBackend::submit_transaction
| | | | | | | + 17.20% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)
| | | | | | | | + 17.20% BlueStore::queue_transactions
| | | | | | | | + 11.80% BlueStore::_txc_add_transaction
| | | | | | | | | + 9.20% BlueStore::_write
| | | | | | | | | + 1.00% BlueStore::Collection::get_onode
| | | | | | | | | + 0.40% BlueStore::_omap_setkeys
| | | | | | | | | + 0.30% ceph::os::Transaction::iterator::decode_attrset_bl
| | | | | | | | | + 0.20% ceph::os::Transaction::begin
| | | | | | | | | + 0.10% std::vector<boost::intrusive_ptr<BlueStore::Onode>, std::allocator<boost::intrusive_ptr<BlueStore::Onode> > >::~vector
| | | | | | | | | + 0.10% std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v14_2_0::ptr, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v14_2_0::ptr> > >::~map
| | | | | | | | | + 0.10% ceph::os::Transaction::iterator::decode_attrset
| | | | | | | | | + 0.10% ceph::buffer::v14_2_0::list::~list
| | | | | | | | | + 0.10% BlueStore::_setattrs
| | | | | | | | | + 0.10% BlueStore::_get_collection
| | | | | | | | + 1.90% BlueStore::_txc_write_nodes
| | | | | | | | + 1.80% BlueStore::BlueStoreThrottle::try_start_transaction
| | | | | | | | + 1.20% BlueStore::_txc_state_proc
| | | | | | | | + 0.30% BlueStore::_txc_create
| | | | | | | | + 0.20% BlueStore::_txc_finalize_kv
| | | | | | | + 3.20% non-virtual thunk to PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)
| | | | | | | + 0.40% generate_transaction
| | | | | | | + 0.20% PrimaryLogPG::apply_stats
| | | | | | | + 0.10% std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >::vector
| | | | | | | + 0.10% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::~vector
| | | | | | | + 0.10% std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >::push_back
| | | | | | | + 0.10% std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >::~unique_ptr
| | | | | | | + 0.10% std::set<pg_shard_t, std::less<pg_shard_t>, std::allocator<pg_shard_t> >::insert<std::_Rb_tree_const_iterator<pg_shard_t> >
| | | | | | | + 0.10% ceph::make_ref<ReplicatedBackend::InProgressOp, unsigned long&, Context*&, boost::intrusive_ptr<OpRequest>&, eversion_t const&>
| | | | | | + 0.20% PGLog::IndexedLog::add
| | | | | | + 0.10% PGTransaction::add_obc
| | | | | + 1.40% PeeringState::update_trim_to
| | | | | + 0.70% PrimaryLogPG::prepare_transaction
| | | | | + 0.10% std::vector<snapid_t, std::allocator<snapid_t> >::operator=
| | | | | + 0.10% boost::intrusive_ptr<OpRequest>::~intrusive_ptr
| | | | + 4.10% PrimaryLogPG::find_object_context
| | | | + 0.70% MOSDOp::finish_decode
| | | | + 0.60% Message::clear_payload
| | | | + 0.20% PGLog::IndexedLog::get_request
| | | | + 0.10% boost::intrusive_ptr<OpRequest>::intrusive_ptr
| | | | + 0.10% PrimaryLogPG::maybe_await_blocked_head
| | | | + 0.10% PrimaryLogPG::check_laggy
| | | | + 0.10% PG::op_has_sufficient_caps
| | | | + 0.10% OSDService::check_failsafe_full
| | | | + 0.10% OSDService::_check_full
| | | | + 0.10% OSD::init_op_flags
| | | | + 0.10% Message::get_source_addr
| | | + 0.10% pg_t::get_hobj_start
| | | + 0.10% PG::can_discard_request
| | + 0.10% PGPeeringItem::run
| + 1.60% PG::lock
| + 0.20% boost::intrusive_ptr<PG>::intrusive_ptr
| + 0.20% WeightedPriorityQueue<OpQueueItem, unsigned long>::dequeue
+ 0.10% ceph::HeartbeatMap::reset_timeout
Thread: 71 (tp_osd_cmd) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% ThreadPool::WorkThread::entry
+ 100.00% ThreadPool::worker
+ 100.00% std::condition_variable::wait_for<long, std::ratio<1l, 1l> >
+ 100.00% std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >
+ 100.00% std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >
+ 100.00% __gthread_cond_timedwait
+ 100.00% pthread_cond_timedwait@@GLIBC_2.3.2
Thread: 72 (osd_srv_heartbt) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% OSD::T_Heartbeat::entry
+ 100.00% OSD::heartbeat_entry
+ 99.70% std::condition_variable::wait_for<unsigned long, std::ratio<1l, 1000000000l> >
| + 99.70% std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >
| + 99.70% std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >
| + 99.70% __gthread_cond_timedwait
| + 99.70% pthread_cond_timedwait@@GLIBC_2.3.2
+ 0.30% OSD::heartbeat
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment