Skip to content

Instantly share code, notes, and snippets.

@Samrose-Ahmed
Created July 11, 2024 20:31
Show Gist options
  • Save Samrose-Ahmed/860f8deb8f5e9f1d6b50106cae0b01ec to your computer and use it in GitHub Desktop.
Save Samrose-Ahmed/860f8deb8f5e9f1d6b50106cae0b01ec to your computer and use it in GitHub Desktop.
Query:
Summary:
- Query ID: 5e2bb5da-3fc3-11ef-abdf-0edc7e62e9bc
- Start Time: 2024-07-11 13:23:00
- End Time: 2024-07-11 13:23:05
- Total: 5s588ms
- Query Type: Query
- Query State: Finished
- StarRocks Version: 3.3.0-19a3f66
- User: root
- Default Db
- Sql Statement: explain analyze select 1 from ice2.matano.aws_cloudtrail where event.id = '1fb803b2-3100-48da-8d30-68d74e8e42e0'
- Variables: parallel_fragment_exec_instance_num=1,max_parallel_scan_instance_num=-1,pipeline_dop=0,enable_adaptive_sink_dop=true,enable_runtime_adaptive_dop=false,runtime_profile_report_interval=10
- NonDefaultSessionVariables: {"io_tasks_per_scan_operator":{"defaultValue":4,"actualValue":640},"connector_io_tasks_per_scan_operator":{"defaultValue":16,"actualValue":640},"default_table_compression":{"defaultValue":"lz4_frame","actualValue":"zstd"},"enable_populate_datacache":{"defaultValue":true,"actualValue":false},"enable_datacache_io_adaptor":{"defaultValue":true,"actualValue":false},"new_planner_optimize_timeout":{"defaultValue":3000,"actualValue":20000},"enable_adaptive_sink_dop":{"defaultValue":false,"actualValue":true},"enable_connector_adaptive_io_tasks":{"defaultValue":true,"actualValue":false},"connector_sink_compression_codec":{"defaultValue":"uncompressed","actualValue":"zstd"},"enable_profile":{"defaultValue":false,"actualValue":true},"enable_async_profile":{"defaultValue":true,"actualValue":false}}
- Collect Profile Time: 65ms
- IsProfileAsync: false
Planner:
- -- Total[1] 72ms
- -- Analyzer[1] 1ms
- -- AnalyzeDatabase[1] 0
- -- AnalyzeTemporaryTable[1] 0
- -- AnalyzeTable[1] 0
- -- Transformer[1] 0
- -- Optimizer[1] 42ms
- -- preprocessMvs[1] 0
- -- chooseCandidates[1] 0
- -- generateMvPlan[1] 0
- -- validateMv[1] 0
- -- mvWithView[1] 0
- -- RuleBaseOptimize[1] 25ms
- -- ICEBERG.processSplit.IcebergFilter{databaseName='matano', tableName='aws_cloudtrail', snapshotId=3058437762871481886, predicate=Subfield([7: event], "id") = 1fb803b2-3100-48da-8d30-68d74e8e42e0}[1] 15ms
- -- ICEBERG.calculateCardinalityIcebergFilter{databaseName='matano', tableName='aws_cloudtrail', snapshotId=3058437762871481886, predicate=Subfield([7: event], "id") = 1fb803b2-3100-48da-8d30-68d74e8e42e0}[2] 1ms
- -- CostBaseOptimize[1] 14ms
- -- PhysicalRewrite[1] 0
- -- PlanValidate[1] 0
- -- InputDependenciesChecker[1] 0
- -- TypeChecker[1] 0
- -- CTEUniqueChecker[1] 0
- -- ColumnReuseChecker[1] 0
- -- ExecPlanBuild[1] 28ms
- -- Pending[1] 0
- -- Prepare[1] 11ms
- -- Deploy[1] 36ms
- -- DeployLockInternalTime[1] 36ms
- -- DeploySerializeConcurrencyTime[2] 1ms
- -- DeployStageByStageTime[6] 1ms
- -- DeployWaitTime[6] 32ms
- -- DeployAsyncSendTime[2] 1ms
- aws_cloudtrail rebalance_bytes: kube-starrocks-cn-0.kube-starrocks-cn-search.default.svc.cluster.local:0,
- aws_cloudtrail scan_range_bytes: kube-starrocks-cn-0.kube-starrocks-cn-search.default.svc.cluster.local:2564375458,
- DeployDataSize: 130488
ICEBERG:
ScanMetrics:
ice2:
matano:
aws_cloudtrail[ref(name="event:
- id") == "1fb803b2-3100-48da-8d30-68d74e8e42e0"]: ScanMetricsResult{totalPlanningDuration=TimerResult{timeUnit=NANOSECONDS, totalDuration=PT0.010917997S, count=1}, resultDataFiles=CounterResult{unit=COUNT, value=604}, resultDeleteFiles=CounterResult{unit=COUNT, value=0}, totalDataManifests=CounterResult{unit=COUNT, value=1}, totalDeleteManifests=CounterResult{unit=COUNT, value=0}, scannedDataManifests=CounterResult{unit=COUNT, value=1}, skippedDataManifests=CounterResult{unit=COUNT, value=0}, totalFileSizeInBytes=CounterResult{unit=BYTES, value=2564375458}, totalDeleteFileSizeInBytes=CounterResult{unit=BYTES, value=0}, skippedDataFiles=CounterResult{unit=COUNT, value=0}, skippedDeleteFiles=CounterResult{unit=COUNT, value=0}, scannedDeleteManifests=CounterResult{unit=COUNT, value=0}, skippedDeleteManifests=CounterResult{unit=COUNT, value=0}, indexedDeleteFiles=CounterResult{unit=COUNT, value=0}, equalityDeleteFiles=CounterResult{unit=COUNT, value=0}, positionalDeleteFiles=CounterResult{unit=COUNT, value=0}}
Reason:
Execution:
- Topology: {"rootId":2,"nodes":[{"id":2,"name":"EXCHANGE","properties":{"sinkIds":[],"displayMem":true},"children":[1]},{"id":1,"name":"PROJECT","properties":{"sinkIds":[2],"displayMem":false},"children":[0]},{"id":0,"name":"ICEBERG_SCAN","properties":{"displayMem":false},"children":[]}]}
- FrontendProfileMergeTime: 4.089ms
- QueryAllocatedMemoryUsage: 7.741 GB
- QueryCumulativeCpuTime: 5s464ms
- QueryCumulativeNetworkTime: 173.153us
- QueryCumulativeOperatorTime: 2s233ms
- QueryCumulativeScanTime: 1s385ms
- QueryDeallocatedMemoryUsage: 7.727 GB
- QueryExecutionWallTime: 5s492ms
- QueryPeakMemoryUsage: 110.577 MB
- QueryPeakScheduleTime: 34.954ms
- QuerySpillBytes: 0.000 B
- ResultDeliverTime: 0ns
Fragment 0:
- BackendAddresses: kube-starrocks-cn-0.kube-starrocks-cn-search.default.svc.cluster.local:9060
- InstanceIds: 5e2bb5da-3fc3-11ef-abdf-0edc7e62e9bd
- BackendNum: 1
- BackendProfileMergeTime: 358.384us
- FragmentInstancePrepareTime: 257.789us
- prepare-fragment-ctx: 1.141us
- prepare-pipeline-driver: 76.365us
- prepare-pipeline-driver-factory: 53.260us
- prepare-query-ctx: 5.793us
- prepare-runtime-state: 118.573us
- InitialProcessDriverCount: 0
- InitialProcessMem: 425.460 MB
- InstanceAllocatedMemoryUsage: 29.289 KB
- InstanceDeallocatedMemoryUsage: 1.492 KB
- InstanceNum: 1
- InstancePeakMemoryUsage: 27.797 KB
- QueryMemoryLimit: -1.000 B
Pipeline (id=0):
- isGroupExecution: false
- ActiveTime: 9.313us
- BlockByInputEmpty: 0
- BlockByOutputFull: 0
- BlockByPrecondition: 0
- DegreeOfParallelism: 1
- DriverPrepareTime: 74.191us
- DriverTotalTime: 5s428ms
- OverheadTime: 0ns
- PeakDriverQueueSize: 0
- PendingTime: 5s428ms
- InputEmptyTime: 5s428ms
- FirstInputEmptyTime: 5s428ms
- FollowupInputEmptyTime: 0ns
- OutputFullTime: 0ns
- PendingFinishTime: 0ns
- PreconditionBlockTime: 0ns
- ScheduleCount: 1
- ScheduleTime: 90.052us
- TotalDegreeOfParallelism: 1
- YieldByLocalWait: 0
- YieldByPreempt: 0
- YieldByTimeLimit: 0
RESULT_SINK (plan_node_id=-1):
CommonMetrics:
- IsFinalSink
- CloseTime: 52.743us
- OperatorAllocatedMemoryUsage: 0.000 B
- OperatorDeallocatedMemoryUsage: 0.000 B
- OperatorPeakMemoryUsage: 0.000 B
- OperatorTotalTime: 52.965us
- PrepareTime: 11.217us
- PullChunkNum: 0
- PullRowNum: 0
- PullTotalTime: 0ns
- PushChunkNum: 0
- PushRowNum: 0
- PushTotalTime: 0ns
- SetFinishedTime: 41ns
- SetFinishingTime: 181ns
UniqueMetrics:
result sink:
- AppendChunkTime: 0ns
- ResultRendTime: 0ns
- TupleConvertTime: 0ns
- NumSentRows: 0
CHUNK_ACCUMULATE (plan_node_id=-1):
CommonMetrics:
- IsSubordinate
- CloseTime: 271ns
- OperatorTotalTime: 444ns
- PrepareTime: 8.123us
- PullChunkNum: 0
- PullRowNum: 0
- PullTotalTime: 0ns
- PushChunkNum: 0
- PushRowNum: 0
- PushTotalTime: 0ns
- SetFinishedTime: 74ns
- SetFinishingTime: 99ns
UniqueMetrics:
EXCHANGE_SOURCE (plan_node_id=2):
CommonMetrics:
- CloseTime: 10.322us
- OperatorAllocatedMemoryUsage: 1.547 KB
- OperatorDeallocatedMemoryUsage: 800.000 B
- OperatorPeakMemoryUsage: 896.000 B
- OperatorTotalTime: 14.104us
- PrepareTime: 29.637us
- PullChunkNum: 0
- PullRowNum: 0
- PullTotalTime: 0ns
- PushChunkNum: 0
- PushRowNum: 0
- PushTotalTime: 0ns
- RuntimeBloomFilterNum: 0
- RuntimeInFilterNum: 0
- SetFinishedTime: 57ns
- SetFinishingTime: 3.725us
UniqueMetrics:
- BufferUnplugCount: 0
- BytesPassThrough: 0.000 B
- BytesReceived: 0.000 B
- ClosureBlockCount: 0
- ClosureBlockTime: 0ns
- DecompressChunkTime: 0ns
- DeserializeChunkTime: 0ns
- PeakBufferMemoryBytes: 0.000 B
- ReceiverProcessTotalTime: 0ns
- RequestReceived: 0
- WaitLockTime: 0ns
Fragment 1:
- BackendAddresses: kube-starrocks-cn-0.kube-starrocks-cn-search.default.svc.cluster.local:9060
- InstanceIds: 5e2bb5da-3fc3-11ef-abdf-0edc7e62e9be
- BackendNum: 1
- BackendProfileMergeTime: 876.876us
- FragmentInstancePrepareTime: 1.223ms
- prepare-fragment-ctx: 1.174us
- prepare-pipeline-driver: 116.373us
- prepare-pipeline-driver-factory: 255.024us
- prepare-query-ctx: 3.741us
- prepare-runtime-state: 844.728us
- InitialProcessDriverCount: 1
- InitialProcessMem: 425.616 MB
- InstanceAllocatedMemoryUsage: 7.741 GB
- InstanceDeallocatedMemoryUsage: 7.727 GB
- InstanceNum: 1
- InstancePeakMemoryUsage: 110.494 MB
- QueryMemoryLimit: -1.000 B
Pipeline (id=0):
- isGroupExecution: false
- ActiveTime: 795.800ms
- BlockByInputEmpty: 409
- BlockByOutputFull: 0
- BlockByPrecondition: 0
- DegreeOfParallelism: 1
- DriverPrepareTime: 114.356us
- DriverTotalTime: 5s424ms
- OverheadTime: 0ns
- PeakDriverQueueSize: 1
- PendingTime: 4s593ms
- InputEmptyTime: 4s594ms
- FirstInputEmptyTime: 464.339ms
- FollowupInputEmptyTime: 4s130ms
- OutputFullTime: 0ns
- PendingFinishTime: 116.218us
- PreconditionBlockTime: 0ns
- ScheduleCount: 410
- ScheduleTime: 34.954ms
- TotalDegreeOfParallelism: 1
- YieldByLocalWait: 0
- YieldByPreempt: 0
- YieldByTimeLimit: 0
EXCHANGE_SINK (plan_node_id=2):
CommonMetrics:
- CloseTime: 14.613us
- OperatorAllocatedMemoryUsage: 3.188 KB
- OperatorDeallocatedMemoryUsage: 960.000 B
- OperatorPeakMemoryUsage: 2.250 KB
- OperatorTotalTime: 80.016us
- PrepareTime: 59.701us
- PullChunkNum: 0
- PullRowNum: 0
- PullTotalTime: 0ns
- PushChunkNum: 0
- PushRowNum: 0
- PushTotalTime: 0ns
- SetFinishedTime: 41ns
- SetFinishingTime: 65.362us
UniqueMetrics:
- ChannelNum: 1
- DestFragments: 5e2bb5da3fc311ef-abdf0edc7e62e9bd
- DestID: 2
- PartType: UNPARTITIONED
- BytesPassThrough: 0.000 B
- BytesSent: 0.000 B
- BytesUnsent: 0.000 B
- CompressTime: 0ns
- CompressedBytes: 0.000 B
- NetworkBandwidth: 0.000 B/sec
- NetworkTime: 173.153us
- OverallThroughput: 0.000 B/sec
- OverallTime: 176.887us
- PassThroughBufferPeakMemoryUsage: 0.000 B
- RequestSent: 0
- RequestUnsent: 0
- RpcAvgTime: 173.153us
- RpcCount: 1
- SenderInputBytes: 0.000 B
- SerializeChunkTime: 0ns
- SerializedBytes: 0.000 B
- ShuffleChunkAppendCounter: 0
- ShuffleChunkAppendTime: 0ns
- ShuffleHashTime: 0ns
- WaitTime: 62.822ms
PROJECT (plan_node_id=1):
CommonMetrics:
- CloseTime: 6.203us
- OperatorAllocatedMemoryUsage: 0.000 B
- OperatorDeallocatedMemoryUsage: 4.719 KB
- OperatorPeakMemoryUsage: 0.000 B
- OperatorTotalTime: 292.145us
- PrepareTime: 6.934us
- PullChunkNum: 0
- PullRowNum: 0
- PullTotalTime: 95.001us
- PushChunkNum: 0
- PushRowNum: 0
- PushTotalTime: 190.736us
- RuntimeBloomFilterNum: 0
- RuntimeInFilterNum: 0
- SetFinishedTime: 82ns
- SetFinishingTime: 123ns
UniqueMetrics:
- CommonSubExprComputeTime: 0ns
- ExprComputeTime: 0ns
CHUNK_ACCUMULATE (plan_node_id=0):
CommonMetrics:
- IsSubordinate
- CloseTime: 295ns
- OperatorTotalTime: 385.495us
- PrepareTime: 6.671us
- PullChunkNum: 0
- PullRowNum: 0
- PullTotalTime: 125.690us
- PushChunkNum: 0
- PushRowNum: 0
- PushTotalTime: 259.263us
- SetFinishedTime: 99ns
- SetFinishingTime: 148ns
UniqueMetrics:
CONNECTOR_SCAN (plan_node_id=0):
CommonMetrics:
- CloseTime: 61.742ms
- OperatorAllocatedMemoryUsage: 7.739 GB
- OperatorDeallocatedMemoryUsage: 7.727 GB
- OperatorPeakMemoryUsage: 108.364 MB
- OperatorTotalTime: 847.356ms
- PrepareTime: 19.865us
- PullChunkNum: 0
- PullRowNum: 0
- PullTotalTime: 785.561ms
- PushChunkNum: 0
- PushRowNum: 0
- PushTotalTime: 0ns
- RuntimeBloomFilterNum: 0
- RuntimeInFilterNum: 0
- SetFinishedTime: 50ns
- SetFinishingTime: 53.006us
UniqueMetrics:
- AdaptiveIOTasks: False
- DataSourceType: HiveDataSource
- MorselQueueType: dynamic_morsel_queue
- Predicates: 7: event.id[true] = '1fb803b2-3100-48da-8d30-68d74e8e42e0'
- SharedScan: False
- Table: aws_cloudtrail
- ChunkBufferCapacity: 64
- DefaultChunkBufferCapacity: 64
- IOTaskExecTime: 138.949ms
- __MAX_OF_IOTaskExecTime: 566.595ms
- __MIN_OF_IOTaskExecTime: 66.029ms
- ColumnConvertTime: 0ns
- ColumnReadTime: 133.401ms
- __MAX_OF_ColumnReadTime: 565.821ms
- __MIN_OF_ColumnReadTime: 0ns
- DataCache:
- DataCacheReadBlockBufferBytes: 0.000 B
- DataCacheReadBlockBufferCounter: 0
- DataCacheReadBytes: 0.000 B
- DataCacheReadDiskBytes: 0.000 B
- DataCacheReadMemBytes: 0.000 B
- DataCacheReadCounter: 0
- DataCacheReadTimer: 0ns
- DataCacheSkipReadBytes: 0.000 B
- DataCacheSkipReadCounter: 0
- DataCacheWriteBytes: 0.000 B
- DataCacheWriteCounter: 0
- DataCacheWriteFailBytes: 0.000 B
- DataCacheWriteFailCounter: 0
- DataCacheWriteTimer: 0ns
- ExprFilterTime: 781.990us
- __MAX_OF_ExprFilterTime: 23.699ms
- __MIN_OF_ExprFilterTime: 0ns
- InputStream:
- AppIOBytesRead: 264.090 MB
- __MAX_OF_AppIOBytesRead: 720.661 KB
- __MIN_OF_AppIOBytesRead: 177.000 B
- AppIOCounter: 604
- __MAX_OF_AppIOCounter: 1
- __MIN_OF_AppIOCounter: 1
- AppIOTime: 127.434ms
- __MAX_OF_AppIOTime: 563.304ms
- __MIN_OF_AppIOTime: 65.774ms
- FSIOBytesRead: 411.853 MB
- __MAX_OF_FSIOBytesRead: 1.000 MB
- __MIN_OF_FSIOBytesRead: 67.873 KB
- FSIOCounter: 604
- __MAX_OF_FSIOCounter: 1
- __MIN_OF_FSIOCounter: 1
- FSIOTime: 127.400ms
- __MAX_OF_FSIOTime: 563.293ms
- __MIN_OF_FSIOTime: 65.768ms
- LateMaterializeSkipRows: 13.197M (13197350)
- __MAX_OF_LateMaterializeSkipRows: 36.940K (36940)
- __MIN_OF_LateMaterializeSkipRows: 0
- MemAllocFailed: 0
- OpenFile: 637.374us
- __MAX_OF_OpenFile: 108.997ms
- __MIN_OF_OpenFile: 64.984us
- Parquet:
- FooterCacheReadCount: 604
- __MAX_OF_FooterCacheReadCount: 1
- __MIN_OF_FooterCacheReadCount: 1
- FooterCacheReadTimer: 11.844us
- __MAX_OF_FooterCacheReadTimer: 569.552us
- __MIN_OF_FooterCacheReadTimer: 5.153us
- FooterCacheWriteBytes: 0.000 B
- FooterCacheWriteCount: 0
- FooterCacheWriteFailCount: 0
- GroupActiveLazyColumnIOCoalesceSeperately: 588
- __MAX_OF_GroupActiveLazyColumnIOCoalesceSeperately: 1
- __MIN_OF_GroupActiveLazyColumnIOCoalesceSeperately: 0
- GroupActiveLazyColumnIOCoalesceTogether: 16
- __MAX_OF_GroupActiveLazyColumnIOCoalesceTogether: 1
- __MIN_OF_GroupActiveLazyColumnIOCoalesceTogether: 0
- GroupChunkRead: 134.523ms
- __MAX_OF_GroupChunkRead: 566.258ms
- __MIN_OF_GroupChunkRead: 164ns
- GroupDictDecode: 0ns
- GroupDictFilter: 0ns
- GroupMinRound: 1
- HasPageStatistics: 6
- __MAX_OF_HasPageStatistics: 1
- __MIN_OF_HasPageStatistics: 0
- IcebergV2FormatTimer:
- DeleteFileBuildFilterTime: 0ns
- DeleteFileBuildTime: 0ns
- DeleteFilesPerScan: 0
- LevelDecodeTime: 3.005us
- __MAX_OF_LevelDecodeTime: 192.884us
- __MIN_OF_LevelDecodeTime: 0ns
- PageIndexTime: 539ns
- __MAX_OF_PageIndexTime: 32.673us
- __MIN_OF_PageIndexTime: 246ns
- PageReadTime: 132.372ms
- __MAX_OF_PageReadTime: 565.308ms
- __MIN_OF_PageReadTime: 65.798ms
- PageSkipCounter: 0
- ReaderInitColumnReaderInit: 65.495us
- __MAX_OF_ReaderInitColumnReaderInit: 31.162ms
- __MIN_OF_ReaderInitColumnReaderInit: 9.206us
- ReaderInitFooterRead: 0ns
- RequestBytesRead: 264.090 MB
- __MAX_OF_RequestBytesRead: 720.661 KB
- __MIN_OF_RequestBytesRead: 177.000 B
- RequestBytesReadUncompressed: 504.079 MB
- __MAX_OF_RequestBytesReadUncompressed: 1.351 MB
- __MIN_OF_RequestBytesReadUncompressed: 159.000 B
- RowsBeforePageIndex: 13.219M (13219153)
- __MAX_OF_RowsBeforePageIndex: 36.940K (36940)
- __MIN_OF_RowsBeforePageIndex: 1
- ValueDecodeTime: 1.322ms
- __MAX_OF_ValueDecodeTime: 34.119ms
- __MIN_OF_ValueDecodeTime: 0ns
- RawRowsRead: 13.197M (13197350)
- __MAX_OF_RawRowsRead: 36.940K (36940)
- __MIN_OF_RawRowsRead: 0
- ReaderInit: 502.211us
- __MAX_OF_ReaderInit: 108.950ms
- __MIN_OF_ReaderInit: 42.675us
- RowsRead: 0
- ScanRanges: 604
- __MAX_OF_ScanRanges: 1
- __MIN_OF_ScanRanges: 1
- ScanRangesSize: 2.388 GB
- __MAX_OF_ScanRangesSize: 5.503 MB
- __MIN_OF_ScanRangesSize: 67.873 KB
- SharedBuffered:
- DirectIOBytes: 0.000 B
- DirectIOCount: 0
- DirectIOTime: 0ns
- SharedAlignIOBytes: 147.727 MB
- __MAX_OF_SharedAlignIOBytes: 486.789 KB
- __MIN_OF_SharedAlignIOBytes: 25.125 KB
- SharedIOBytes: 411.853 MB
- __MAX_OF_SharedIOBytes: 1.000 MB
- __MIN_OF_SharedIOBytes: 67.873 KB
- SharedIOCount: 604
- __MAX_OF_SharedIOCount: 1
- __MIN_OF_SharedIOCount: 1
- SharedIOTime: 127.419ms
- __MAX_OF_SharedIOTime: 563.297ms
- __MIN_OF_SharedIOTime: 65.770ms
- IOTaskWaitTime: 399.670ms
- __MAX_OF_IOTaskWaitTime: 1s99ms
- __MIN_OF_IOTaskWaitTime: 2.422ms
- MorselsCount: 604
- PeakChunkBufferMemoryUsage: 0.000 B
- PeakChunkBufferSize: 64
- PeakIOTasks: 65
- PeakScanTaskQueueSize: 63
- PrepareChunkSourceTime: 2.881ms
- ScanTime: 538.619ms
- __MAX_OF_ScanTime: 1s385ms
- __MIN_OF_ScanTime: 99.532ms
- SubmitTaskCount: 604
- SubmitTaskTime: 981.336us
- TabletCount: 604
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment