clickhouse crash 2

2020.11.10 16:28:04.174076 [ 48902 ] {} <Trace> graphite.data_rev: Renaming temporary part tmp_fetch_20201110_214607_214607_0 to 20201110_214607_214607_0.
2020.11.10 16:28:04.174333 [ 49049 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.021112294 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.174337 [ 34427 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.021028329 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.174381 [ 34423 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.021097418 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.174381 [ 4333 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.021125629 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.174670 [ 34432 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.021441782 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.175914 [ 49112 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregating
2020.11.10 16:28:04.175943 [ 49112 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> Aggregator: Aggregation method: key_string
2020.11.10 16:28:04.175959 [ 49112 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> Aggregator: Converting aggregation data to two-level.
2020.11.10 16:28:04.176519 [ 48902 ] {} <Debug> graphite.data_rev: Fetched part 20201110_214607_214607_0 from /clickhouse/tables/shared_replicator_common/graphite.data_rev/replicas/r6
2020.11.10 16:28:04.177364 [ 49050 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.024086511 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.177626 [ 67705 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.024409498 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.177987 [ 49112 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 2 to 2 rows (from 112.00 B) in 0.024646393 sec. (81.14777687753336 rows/sec., 4.44 KiB/sec.)
2020.11.10 16:28:04.179201 [ 67712 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.02599501 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.180970 [ 49124 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.027678241 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.180978 [ 49118 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.027679449 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.181371 [ 49016 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.028108153 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.181564 [ 34426 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.028313671 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.181825 [ 49079 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.028499646 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.182598 [ 48996 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.029338718 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.182850 [ 60960 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.029519734 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.183018 [ 34436 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.029795217 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.183107 [ 49052 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.029881933 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.183478 [ 60942 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.03020196 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.184248 [ 48952 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.030945316 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.184932 [ 49004 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.031712424 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.185189 [ 49069 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.031877772 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.185863 [ 24601 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.032559165 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.186023 [ 60981 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> AggregatingTransform: Aggregated. 0 to 0 rows (from 0.00 B) in 0.032756224 sec. (0.0 rows/sec., 0.00 B/sec.)
2020.11.10 16:28:04.186040 [ 60981 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Trace> Aggregator: Merging aggregated data
2020.11.10 16:28:04.190728 [ 52103 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Information> executeQuery: Read 11388681 rows, 976.87 MiB in 0.041289841 sec., 275822834 rows/sec., 23.10 GiB/sec.
2020.11.10 16:28:04.191333 [ 48921 ] {} <Debug> graphite.data_rev (ReplicatedMergeTreeQueue): Pulling 1 entries to queue: log-0123526161 - log-0123526161
2020.11.10 16:28:04.191604 [ 52103 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Information> DynamicQueryHandler: Done processing query
2020.11.10 16:28:04.191629 [ 52103 ] {759b7ef42d2013d01c2fb1d063027446::7e601074b7cab507} <Debug> MemoryTracker: Peak memory usage (for query): 167.47 MiB.
2020.11.10 16:28:04.191648 [ 52103 ] {} <Debug> MemoryTracker: Peak memory usage (for query): 167.47 MiB.
2020.11.10 16:28:04.192315 [ 60964 ] {} <Trace> HTTPHandler-factory: HTTP Request for HTTPHandler-factory. Method: POST, Address: 127.0.0.1:35872, User-Agent: graphite-clickhouse/0.8.26 (table:graphite.data_rev), Length: 318, Content Type: , Transfer Encoding: identity
2020.11.10 16:28:04.192442 [ 60964 ] {} <Trace> DynamicQueryHandler: Request URI: /?max_execution_time=30&max_memory_usage=107374182400&max_query_size=2000000&query_id=759b7ef42d2013d01c2fb1d063027446%3A%3Aa98bbbaf3a8bd8db
2020.11.10 16:28:04.192812 [ 60964 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Debug> executeQuery: (from 127.0.0.1:35872)  SELECT Path, Time, Value, Timestamp FROM graphite.data_rev PREWHERE ((Date >='2020-11-10' AND Date <= '2020-11-10')) WHERE ((Path in ('speed.eno4.ifstat.collectd.avi-ix-kub29.servers','speed.eno3.ifstat.collectd.avi-ix-kub29.servers')) AND (Time >= 1605014704 AND Time <= 1605014909)) FORMAT RowBinary
2020.11.10 16:28:04.192934 [ 48921 ] {} <Debug> graphite.data_rev (ReplicatedMergeTreeQueue): Pulled 1 entries to queue.
2020.11.10 16:28:04.194003 [ 60964 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Trace> ContextAccess (default): Access granted: SELECT(Path, Value, Time, Date, Timestamp) ON graphite.data_rev
2020.11.10 16:28:04.194332 [ 60964 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Debug> graphite.data_rev (SelectExecutor): Key condition: unknown, unknown, and, (column 0 in 2-element set), (column 1 in [1605014704, +inf)), (column 1 in (-inf, 1605014909]), and, and, and, unknown, unknown, and, and
2020.11.10 16:28:04.194355 [ 60964 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Debug> graphite.data_rev (SelectExecutor): MinMax index condition: (column 0 in [18576, +inf)), (column 0 in (-inf, 18576]), and, unknown, unknown, unknown, and, and, and, (column 0 in [18576, +inf)), (column 0 in (-inf, 18576]), and, and
2020.11.10 16:28:04.194686 [ 67733 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Trace> graphite.data_rev (SelectExecutor): Used generic exclusion search over index for part 20201110_214544_214578_2 with 27 steps
2020.11.10 16:28:04.194690 [ 15730 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Trace> graphite.data_rev (SelectExecutor): Used generic exclusion search over index for part 20201110_214605_214605_0 with 1 steps
2020.11.10 16:28:04.194708 [ 49087 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Trace> graphite.data_rev (SelectExecutor): Used generic exclusion search over index for part 20201110_214365_214543_4 with 32 steps
2020.11.10 16:28:04.194735 [ 67707 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Trace> graphite.data_rev (SelectExecutor): Used generic exclusion search over index for part 20201110_214579_214589_1 with 20 steps
2020.11.10 16:28:04.194750 [ 49029 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Trace> graphite.data_rev (SelectExecutor): Used generic exclusion search over index for part 20201110_214294_214364_3 with 30 steps
2020.11.10 16:28:04.194760 [ 48920 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Trace> graphite.data_rev (SelectExecutor): Used generic exclusion search over index for part 20201110_214607_214607_0 with 16 steps
2020.11.10 16:28:04.194712 [ 48995 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Trace> graphite.data_rev (SelectExecutor): Used generic exclusion search over index for part 20201110_213963_214293_5 with 34 steps
2020.11.10 16:28:04.194800 [ 34429 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Trace> graphite.data_rev (SelectExecutor): Used generic exclusion search over index for part 20201110_214598_214603_1 with 18 steps
2020.11.10 16:28:04.194750 [ 67710 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Trace> graphite.data_rev (SelectExecutor): Used generic exclusion search over index for part 20201110_214606_214606_0 with 13 steps
2020.11.10 16:28:04.194792 [ 48893 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Trace> graphite.data_rev (SelectExecutor): Used generic exclusion search over index for part 20201110_214604_214604_0 with 14 steps
2020.11.10 16:28:04.194840 [ 49081 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Trace> graphite.data_rev (SelectExecutor): Used generic exclusion search over index for part 20201110_0_184503_330 with 76 steps
2020.11.10 16:28:04.194805 [ 49098 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Trace> graphite.data_rev (SelectExecutor): Used generic exclusion search over index for part 20201110_214590_214597_1 with 18 steps
2020.11.10 16:28:04.194929 [ 49113 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Trace> graphite.data_rev (SelectExecutor): Used generic exclusion search over index for part 20201110_195180_205003_8 with 53 steps
2020.11.10 16:28:04.194765 [ 15717 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Trace> graphite.data_rev (SelectExecutor): Used generic exclusion search over index for part 20201110_205004_208699_7 with 44 steps
2020.11.10 16:28:04.194805 [ 48933 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Trace> graphite.data_rev (SelectExecutor): Used generic exclusion search over index for part 20201110_184504_195179_9 with 55 steps
2020.11.10 16:28:04.194863 [ 49025 ] {759b7ef42d2013d01c2fb1d063027446::a98bbbaf3a8bd8db} <Trace> graphite.data_rev (SelectExecutor): Used generic exclusion search over index for part 20201110_208700_212645_7 with 48 steps
2020.11.10 16:28:35.979741 [ 62818 ] {} <Information> SentryWriter: Sending crash reports is disabled
2020.11.10 16:28:35.982206 [ 62818 ] {} <Trace> Pipe: Pipe capacity is 1.00 MiB
2020.11.10 16:28:36.020064 [ 62818 ] {} <Information> : Starting ClickHouse 20.10.3.30 with revision 54441, build id: 4EAF0337F53BC2B4, PID 62818
2020.11.10 16:28:36.020298 [ 62818 ] {} <Information> Application: starting up
2020.11.10 16:28:36.025385 [ 62818 ] {} <Information> StatusFile: Status file /var/lib/clickhouse/status already exists - unclean restart. Contents:




PID: 48858
Started at: 2020-11-10 14:36:36
Revision: 54441

2020.11.10 16:28:36.025450 [ 62818 ] {} <Debug> Application: rlimit on number of file descriptors is 500000
2020.11.10 16:28:36.025464 [ 62818 ] {} <Debug> Application: Initializing DateLUT.
2020.11.10 16:28:36.025478 [ 62818 ] {} <Trace> Application: Initialized DateLUT with time zone 'Europe/Moscow'.
2020.11.10 16:28:36.025493 [ 62818 ] {} <Debug> Application: Setting up /var/lib/clickhouse/tmp/ to store temporary data in it
2020.11.10 16:28:36.025525 [ 62818 ] {} <Debug> Application: Skipped file in temporary path /var/lib/clickhouse/tmp/data
2020.11.10 16:28:36.025566 [ 62818 ] {} <Debug> Application: Configuration parameter 'interserver_http_host' doesn't exist or exists and empty. Will use 'graphite-clickhouse08.msk.avito.ru' as replica host.
2020.11.10 16:28:36.026360 [ 62818 ] {} <Debug> ConfigReloader: Loading config '/etc/clickhouse-server/users.xml'
2020.11.10 16:28:36.026878 [ 62818 ] {} <Debug> ConfigReloader: Loaded config '/etc/clickhouse-server/users.xml', performing update on configuration
2020.11.10 16:28:36.027428 [ 62818 ] {} <Debug> ConfigReloader: Loaded config '/etc/clickhouse-server/users.xml', performed update on configuration
2020.11.10 16:28:36.028018 [ 62818 ] {} <Information> Application: Setting max_server_memory_usage was set to 338.89 GiB (376.55 GiB available * 0.90 max_server_memory_usage_to_ram_ratio)
2020.11.10 16:28:36.028033 [ 62818 ] {} <Information> Application: Loading metadata from /var/lib/clickhouse/
2020.11.10 16:28:36.029380 [ 62818 ] {} <Information> DatabaseOrdinary (system): Total 4 tables and 0 dictionaries.
2020.11.10 16:28:36.032760 [ 62825 ] {} <Information> BackgroundProcessingPool: Create BackgroundProcessingPool with 16 threads
2020.11.10 16:28:36.033515 [ 62825 ] {} <Debug> system.query_log_1: Loading data parts
2020.11.10 16:28:36.033521 [ 62822 ] {} <Debug> system.query_log: Loading data parts
2020.11.10 16:28:36.033524 [ 62824 ] {} <Debug> system.query_log_0: Loading data parts
2020.11.10 16:28:36.033531 [ 62826 ] {} <Debug> system.query_thread_log: Loading data parts
2020.11.10 16:28:36.034147 [ 62824 ] {} <Debug> system.query_log_0: Loaded data parts (0 items)
2020.11.10 16:28:36.036335 [ 62826 ] {} <Debug> system.query_thread_log: Loaded data parts (0 items)
2020.11.10 16:28:36.072848 [ 62822 ] {} <Debug> system.query_log: Loaded data parts (82 items)
2020.11.10 16:28:36.073144 [ 62825 ] {} <Debug> system.query_log_1: Loaded data parts (10 items)
2020.11.10 16:28:36.073325 [ 62818 ] {} <Information> DatabaseOrdinary (system): Starting up tables.
2020.11.10 16:28:36.073375 [ 62824 ] {} <Trace> system.query_log: Found 8 old parts to remove.
2020.11.10 16:28:36.073397 [ 62824 ] {} <Debug> system.query_log: Removing part from filesystem 202011_2683_2717_3
2020.11.10 16:28:36.077886 [ 62824 ] {} <Debug> system.query_log: Removing part from filesystem 202011_2718_2746_3
2020.11.10 16:28:36.081538 [ 62824 ] {} <Debug> system.query_log: Removing part from filesystem 202011_2747_2763_2
2020.11.10 16:28:36.084598 [ 62824 ] {} <Debug> system.query_log: Removing part from filesystem 202011_2764_2764_0
2020.11.10 16:28:36.084875 [ 62824 ] {} <Debug> system.query_log: Removing part from filesystem 202011_2765_2765_0
2020.11.10 16:28:36.085225 [ 62824 ] {} <Debug> system.query_log: Removing part from filesystem 202011_2766_2766_0