Skip to content

Instantly share code, notes, and snippets.

@tyriis

tyriis/found-the-issue

Created August 16, 2025 14:37
Show Gist options
  • Save tyriis/8b44a4cd705303ead62d6a14c0f5dd09 to your computer and use it in GitHub Desktop.
Save tyriis/8b44a4cd705303ead62d6a14c0f5dd09 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
found-the-issue
1 NIC is running with 100Mbits only :(
```
Connecting to host 192.168.100.101, port 5201
[ 5] local 192.168.100.103 port 42392 connected to 192.168.100.101 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 11.9 MBytes 99.5 Mbits/sec 0 42.4 KBytes
[ 5] 1.00-2.00 sec 11.1 MBytes 93.3 Mbits/sec 0 45.2 KBytes
[ 5] 2.00-3.00 sec 11.2 MBytes 94.4 Mbits/sec 0 45.2 KBytes
[ 5] 3.00-4.00 sec 11.1 MBytes 93.3 Mbits/sec 0 48.1 KBytes
[ 5] 4.00-5.00 sec 10.6 MBytes 89.1 Mbits/sec 0 45.2 KBytes
[ 5] 5.00-6.00 sec 10.6 MBytes 89.1 Mbits/sec 0 41.0 KBytes
[ 5] 6.00-7.00 sec 11.1 MBytes 93.3 Mbits/sec 0 50.9 KBytes
[ 5] 7.00-8.00 sec 11.2 MBytes 94.4 Mbits/sec 0 50.9 KBytes
[ 5] 8.00-9.00 sec 11.1 MBytes 93.3 Mbits/sec 0 45.2 KBytes
[ 5] 9.00-10.00 sec 11.2 MBytes 94.4 Mbits/sec 0 5.66 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 111 MBytes 93.4 Mbits/sec 0 sender
[ 5] 0.00-10.01 sec 110 MBytes 92.0 Mbits/sec receiver
iperf Done.
```
this was the reason for low rook-ceph performance:
after fix
```
bash-5.1$ rados bench -p ceph-blockpool 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_talos01_1168
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 2 2 0 0 0 - 0
1 16 66 50 199.967 200 0.375406 0.201107
2 16 103 87 173.977 148 0.0129707 0.310708
3 16 144 128 170.642 164 0.0156619 0.337213
4 16 187 171 170.975 172 0.715322 0.33933
5 16 222 206 164.778 140 0.929181 0.351855
6 16 266 250 166.645 176 0.0181203 0.359934
7 16 307 291 166.263 164 0.508394 0.36839
8 16 352 336 167.975 180 0.394201 0.366298
9 16 393 377 167.531 164 0.512105 0.369183
10 16 431 415 165.975 152 0.435443 0.372946
Total time run: 10.6541
Total writes made: 431
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 161.815
Stddev Bandwidth: 17.3077
Max bandwidth (MB/sec): 200
Min bandwidth (MB/sec): 140
Average IOPS: 40
Stddev IOPS: 4.32692
Max IOPS: 50
Min IOPS: 35
Average Latency(s): 0.385355
Stddev Latency(s): 0.315917
Max latency(s): 1.03777
Min latency(s): 0.0125966
```
@tyriis
Copy link
Author

tyriis commented Aug 16, 2025

mixed public network thunderbolt + 10GB NIC

bash-5.1$ rados bench -p ceph-blockpool 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_talos01_141
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0      16        16         0         0         0           -           0
    1      16       307       291   1163.73      1164    0.032906   0.0473561
    2      16       576       560   1119.75      1076   0.0136398   0.0535368
    3      16       791       775   1033.14       860   0.0345474   0.0617631
    4      16      1128      1112   1111.81      1348   0.0138425   0.0563346
    5      16      1530      1514   1211.01      1608   0.0197638   0.0526762
    6      16      1764      1748   1165.16       936   0.0167593   0.0546279
    7      16      2136      2120   1211.25      1488    0.228938    0.051471
    8      16      2510      2494   1246.83      1496   0.0162949   0.0512529
    9      16      2830      2814   1250.49      1280   0.0116342   0.0506672
   10      16      3203      3187   1274.61      1492   0.0170389   0.0498197
Total time run:         10.1559
Total writes made:      3203
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     1261.53
Stddev Bandwidth:       257.468
Max bandwidth (MB/sec): 1608
Min bandwidth (MB/sec): 860
Average IOPS:           315
Stddev IOPS:            64.3671
Max IOPS:               402
Min IOPS:               215
Average Latency(s):     0.0506837
Stddev Latency(s):      0.0632691
Max latency(s):         0.261037
Min latency(s):         0.0108483

@tyriis
Copy link
Author

tyriis commented Aug 16, 2025 •
edited
Loading

rook-ceph with 10G NIC node01

TEST_FILE: /volume/test                                                                                                                                                                                                                                     
TEST_OUTPUT_PREFIX: test_device                                                                                                                                                                                                                             
TEST_SIZE: 30G                                                                                                                                                                                                                                              
Benchmarking iops.fio into test_device-iops.json                                                                                                                                                                                                            
Benchmarking bandwidth.fio into test_device-bandwidth.json                                                                                                                                                                                                  
Benchmarking latency.fio into test_device-latency.json                                                                                                                                                                                                      
                                                                                                                                                                                                                                                            
=========================                                                                                                                                                                                                                                   
FIO Benchmark Summary                                                                                                                                                                                                                                       
For: test_device                                                                                                                                                                                                                                            
CPU Idleness Profiling: disabled                                                                                                                                                                                                                            
Size: 30G                                                                                                                                                                                                                                                   
Quick Mode: disabled                                                                                                                                                                                                                                        
=========================                                                                                                                                                                                                                                   
IOPS (Read/Write)                                                                                                                                                                                                                                           
        Random:           69,040 / 5,700                                                                                                                                                                                                                    
    Sequential:            7,825 / 6,705                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
Bandwidth in KiB/sec (Read/Write)                                                                                                                                                                                                                           
        Random:        988,501 / 564,709                                                                                                                                                                                                                    
    Sequential:        766,049 / 542,822                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                            
Latency in ns (Read/Write)                                                                                                                                                                                                                                  
        Random:      292,516 / 5,599,995                                                                                                                                                                                                                    
    Sequential:       62,989 / 4,561,601

rook-ceph with 10G NIC node02

TEST_FILE: /volume/test                                                                                                                                                                                                                                     
TEST_OUTPUT_PREFIX: test_device                                                                                                                                                                                                                             
TEST_SIZE: 30G                                                                                                                                                                                                                                              
Benchmarking iops.fio into test_device-iops.json                                                                                                                                                                                                            
Benchmarking bandwidth.fio into test_device-bandwidth.json                                                                                                                                                                                                  
Benchmarking latency.fio into test_device-latency.json                                                                                                                                                                                                      
                                                                                                                                                                                                                                                            
=========================                                                                                                                                                                                                                                   
FIO Benchmark Summary                                                                                                                                                                                                                                       
For: test_device                                                                                                                                                                                                                                            
CPU Idleness Profiling: disabled                                                                                                                                                                                                                            
Size: 30G                                                                                                                                                                                                                                                   
Quick Mode: disabled                                                                                                                                                                                                                                        
=========================                                                                                                                                                                                                                                   
IOPS (Read/Write)                                                                                                                                                                                                                                           
        Random:           70,899 / 5,680                                                                                                                                                                                                                    
    Sequential:           36,895 / 6,668                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
Bandwidth in KiB/sec (Read/Write)                                                                                                                                                                                                                           
        Random:        943,277 / 539,948                                                                                                                                                                                                                    
    Sequential:      1,014,213 / 574,627                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                            
Latency in ns (Read/Write)                                                                                                                                                                                                                                  
        Random:       98,570 / 5,580,540                                                                                                                                                                                                                    
    Sequential:       61,411 / 4,389,462

rook-ceph with 10G NIC node03

TEST_FILE: /volume/test                                                                                                                                                                                                                                     
TEST_OUTPUT_PREFIX: test_device                                                                                                                                                                                                                             
TEST_SIZE: 30G                                                                                                                                                                                                                                              
Benchmarking iops.fio into test_device-iops.json                                                                                                                                                                                                            
Benchmarking bandwidth.fio into test_device-bandwidth.json                                                                                                                                                                                                  
Benchmarking latency.fio into test_device-latency.json                                                                                                                                                                                                      
                                                                                                                                                                                                                                                            
=========================                                                                                                                                                                                                                                   
FIO Benchmark Summary                                                                                                                                                                                                                                       
For: test_device                                                                                                                                                                                                                                            
CPU Idleness Profiling: disabled                                                                                                                                                                                                                            
Size: 30G                                                                                                                                                                                                                                                   
Quick Mode: disabled                                                                                                                                                                                                                                        
=========================                                                                                                                                                                                                                                   
IOPS (Read/Write)                                                                                                                                                                                                                                           
        Random:           66,197 / 5,694                                                                                                                                                                                                                    
    Sequential:            6,504 / 6,594                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
Bandwidth in KiB/sec (Read/Write)                                                                                                                                                                                                                           
        Random:        939,746 / 545,931                                                                                                                                                                                                                    
    Sequential:        739,534 / 417,548                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                            
Latency in ns (Read/Write)                                                                                                                                                                                                                                  
        Random:      301,277 / 9,419,883                                                                                                                                                                                                                    
    Sequential:      111,137 / 8,106,560

@tyriis
Copy link
Author

tyriis commented Aug 16, 2025 •
edited
Loading

rook-ceph thunderbolt (cluster) + 10G (public)

node01

TEST_FILE: /volume/test                                                                                                                                                                                                                                     
TEST_OUTPUT_PREFIX: test_device                                                                                                                                                                                                                             
TEST_SIZE: 30G                                                                                                                                                                                                                                              
Benchmarking iops.fio into test_device-iops.json                                                                                                                                                                                                            
Benchmarking bandwidth.fio into test_device-bandwidth.json                                                                                                                                                                                                  
Benchmarking latency.fio into test_device-latency.json                                                                                                                                                                                                      
                                                                                                                                                                                                                                                            
=========================                                                                                                                                                                                                                                   
FIO Benchmark Summary                                                                                                                                                                                                                                       
For: test_device                                                                                                                                                                                                                                            
CPU Idleness Profiling: disabled                                                                                                                                                                                                                            
Size: 30G                                                                                                                                                                                                                                                   
Quick Mode: disabled                                                                                                                                                                                                                                        
=========================                                                                                                                                                                                                                                   
IOPS (Read/Write)                                                                                                                                                                                                                                           
        Random:           71,967 / 5,736                                                                                                                                                                                                                    
    Sequential:            7,534 / 6,774                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
Bandwidth in KiB/sec (Read/Write)                                                                                                                                                                                                                           
        Random:        984,150 / 580,090                                                                                                                                                                                                                    
    Sequential:        788,600 / 368,704                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                            
Latency in ns (Read/Write)                                                                                                                                                                                                                                  
        Random:      286,365 / 4,812,697                                                                                                                                                                                                                    
    Sequential:       60,550 / 4,528,310

node02

TEST_FILE: /volume/test                                                                                                                                                                                                                                     
TEST_OUTPUT_PREFIX: test_device                                                                                                                                                                                                                             
TEST_SIZE: 30G                                                                                                                                                                                                                                              
Benchmarking iops.fio into test_device-iops.json                                                                                                                                                                                                            
Benchmarking bandwidth.fio into test_device-bandwidth.json                                                                                                                                                                                                  
Benchmarking latency.fio into test_device-latency.json                                                                                                                                                                                                      
                                                                                                                                                                                                                                                            
=========================                                                                                                                                                                                                                                   
FIO Benchmark Summary                                                                                                                                                                                                                                       
For: test_device                                                                                                                                                                                                                                            
CPU Idleness Profiling: disabled                                                                                                                                                                                                                            
Size: 30G                                                                                                                                                                                                                                                   
Quick Mode: disabled                                                                                                                                                                                                                                        
=========================                                                                                                                                                                                                                                   
IOPS (Read/Write)                                                                                                                                                                                                                                           
        Random:           74,968 / 5,695                                                                                                                                                                                                                    
    Sequential:           36,010 / 6,543                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
Bandwidth in KiB/sec (Read/Write)                                                                                                                                                                                                                           
        Random:        993,236 / 591,083                                                                                                                                                                                                                    
    Sequential:      1,043,011 / 658,092                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                            
Latency in ns (Read/Write)                                                                                                                                                                                                                                  
        Random:       97,216 / 6,354,677                                                                                                                                                                                                                    
    Sequential:       62,772 / 4,478,008

node03

TEST_FILE: /volume/test                                                                                                                                                                                                                                     
TEST_OUTPUT_PREFIX: test_device                                                                                                                                                                                                                             
TEST_SIZE: 30G                                                                                                                                                                                                                                              
Benchmarking iops.fio into test_device-iops.json                                                                                                                                                                                                            
Benchmarking bandwidth.fio into test_device-bandwidth.json                                                                                                                                                                                                  
Benchmarking latency.fio into test_device-latency.json                                                                                                                                                                                                      
                                                                                                                                                                                                                                                            
=========================                                                                                                                                                                                                                                   
FIO Benchmark Summary                                                                                                                                                                                                                                       
For: test_device                                                                                                                                                                                                                                            
CPU Idleness Profiling: disabled                                                                                                                                                                                                                            
Size: 30G                                                                                                                                                                                                                                                   
Quick Mode: disabled                                                                                                                                                                                                                                        
=========================                                                                                                                                                                                                                                   
IOPS (Read/Write)                                                                                                                                                                                                                                           
        Random:           70,447 / 5,523                                                                                                                                                                                                                    
    Sequential:            6,457 / 5,771                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
Bandwidth in KiB/sec (Read/Write)                                                                                                                                                                                                                           
        Random:        949,631 / 572,395                                                                                                                                                                                                                    
    Sequential:        761,615 / 450,329                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                            
Latency in ns (Read/Write)                                                                                                                                                                                                                                  
        Random:     293,823 / 10,886,182                                                                                                                                                                                                                    
    Sequential:      118,248 / 8,036,728

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment