found-the-issue

1 NIC is running with 100Mbits only :(

```
Connecting to host 192.168.100.101, port 5201
[  5] local 192.168.100.103 port 42392 connected to 192.168.100.101 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  11.9 MBytes  99.5 Mbits/sec    0   42.4 KBytes       
[  5]   1.00-2.00   sec  11.1 MBytes  93.3 Mbits/sec    0   45.2 KBytes       
[  5]   2.00-3.00   sec  11.2 MBytes  94.4 Mbits/sec    0   45.2 KBytes       
[  5]   3.00-4.00   sec  11.1 MBytes  93.3 Mbits/sec    0   48.1 KBytes       
[  5]   4.00-5.00   sec  10.6 MBytes  89.1 Mbits/sec    0   45.2 KBytes       
[  5]   5.00-6.00   sec  10.6 MBytes  89.1 Mbits/sec    0   41.0 KBytes       
[  5]   6.00-7.00   sec  11.1 MBytes  93.3 Mbits/sec    0   50.9 KBytes       
[  5]   7.00-8.00   sec  11.2 MBytes  94.4 Mbits/sec    0   50.9 KBytes       
[  5]   8.00-9.00   sec  11.1 MBytes  93.3 Mbits/sec    0   45.2 KBytes       
[  5]   9.00-10.00  sec  11.2 MBytes  94.4 Mbits/sec    0   5.66 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   111 MBytes  93.4 Mbits/sec    0            sender
[  5]   0.00-10.01  sec   110 MBytes  92.0 Mbits/sec                  receiver

iperf Done.
```

this was the reason for low rook-ceph performance:

after fix

```
bash-5.1$ rados bench -p ceph-blockpool 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_talos01_1168
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       2         2         0         0         0           -           0
    1      16        66        50   199.967       200    0.375406    0.201107
    2      16       103        87   173.977       148   0.0129707    0.310708
    3      16       144       128   170.642       164   0.0156619    0.337213
    4      16       187       171   170.975       172    0.715322     0.33933
    5      16       222       206   164.778       140    0.929181    0.351855
    6      16       266       250   166.645       176   0.0181203    0.359934
    7      16       307       291   166.263       164    0.508394     0.36839
    8      16       352       336   167.975       180    0.394201    0.366298
    9      16       393       377   167.531       164    0.512105    0.369183
   10      16       431       415   165.975       152    0.435443    0.372946
Total time run:         10.6541
Total writes made:      431
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     161.815
Stddev Bandwidth:       17.3077
Max bandwidth (MB/sec): 200
Min bandwidth (MB/sec): 140
Average IOPS:           40
Stddev IOPS:            4.32692
Max IOPS:               50
Min IOPS:               35
Average Latency(s):     0.385355
Stddev Latency(s):      0.315917
Max latency(s):         1.03777
Min latency(s):         0.0125966
```

rook-ceph thunderbolt (cluster) + 10G (public)

node01

TEST_FILE: /volume/test                                                                                                                                                                                                                                     
TEST_OUTPUT_PREFIX: test_device                                                                                                                                                                                                                             
TEST_SIZE: 30G                                                                                                                                                                                                                                              
Benchmarking iops.fio into test_device-iops.json                                                                                                                                                                                                            
Benchmarking bandwidth.fio into test_device-bandwidth.json                                                                                                                                                                                                  
Benchmarking latency.fio into test_device-latency.json                                                                                                                                                                                                      
                                                                                                                                                                                                                                                            
=========================                                                                                                                                                                                                                                   
FIO Benchmark Summary                                                                                                                                                                                                                                       
For: test_device                                                                                                                                                                                                                                            
CPU Idleness Profiling: disabled                                                                                                                                                                                                                            
Size: 30G                                                                                                                                                                                                                                                   
Quick Mode: disabled                                                                                                                                                                                                                                        
=========================                                                                                                                                                                                                                                   
IOPS (Read/Write)                                                                                                                                                                                                                                           
        Random:           71,967 / 5,736                                                                                                                                                                                                                    
    Sequential:            7,534 / 6,774                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
Bandwidth in KiB/sec (Read/Write)                                                                                                                                                                                                                           
        Random:        984,150 / 580,090                                                                                                                                                                                                                    
    Sequential:        788,600 / 368,704                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                            
Latency in ns (Read/Write)                                                                                                                                                                                                                                  
        Random:      286,365 / 4,812,697                                                                                                                                                                                                                    
    Sequential:       60,550 / 4,528,310 

node02

TEST_FILE: /volume/test                                                                                                                                                                                                                                     
TEST_OUTPUT_PREFIX: test_device                                                                                                                                                                                                                             
TEST_SIZE: 30G                                                                                                                                                                                                                                              
Benchmarking iops.fio into test_device-iops.json                                                                                                                                                                                                            
Benchmarking bandwidth.fio into test_device-bandwidth.json                                                                                                                                                                                                  
Benchmarking latency.fio into test_device-latency.json                                                                                                                                                                                                      
                                                                                                                                                                                                                                                            
=========================                                                                                                                                                                                                                                   
FIO Benchmark Summary                                                                                                                                                                                                                                       
For: test_device                                                                                                                                                                                                                                            
CPU Idleness Profiling: disabled                                                                                                                                                                                                                            
Size: 30G                                                                                                                                                                                                                                                   
Quick Mode: disabled                                                                                                                                                                                                                                        
=========================                                                                                                                                                                                                                                   
IOPS (Read/Write)                                                                                                                                                                                                                                           
        Random:           74,968 / 5,695                                                                                                                                                                                                                    
    Sequential:           36,010 / 6,543                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
Bandwidth in KiB/sec (Read/Write)                                                                                                                                                                                                                           
        Random:        993,236 / 591,083                                                                                                                                                                                                                    
    Sequential:      1,043,011 / 658,092                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                            
Latency in ns (Read/Write)                                                                                                                                                                                                                                  
        Random:       97,216 / 6,354,677                                                                                                                                                                                                                    
    Sequential:       62,772 / 4,478,008 

node03

TEST_FILE: /volume/test                                                                                                                                                                                                                                     
TEST_OUTPUT_PREFIX: test_device                                                                                                                                                                                                                             
TEST_SIZE: 30G                                                                                                                                                                                                                                              
Benchmarking iops.fio into test_device-iops.json                                                                                                                                                                                                            
Benchmarking bandwidth.fio into test_device-bandwidth.json                                                                                                                                                                                                  
Benchmarking latency.fio into test_device-latency.json                                                                                                                                                                                                      
                                                                                                                                                                                                                                                            
=========================                                                                                                                                                                                                                                   
FIO Benchmark Summary                                                                                                                                                                                                                                       
For: test_device                                                                                                                                                                                                                                            
CPU Idleness Profiling: disabled                                                                                                                                                                                                                            
Size: 30G                                                                                                                                                                                                                                                   
Quick Mode: disabled                                                                                                                                                                                                                                        
=========================                                                                                                                                                                                                                                   
IOPS (Read/Write)                                                                                                                                                                                                                                           
        Random:           70,447 / 5,523                                                                                                                                                                                                                    
    Sequential:            6,457 / 5,771                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
Bandwidth in KiB/sec (Read/Write)                                                                                                                                                                                                                           
        Random:        949,631 / 572,395                                                                                                                                                                                                                    
    Sequential:        761,615 / 450,329                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                            
Latency in ns (Read/Write)                                                                                                                                                                                                                                  
        Random:     293,823 / 10,886,182                                                                                                                                                                                                                    
    Sequential:      118,248 / 8,036,728