Pin mdraid kernel threads to a CPU

gistfile1.txt

The kernel creates a thread for every md raid configured:
# ps -ef|grep raid
root         912       2  0 15:24 ?        00:00:00 [raid5wq]
root        3383       2  0 15:24 ?        00:00:00 [md0_raid5]
root        3402       2  0 15:24 ?        00:00:03 [md1_raid5]


Being kernel threads, its management is limited, they cannot be pinned to CPU therefore they can move accross all available CPUs, they can get hammered by IRQ, they might have a 'busy' sibling…
Probably dedicated CPU will provide better performance (testing will follow!).

time for a Proof of Concept!

I had a look at the kernel source, the file drivers/md/raid5.c implement the raid stripe management  (line 613). I decided to pin this thread to a cpu I can specify via command line like mdraid_cpu=17, see below:
# cat /proc/cmdline
BOOT_IMAGE=(hd2,gpt2)/vmlinuz-6.5.7-raid5-t root=/dev/mapper/rl-root ro rd.lvm.lv=rl/root mdraid_cpu=17 isolcpus=17

if I check which processor is being used, it uses the selected one:
# for p in `pgrep raid5`; do ps -p $p -h -o cmd; cat /proc/$p/stat|cut -d ' ' -f 38; done
[raid5wq]
17
[md0_raid5]
17
[md1_raid5]
17

 

here the code added in the raid5.c module (idea taken from 

 ):
static int __init mdraid_setup(char *str)
{
        int ret = 0;
        ret = kstrtoint(str,10,&reserved_cpu);
        printk(KERN_INFO "CPU reserved for RAID5 Kthread: %d\n", reserved_cpu );
        return ret;
}

__setup("mdraid_cpu=", mdraid_setup);

 

Then I wanted to implement a policy to have a dedicated processor per raid device and I decided to use the processor specified in the command line plus the md minor numer (ie for md1 drive is 1, for md2 is 2 and so on )
        // custom CPU assigment mdadm_cpu + md device number
        int right_cpu = reserved_cpu + conf->mddev->md_minor;
        //printk(KERN_INFO "RIGHT CPU: %d\n", right_cpu );
        sh->cpu = right_cpu;

On idle all processes stay on the default processor, but with a printk I proved the run in the designed cpu.
# dmesg|grep RIGH|tail
[ 2620.511246] RIGHT CPU: 18
[ 2620.511249] RIGHT CPU: 18
[ 2620.511252] RIGHT CPU: 18
[ 2620.511255] RIGHT CPU: 18
[ 2620.511258] RIGHT CPU: 18


I also added isolcpus=17 in the command line so it stays out of the scheduler and no other process is allowed to run on it.