Improve linux for performance

readme.md

CPU Cache Optimization

• Set process affinity to prevent cache thrashing: taskset -c 0-3 jq ...
• Align buffers to cache lines with compiler flags: -O3 -march=native -falign-functions=64
• Compile jq from source with profile-guided optimization:

  ./configure --enable-lto
  make CFLAGS="-O3 -march=native -flto -fprofile-generate"
  # Run benchmarks
  make clean && make CFLAGS="-O3 -march=native -flto -fprofile-use"
  

Memory Tuning

• Configure huge pages for large datasets: echo 20 > /proc/sys/vm/nr_hugepages
• Adjust transparent huge pages for pipeline workloads: echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
• Fine-tune virtual memory parameters:

  # Reduce swappiness to keep hot data in RAM
  sysctl -w vm.swappiness=1
  # Aggressive dirty page flushing for streaming IO
  sysctl -w vm.dirty_bytes=134217728
  sysctl -w vm.dirty_background_bytes=67108864
  # Optimize page cache reclaim
  sysctl -w vm.vfs_cache_pressure=50
  

IO Subsystem Tuning

• Adjust IO scheduler for SSD/NVMe: echo kyber > /sys/block/nvme0n1/queue/scheduler
• Increase readahead for sequential operations: blockdev --setra 8192 /dev/nvme0n1
• Tune IO queue depth: echo 64 > /sys/block/nvme0n1/queue/nr_requests
• Pre-allocate files with fallocate for contiguous allocation

Pipeline Optimization

• Add buffer size tuning to maximize throughput:

  mkfifo -m 600 pipe
  dd bs=1M if=large.json of=pipe & jq -c 'select(.field)' < pipe | ...
  

• Adjust ionice priority: ionice -c1 -n0 jq ...
• Exploit RAM disk for intermediate results: mount -t tmpfs -o size=16G tmpfs /mnt/tmpfs

Advanced jq Usage

• Pre-compile jq filters: jq -c --arg v "value" -f filter.jq
• Use binary mode for reduced UTF-8 processing: jq -c -b
• Split processing with named pipes and parallel workers