Harshavardhana harshavardhana

    Outbound
    ┌───────────────────┐             ┌────────────────┐               ┌───────────────┐                      ┌────────────────┐
    │                   │   Parr.     │                │  (http body)  │               │                      │                │
    │ Bitrot Hash       │     Write   │      Pipe      │      Read     │  HTTP buffer  │    Write (syscall)   │  TCP Buffer    │
    │ Erasure Shard     │ ──────────► │  (unbuffered)  │ ────────────► │   (64K Max)   │ ───────────────────► │    (4MB)       │
    │                   │             │                │               │  (io.Copy)    │                      │                │

Spark-shell with S3A based checkpointing

Spark context Web UI available at http://nirisvara:4040
Spark context available as 'sc' (master = local[*], app id = local-1677271782301).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
 _\ \/ _ \/ _ `/ __/ '_/

Kernel Tuning for MinIO Production Deployment on Linux Servers

List of kernel tuning recommendations for MinIO server deployments, a copy of this script is available here that can be applied on the servers.

NOTE: These are common recommendations on any Linux servers, users must be careful on any premature optimization. These settings are not mandatory, these settings do not fix any hardware issues and should not be considered as an alternative to boost performance to mask underlying hardware problems. Under all circumstances it is recommended to perform this tuning only after performing benchmarks for the hardware with expected baseline results.

Google Public NTP [AS15169]:

time.google.com

time1.google.com

time2.google.com

time3.google.com

	## template:jinja
	{#
	This file (/etc/cloud/templates/hosts.debian.tmpl) is only utilized
	if enabled in cloud-config. Specifically, in order to enable it
	you need to add the following to config:
	manage_etc_hosts: True
	-#}
	# Your system has configured 'manage_etc_hosts' as True.
	# As a result, if you wish for changes to this file to persist
	# then you will need to either

	version: '3.7'

	# Settings and configurations that are common for all containers
	x-minio-common: &minio-common
	image: quay.io/minio/minio:${RELEASE}
	command: server http://site1-minio{1...4}/data{1...2}
	environment:
	- MINIO_PROMETHEUS_AUTH_TYPE=public
	- CI=true

	"""
	FTP benchmark.

	Usage:
	ftpbench --help
	ftpbench -h <host> -u <user> -p <password> [options] login
	ftpbench -h <host> -u <user> -p <password> [options] upload <workdir> [-s <size>]
	ftpbench -h <host> -u <user> -p <password> [options] download <workdir> [-s <size>] [--files <count>]

	Connection options:

	#!/bin/bash \

	i=1

	sudo rm -f /tmp/fstab
	for disk in $(lsblk -i -p -n -o NAME \| grep -v 'loop\\|nvme0\\|nvme1\\|nvme2\\|nvme3\\|ubuntu\\|md\\|sda' \| sort); do
	sudo mkdir -p /disk${i}
	sudo mkfs.xfs -f -L disk${i} $disk
	echo "LABEL=disk${i} /disk${i} xfs defaults,noatime 0 0" \| sudo tee -a /tmp/fstab
	i=$(( $i + 1 ))

	package deadlineconn

	// DeadlineConn - is a generic stream-oriented network connection supporting buffered reader and read/write timeout.
	type DeadlineConn struct {
	net.Conn
	readTimeout time.Duration // sets the read timeout in the connection.
	writeTimeout time.Duration // sets the write timeout in the connection.
	}

	// Sets read timeout

	#!/bin/bash

	docker system prune -af --filter "until=8h"