harshavardhana

[pvs]
# hostname local to this system on k8s
node=host1
# capacity for all devices this is common for all devices.
capacity=400Gi
# list of devices to format in command separated list.
devices=/dev/sdb

[minio]
release=RELEASE.2022-07-15T03-44-22Z

harshavardhana

package main

import (
	"crypto/tls"
	"fmt"
	"net"
	"net/http"
	"os"
	"strconv"
	"time"

harshavardhana

#!/bin/bash

ENGINE=sync ## can be changed to psync as well 
RESULT_FILE=/tmp/result_$$

rm -rf $HOME/tmp-fio
mkdir -p $HOME/tmp-fio && cd $HOME/tmp-fio

echo "Starting storage tests"
echo "Results will be saved in $RESULT_FILE"

harshavardhana

API	AWS S3	MinIO	GCS (S3 compatibility)
ListObjectVersions	✔️	✔️	❗

GCS fails in this manner while they succeed on AWS S3 and MinIO fine.

~ mc ls --versions gcs/harshavardhana
mc: <ERROR> Unable to list folder. unrecognized option:Marker

harshavardhana

import org.apache.spark.sql.DataFrame

def dataFrameToDDL(dataFrame: DataFrame, tableName: String): String = {
    val columns = dataFrame.schema.map { field =>
        "  " + field.name + " " + field.dataType.simpleString.toUpperCase
    }
    s"CREATE TABLE $tableName (\n${columns.mkString(",\n")}\n)"
}

import spark.sqlContext.implicits._

harshavardhana

#!/bin/sh
#
# Resource script for Minio
#
# Description:  Manages Minio as an OCF resource in
#               an Active-Passive High Availability setup.
#
# Author:       Ricardo Branco <[email protected]> : Initial script for minio server
# License:      GNU General Public License (GPL)
#

harshavardhana

Erasure code sizing guide

Toy Setup

Capacity constrainted environments, works but not recommended for production.

servers	drives	stripe_size	parity
1	1	1	0
1	4	4	2

harshavardhana

apiVersion: v1
kind: Secret
metadata:
  name: test-minio-creds-secret
type: Opaque
data:
  accesskey: bWluaW8= # base 64 encoded "minio" (echo -n 'minio' | base64)
  secretkey: bWluaW8xMjM= # based 64 encoded "minio123" (echo -n 'minio123' | base64)
---
apiVersion: minio.min.io/v1

harshavardhana

# This version makes a GET request and passes the signature
# in the Authorization header.
import sys, os, base64, datetime, hashlib, hmac, urllib

import requests
import logging

from minio.signer import presign_v4
from minio.helpers import get_target_url
from minio.credentials import Credentials, Static

harshavardhana

# You can set MTU value here. If left undefined or empty, it will
# not be specified in calico CNI config, so Calico will use built-in
# defaults. The value should be a number, not a string.
calico_mtu: 1500

# Configure the MTU to use for workload interfaces and tunnels.
# - If Wireguard is enabled, set to your network MTU - 60
# - Otherwise, if VXLAN or BPF mode is enabled, set to your network MTU - 50
# - Otherwise, if IPIP is enabled, set to your network MTU - 20
# - Otherwise, if not using any encapsulation, set to your network MTU.