harshavardhana

#!/bin/bash

ENGINE=sync ## can be changed to psync as well 
RESULT_FILE=/tmp/result_$$

rm -rf $HOME/tmp-fio
mkdir -p $HOME/tmp-fio && cd $HOME/tmp-fio

echo "Starting storage tests"
echo "Results will be saved in $RESULT_FILE"

harshavardhana

API	AWS S3	MinIO	GCS (S3 compatibility)
ListObjectVersions	✔️	✔️	❗

GCS fails in this manner while they succeed on AWS S3 and MinIO fine.

~ mc ls --versions gcs/harshavardhana
mc: <ERROR> Unable to list folder. unrecognized option:Marker

harshavardhana

import org.apache.spark.sql.DataFrame

def dataFrameToDDL(dataFrame: DataFrame, tableName: String): String = {
    val columns = dataFrame.schema.map { field =>
        "  " + field.name + " " + field.dataType.simpleString.toUpperCase
    }
    s"CREATE TABLE $tableName (\n${columns.mkString(",\n")}\n)"
}

import spark.sqlContext.implicits._

harshavardhana

#!/bin/sh
#
# Resource script for Minio
#
# Description:  Manages Minio as an OCF resource in
#               an Active-Passive High Availability setup.
#
# Author:       Ricardo Branco <[email protected]> : Initial script for minio server
# License:      GNU General Public License (GPL)
#

harshavardhana

Erasure code sizing guide

Toy Setup

Capacity constrainted environments, works but not recommended for production.

servers	drives	stripe_size	parity
1	1	1	0
1	4	4	2

harshavardhana

apiVersion: v1
kind: Secret
metadata:
  name: test-minio-creds-secret
type: Opaque
data:
  accesskey: bWluaW8= # base 64 encoded "minio" (echo -n 'minio' | base64)
  secretkey: bWluaW8xMjM= # based 64 encoded "minio123" (echo -n 'minio123' | base64)
---
apiVersion: minio.min.io/v1

harshavardhana

# This version makes a GET request and passes the signature
# in the Authorization header.
import sys, os, base64, datetime, hashlib, hmac, urllib

import requests
import logging

from minio.signer import presign_v4
from minio.helpers import get_target_url
from minio.credentials import Credentials, Static

harshavardhana

# You can set MTU value here. If left undefined or empty, it will
# not be specified in calico CNI config, so Calico will use built-in
# defaults. The value should be a number, not a string.
calico_mtu: 1500

# Configure the MTU to use for workload interfaces and tunnels.
# - If Wireguard is enabled, set to your network MTU - 60
# - Otherwise, if VXLAN or BPF mode is enabled, set to your network MTU - 50
# - Otherwise, if IPIP is enabled, set to your network MTU - 20
# - Otherwise, if not using any encapsulation, set to your network MTU.

harshavardhana

Creating PersistentVolumes

We recommend using local Persistent Volumes as the backing storage for MinIO StatefulSets. MinIO creates a distributed, resilient, failure tolerant storage system on local PVs. There is no explicit dependency on external distributed file systems or CSI vendors.

This document explains how to create local PVs. Once you create these PVs, MinIOInstance PVCs should automatically bind to these PVs.

harshavardhana

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,