nitisht

from time import time
from pyspark import SparkContext,SparkConf
import pyrosbag
from functools import partial
import pandas as pd
import numpy as np
from PIL import Image
from io import BytesIO
import rosbag
import cv2

nitisht

Delete all the objects in a bucket

while read -r key; do
    mc rm myminio/testb/${key}
done< <(mc ls --json myminio/testb | jq --raw-output '"\(.key)"')

Count objects in a bucket

nitisht

Minio Healthcheck

Liveness probe definition

Used to identify situations where server is running but may not behave optimally, i.e. sluggish response or corrupt backend. Such situations can be generally only be fixed after a restart.

Kubernetes kills the container and restarts it in case of liveness probe responding with a failure code.

Readiness probe definition

Used to identify situations where server is not ready to accept requests yet. Such situations are generally recovered after waiting for some time.

nitisht

CoreDNS vs kubedns

• On-the-fly DNSSEC signing of served data in CoreDNS.
• kube-dns supports only etcd as the backend, CoreDNS on the other hand has several supported backends.
• kube-dns records do not reflect the state of the cluster. Any query to w-x-y-z.namespace.pod.cluster.local will return an A record with w.x.y.z, even if that IP does not belong to specified namespace or even to the cluster address space. CoreDNS integration offers the option pods verified, which will verify that the IP address w.x.y.z returned is in fact the IP of a pod in the specified namespace.
• Plugin chaining and pluggable architecture makes CoreDNS better suited to adapt to various backends, as compared to kubedns.

CoreDNS plugins

nitisht

Nginx SSL termination for Minio server load balanced setup

This document explains the steps required to set up Nginx proxy and SSL termination for Minio servers running in the backgronud.

Generate self signed certificate

Create a directory /etc/nginx/ssl/domain.abc, here domain.abc is the name of your website domain. Then use the below commands

sudo openssl genrsa -out private.key 2048
sudo openssl req -new -x509 -days 3650 -key private.key -out public.crt -subj "/C=US/ST=state/L=location/O=organization/CN=domain.abc"

nitisht

Minio Storage Class

Storage Classes provide administrators an option to set number of parity disks used for erasure encoded objects.

Current behaviour

By default Minio server uses half of the available disks to store data and rest for parity. This means if you have N disks, Minio server will erasure-encode data across as N/2 data disks and N/2 parity disks.

The write quorum is (N/2) + 1 and read quorum is N/2.

With Storage class support

Before starting Minio server, administrator can set these environment variables.

nitisht

Mint

Mint is a Docker image based platform to run tests against Minio server using various Minio, AWS SDKs and Tools. Read more on Mint here.

Requirements of Mint automation pod

Following are the steps we need to follow

• Once a PR is opened on minio/minio repository, GitHub sends webhook notifications to our web server.
• Web server parses the incoming PR information and passes on this to an automation script. We can use https://github.com/adnanh/webhook or https://github.com/ceejbot/jthoober as the webhook receiver.
• The automation script checks out the PR locally and builds Minio Docker image.
• Push the Docker image to play.minio.io Docker registry.

nitisht

{
  "type": "file",
  "enabled": true,
  "connection": "s3a://lake",
  "config": null,
  "workspaces": {
    "root": {
      "location": "/",
      "writable": false,
      "defaultInputFormat": null

nitisht

<?xml version="1.0" encoding="UTF-8" ?>
<!--
 Licensed to the Apache Software Foundation (ASF) under one or more
 contributor license agreements.  See the NOTICE file distributed with
 this work for additional information regarding copyright ownership.
 The ASF licenses this file to You under the Apache License, Version 2.0
 (the "License"); you may not use this file except in compliance with
 the License.  You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

nitisht

apiVersion: v1
entries:
  minio:
  - apiVersion: v1
    created: 2017-06-15T17:48:36.895822482Z
    description: Distributed object storage server built for cloud applications and
      devops.
    digest: 75ff1e3d779d8937cff57c28a102da97a520245d50e22c1a2763cbea064a76cd
    home: https://minio.io
    icon: https://www.minio.io/img/logo_160x160.png