Skip to content

Instantly share code, notes, and snippets.

View prashant-shahi's full-sized avatar

Prashant Shahi prashant-shahi

View GitHub Profile
@prashant-shahi
prashant-shahi / sidecar.md
Last active February 6, 2019 08:09
Sidecar and its advantages

Co-locate a cohesive set of tasks with the primary application, but place them inside their own process or container, providing a homogeneous interface for platform services across languages.

sidecar

A sidecar service is not necessarily part of the application, but is connected to it. It goes wherever the parent application goes. Sidecars are supporting processes or services that are deployed with the primary application.

Advantages of using a sidecar pattern include:

  • independent from its primary application in terms of runtime environment and programming language, so you don't need to develop one sidecar per language.
@prashant-shahi
prashant-shahi / json-styled-logger.py
Created February 6, 2019 09:57
JSON style logging of stdout/stderr of a Python application for Fluentd
import logging
import sys
from pythonjsonlogger import jsonlogger
class StackdriverJsonFormatter(jsonlogger.JsonFormatter, object):
def __init__(self, fmt="%(asctime) %(levelname) %(message)", style='%', *args, **kwargs):
jsonlogger.JsonFormatter.__init__(self, fmt=fmt, *args, **kwargs)
def process_log_record(self, log_record):
log_record['timestamp'] = log_record['asctime']
@prashant-shahi
prashant-shahi / instruction.md
Last active February 11, 2019 14:07
Regular Expression(RegEx) to obtain Dimensions Spec and Flatten Spec of ingestionSpec for Kafka in Druid

Dimension Spec

Note: Assuming that you have a flattened data sample to begin with.

For string

Find:

"([a-zA-Z\_]+)": ".+",

Replace:

@prashant-shahi
prashant-shahi / gist:c144e4142a33e0c8e4c272d13a45c230
Created February 15, 2019 10:13
Java parameters' manual "java -x"
$ java -X
-Xmixed mixed mode execution (default)
-Xint interpreted mode execution only
-Xbootclasspath:<directories and zip/jar files separated by ;>
set search path for bootstrap classes and resources
-Xbootclasspath/a:<directories and zip/jar files separated by ;>
append to end of bootstrap class path
-Xbootclasspath/p:<directories and zip/jar files separated by ;>
prepend in front of bootstrap class path
-Xdiag show additional diagnostic messages
@prashant-shahi
prashant-shahi / Golang-StatusCodes.md
Last active February 25, 2019 19:41
net/http HTTP status code constants in GoLang
Description Status Code
StatusContinue 100
StatusSwitchingProtocols 101
StatusProcessing 102
StatusOK 200
StatusCreated 201
StatusAccepted 202
StatusNonAuthoritativeInfo 203
StatusNoContent 204
@prashant-shahi
prashant-shahi / instruction.md
Created March 4, 2019 08:02
Increase or Decrease size of Virtual Machine (VM) in Google Cloud

Follow the steps to increase or decrease the size of VM in Google Cloud :

  1. Goto https://console.cloud.google.com
  2. Goto Compute Engine tab, followed by VM Instances.
  3. Select the VM for which you want to increase/decrease the disk size.
  4. Under Boot disk, you will see one(or more) disks for the VM. Click on one for which you desire to update disk size.
  5. Click Edit in the top. Then, modify the Size. At last, click on save.
  6. Now, we just need to tell the OS to grow the partition to the total size. This can be done by executing following commands :
fdisk -l
growpart /dev/sda 1
@prashant-shahi
prashant-shahi / core-site.xml
Last active March 23, 2019 07:44
Python Application which uses SQL-like SELECT Expression to read gzip-compressed CSV in S3 and output result in CSV
<!--
Then, open the file $HADOOP_HOME/etc/hadoop/core-site.xml for editing.
In this example Minio server is running at http://127.0.0.1:9000 with access key minio and secret key minio123.
Make sure to update relevant sections with valid Minio server endpoint and credentials.
-->
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
@prashant-shahi
prashant-shahi / natality-0-schema.md
Last active April 2, 2019 01:24
Natality Schema
Field name Type Mode Description
source_year INTEGER REQUIRED Four-digit year of the birth. Example: 1975.
year INTEGER NULLABLE Four-digit year of the birth. Example: 1975.
month INTEGER NULLABLE Month index of the date of birth, where 1=January.
day INTEGER NULLABLE Day of birth, starting from 1.
wday INTEGER NULLABLE Day of the week, where 1 is Sunday and 7 is Saturday.
state STRING NULLABLE The two character postal code for the state. Entries after 2004 do not include this value.
is_male BOOLEAN REQUIRED TRUE if the child is male, FALSE i
import findspark
findspark.init("/opt/spark")
import random
from pyspark import SparkContext
sc = SparkContext(appName="EstimatePi")
def inside(p):
x, y = random.random(), random.random()
return x*x + y*y < 1
NUM_SAMPLES = 1000000
count = sc.parallelize(range(0, NUM_SAMPLES)) \
@prashant-shahi
prashant-shahi / colormaps.md
Created April 1, 2019 23:03
matplotlib : All supported colormaps

List of all the colormaps supported by matplotlib :

  • Accent
  • Accent_r
  • Blues
  • Blues_r
  • BrBG
  • BrBG_r
  • BuGn
  • BuGn_r