Spark Training Environment Setup

Anaconda

# Download installer
wget https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh

bash Anaconda3-2020.02-Linux-x86_64.sh

Kalki5 / docker-compose.yml

Created April 23, 2020 11:57

Temporary

	version: "3"
	services:
	web:
	image: YOUR_DOCKER_ID/pyapp:latest
	deploy:
	replicas: 2
	ports:
	- "80:80"
	networks:
	- webnet

Kalki5 / lazyloading.html

Created March 13, 2020 11:16

Lazy Loading CSS and JS libraries


	<!-- Preloading Files -->
	<script>
	function lazyLoadLibraries() {
	const csslibraries = [
	'https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.18.1/styles/monokai-sublime.min.css',
	]
	const jslibraries = [
	'https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.18.1/highlight.min.js',
	]

Kalki5 / tcp_monitor.py

Created March 12, 2020 13:21

Monitor the changes in Listening TCP connections

	from socket import SOCK_STREAM
	from time import sleep
	import psutil


	standard_ports = {
	# Connections
	21, 22, 25, 53, 3389,
	# Databases
	3306, 5432, 1433, 6379, 11211,

Kalki5 / .kitchen.yml

Last active February 18, 2020 04:56

	---
	driver:
	name: azurerm

	driver_config:
	subscription_id: '91d5ad1d-29c6-442e-9f1b-a8bb6fef2189'
	location: 'East US'
	machine_size: 'Standard_B1s'
	username: 'TestAdmin'
	password: 'TestPassword@123'

Kalki5 / server.js

Created December 12, 2019 11:07

Simple Node UDP Server and Client

	var PORT = 33333;
	var HOST = '0.0.0.0';

	var dgram = require('dgram');
	var server = dgram.createSocket('udp4');

	server.on('listening', function() {
	var address = server.address();
	console.log('UDP Server listening on ' + address.address + ':' + address.port);
	});

Kalki5 / cert.yaml

Last active April 2, 2021 11:41

Nginx Ingress Controller

	---
	apiVersion: v1
	kind: Secret
	metadata:
	name: default-server-secret
	type: Opaque
	data:
	tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZLakNDQkJLZ0F3SUJBZ0lTQkZlMEI3cEExZm5kYjE1MzNGalpoM2dMTUEwR0NTcUdTSWIzRFFFQkN3VUEKTURJeEN6QUpCZ05WQkFZVEFsVlRNUll3RkFZRFZRUUtFdzFNWlhRbmN5QkZibU55ZVhCME1Rc3dDUVlEVlFRRApFd0pTTXpBZUZ3MHlNVEEwTURJeE1ETTNNemhhRncweU1UQTNNREV4TURNM016aGFNQjB4R3pBWkJnTlZCQU1NCkVpb3VhWEF1YzJ0aFlYTnNZV0p6TG1OdmJUQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0MKZ2dFQkFORmVhbkNtVmN3YzhoNE96ekIya2FuQ1Jtbjdubi8xQ3JzT3p2dER3TXI1RGcxOU0ybE5oVGM2M01ETQpjK0E1TzFzQ2I5WTQvV3dka0tTaWRrRHpPS0pNMUpiVmxBNGs0Y003OUoyZXFZSGJ6aUF2bUtTNGJDQmtqTTlaCkk4citweVN6L2xaZjZnT29KbWFDYU8rNzdQSkk3QVdzai82QlBqQUNGbmYzN05MWFB0UUt1cXR6WHh1aGl4UlcKQjhVWWFSM0tMeFpaU3g2ZDlsRzg5enR4clBTa1VTZDJLZFVlZXNLdGlVYmtLeEdTYlpBUXJ5SDhINjRoYlV1KwpVQ0N3RVpqc3M4aEp4TWdnTU1zM1ZHaWVuQWlIdVc5SDZidGdPSGVpa2xVT1hITllkbTh5WHpjRlRCTkxrMCtzClhEVEZFWTZBdVVaUHBDOUUvMCtKQmpKSzh6TUNBd0VBQWFPQ0FrMHdnZ0pKTUE0R0ExVWREd0VCL3dRRUF3SUYKb0RBZEJnTlZIU1VFRm

Kalki5 / forecasting_metrics.py

Created October 30, 2019 08:08 — forked from bshishov/forecasting_metrics.py

Python Numpy functions for most common forecasting metrics

	import numpy as np

	EPSILON = 1e-10


	def _error(actual: np.ndarray, predicted: np.ndarray):
	""" Simple error """
	return actual - predicted

Kalki5 / script.sh

Created June 26, 2019 12:24

Add Swap to multiple machines in bulk

	sudo su

	apt-get update

	apt-get install -y ansible

	echo ANSIBLE_HOST_KEY_CHECKING=False >> /etc/environment

	echo "
	[machines]

Kalki5 / add_policy.py

Created March 19, 2019 06:22

Add own policy to a lambda function

	import boto3

	lamba_client = boto3.client('lambda', region_name='REGION_NAME')

	lamba_client.add_permission(
	FunctionName='create_lab',
	StatementId='AWSEventsRule',
	Action='lambda:InvokeFunction',
	Principal='events.amazonaws.com',
	SourceArn='arn:aws:events:REGION_NAME:ACCOUNT_NUMBER:rule/*',