Stana mathsigit

The Best of the Best Practices (BOBP) Guide for Python

A "Best of the Best Practices" (BOBP) guide to developing in Python.

In General

Values

"Build tools for others that you want to be built for you." - Kenneth Reitz
"Simplicity is alway better than functionality." - Pieter Hintjens

Spark On K8s

Requirement

spark 2.4.7 or 3.0.1
minikube 1.15.3 or 1.19
kubectl 1.19

Checking the compatibility matrix of Kubernetes Client: https://github.com/fabric8io/kubernetes-client/blob/master/README.md#compatibility-matrix

Minikube

Install

macOS

Before you begin

To check if virtualization is supported on macOS, run the following command on your terminal.

Virtualenv

Commands

Create virtualenv

virtualenv -p ${PYTHON_VERSION} ${ENV_FOLDER_PATH}

#For instance

Intellij on jetbrains - Ultimate feature

How to setup, to see how-can-pyspark-be-called-in-debug-mode - stackoverflow

Something about Pyenv On MacOS

How to install pyenv

Installing pyenv

#Via homebrew
brew update
brew install pyenv

如何在 kafka connector 的 Restful Server 建立 https

1.使用 openssl 建立 SSL key, 整個 script 如下

#!/bin/bash
#Step 1
keytool -keystore kafka.server.keystore.jks -alias localhost -validity 365 -genkey
#Step 2
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert

function docker_container_ip() {
  container_ids=$(docker ps -aq)
  for cid in ${container_ids}
  do
    info=$(docker inspect -f 'name={{.Name}}  ip={{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ${cid})
    echo -e "$cid \t $info"
  done
}