StudioEtrange StudioEtrange

To invoke JupyterLab with Spark capabilities there are two ways. An ad hoc method is to just state on the command line that JupyterLab should use pyspark as kernel. For instance starting JupyterLab with Python 3.6 (needs to be consistent with your Spark distribution), 20 executors each having 5 cores might look like this:

PYSPARK_PYTHON=python3.6 PYSPARK_DRIVER_PYTHON="jupyter" PYSPARK_DRIVER_PYTHON_OPTS="notebook --no-browser --port=8899" /usr/bin/pyspark2 --master yarn --deploy-mode client --num-executors 20 --executor-memory 10g --executor-cores 5 --conf spark.dynamicAllocation.enabled=false

In order to be able to create notebooks with a specific PySpark kernel directly from JupyterLab, just create a file ~/.local/share/jupyter/kernels/pyspark/kernel.json holding:

{
 "display_name": "PySpark",
 "language": "python",

BIND MOUNT INSIDE CONTAINER

docker run -v /path/on/host:/data busybox ls /data

Equivalent way but managed with "docker volume" - This do not work with file, only with directory

docker volume create --opt type=none --opt device=/path/on/host --opt o=bind my_vol
docker run -v my_vol:/data busybox ls /data

The user lauching alpine docker image is member of docker group. As he does not need sudo to use docker

sudo ls -al /root
drwx------  3 root root 4096 aoÃ»t   9  2015 .
drwxr-xr-x 25 root root 4096 fÃ©vr. 11  2017 ..
-rw-r--r--  1 root root 3106 fÃ©vr. 20  2014 .bashrc
-rw-r--r--  1 root root  140 fÃ©vr. 20  2014 .profile
-rw-r--r-- 1 root root 36 janv. 5 2015 .smbcredentials

	#https://stackoverflow.com/a/10965292

	[ -n "$var" ] && echo "var is set and not empty"
	[ -z "$var" ] && echo "var is unset or empty"
	[ "${var+x}" = "x" ] && echo "var is set" # may or may not be empty
	[ -n "${var+x}" ] && echo "var is set" # may or may not be empty
	[ -z "${var+x}" ] && echo "var is unset"
	[ -z "${var-x}" ] && echo "var is set and empty"

	Copy FOLDER_1 into PATH_B and preserve all attributes

	sudo rsync -avihXP --info=progress2 --stats /PATH_A/FOLDER_1/ /PATH_B/


	* use --dry-run to test
	* for encoding char problems between Mac/Linux : https://serverfault.com/questions/638316/rsync-iconv-option-on-mac-not-working-sync-from-remote-linux-server-to-local
	** rsync initiated from Mac : --iconv=utf-8-mac,utf-8
	** rsync initiated from Linux : --iconv=utf-8,utf-8-mac

	A note about using gmail as a relay
	Gmail by default does not allow email clients that don't use OAUTH 2 for authentication (like Thunderbird or Outlook).
	First you need to enable access to "Less secure apps" on your google settings.


	Serveur de courrier entrant (IMAP) imap.gmail.com

	SSL requis : oui
	Port : 993

	# https://stackoverflow.com/a/28776166
	sourced=0
	if [ -n "$ZSH_EVAL_CONTEXT" ]; then
	case $ZSH_EVAL_CONTEXT in *:file) sourced=1;; esac
	elif [ -n "$KSH_VERSION" ]; then
	[ "$(cd $(dirname -- $0) && pwd -P)/$(basename -- $0)" != "$(cd $(dirname -- ${.sh.file}) && pwd -P)/$(basename -- ${.sh.file})" ] && sourced=1
	elif [ -n "$BASH_VERSION" ]; then
	[ "$0" != "$BASH_SOURCE" ] && sourced=1
	else # All other shells: examine $0 for known shell binary filenames
	# Detects `sh` and `dash`; add additional shell filenames as needed.

	# List block devices
	lsblk
	NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
	sda 8:0 0 238.5G 0 disk
	├─sda1 8:1 0 200M 0 part /boot/efi
	├─sda2 8:2 0 500M 0 part /boot
	└─sda3 8:3 0 237.8G 0 part
	├─fedora-root 253:0 0 50G 0 lvm /
	├─fedora-swap 253:1 0 2G 0 lvm [SWAP]
	└─fedora-home 253:2 0 185.9G 0 lvm

	#!/usr/bin/env bash

	# Will scan a github repo to compute some statistics on PR
	# requires curl and bc (presents on most unix like system)


	# PARAMETERS ----------------------------
	# Github repo to scan
	GITHUB_REPO="status-im/status-react"

	#!/usr/bin/env bash

	# bash script to generate tree structure of a directory
	# Pravendra Singh (@hackpravj)

	pwd=$(pwd)
	find $pwd -print \| sed -e "s;$pwd;\.;g;s;[^/]*\/;\|__;g;s;__\|; \|;g"