Martin Grigorov martin-g

🏠

Working from home

Open for work! Freelancing in Rust/Scala/Kotlin/Java

martin-g / successful-logs-spark-3.2.1-on-k8s.txt

Created March 15, 2022 13:04

	22/03/15 13:01:05 INFO LoggingPodStatusWatcherImpl: State changed, new state:
	pod name: spark-on-k8s-app-b336777f8daa93b6-driver
	namespace: spark-on-k8s
	labels: spark-app-selector -> spark-9e850f59cfa54f77bec1a2f30f4753a7, spark-role -> driver
	pod uid: 48088478-9896-4019-b263-47866214cbb9
	creation time: 2022-03-15T13:00:41Z
	service account name: spark-account-name
	volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-slmsc
	node name: minikube
	start time: 2022-03-15T13:00:41Z

martin-g / run-spark-pi-on-k8s.bash

Last active March 15, 2022 13:58

	#!/usr/bin/env bash

	DRIVER_CORES=1
	DRIVER_MEM=2048m
	EXECUTOR_INSTANCES=3
	EXECUTOR_CORES=2
	EXECUTOR_MEM=2048m
	SPARK_HOME="$HOME/spark/spark-3.2.1-bin-hadoop3.2"

	PATH="$JAVA_8_HOME/bin:$SPARK_HOME/bin:$PATH"

martin-g / kind-cluster.yaml

Created March 15, 2022 09:24

Kubernetes description for creating Kind cluster

	kind: Cluster
	apiVersion: kind.x-k8s.io/v1alpha4
	name: my-kind-cluster-name
	nodes:
	- role: control-plane
	- role: worker

martin-g / concourse-execute-pipeline.sh

Last active January 20, 2021 09:43

	#!/bin/bash

	set -eu

	DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
	export fly_target=${fly_target:-tutorial}
	echo "Concourse API target ${fly_target}"
	echo "Tutorial $(basename $DIR)"

	pushd $DIR

martin-g / concourse-basic-pipeline.yml

Created January 20, 2021 09:40

A ConcourseCI pipeline definition that runs the hello-world task only on worker nodes tagged with 'arm64'

martin-g / councourse-task-hello-world.yml

Created January 20, 2021 09:33

martin-g / concourse-worker-node.fish

Last active January 20, 2021 11:37

Fish shell script that starts concourse-arm-worker Docker container

	#!/usr/bin/fish

	docker run \
	-it \
	--rm \
	--privileged \
	-p 7777:7777 \
	-p 7788:7788 \
	-p 8888:8888 \
	-v /home/ubuntu/devel/concourse-work-dir:/concourse \

martin-g / concourse-docker-compose.yml

Created January 20, 2021 09:10

	version: '3'

	services:
	concourse-db:
	image: postgres
	ports: ["5432:5432"]
	environment:
	POSTGRES_DB: concourse
	POSTGRES_USER: concourse_user
	POSTGRES_PASSWORD: concourse_pass

martin-g / web-node.fish

Created January 20, 2021 08:51

Fish shell script to start ConcourseCI web node

	#!/usr/bin/env fish

	set -x CONCOURSE_POSTGRES_HOST localhost
	set -x CONCOURSE_POSTGRES_USER concourse_user
	set -x CONCOURSE_POSTGRES_PASSWORD concourse_pass
	set -x CONCOURSE_POSTGRES_DATABASE concourse
	set -x CONCOURSE_EXTERNAL_URL http://web-node:8080
	set -x CONCOURSE_X_FRAME_OPTIONS deny
	set -x CONCOURSE_ADD_LOCAL_USER user:password
	set -x CONCOURSE_MAIN_TEAM_LOCAL_USER user

martin-g / build.sh-concourse-arm-worker

Created January 20, 2021 08:40

	./build.sh
	+ set -e
	+ docker build --build-arg http_proxy --build-arg https_proxy -t registry-image . -f Dockerfile-registry-image
	Sending build context to Docker daemon 21.79MB
	Step 1/15 : FROM golang:1.14.3-alpine3.11 as builder
	---> 6bf3dc21cd79
	Step 2/15 : RUN apk add git
	---> Using cache
	---> f7e0eca88239
	Step 3/15 : RUN git clone https://github.com/concourse/registry-image-resource.git /src/registry-image-resource