marcintustin

Data engineer, data strategy and engineering management consultant

marcintustin / pgerr.go

Created October 30, 2024 13:53

Handling a pg error and getting the stacktrace in go

	import (
	"github.com/jackc/pgconn"
	"github.com/pkg/errors"
	)

	func main() {
	if err != nil {
	var pgerr *pgconn.PgError
	if errors.As(err, &pgerr) && pgerr.SQLState() == database.DUPLICATE {
	t.Log(err)

marcintustin / postgres_queries_and_commands.sql

Last active February 9, 2023 19:19 — forked from rgreenjr/postgres_queries_and_commands.sql

Useful PostgreSQL Queries and Commands

	-- show blocked queries (not the same as idle)
	SELECT
	pid,
	usename,
	pg_stat_activity.query_start,
	now() - pg_stat_activity.query_start AS query_time,
	query,
	state,
	wait_event_type,
	wait_event

marcintustin / learnada.gpr

Created January 29, 2022 19:35

Make building work with alr on mac

	with "config/learnada_config.gpr";
	project Learnada is

	-- This is the magic
	package Linker is
	for Switches ("Ada") use ("-L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/");
	end Linker;

	for Source_Dirs use ("src");
	for Object_Dir use "obj";

marcintustin / duplicates.py

Created September 29, 2021 14:28

Find duplicates python exercise

	"""
	Write functions that:
	a. detects if an array/slice has duplicates
	b. detects what the duplicates are
	c. counts the duplicates
	You may tackle these in any order
	You may alter the signatures of the skeletons to meet your needs
	"""
	from typing import List, Mapping, TypeVar, Iterable
	import pytest

marcintustin / aggregator.scala

Created August 17, 2019 18:38

Using Aggregator in Spark

	import org.apache.spark.sql.functions._
	import org.apache.spark.sql.{Dataset, Encoder, Encoders}
	import org.apache.spark.sql.expressions.Aggregator
	import org.apache.spark.sql.TypedColumn

	case class Food(key: String, date: String, numeric: long, text: String)

	object AggregateLatestFoods {
	/ Load the data elsewhere /
	def aggregateLatestFoods(in: Dataset[Food]): Dataset[Food] =

marcintustin / 0Mandelbrot

Last active May 14, 2018 03:35

Mandelbrot set in p5.js (see sketch.js) with grayscale shading

marcintustin / index.html

Created March 10, 2018 21:05

Basic mandelbrot set with p5.js

	<html>
	<head>
	<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.5.11/p5.min.js"></script>
	<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.5.11/addons/p5.dom.min.js"></script>
	<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.5.11/addons/p5.sound.min.js"></script>
	<script src="https://cdnjs.cloudflare.com/ajax/libs/mathjs/4.0.0/math.min.js" integrity="sha256-SXLRPQMQE3pP06076EMmcA5x6Qv5Wu0rfyH51xcMd8c=" crossorigin="anonymous"></script>
	<link rel="stylesheet" type="text/css" href="main.css">
	</head>
	<body>
	<script src="sketch.js"></script>

marcintustin / genericSpark.scala

Last active January 1, 2018 20:36

Generic Spark Code

marcintustin / genericSparkBroken.scala

Created January 1, 2018 20:00

A somewhat working attempt at some generic spark code

	trait TimeColumns {
	val date: Date
	val segment_time: Timestamp
	val station_id: Integer
	val start_time: Timestamp
	val end_time: Timestamp
	}

	case class Viewership(
	date: Date, segment_time: Timestamp, station_id: Integer, start_time: Timestamp, end_time: Timestamp,

marcintustin / parametricStatic.scala

Created January 1, 2018 19:05

Spark code with explicit schemas - dangerously parametrised

	def matchViewershipWithSchedules[T, RESULT](
	viewership: Dataset[T], schedules: Dataset[Schedule])(
	// You need this so that RESULT can be used as a type parameter
	// inside the body of the function. A quirk of how Scala conforms
	// to Java's type erasure rules
	implicit ev: TypeTag[RESULT]): Dataset[RESULT] = {
	import session.implicits._

	viewershipWithQHAndDate.toDF.as('viewership).join(schedules.toDF.as('schedules),
	$"viewership.date" === $"schedules.date"