Skip to content

Instantly share code, notes, and snippets.

@jobliz
jobliz / gist:f31b5eeac16f8b6649907fa00243d9c2
Created May 4, 2020 09:15
Bash function to set terminal title. Put it in .bashrc
function set-title(){
if [[ -z "$ORIG" ]]; then
ORIG=$PS1
fi
TITLE="\[\e]2;$*\a\]"
PS1=${ORIG}${TITLE}
}
@jobliz
jobliz / lftp_sync.sh
Last active May 25, 2019 19:02
A script to sync a local directory with a remote directory through FTP.
#!/bin/bash
#
# A script to sync a local directory with a remote directory through FTP. The local directory contents
# will overwrite the remote directory. If a file was deleted locally, it will be deleted remotely.
# Notes:
#
# - It excludes the content of the .git directory.
# - The -P flag is for parallelizing work.
# - 'set ssl:verify-certificate false' might not be necessary. YMMV.
#
@jobliz
jobliz / hb.c
Created May 12, 2019 01:49
Himmelblau function with simulated annealing
// gcc himmelblau_simulated_annealing.c -lm && ./a.out
#include <math.h>
#include <stdio.h>
#include <stdlib.h> // RAND_MAX
double rand01(void)
{
return rand() / ((double) RAND_MAX);
}
@jobliz
jobliz / keybase.md
Created March 13, 2019 19:22
Keybase proof

Keybase proof

I hereby claim:

  • I am jobliz on github.
  • I am jobliz (https://keybase.io/jobliz) on keybase.
  • I have a public key ASDvrV7KVZ6FJMIq3ZP0e-vmMPhZdadkN9hMuZaE3sFEtwo

To claim this, I am signing this object:

@jobliz
jobliz / create_elasticsearch_index.py
Created August 10, 2018 02:06
Two step processing for loading the goodbooks-10k dataset into elasticsearch
import sys
import csv
from elasticsearch_dsl.connections import connections
from elasticsearch_dsl import DocType, Text, Date, Search
from elasticsearch import Elasticsearch
connections.create_connection(hosts=['localhost'], timeout=20)
es = Elasticsearch()
ess = Search(using=es)
@jobliz
jobliz / tag_search_with_elasticsearch_6.md
Created July 28, 2018 15:02
Tag Search with ElasticSearch 6

Tag Search with ElasticSearch 6

This is a modified version of this tutorial, the queries have been modified so that they work with ES6. Many backwards incompatible changes happened between previous verions and 6, so many how-to's on the internet are outdated. If you're just starting learning ElasticSearch with version 6 then you should read these links and keep a mental note of them.

@jobliz
jobliz / black_white_css_sprites.py
Created July 16, 2018 04:27
Create CSS spritesheet with mogrify and glue-sprite
import os
import sys
import atexit
import shutil
import subprocess
def main(width, height):
# get image paths in full
dir_path = os.path.dirname(os.path.realpath(__file__))
dir_images = os.path.join(dir_path, 'sprite-src/')
@jobliz
jobliz / posgresql_configuration_with_ltree.md
Created June 22, 2018 19:08
PostgreSQL database configuration with ltree extension

PostgreSQL database configuration with ltree extension

  1. Install PostgreSQL database (apt install postgresql postgresql-contrib under Ubuntu)
  2. Log in to the postgres account: sudo --login --user postgres
  3. Start the PostgreSQL interactive terminal: psql
  4. Create the database: postgres=# CREATE DATABASE my_database;
  5. Create a user: postgres=# CREATE USER my_user WITH PASSWORD 'my_password';
  6. Grant the user access to the database. postgres=# GRANT ALL PRIVILEGES ON DATABASE my_database to my_user;
  7. Switch to the database: \connect my_database;
@jobliz
jobliz / pyspark_commands_from_list.py
Created June 19, 2018 00:13
Running PySpark commands from a list
from typing import List, Tuple
from pyspark import SparkContext
from pyspark.sql import SparkSession
commands = [
['read'],
['option', ("inferSchema", "true")],
['option', ("header", "true")],
['option', ("dateFormat", "dd/MM/yyyy H:m")],
@jobliz
jobliz / pandas_spark_conversion.py
Created June 18, 2018 21:55
Pandas/Spark dataframe conversion with retail dataset
"""
CSV Dataset
Download XLSX dataset from http://archive.ics.uci.edu/ml/datasets/online+retail
Convert to CSV with LibreOffice Calc
Spark timestamp and datetime conversion from string:
https://stackoverflow.com/questions/46295879/how-to-read-date-in-custom-format-from-csv-file
https://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html
Pandas dataframe to spark