Skip to content

Instantly share code, notes, and snippets.

View YordanGeorgiev's full-sized avatar

Yordan Georgiev YordanGeorgiev

16 followers · 2 following
View GitHub Profile
@YordanGeorgiev
YordanGeorgiev / perl-iterate-over-hash-ref-of-hash-refs.pm
Created February 17, 2018 11:15
sort hash ref of hash refs by a key on the second level in perl #perl
@$rs = sort { $a->{ 'SeqId' } <=> $b->{ 'SeqId' } } @$rs;
foreach my $row ( @$rs ) {
# dof stuff
my $var = $row->{'col1'}
}
@YordanGeorgiev
YordanGeorgiev / perl-constuctor
Created February 17, 2018 11:18
[constructor in Perl] how-to add an OO constructor in Perl #perl #perl-OO
# -----------------------------------------------------------------------------
# the constructor
# -----------------------------------------------------------------------------
sub new {
my $class = shift; # Class name is in the first parameter
$appConfig = ${ shift @_ } || { 'foo' => 'bar' ,} ;
my $self = {}; # Anonymous hash reference holds instance attributes
bless( $self, $class ); # Say: $self is a $class
@YordanGeorgiev
YordanGeorgiev / parse-cmd-args.sh
Created February 17, 2018 12:14
[parse cmd args in bash] how-to parse cmd args in bash #bash #shell
#------------------------------------------------------------------------------
# parse the single letter command line args
#------------------------------------------------------------------------------
doParseCmdArgs(){
# traverse all the possible cmd args
while getopts ":a:c:i:h:" opt; do
case $opt in
a)
@YordanGeorgiev
YordanGeorgiev / log-output
Last active July 27, 2020 10:53
[log output in bash] how-to log output in bash #bash #functions #bash-funcs
#------------------------------------------------------------------------------
# echo pass params and print them to a log file and terminal
# with timestamp and $host_name and $0 PID
# usage:
# doLog "INFO some info message"
# doLog "DEBUG some debug message"
#------------------------------------------------------------------------------
doLog(){
type_of_msg=$(echo $*|cut -d" " -f1)
@YordanGeorgiev
YordanGeorgiev / create-table-item.sql
Last active February 17, 2018 12:21
[create table ddl in postgres] how-to create table in postgres pgsql #sql #pgsql #postgres
-- DROP TABLE IF EXISTS item ;
SELECT 'create the "item" table'
;
CREATE TABLE item (
guid UUID NOT NULL DEFAULT gen_random_uuid()
, level integer NULL
, seq integer NULL
, prio integer NULL
, name varchar (200) NOT NULL
@YordanGeorgiev
YordanGeorgiev / scala-spark-chain-transformations.scala
Created February 19, 2018 13:41
[scala spark chaining transformations] how-to chain transformations in scala spark on a DataFrame obj #scala #spark #dataframe #transformations
import org.apache.spark.sql.SparkSession
trait Phase {
implicit val spark: SparkSession = SparkSession.builder().getOrCreate()
def process(df: DataFrame): DataFrame
}
class Level1Phase ( cnf: Configuration) extends Phase {
override def process(df: DataFrame): DataFrame = {
@YordanGeorgiev
YordanGeorgiev / scala-spark-create-hardcoded-schema.scala
Last active February 21, 2018 09:14
[scala spark create harcoded schema] how-to create hardcoded schema in scala spark for a dataframe #scala #spark #dataframe #schema
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.Row
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.types.StructField
import org.apache.spark.sql.types.DoubleType
import org.apache.spark.sql.types.StringType
val objSchema: StructType = StructType(
@YordanGeorgiev
YordanGeorgiev / scala-spark-create-nullable-cols-dataframe.scala
Created February 19, 2018 15:43
[scala spark create hardcoded dataframe with nullable cols] how-to create a dataframe with nullable columns in scala spark #scala #spark #dataframe #hardcoded
val spark = SparkSession.builder().getOrCreate()
import spark.implicits._
// format: off
// obs first and second row hardcoded vals are for "teaching" schema !!!
val ds = Seq(
// foo comments
(1,"foo",Some("bar"),Some(1850)) ,
// bar comments
(2,"foo",None,None)
@YordanGeorgiev
YordanGeorgiev / perl-oneliner-add-remove-file-first-line.sh
Created February 20, 2018 10:58
[add and remove the first line of a file oneliner] how-to to add or remove the first line of a file with a perl oneliner #perl #oneliners #shell
# add the "sep=," for xls to open quickly csv files
find $dir -type f -exec perl -pi -e 'print "sep=,\n" if $.==1' {} \;
# remove it
find $dir -type f -exec perl -pi -e '$_ = "" if $. == 1' {} \;
@YordanGeorgiev
YordanGeorgiev / to-from-xls.sh
Created February 20, 2018 17:08
[edit-csv-files-in-xls] how-to edit csv files in Excel with perl oneliners and bash #bash #perl #oneliners #excel #xls
# usage:
# source zeppelin/sh/funcs/to-from-xls.sh
# export dir=<<path-to-the-root-dir-holding-the-csv-files-obs-recursive!!!!>
# toXls
# fromXls
# use BEFORE you wanto to open the files in xls
toXls(){
set -u -o pipefail