Skip to content

Instantly share code, notes, and snippets.

View absognety's full-sized avatar
🎯
Focusing

Vikas Chitturi - Open Source Contributor absognety

🎯
Focusing
175 followers · 561 following
View GitHub Profile
@rnbtechnology
rnbtechnology / write_to_s3.py
Last active November 8, 2021 08:39
import copy
# write to a path using the Hudi format
def hudi_write(df, schema, table, path, mode, hudi_options):
hudi_options = {
"hoodie.datasource.write.recordkey.field": "recordkey",
"hoodie.datasource.write.precombine.field": "precombine_field",
"hoodie.datasource.write.partitionpath.field": "partitionpath_field",
"hoodie.datasource.write.operation": "write_operaion",
"hoodie.datasource.write.table.type": "table_type",
@RobertAKARobin
RobertAKARobin / python.md
Last active August 23, 2026 08:08
Python Is Not A Great Programming Language
@shravan-kuchkula
shravan-kuchkula / apache-airflow.md
Last active September 12, 2022 07:27
Install apache-airflow locally on mac

Using Docker and docker-compose to manage Apache Airflow on mac

Using our beloved docker and docker-compose, we can very quickly bring up an Apache Airflow instance on our mac.

Contents of docker-compose.yml

About the only thing you need to customize in this docker-compose.yml file is the volumes section. This will tell docker to map the given directory containing your Airflow DAGs/plugins to container file system.

version: '3'
services:
@tamiroze
tamiroze / sql2sf.py
Last active September 22, 2025 13:15
Converts Oracle, SQL-Server, and other DDL to Snowflake DDL
#!/usr/bin/python
# $Id: $
# Converts Oracle, SQL-Server, and other DDL to Snowflake DDL
def usage():
print """\
# Usage: sql2sf.py input-file [output-file]
"""
@jph00
jph00 / install_basic.sh
Last active March 7, 2021 03:53
sudo add-apt-repository -y ppa:apt-fast/stable
sudo add-apt-repository -y ppa:graphics-drivers/ppa
sudo apt-get update
sudo apt-get -y install apt-fast
# prompts
sudo apt-fast -y upgrade
sudo apt-fast install -y python3-pip ubuntu-drivers-common libvorbis-dev libflac-dev libsndfile-dev cmake build-essential libgflags-dev libgoogle-glog-dev libgtest-dev google-mock zlib1g-dev libeigen3-dev libboost-all-dev libasound2-dev libogg-dev libtool libfftw3-dev libbz2-dev liblzma-dev libgoogle-glog0v5 gcc-6 gfortran-6 g++-6 doxygen graphviz libsox-fmt-all parallel exuberant-ctags vim-nox python-powerline python3-pip ack lsyncd
sudo apt-fast install -y tigervnc-standalone-server firefox mesa-common-dev
@absognety
absognety / scala_env_setup.md
Last active April 11, 2023 06:22
Setup Scala environment in Linux/RHEL/CentOS/Debian/Ubuntu/Fedora releases
Check for Java Development Kit (JDK) version
$ java -version
java version "1.7.0_171"
OpenJDK Runtime Environment (rhel-2.6.13.0.el7_4-x86_64 u171-b01)
OpenJDK 64-Bit Server VM (build 24.171-b01, mixed mode)

For scala to be set up JDK 8 or greater version is required
if JDK/OpenJDK version is less than 1.8 then follow the below steps

@bgauduch
bgauduch / multiple-repository-and-identities-git-configuration.md
Last active August 23, 2026 16:11
Multiple git identities on one machine: config per folder and per provider (personal vs work). Every command verified.

Multiple git identities on one machine

TL;DR

One machine, several git identities: typically personal vs work, GitHub vs GitLab, or two accounts on the same provider. This setup makes every repository pick the right name, email, and keys on its own, so no work commit ever goes out under your personal address (or the reverse). In a hurry: run the Appendix B script once per identity.

@asmaier
asmaier / load_parquet_s3.py
Last active March 5, 2021 03:43
Pyspark script for downloading a single parquet file from Amazon S3 via the s3a protocol. It also reads the credentials from the "~/.aws/credentials", so we don't need to hardcode them. See also https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html .
#
# Some constants
#
aws_profile = "your_profile"
aws_region = "your_region"
s3_bucket = "your_bucket"
#
# Reading environment variables from aws credential file
#
@xgenvn
xgenvn / books.md
Created August 19, 2017 15:15 — forked from abstractart/books.md
Download free O'Reilly books
@mGalarnyk
mGalarnyk / RStudioUbuntu1404.sh
Last active July 29, 2020 15:46
How to Install R and RStudio on Ubuntu 14.04 (trusty) for the blog https://medium.com/@GalarnykMichael/install-r-and-rstudio-on-ubuntu-12-04-14-04-16-04-b6b3107f7779#.8bwdcvjxv
# Install R + RStudio on Ubuntu 14.04
sudo apt-key adv –keyserver keyserver.ubuntu.com –recv-keys E084DAB9
# Ubuntu 12.04: precise
# Ubuntu 14.04: trusty
# Ubuntu 16.04: xenial
# Basic format of next line deb https://<my.favorite.cran.mirror>/bin/linux/ubuntu <enter your ubuntu version>/
sudo add-apt-repository 'deb https://ftp.ussg.iu.edu/CRAN/bin/linux/ubuntu trusty/'
sudo apt-get update