Skip to content

Instantly share code, notes, and snippets.

@monocongo
monocongo / xarray_stack_groupby_apply_unstack_spi.ipynb
Last active January 3, 2023 23:26
Use xarray for parallel computation with stack-groupby-apply-unstack
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@monocongo
monocongo / start_jupyter_pyspark.sh
Created July 29, 2022 01:06 — forked from BryanCutler/start_jupyter_pyspark.sh
How to start a Jupyter Notebook with PySpark Kernel
#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
@gordthompson
gordthompson / mssql_df_upsert.py
Last active October 5, 2024 23:34
Build a T-SQL MERGE statement and upsert a DataFrame
# Copyright 2024 Gordon D. Thompson, [email protected]
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
@bennyistanto
bennyistanto / SPI_based_on_CHIRPS_GeoTIFF.md
Last active August 9, 2024 17:56
Calculate SPI using CHIRPS data

Calculate SPI using monthly rainfall data in GeoTIFF format

These last few months, I have tried a lot of difference formulation to calculate Standardized Precipitation Index (SPI) based on rainfall data in netCDF format, check below files as a background:

The reason why I use rainfall in netCDF format in above files because the software to calculate SPI: climate-indices python package will only accept single netCDF as input, and the SPI script will read the netCDF input file based on time dimension.

Converting raster files into netCDF is easy using GDAL or other GIS software, but to make the time dimension enabled n

@BryanCutler
BryanCutler / start_jupyter_pyspark.sh
Last active July 29, 2022 01:06
How to start a Jupyter Notebook with PySpark Kernel
#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
@mbaitelman
mbaitelman / README.md
Last active February 20, 2024 22:31
Automated Terraform Deployments Using Bitbucket Pipelines
@joshbode
joshbode / logging.py
Last active September 12, 2024 05:38
Colour Logging - Works in Jupyter Lab/Notebook
import sys
import logging
from typing import Optional, Dict
from colorama import Fore, Back, Style
class ColoredFormatter(logging.Formatter):
"""Colored log formatter."""
@keithweaver
keithweaver / split-video-by-frame.py
Created May 10, 2017 19:30
Using OpenCV takes a mp4 video and produces a number of images.
'''
Using OpenCV takes a mp4 video and produces a number of images.
Requirements
----
You require OpenCV 3.2 to be installed.
Run
----
Open the main.py and edit the path to the video. Then run:
@cameres
cameres / compute_correlation_matrix.py
Last active November 22, 2022 14:19
Compute Pandas Correlation Matrix of a Spark Data Frame
from pyspark.mllib.stat import Statistics
import pandas as pd
# result can be used w/ seaborn's heatmap
def compute_correlation_matrix(df, method='pearson'):
# wrapper around
# https://forums.databricks.com/questions/3092/how-to-calculate-correlation-matrix-with-all-colum.html
df_rdd = df.rdd.map(lambda row: row[0:])
corr_mat = Statistics.corr(df_rdd, method=method)
corr_mat_df = pd.DataFrame(corr_mat,
@bikz05
bikz05 / opencv-3-installation
Last active April 22, 2024 20:13
OpenCV 3 Installation
sudo apt-get update
sudo apt-get upgrade
# Added by me
sudo apt-get install freeglut3 freeglut3-dev libtbb-dev libqt4-dev
# Copied from pyimagesearch.com
sudo apt-get install build-essential cmake git pkg-config
sudo apt-get install libjpeg8-dev libtiff4-dev libjasper-dev libpng12-dev
sudo apt-get install libgtk2.0-dev
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev