Skip to content

Instantly share code, notes, and snippets.

@oliverangelil
oliverangelil / databricks_scan_uc.py
Created November 8, 2024 21:21
Recursively Scan Unity Catalog Tables
import pandas as pd
def get_table_details(catalogs: list[str]):
"""
Retrieve detailed information about tables across multiple catalogs and schemas, including row count and size statistics.
This function uses Spark SQL to iterate over a list of catalogs, fetching details for each schema and table within
each catalog. For each table, it calculates the number of rows, the total size in bytes, and the average size per row
@oliverangelil
oliverangelil / maize_yield.py
Created June 17, 2018 04:42
Maize Yield for Siya
import os
import pandas as pd
import matplotlib as mpl
mpl.use('Agg')
import matplotlib.pyplot as plt
path_maize = '/home/siyabusa/terra/Forecast_ProcessedFiles/Outputs/Maize/'
files_maize = sorted([i for i in os.listdir(path_maize) if i.endswith('ec1.csv')])
names = []
@oliverangelil
oliverangelil / for_henri.txt
Created June 14, 2018 08:06
log file of luatex breaking in AUCTEX
This is LuaTeX, Version beta-0.80.0 (TeX Live 2015) (rev 5238) (format=lualatex 2015.11.17) 14 JUN 2018 18:00
restricted \write18 enabled.
file:line:error style messages enabled.
**\input .tex
(./.tex)
! Emergency stop.
<*> \input .tex
*** (job aborted, no legal \end found)
@oliverangelil
oliverangelil / for_henri.tex
Last active June 14, 2018 08:10
simple luatex
\documentclass{article}
\usepackage{fontawesome}
\begin{document}
\begin{center}
\textbf{\faEnvelope John Smith}\\
\end{center}
\end{document}
@oliverangelil
oliverangelil / note.txt
Created June 4, 2018 20:40
port install mysql57 note
mysql57 has the following notes:
On activation if no /opt/local/etc/mysql57/my.cnf file exists one
will be created which loads
/opt/local/etc/mysql57/macports-default.cnf.
If a /opt/local/etc/mysql57/my.cnf file exists MacPorts does not
touch it and any changes you make to /opt/local/etc/mysql57/my.cnf
will be preserved (e.g., during port upgrades, deactivations or
activations). /opt/local/etc/mysql57/my.cnf is a good place to
customize your mysql57 installation.
@oliverangelil
oliverangelil / ridge_bias_variance.py
Last active October 13, 2021 09:15
generate dummy data and apply ridge with varying levels of regularisation. Plots train and test results. This code goes with my blog post: https://oliverangelil.github.io/posts/bias-variance
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_squared_error
import numpy as np
import matplotlib.pyplot as plt
# generate data
X, y, w = make_regression(n_samples=1000, n_features=200, coef=True,
random_state=1, bias=0, noise=3, tail_strength=0.9, effective_rank=10)