Skip to content

Instantly share code, notes, and snippets.

View Priyansh121096's full-sized avatar
♟️

Priyansh Agrawal Priyansh121096

♟️
View GitHub Profile
@Priyansh121096
Priyansh121096 / local_iceberg_table_using_trino.txt
Last active October 10, 2023 01:24
How to setup a local Iceberg table using Trino?
Requirements: Linux, Trino
Topics:
- Apache Iceberg (https://iceberg.apache.org/)
- Trino (https://trino.io/)
- MySQL (https://www.mysql.com/)
- Hive metastore (https://hive.apache.org/)
# Install and configure mysql
# https://www.digitalocean.com/community/tutorials/how-to-install-mysql-on-ubuntu-20-04
@Priyansh121096
Priyansh121096 / ipython_memory_profile.py
Created October 5, 2024 13:19
How to profile memory usage with ipython / jupyter
In [1]: import ipython_memory_usage.ipython_memory_usage as imu
In [2]: imu.start_watching_memory()
In [2] used 0.0000 MiB RAM in 5.31s, peaked 0.00 MiB above current, total RAM usage 15.57 MiB
In [3]: %cat slots.py
class MyClass(object):
__slots__ = ['name', 'identifier']
def __init__(self, name, identifier):
self.name = name
from pyspark.sql import SparkSession
from pyspark.sql.functions import rand, expr, from_unixtime
import time
# Initialize Spark session
spark = SparkSession.builder.appName("RandomDataFrame").getOrCreate()
# Number of rows
n_rows = 1_000_000
request_id event_time status metadata
eca6a956-9cea-4129-baf3-4c10ad7df539 2024-11-08 05:45:26.399 foo
37573810-cb4e-458a-9a28-d7a8dfcbb4d3 2024-11-08 05:45:27.399 bar
960bf05b-1d57-41ff-b781-860119b00a93 2024-11-08 05:45:28.399 baz ""
7f9ac1c1-fe20-48b0-81bb-fee7da270cde 2024-11-08 05:45:29.399 bam