Skip to content

Instantly share code, notes, and snippets.

View rjzamora's full-sized avatar

Richard (Rick) Zamora rjzamora

View GitHub Profile
import argparse
import random
from functools import partial
from time import perf_counter as clock
import fsspec
import numpy as np
import pandas as pd
import dask
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@rjzamora
rjzamora / parquet_bench.py
Last active December 6, 2024 18:56
Simple multi-file parquet benchmark for S3 storage
# # KvikIO S3 Env Variables
# os.environ["CUDF_NATIVE_S3_IO"] = "True"
# if os.environ.get("CUDF_NATIVE_S3_IO", "False") == "True":
# os.environ["KVIKIO_NTHREADS"] = f"{os.cpu_count()}"
import os
import math
import argparse
import random
import time
@rjzamora
rjzamora / resources.ipynb
Last active July 25, 2024 20:08
Splitting IO and compute onto different resources.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@rjzamora
rjzamora / multi-file-json.ipynb
Created May 30, 2024 18:24
Multi-file json read experiments
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@rjzamora
rjzamora / ray_shuffle.ipynb
Last active April 18, 2024 16:02
Simple shuffling example with `cudf` and `ray`
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@rjzamora
rjzamora / parquet_delayed_mapping.ipynb
Created December 20, 2023 18:01
Experimenting with simpler ``blocksize`` logic for ``read_parquet``
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@rjzamora
rjzamora / ray_exploration.ipynb
Created November 21, 2023 17:56
Exploring Ray and Dask on Ray with GPUs
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@rjzamora
rjzamora / dask_expr_dask-demo-day.ipynb
Last active May 18, 2023 14:58
Demo: Dask Expressions
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.