Skip to content

Instantly share code, notes, and snippets.

View RaczeQ's full-sized avatar
🌍
Crafting new Open Source tools 🔮

Kamil Raczycki RaczeQ

🌍
Crafting new Open Source tools 🔮
View GitHub Profile
@RaczeQ
RaczeQ / pyarrow_multiprocessing_streaming.py
Last active May 19, 2024 19:41
Pyarrow Multiprocessing with streaming the result
import multiprocessing
from pathlib import Path
from queue import Queue
from time import sleep
from typing import Callable
import pyarrow as pa
import pyarrow.parquet as pq
from tqdm import tqdm