Skip to content

Instantly share code, notes, and snippets.

View ianmcook's full-sized avatar

Ian Cook ianmcook

View GitHub Profile
@ianmcook
ianmcook / request-body.json
Last active February 25, 2025 15:28
Use the Snowflake SQL REST API from a shell script with curl and jq to execute a query and download the result partitions in Arrow format
{
"statement": "SELECT * FROM MYTABLE",
"resultSetMetaData": {
"format": "arrowv1"
},
"timeout": 60,
"database": "MYDATABASE",
"schema": "MYSCHEMA",
"warehouse": "MYWAREHOUSE",
"role": "MYROLE"
@ianmcook
ianmcook / write_arrow_ipc_formats.py
Created April 2, 2025 16:48
Create sample data and write it to two files in Arrow IPC stream format and file format
import pandas as pd
import pyarrow as pa
file_path = 'fruit.arrow'
stream_path = 'fruit.arrows'
df = pd.DataFrame(data={'fruit': ['apple', 'apple', 'apple', 'orange', 'orange', 'orange'],
'variety': ['gala', 'honeycrisp', 'fuji', 'navel', 'valencia', 'cara cara'],
'weight': [134.2 , 158.6, None, 142.1, 96.7, None]})
@ianmcook
ianmcook / curl_arrow_pipe_python.md
Last active May 14, 2025 20:37
Pipe Arrow IPC stream from curl to Python

First start an HTTP server to serve Arrow IPC stream data. You can do this using one of the server examples in HTTP GET Arrow Data: Simple Examples or simply by starting a Python HTTP server in the same directory where you have an Arrow IPC stream file (named file.arrows in this example).

python -m http.server 8008

Download the attached Python script script.py. You might need to do chmod +x script.py to make it executable.