Skip to content

Instantly share code, notes, and snippets.

View bnaul's full-sized avatar

Brett Naul bnaul

View GitHub Profile
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file has been truncated, but you can view the full file.
<!DOCTYPE html>
<html lang="en" >
This file has been truncated, but you can view the full file.
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8"/>
<title>Kepler.gl embedded map</title>
<!--Uber Font-->
<link rel="stylesheet" href="https://d1a3f4spazzrp4.cloudfront.net/kepler.gl/uber-fonts/4.0.0/superfine.css">
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@bnaul
bnaul / dask_bigquery.py
Created July 15, 2021 17:23
Dask <-> BigQuery helpers
def _stream_to_dfs(bqs_client, stream_name, schema, timeout):
"""Given a Storage API client and a stream name, yield all dataframes."""
return [
pyarrow.ipc.read_record_batch(
pyarrow.py_buffer(message.arrow_record_batch.serialized_record_batch), schema
).to_pandas()
for message in bqs_client.read_rows(name=stream_name, offset=0, timeout=timeout)
]
This file has been truncated, but you can view the full file.
{
"tcp://10.125.88.48:37099": {
"type": "Worker",
"id": "Worker-4363f614-359f-4aed-af17-35e08650eb47",
"scheduler": "tcp://brett-2ad617c8-daskscheduler:8786",
"nthreads": 4,
"memory_limit": 8400000000,
"address": "tcp://10.125.88.48:37099",
"status": "<Status.running: 'running'>",
"thread_id": 140597733148480,