Skip to content

Instantly share code, notes, and snippets.

@TomNicholas

TomNicholas/quadratic_means_coiled_executor.ipynb

Last active July 17, 2023 20:29
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save TomNicholas/15e60a591827fd602251144cb30050ee to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save TomNicholas/15e60a591827fd602251144cb30050ee to your computer and use it in GitHub Desktop.
Download ZIP
Coiled functions executor for Cubed
Display the source blob
Display the rendered blob
Raw
quadratic_means_coiled_executor.ipynb
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "144e8663-f783-496b-99a4-07de3e92f172",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import xarray as xr"
]
},
{
"cell_type": "markdown",
"id": "a0b0b052-ea72-4b0c-885f-e6ccf39ebfac",
"metadata": {},
"source": [
"## Means of quadratic quantities\n",
"\n",
"**Problem:** We have 2 time-varying scalar fields $U(t)$ and $V(t)$. \n",
"We want to calculate the time-averages of all the quadratic combinations of these fields, i.e. $<U^2>_t$, $<V^2>_t$, and $<UV>_t$ all simultaneously.\n",
"\n",
"If these fields are chunked along time, then computing the mean of any one quadratic field should process only require loading one time chunk at a time.\n",
"\n",
"(From https://github.com/pangeo-data/distributed-array-examples/issues/2)"
]
},
{
"cell_type": "markdown",
"id": "b9015e8b-ae03-4c62-be0c-b661269838c0",
"metadata": {},
"source": [
"# Using Cubed"
]
},
{
"cell_type": "markdown",
"id": "5425723c-b274-4e6a-87e3-0346efa57b9a",
"metadata": {},
"source": [
"With cubed we can try the same problem, but state the maximum memory we will allow the computation to consume.\n",
"\n",
"With perfect allocation of resources we might we expect the maximum memory to be the 5x a single chunk, for the 5 fields $U$,$V$, $U^2$, $V^2$, & $UV$. We could then add an additional factor x4 for the [4x overhead that cubed's blockwise operation assumes](https://github.com/tomwhite/cubed/issues/160#issuecomment-1533648472). For a ~150MB chunk, that gives 5x4x150MB ~= 3GB RAM required.\n",
"\n",
"Because right now Cubed won't actually co-allocate these related chunks to the same worker ([neither will dask](https://github.com/dask/distributed/issues/7555)), we will instead just use the [recommended default spec size](https://tom-e-white.com/cubed/user-guide/memory.html#chunk-sizes) of 2GB."
]
},
{
"cell_type": "markdown",
"id": "5d4fa394-56dd-486b-8342-e1cd293a89ef",
"metadata": {},
"source": [
"## Imports"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "8626ced5-02b6-4a87-96d2-6cec0b1d0ce9",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import cubed\n",
"import cubed.random"
]
},
{
"cell_type": "markdown",
"id": "656298f4-0815-4ddc-8a5b-9dfd3c13b0e0",
"metadata": {},
"source": [
"## Configuration"
]
},
{
"cell_type": "markdown",
"id": "11eb0998-ba7d-44c8-b02a-c584c7d394e6",
"metadata": {
"tags": []
},
"source": [
"We have to give the necessary permissions to create Zarr data in our cloud provider (here GCS), and also give this information to Lithops via a configuration file."
]
},
{
"cell_type": "markdown",
"id": "9fc78c35-81be-41d3-a351-2efb95667107",
"metadata": {},
"source": [
"Notice we increased the number of `max_workers` (up from the Lithops default of 1000) to ensure maximum parallelism on our largest workload."
]
},
{
"cell_type": "markdown",
"id": "a6736597-aa51-41b3-be73-0ea62ddb9573",
"metadata": {},
"source": [
"## Prevent logging warnings"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "1800c3f3-252b-4121-b0ed-45f7c2a0096f",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import logging\n",
"import sys\n",
"\n",
"from tqdm.contrib.logging import logging_redirect_tqdm"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "033e053b-c421-42db-86b4-d335c0983c50",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"logging.basicConfig(level=logging.INFO)\n",
"# suppress harmless connection pool warnings\n",
"logging.getLogger(\"urllib3.connectionpool\").setLevel(logging.ERROR)"
]
},
{
"cell_type": "markdown",
"id": "8e8ce461-fdb0-4d09-8ba0-74b541a78597",
"metadata": {},
"source": [
"## Choose Coiled executor"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "ccd5de28-e439-439e-93f5-10bc1e016225",
"metadata": {},
"outputs": [],
"source": [
"from cubed.runtime.executors.coiled import CoiledFunctionsDagExecutor\n",
"\n",
"coiledexecutor = CoiledFunctionsDagExecutor()"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "e26027dc-28ab-461d-b1e3-84f2501fe8a6",
"metadata": {},
"outputs": [],
"source": [
"import coiled"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "f7930853-e66e-43f1-8527-a393728ff384",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'t4g.small': {'name': 't4g.small',\n",
" 'cores': 2,\n",
" 'gpus': 0,\n",
" 'gpu_name': None,\n",
" 'memory': 2048,\n",
" 'backend_type': 'vm_aws',\n",
" 'coiled_credits_per_hour': 2.0},\n",
" 't3.small': {'name': 't3.small',\n",
" 'cores': 2,\n",
" 'gpus': 0,\n",
" 'gpu_name': None,\n",
" 'memory': 2048,\n",
" 'backend_type': 'vm_aws',\n",
" 'coiled_credits_per_hour': 2.0},\n",
" 't3a.small': {'name': 't3a.small',\n",
" 'cores': 2,\n",
" 'gpus': 0,\n",
" 'gpu_name': None,\n",
" 'memory': 2048,\n",
" 'backend_type': 'vm_aws',\n",
" 'coiled_credits_per_hour': 2.0},\n",
" 'c6g.medium': {'name': 'c6g.medium',\n",
" 'cores': 1,\n",
" 'gpus': 0,\n",
" 'gpu_name': None,\n",
" 'memory': 2048,\n",
" 'backend_type': 'vm_aws',\n",
" 'coiled_credits_per_hour': 1.0},\n",
" 'c7g.medium': {'name': 'c7g.medium',\n",
" 'cores': 1,\n",
" 'gpus': 0,\n",
" 'gpu_name': None,\n",
" 'memory': 2048,\n",
" 'backend_type': 'vm_aws',\n",
" 'coiled_credits_per_hour': 1.0},\n",
" 'c6gd.medium': {'name': 'c6gd.medium',\n",
" 'cores': 1,\n",
" 'gpus': 0,\n",
" 'gpu_name': None,\n",
" 'memory': 2048,\n",
" 'backend_type': 'vm_aws',\n",
" 'coiled_credits_per_hour': 1.0},\n",
" 'c6gn.medium': {'name': 'c6gn.medium',\n",
" 'cores': 1,\n",
" 'gpus': 0,\n",
" 'gpu_name': None,\n",
" 'memory': 2048,\n",
" 'backend_type': 'vm_aws',\n",
" 'coiled_credits_per_hour': 1.0},\n",
" 'c7gn.medium': {'name': 'c7gn.medium',\n",
" 'cores': 1,\n",
" 'gpus': 0,\n",
" 'gpu_name': None,\n",
" 'memory': 2048,\n",
" 'backend_type': 'vm_aws',\n",
" 'coiled_credits_per_hour': 1.0}}"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"coiled.list_instance_types(\n",
" backend=\"aws\", \n",
" #cores=1, \n",
" memory=\"2 Gib\",\n",
")"
]
},
{
"cell_type": "markdown",
"id": "c1baa7d0-fe64-4082-8034-4e0acbf0a2b0",
"metadata": {},
"source": [
"I have no idea what the difference between these is, but we can just tell Coiled not to be picky and run using any of these that are available."
]
},
{
"cell_type": "markdown",
"id": "60473aba-c852-46fd-90c2-2eab75fb92c4",
"metadata": {},
"source": [
"## Measure reserved memory"
]
},
{
"cell_type": "markdown",
"id": "dff816a7-0f1d-4412-ab86-e91815e4feca",
"metadata": {
"tags": []
},
"source": [
"Before running the real calculation, we should measure the memory required by the Lithops framework itself."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "689090f7-921f-4dbe-872d-8b4f85f60e6e",
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "449e36e382b840cfba40ad5bc7d891d1",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:coiled:Resolving your local Python environment...\n",
"INFO:coiled.package_sync:No existing wheel, creating a wheel for cubed @ /home/tom/Documents/Work/Code/cubed\n",
"INFO:coiled.package_sync:Using wheel @ /tmp/tmpgfw1pup1/cubed/cubed-0.9.0-py3-none-any.whl\n",
"INFO:coiled.package_sync:No existing wheel, creating a wheel for cubed-xarray @ /home/tom/Documents/Work/Code/cubed-xarray\n",
"INFO:coiled.package_sync:Using wheel @ /tmp/tmp0iu327o_/cubed-xarray/cubed_xarray-0.0.4-py3-none-any.whl\n",
"INFO:coiled:Package - cubed, Wheel built from /home/tom/Documents/Work/Code/cubed is missing conftest.py, examples/dataflow/setup.py, setup.py, venv/bin/activate_this.py\n",
"INFO:coiled:Package - cubed-xarray, Wheel built from /home/tom/Documents/Work/Code/cubed-xarray is missing setup.py\n",
"INFO:coiled:Starting upload for cubed-0.9.0-py3-none-any.whl\n",
"INFO:coiled:Starting upload for cubed_xarray-0.0.4-py3-none-any.whl\n"
]
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">╭────────────────────────────────────────── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">Package Info</span> ──────────────────────────────────────────╮\n",
"│ ╷ │\n",
"│ <span style=\"font-weight: bold\"> Package </span>│<span style=\"font-weight: bold\"> Note </span> │\n",
"│ ╶──────────────┼───────────────────────────────────────────────────────────────────────────────╴ │\n",
"│ cubed │ Wheel built from /home/tom/Documents/Work/Code/cubed is missing conftest.py, │\n",
"│ │ examples/dataflow/setup.py, setup.py, venv/bin/activate_this.py │\n",
"│ cubed-xarray │ Wheel built from /home/tom/Documents/Work/Code/cubed-xarray is missing │\n",
"│ │ setup.py │\n",
"│ ╵ │\n",
"╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
"</pre>\n"
],
"text/plain": [
"╭────────────────────────────────────────── \u001b[1;32mPackage Info\u001b[0m ──────────────────────────────────────────╮\n",
"│ ╷ │\n",
"│ \u001b[1m \u001b[0m\u001b[1mPackage \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1mNote \u001b[0m\u001b[1m \u001b[0m │\n",
"│ ╶──────────────┼───────────────────────────────────────────────────────────────────────────────╴ │\n",
"│ cubed │ Wheel built from /home/tom/Documents/Work/Code/cubed is missing conftest.py, │\n",
"│ │ examples/dataflow/setup.py, setup.py, venv/bin/activate_this.py │\n",
"│ cubed-xarray │ Wheel built from /home/tom/Documents/Work/Code/cubed-xarray is missing │\n",
"│ │ setup.py │\n",
"│ ╵ │\n",
"╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:coiled:Creating Cluster (name: run-431e4165, https://cloud.coiled.io/clusters/244098?account=dask ). This usually takes 1-2 minutes...\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "3cc175c55e294a16a598b4d5abaa3be5",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials\n",
"INFO:distributed.deploy.adaptive:Adaptive scaling started: minimum=0 maximum=100\n",
"INFO:coiled:Adaptive scaling up to 1 workers.\n",
"INFO:aiobotocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials\n"
]
}
],
"source": [
"measured_coiled_mem = cubed.measure_reserved_mem(\n",
" executor=coiledexecutor, \n",
" work_dir=\"s3://test-bucket-for-tom\", # bucket coiled people made for me\n",
" memory='2 GiB', # must be greater than allowed_mem, see cubed issue #220\n",
" #cpu=1, # only need one CPU to process one chunk\n",
" account='dask', # use dask maintainers account\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "6108be5d-87ba-4daa-8aa9-2586ec6a118e",
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"We should set the reserved_mem ~= 210.23 MB\n"
]
}
],
"source": [
"print(f\"We should set the reserved_mem ~= {measured_coiled_mem / 1e6:.2f} MB\")"
]
},
{
"cell_type": "markdown",
"id": "223440ef-71aa-44dc-a4e3-29398b9a991a",
"metadata": {},
"source": [
"## Specify allowed memory"
]
},
{
"cell_type": "markdown",
"id": "319e83be-7449-485d-8519-9c09b54a5b3f",
"metadata": {},
"source": [
"Now we can specify the maximum memory we are allowing our calculation to use."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "45507fff-76b7-4376-ae0f-c83be28d4bf3",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"spec = cubed.Spec(\n",
" work_dir=\"s3://test-bucket-for-tom\", # bucket coiled people made for me\n",
" allowed_mem='2GB',\n",
" reserved_mem='210MB',\n",
")"
]
},
{
"cell_type": "markdown",
"id": "3f30105b-9676-45c6-b235-44205b2dff3e",
"metadata": {},
"source": [
"## Define problem"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "8f55d7d7-88ca-4ede-b70e-e664f3672975",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import xarray as xr"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "e496814d-2e69-4b1c-8eb4-e43f56ee3b35",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"def create_cubed_data(t_length):\n",
" return xr.Dataset(\n",
" dict(\n",
" anom_u=([\"time\", \"face\", \"j\", \"i\"], cubed.random.random((t_length, 1, 987, 1920), chunks=(10, 1, -1, -1), spec=spec)),\n",
" anom_v=([\"time\", \"face\", \"j\", \"i\"], cubed.random.random((t_length, 1, 987, 1920), chunks=(10, 1, -1, -1), spec=spec)),\n",
" )\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "dfad6f0b-eb36-47fb-9642-3188b9724b58",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"datasets = {\n",
" '1.5GB': create_cubed_data(50),\n",
" '15GB': create_cubed_data(500),\n",
" '150GB': create_cubed_data(5000),\n",
" '1.5TB': create_cubed_data(50000),\n",
" #'15TB': create_cubed_data(500000),\n",
" #'1.5PB': create_cubed_data(50000000),\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "c8bd3aa9-127c-46d4-a86d-9bd66ada5333",
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1.5 GB dataset, \n",
"1.5e+01 GB dataset, \n",
"1.5e+02 GB dataset, \n",
"1.5e+03 GB dataset, \n"
]
}
],
"source": [
"for scale, ds in datasets.items():\n",
" print(f'{ds.nbytes / 1e9:.2} GB dataset, ')"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "d461c3cd-2846-4c92-9eeb-23da706692ed",
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"<div><svg style=\"position: absolute; width: 0; height: 0; overflow: hidden\">\n",
"<defs>\n",
"<symbol id=\"icon-database\" viewBox=\"0 0 32 32\">\n",
"<path d=\"M16 0c-8.837 0-16 2.239-16 5v4c0 2.761 7.163 5 16 5s16-2.239 16-5v-4c0-2.761-7.163-5-16-5z\"></path>\n",
"<path d=\"M16 17c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z\"></path>\n",
"<path d=\"M16 26c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z\"></path>\n",
"</symbol>\n",
"<symbol id=\"icon-file-text2\" viewBox=\"0 0 32 32\">\n",
"<path d=\"M28.681 7.159c-0.694-0.947-1.662-2.053-2.724-3.116s-2.169-2.030-3.116-2.724c-1.612-1.182-2.393-1.319-2.841-1.319h-15.5c-1.378 0-2.5 1.121-2.5 2.5v27c0 1.378 1.122 2.5 2.5 2.5h23c1.378 0 2.5-1.122 2.5-2.5v-19.5c0-0.448-0.137-1.23-1.319-2.841zM24.543 5.457c0.959 0.959 1.712 1.825 2.268 2.543h-4.811v-4.811c0.718 0.556 1.584 1.309 2.543 2.268zM28 29.5c0 0.271-0.229 0.5-0.5 0.5h-23c-0.271 0-0.5-0.229-0.5-0.5v-27c0-0.271 0.229-0.5 0.5-0.5 0 0 15.499-0 15.5 0v7c0 0.552 0.448 1 1 1h7v19.5z\"></path>\n",
"<path d=\"M23 26h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
"<path d=\"M23 22h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
"<path d=\"M23 18h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
"</symbol>\n",
"</defs>\n",
"</svg>\n",
"<style>/* CSS stylesheet for displaying xarray objects in jupyterlab.\n",
" *\n",
" */\n",
"\n",
":root {\n",
" --xr-font-color0: var(--jp-content-font-color0, rgba(0, 0, 0, 1));\n",
" --xr-font-color2: var(--jp-content-font-color2, rgba(0, 0, 0, 0.54));\n",
" --xr-font-color3: var(--jp-content-font-color3, rgba(0, 0, 0, 0.38));\n",
" --xr-border-color: var(--jp-border-color2, #e0e0e0);\n",
" --xr-disabled-color: var(--jp-layout-color3, #bdbdbd);\n",
" --xr-background-color: var(--jp-layout-color0, white);\n",
" --xr-background-color-row-even: var(--jp-layout-color1, white);\n",
" --xr-background-color-row-odd: var(--jp-layout-color2, #eeeeee);\n",
"}\n",
"\n",
"html[theme=dark],\n",
"body[data-theme=dark],\n",
"body.vscode-dark {\n",
" --xr-font-color0: rgba(255, 255, 255, 1);\n",
" --xr-font-color2: rgba(255, 255, 255, 0.54);\n",
" --xr-font-color3: rgba(255, 255, 255, 0.38);\n",
" --xr-border-color: #1F1F1F;\n",
" --xr-disabled-color: #515151;\n",
" --xr-background-color: #111111;\n",
" --xr-background-color-row-even: #111111;\n",
" --xr-background-color-row-odd: #313131;\n",
"}\n",
"\n",
".xr-wrap {\n",
" display: block !important;\n",
" min-width: 300px;\n",
" max-width: 700px;\n",
"}\n",
"\n",
".xr-text-repr-fallback {\n",
" /* fallback to plain text repr when CSS is not injected (untrusted notebook) */\n",
" display: none;\n",
"}\n",
"\n",
".xr-header {\n",
" padding-top: 6px;\n",
" padding-bottom: 6px;\n",
" margin-bottom: 4px;\n",
" border-bottom: solid 1px var(--xr-border-color);\n",
"}\n",
"\n",
".xr-header > div,\n",
".xr-header > ul {\n",
" display: inline;\n",
" margin-top: 0;\n",
" margin-bottom: 0;\n",
"}\n",
"\n",
".xr-obj-type,\n",
".xr-array-name {\n",
" margin-left: 2px;\n",
" margin-right: 10px;\n",
"}\n",
"\n",
".xr-obj-type {\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-sections {\n",
" padding-left: 0 !important;\n",
" display: grid;\n",
" grid-template-columns: 150px auto auto 1fr 20px 20px;\n",
"}\n",
"\n",
".xr-section-item {\n",
" display: contents;\n",
"}\n",
"\n",
".xr-section-item input {\n",
" display: none;\n",
"}\n",
"\n",
".xr-section-item input + label {\n",
" color: var(--xr-disabled-color);\n",
"}\n",
"\n",
".xr-section-item input:enabled + label {\n",
" cursor: pointer;\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-section-item input:enabled + label:hover {\n",
" color: var(--xr-font-color0);\n",
"}\n",
"\n",
".xr-section-summary {\n",
" grid-column: 1;\n",
" color: var(--xr-font-color2);\n",
" font-weight: 500;\n",
"}\n",
"\n",
".xr-section-summary > span {\n",
" display: inline-block;\n",
" padding-left: 0.5em;\n",
"}\n",
"\n",
".xr-section-summary-in:disabled + label {\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-section-summary-in + label:before {\n",
" display: inline-block;\n",
" content: '►';\n",
" font-size: 11px;\n",
" width: 15px;\n",
" text-align: center;\n",
"}\n",
"\n",
".xr-section-summary-in:disabled + label:before {\n",
" color: var(--xr-disabled-color);\n",
"}\n",
"\n",
".xr-section-summary-in:checked + label:before {\n",
" content: '▼';\n",
"}\n",
"\n",
".xr-section-summary-in:checked + label > span {\n",
" display: none;\n",
"}\n",
"\n",
".xr-section-summary,\n",
".xr-section-inline-details {\n",
" padding-top: 4px;\n",
" padding-bottom: 4px;\n",
"}\n",
"\n",
".xr-section-inline-details {\n",
" grid-column: 2 / -1;\n",
"}\n",
"\n",
".xr-section-details {\n",
" display: none;\n",
" grid-column: 1 / -1;\n",
" margin-bottom: 5px;\n",
"}\n",
"\n",
".xr-section-summary-in:checked ~ .xr-section-details {\n",
" display: contents;\n",
"}\n",
"\n",
".xr-array-wrap {\n",
" grid-column: 1 / -1;\n",
" display: grid;\n",
" grid-template-columns: 20px auto;\n",
"}\n",
"\n",
".xr-array-wrap > label {\n",
" grid-column: 1;\n",
" vertical-align: top;\n",
"}\n",
"\n",
".xr-preview {\n",
" color: var(--xr-font-color3);\n",
"}\n",
"\n",
".xr-array-preview,\n",
".xr-array-data {\n",
" padding: 0 5px !important;\n",
" grid-column: 2;\n",
"}\n",
"\n",
".xr-array-data,\n",
".xr-array-in:checked ~ .xr-array-preview {\n",
" display: none;\n",
"}\n",
"\n",
".xr-array-in:checked ~ .xr-array-data,\n",
".xr-array-preview {\n",
" display: inline-block;\n",
"}\n",
"\n",
".xr-dim-list {\n",
" display: inline-block !important;\n",
" list-style: none;\n",
" padding: 0 !important;\n",
" margin: 0;\n",
"}\n",
"\n",
".xr-dim-list li {\n",
" display: inline-block;\n",
" padding: 0;\n",
" margin: 0;\n",
"}\n",
"\n",
".xr-dim-list:before {\n",
" content: '(';\n",
"}\n",
"\n",
".xr-dim-list:after {\n",
" content: ')';\n",
"}\n",
"\n",
".xr-dim-list li:not(:last-child):after {\n",
" content: ',';\n",
" padding-right: 5px;\n",
"}\n",
"\n",
".xr-has-index {\n",
" font-weight: bold;\n",
"}\n",
"\n",
".xr-var-list,\n",
".xr-var-item {\n",
" display: contents;\n",
"}\n",
"\n",
".xr-var-item > div,\n",
".xr-var-item label,\n",
".xr-var-item > .xr-var-name span {\n",
" background-color: var(--xr-background-color-row-even);\n",
" margin-bottom: 0;\n",
"}\n",
"\n",
".xr-var-item > .xr-var-name:hover span {\n",
" padding-right: 5px;\n",
"}\n",
"\n",
".xr-var-list > li:nth-child(odd) > div,\n",
".xr-var-list > li:nth-child(odd) > label,\n",
".xr-var-list > li:nth-child(odd) > .xr-var-name span {\n",
" background-color: var(--xr-background-color-row-odd);\n",
"}\n",
"\n",
".xr-var-name {\n",
" grid-column: 1;\n",
"}\n",
"\n",
".xr-var-dims {\n",
" grid-column: 2;\n",
"}\n",
"\n",
".xr-var-dtype {\n",
" grid-column: 3;\n",
" text-align: right;\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-var-preview {\n",
" grid-column: 4;\n",
"}\n",
"\n",
".xr-index-preview {\n",
" grid-column: 2 / 5;\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-var-name,\n",
".xr-var-dims,\n",
".xr-var-dtype,\n",
".xr-preview,\n",
".xr-attrs dt {\n",
" white-space: nowrap;\n",
" overflow: hidden;\n",
" text-overflow: ellipsis;\n",
" padding-right: 10px;\n",
"}\n",
"\n",
".xr-var-name:hover,\n",
".xr-var-dims:hover,\n",
".xr-var-dtype:hover,\n",
".xr-attrs dt:hover {\n",
" overflow: visible;\n",
" width: auto;\n",
" z-index: 1;\n",
"}\n",
"\n",
".xr-var-attrs,\n",
".xr-var-data,\n",
".xr-index-data {\n",
" display: none;\n",
" background-color: var(--xr-background-color) !important;\n",
" padding-bottom: 5px !important;\n",
"}\n",
"\n",
".xr-var-attrs-in:checked ~ .xr-var-attrs,\n",
".xr-var-data-in:checked ~ .xr-var-data,\n",
".xr-index-data-in:checked ~ .xr-index-data {\n",
" display: block;\n",
"}\n",
"\n",
".xr-var-data > table {\n",
" float: right;\n",
"}\n",
"\n",
".xr-var-name span,\n",
".xr-var-data,\n",
".xr-index-name div,\n",
".xr-index-data,\n",
".xr-attrs {\n",
" padding-left: 25px !important;\n",
"}\n",
"\n",
".xr-attrs,\n",
".xr-var-attrs,\n",
".xr-var-data,\n",
".xr-index-data {\n",
" grid-column: 1 / -1;\n",
"}\n",
"\n",
"dl.xr-attrs {\n",
" padding: 0;\n",
" margin: 0;\n",
" display: grid;\n",
" grid-template-columns: 125px auto;\n",
"}\n",
"\n",
".xr-attrs dt,\n",
".xr-attrs dd {\n",
" padding: 0;\n",
" margin: 0;\n",
" float: left;\n",
" padding-right: 10px;\n",
" width: auto;\n",
"}\n",
"\n",
".xr-attrs dt {\n",
" font-weight: normal;\n",
" grid-column: 1;\n",
"}\n",
"\n",
".xr-attrs dt:hover span {\n",
" display: inline-block;\n",
" background: var(--xr-background-color);\n",
" padding-right: 10px;\n",
"}\n",
"\n",
".xr-attrs dd {\n",
" grid-column: 2;\n",
" white-space: pre-wrap;\n",
" word-break: break-all;\n",
"}\n",
"\n",
".xr-icon-database,\n",
".xr-icon-file-text2,\n",
".xr-no-icon {\n",
" display: inline-block;\n",
" vertical-align: middle;\n",
" width: 1em;\n",
" height: 1.5em !important;\n",
" stroke-width: 0;\n",
" stroke: currentColor;\n",
" fill: currentColor;\n",
"}\n",
"</style><pre class='xr-text-repr-fallback'>&lt;xarray.Dataset&gt;\n",
"Dimensions: (time: 50000, face: 1, j: 987, i: 1920)\n",
"Dimensions without coordinates: time, face, j, i\n",
"Data variables:\n",
" anom_u (time, face, j, i) float64 cubed.Array&lt;chunksize=(10, 1, 987, 1920)&gt;\n",
" anom_v (time, face, j, i) float64 cubed.Array&lt;chunksize=(10, 1, 987, 1920)&gt;</pre><div class='xr-wrap' style='display:none'><div class='xr-header'><div class='xr-obj-type'>xarray.Dataset</div></div><ul class='xr-sections'><li class='xr-section-item'><input id='section-aa031435-a423-4286-96fd-50e7098cc717' class='xr-section-summary-in' type='checkbox' disabled ><label for='section-aa031435-a423-4286-96fd-50e7098cc717' class='xr-section-summary' title='Expand/collapse section'>Dimensions:</label><div class='xr-section-inline-details'><ul class='xr-dim-list'><li><span>time</span>: 50000</li><li><span>face</span>: 1</li><li><span>j</span>: 987</li><li><span>i</span>: 1920</li></ul></div><div class='xr-section-details'></div></li><li class='xr-section-item'><input id='section-1c577d3e-46f6-4907-b25a-7d3ebd1ede88' class='xr-section-summary-in' type='checkbox' disabled ><label for='section-1c577d3e-46f6-4907-b25a-7d3ebd1ede88' class='xr-section-summary' title='Expand/collapse section'>Coordinates: <span>(0)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'></ul></div></li><li class='xr-section-item'><input id='section-15c867e9-4203-4428-9373-355a669b0c4e' class='xr-section-summary-in' type='checkbox' checked><label for='section-15c867e9-4203-4428-9373-355a669b0c4e' class='xr-section-summary' >Data variables: <span>(2)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'><li class='xr-var-item'><div class='xr-var-name'><span>anom_u</span></div><div class='xr-var-dims'>(time, face, j, i)</div><div class='xr-var-dtype'>float64</div><div class='xr-var-preview xr-preview'>cubed.Array&lt;chunksize=(10, 1, 987, 1920)&gt;</div><input id='attrs-b396b116-61de-418b-85dc-9322a5044256' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-b396b116-61de-418b-85dc-9322a5044256' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-009ad6b6-b0bd-4cdf-86bc-0c0a279a5bab' class='xr-var-data-in' type='checkbox'><label for='data-009ad6b6-b0bd-4cdf-86bc-0c0a279a5bab' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><table>\n",
" <tr>\n",
" <td>\n",
" <table>\n",
" <thead>\n",
" <tr>\n",
" <td> </td>\n",
" <th> Array </th>\n",
" <th> Chunk </th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" \n",
" <tr>\n",
" <th> Bytes </th>\n",
" <td> 758.0 GB </td>\n",
" <td> 151.6 MB </td>\n",
" </tr>\n",
" \n",
" <tr>\n",
" <th> Shape </th>\n",
" <td> (50000, 1, 987, 1920) </td>\n",
" <td> (10, 1, 987, 1920) </td>\n",
" </tr>\n",
" <tr>\n",
" <th> Count </th>\n",
" <td> 3 arrays in Plan </td>\n",
" <td> 5000 Chunks </td>\n",
" </tr>\n",
" <tr>\n",
" <th> Type </th>\n",
" <td> float64 </td>\n",
" <td> np.ndarray </td>\n",
" </tr>\n",
" </tbody>\n",
" </table>\n",
" </td>\n",
" <td>\n",
" <svg width=\"477\" height=\"93\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
"\n",
" <!-- Horizontal lines -->\n",
" <line x1=\"0\" y1=\"0\" x2=\"120\" y2=\"0\" style=\"stroke-width:2\" />\n",
" <line x1=\"0\" y1=\"25\" x2=\"120\" y2=\"25\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Vertical lines -->\n",
" <line x1=\"0\" y1=\"0\" x2=\"0\" y2=\"25\" style=\"stroke-width:2\" />\n",
" <line x1=\"3\" y1=\"0\" x2=\"3\" y2=\"25\" />\n",
" <line x1=\"7\" y1=\"0\" x2=\"7\" y2=\"25\" />\n",
" <line x1=\"11\" y1=\"0\" x2=\"11\" y2=\"25\" />\n",
" <line x1=\"15\" y1=\"0\" x2=\"15\" y2=\"25\" />\n",
" <line x1=\"18\" y1=\"0\" x2=\"18\" y2=\"25\" />\n",
" <line x1=\"22\" y1=\"0\" x2=\"22\" y2=\"25\" />\n",
" <line x1=\"26\" y1=\"0\" x2=\"26\" y2=\"25\" />\n",
" <line x1=\"30\" y1=\"0\" x2=\"30\" y2=\"25\" />\n",
" <line x1=\"33\" y1=\"0\" x2=\"33\" y2=\"25\" />\n",
" <line x1=\"37\" y1=\"0\" x2=\"37\" y2=\"25\" />\n",
" <line x1=\"41\" y1=\"0\" x2=\"41\" y2=\"25\" />\n",
" <line x1=\"45\" y1=\"0\" x2=\"45\" y2=\"25\" />\n",
" <line x1=\"48\" y1=\"0\" x2=\"48\" y2=\"25\" />\n",
" <line x1=\"52\" y1=\"0\" x2=\"52\" y2=\"25\" />\n",
" <line x1=\"56\" y1=\"0\" x2=\"56\" y2=\"25\" />\n",
" <line x1=\"60\" y1=\"0\" x2=\"60\" y2=\"25\" />\n",
" <line x1=\"63\" y1=\"0\" x2=\"63\" y2=\"25\" />\n",
" <line x1=\"67\" y1=\"0\" x2=\"67\" y2=\"25\" />\n",
" <line x1=\"71\" y1=\"0\" x2=\"71\" y2=\"25\" />\n",
" <line x1=\"75\" y1=\"0\" x2=\"75\" y2=\"25\" />\n",
" <line x1=\"78\" y1=\"0\" x2=\"78\" y2=\"25\" />\n",
" <line x1=\"82\" y1=\"0\" x2=\"82\" y2=\"25\" />\n",
" <line x1=\"86\" y1=\"0\" x2=\"86\" y2=\"25\" />\n",
" <line x1=\"90\" y1=\"0\" x2=\"90\" y2=\"25\" />\n",
" <line x1=\"93\" y1=\"0\" x2=\"93\" y2=\"25\" />\n",
" <line x1=\"97\" y1=\"0\" x2=\"97\" y2=\"25\" />\n",
" <line x1=\"101\" y1=\"0\" x2=\"101\" y2=\"25\" />\n",
" <line x1=\"105\" y1=\"0\" x2=\"105\" y2=\"25\" />\n",
" <line x1=\"108\" y1=\"0\" x2=\"108\" y2=\"25\" />\n",
" <line x1=\"112\" y1=\"0\" x2=\"112\" y2=\"25\" />\n",
" <line x1=\"116\" y1=\"0\" x2=\"116\" y2=\"25\" />\n",
" <line x1=\"120\" y1=\"0\" x2=\"120\" y2=\"25\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Colored Rectangle -->\n",
" <polygon points=\"0.0,0.0 120.0,0.0 120.0,25.412616514582485 0.0,25.412616514582485\" style=\"fill:#8B4903A0;stroke-width:0\"/>\n",
"\n",
" <!-- Text -->\n",
" <text x=\"60.000000\" y=\"45.412617\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >50000</text>\n",
" <text x=\"140.000000\" y=\"12.706308\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(0,140.000000,12.706308)\">1</text>\n",
"\n",
"\n",
" <!-- Horizontal lines -->\n",
" <line x1=\"190\" y1=\"0\" x2=\"204\" y2=\"14\" style=\"stroke-width:2\" />\n",
" <line x1=\"190\" y1=\"28\" x2=\"204\" y2=\"43\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Vertical lines -->\n",
" <line x1=\"190\" y1=\"0\" x2=\"190\" y2=\"28\" style=\"stroke-width:2\" />\n",
" <line x1=\"204\" y1=\"14\" x2=\"204\" y2=\"43\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Colored Rectangle -->\n",
" <polygon points=\"190.0,0.0 204.9485979497544,14.948597949754403 204.9485979497544,43.90568081847956 190.0,28.95708286872516\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
"\n",
" <!-- Horizontal lines -->\n",
" <line x1=\"190\" y1=\"0\" x2=\"222\" y2=\"0\" style=\"stroke-width:2\" />\n",
" <line x1=\"204\" y1=\"14\" x2=\"237\" y2=\"14\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Vertical lines -->\n",
" <line x1=\"190\" y1=\"0\" x2=\"204\" y2=\"14\" style=\"stroke-width:2\" />\n",
" <line x1=\"222\" y1=\"0\" x2=\"237\" y2=\"14\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Colored Rectangle -->\n",
" <polygon points=\"190.0,0.0 222.88444295879845,0.0 237.83304090855285,14.948597949754403 204.9485979497544,14.948597949754403\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
"\n",
" <!-- Horizontal lines -->\n",
" <line x1=\"204\" y1=\"14\" x2=\"237\" y2=\"14\" style=\"stroke-width:2\" />\n",
" <line x1=\"204\" y1=\"43\" x2=\"237\" y2=\"43\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Vertical lines -->\n",
" <line x1=\"204\" y1=\"14\" x2=\"204\" y2=\"43\" style=\"stroke-width:2\" />\n",
" <line x1=\"237\" y1=\"14\" x2=\"237\" y2=\"43\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Colored Rectangle -->\n",
" <polygon points=\"204.9485979497544,14.948597949754403 237.83304090855285,14.948597949754403 237.83304090855285,43.90568081847956 204.9485979497544,43.90568081847956\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
"\n",
" <!-- Text -->\n",
" <text x=\"221.390819\" y=\"63.905681\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >1920</text>\n",
" <text x=\"257.833041\" y=\"29.427139\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(-90,257.833041,29.427139)\">987</text>\n",
" <text x=\"187.474299\" y=\"56.431382\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(45,187.474299,56.431382)\">1</text>\n",
"</svg>\n",
" </td>\n",
" </tr>\n",
"</table></div></li><li class='xr-var-item'><div class='xr-var-name'><span>anom_v</span></div><div class='xr-var-dims'>(time, face, j, i)</div><div class='xr-var-dtype'>float64</div><div class='xr-var-preview xr-preview'>cubed.Array&lt;chunksize=(10, 1, 987, 1920)&gt;</div><input id='attrs-9cf410b2-45f2-4f9e-baa8-0fa9e3889265' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-9cf410b2-45f2-4f9e-baa8-0fa9e3889265' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-1d3a46c4-49db-41e7-bc23-ca92cefce9ff' class='xr-var-data-in' type='checkbox'><label for='data-1d3a46c4-49db-41e7-bc23-ca92cefce9ff' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><table>\n",
" <tr>\n",
" <td>\n",
" <table>\n",
" <thead>\n",
" <tr>\n",
" <td> </td>\n",
" <th> Array </th>\n",
" <th> Chunk </th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" \n",
" <tr>\n",
" <th> Bytes </th>\n",
" <td> 758.0 GB </td>\n",
" <td> 151.6 MB </td>\n",
" </tr>\n",
" \n",
" <tr>\n",
" <th> Shape </th>\n",
" <td> (50000, 1, 987, 1920) </td>\n",
" <td> (10, 1, 987, 1920) </td>\n",
" </tr>\n",
" <tr>\n",
" <th> Count </th>\n",
" <td> 3 arrays in Plan </td>\n",
" <td> 5000 Chunks </td>\n",
" </tr>\n",
" <tr>\n",
" <th> Type </th>\n",
" <td> float64 </td>\n",
" <td> np.ndarray </td>\n",
" </tr>\n",
" </tbody>\n",
" </table>\n",
" </td>\n",
" <td>\n",
" <svg width=\"477\" height=\"93\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
"\n",
" <!-- Horizontal lines -->\n",
" <line x1=\"0\" y1=\"0\" x2=\"120\" y2=\"0\" style=\"stroke-width:2\" />\n",
" <line x1=\"0\" y1=\"25\" x2=\"120\" y2=\"25\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Vertical lines -->\n",
" <line x1=\"0\" y1=\"0\" x2=\"0\" y2=\"25\" style=\"stroke-width:2\" />\n",
" <line x1=\"3\" y1=\"0\" x2=\"3\" y2=\"25\" />\n",
" <line x1=\"7\" y1=\"0\" x2=\"7\" y2=\"25\" />\n",
" <line x1=\"11\" y1=\"0\" x2=\"11\" y2=\"25\" />\n",
" <line x1=\"15\" y1=\"0\" x2=\"15\" y2=\"25\" />\n",
" <line x1=\"18\" y1=\"0\" x2=\"18\" y2=\"25\" />\n",
" <line x1=\"22\" y1=\"0\" x2=\"22\" y2=\"25\" />\n",
" <line x1=\"26\" y1=\"0\" x2=\"26\" y2=\"25\" />\n",
" <line x1=\"30\" y1=\"0\" x2=\"30\" y2=\"25\" />\n",
" <line x1=\"33\" y1=\"0\" x2=\"33\" y2=\"25\" />\n",
" <line x1=\"37\" y1=\"0\" x2=\"37\" y2=\"25\" />\n",
" <line x1=\"41\" y1=\"0\" x2=\"41\" y2=\"25\" />\n",
" <line x1=\"45\" y1=\"0\" x2=\"45\" y2=\"25\" />\n",
" <line x1=\"48\" y1=\"0\" x2=\"48\" y2=\"25\" />\n",
" <line x1=\"52\" y1=\"0\" x2=\"52\" y2=\"25\" />\n",
" <line x1=\"56\" y1=\"0\" x2=\"56\" y2=\"25\" />\n",
" <line x1=\"60\" y1=\"0\" x2=\"60\" y2=\"25\" />\n",
" <line x1=\"63\" y1=\"0\" x2=\"63\" y2=\"25\" />\n",
" <line x1=\"67\" y1=\"0\" x2=\"67\" y2=\"25\" />\n",
" <line x1=\"71\" y1=\"0\" x2=\"71\" y2=\"25\" />\n",
" <line x1=\"75\" y1=\"0\" x2=\"75\" y2=\"25\" />\n",
" <line x1=\"78\" y1=\"0\" x2=\"78\" y2=\"25\" />\n",
" <line x1=\"82\" y1=\"0\" x2=\"82\" y2=\"25\" />\n",
" <line x1=\"86\" y1=\"0\" x2=\"86\" y2=\"25\" />\n",
" <line x1=\"90\" y1=\"0\" x2=\"90\" y2=\"25\" />\n",
" <line x1=\"93\" y1=\"0\" x2=\"93\" y2=\"25\" />\n",
" <line x1=\"97\" y1=\"0\" x2=\"97\" y2=\"25\" />\n",
" <line x1=\"101\" y1=\"0\" x2=\"101\" y2=\"25\" />\n",
" <line x1=\"105\" y1=\"0\" x2=\"105\" y2=\"25\" />\n",
" <line x1=\"108\" y1=\"0\" x2=\"108\" y2=\"25\" />\n",
" <line x1=\"112\" y1=\"0\" x2=\"112\" y2=\"25\" />\n",
" <line x1=\"116\" y1=\"0\" x2=\"116\" y2=\"25\" />\n",
" <line x1=\"120\" y1=\"0\" x2=\"120\" y2=\"25\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Colored Rectangle -->\n",
" <polygon points=\"0.0,0.0 120.0,0.0 120.0,25.412616514582485 0.0,25.412616514582485\" style=\"fill:#8B4903A0;stroke-width:0\"/>\n",
"\n",
" <!-- Text -->\n",
" <text x=\"60.000000\" y=\"45.412617\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >50000</text>\n",
" <text x=\"140.000000\" y=\"12.706308\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(0,140.000000,12.706308)\">1</text>\n",
"\n",
"\n",
" <!-- Horizontal lines -->\n",
" <line x1=\"190\" y1=\"0\" x2=\"204\" y2=\"14\" style=\"stroke-width:2\" />\n",
" <line x1=\"190\" y1=\"28\" x2=\"204\" y2=\"43\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Vertical lines -->\n",
" <line x1=\"190\" y1=\"0\" x2=\"190\" y2=\"28\" style=\"stroke-width:2\" />\n",
" <line x1=\"204\" y1=\"14\" x2=\"204\" y2=\"43\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Colored Rectangle -->\n",
" <polygon points=\"190.0,0.0 204.9485979497544,14.948597949754403 204.9485979497544,43.90568081847956 190.0,28.95708286872516\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
"\n",
" <!-- Horizontal lines -->\n",
" <line x1=\"190\" y1=\"0\" x2=\"222\" y2=\"0\" style=\"stroke-width:2\" />\n",
" <line x1=\"204\" y1=\"14\" x2=\"237\" y2=\"14\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Vertical lines -->\n",
" <line x1=\"190\" y1=\"0\" x2=\"204\" y2=\"14\" style=\"stroke-width:2\" />\n",
" <line x1=\"222\" y1=\"0\" x2=\"237\" y2=\"14\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Colored Rectangle -->\n",
" <polygon points=\"190.0,0.0 222.88444295879845,0.0 237.83304090855285,14.948597949754403 204.9485979497544,14.948597949754403\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
"\n",
" <!-- Horizontal lines -->\n",
" <line x1=\"204\" y1=\"14\" x2=\"237\" y2=\"14\" style=\"stroke-width:2\" />\n",
" <line x1=\"204\" y1=\"43\" x2=\"237\" y2=\"43\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Vertical lines -->\n",
" <line x1=\"204\" y1=\"14\" x2=\"204\" y2=\"43\" style=\"stroke-width:2\" />\n",
" <line x1=\"237\" y1=\"14\" x2=\"237\" y2=\"43\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Colored Rectangle -->\n",
" <polygon points=\"204.9485979497544,14.948597949754403 237.83304090855285,14.948597949754403 237.83304090855285,43.90568081847956 204.9485979497544,43.90568081847956\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
"\n",
" <!-- Text -->\n",
" <text x=\"221.390819\" y=\"63.905681\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >1920</text>\n",
" <text x=\"257.833041\" y=\"29.427139\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(-90,257.833041,29.427139)\">987</text>\n",
" <text x=\"187.474299\" y=\"56.431382\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(45,187.474299,56.431382)\">1</text>\n",
"</svg>\n",
" </td>\n",
" </tr>\n",
"</table></div></li></ul></div></li><li class='xr-section-item'><input id='section-3ad67333-463d-47db-9cbc-22b444f64a6b' class='xr-section-summary-in' type='checkbox' disabled ><label for='section-3ad67333-463d-47db-9cbc-22b444f64a6b' class='xr-section-summary' title='Expand/collapse section'>Indexes: <span>(0)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'></ul></div></li><li class='xr-section-item'><input id='section-4c4c5f65-8441-4baf-967b-2e4643d4ef2b' class='xr-section-summary-in' type='checkbox' disabled ><label for='section-4c4c5f65-8441-4baf-967b-2e4643d4ef2b' class='xr-section-summary' title='Expand/collapse section'>Attributes: <span>(0)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><dl class='xr-attrs'></dl></div></li></ul></div></div>"
],
"text/plain": [
"<xarray.Dataset>\n",
"Dimensions: (time: 50000, face: 1, j: 987, i: 1920)\n",
"Dimensions without coordinates: time, face, j, i\n",
"Data variables:\n",
" anom_u (time, face, j, i) float64 cubed.Array<chunksize=(10, 1, 987, 1920)>\n",
" anom_v (time, face, j, i) float64 cubed.Array<chunksize=(10, 1, 987, 1920)>"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"datasets['1.5TB']"
]
},
{
"cell_type": "markdown",
"id": "a702c2a3-88b7-418f-a0bf-a00587b6bb5e",
"metadata": {},
"source": [
"You can see we have an xarray object backed by `cubed.Array` objects."
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "03887772-1c62-417f-ba0d-52908385079d",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"def quadratic_means(ds):\n",
" quadratic_products = ds**2\n",
" quadratic_products[\"uv\"] = ds.anom_u * ds.anom_v\n",
" mean = quadratic_products.mean(\"time\", skipna=False) # skipna=False to avoid eager load, see xarray issue #7243\n",
" return mean"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "fab38a76-86ef-4696-8c50-af9dc11af641",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"workloads = {scale: quadratic_means(ds) for scale, ds in datasets.items()}"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "d2d5d639-f5c2-43a4-aa07-619c9a225d07",
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"image/svg+xml": [
"<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"420pt\" height=\"366pt\" viewBox=\"0.00 0.00 420.25 365.50\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 361.5)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-361.5 416.25,-361.5 416.25,4 -4,4\"/>\n",
"<text text-anchor=\"middle\" x=\"86.75\" y=\"-20\" font-family=\"Times,serif\" font-size=\"10.00\">num tasks: 24</text>\n",
"<text text-anchor=\"middle\" x=\"86.75\" y=\"-7.25\" font-family=\"Times,serif\" font-size=\"10.00\">max projected memory: 2.0 GB</text>\n",
"<!-- array&#45;003 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>array-003</title>\n",
"<g id=\"a_node1\"><a xlink:title=\"name: array-003\n",
"shape: (50, 1, 987, 1920)\n",
"chunks: (10, 1, 987, 1920)\n",
"dtype: float64\n",
"chunk memory: 151.6 MB\n",
"\n",
"calls: &lt;module&gt; -&gt; create_cubed_data -&gt; random -&gt; map_direct -&gt; empty -&gt; full\n",
"line: 4 in create_cubed_data\">\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"63.25,-357.5 0,-357.5 0,-321.5 63.25,-321.5 63.25,-357.5\"/>\n",
"<text text-anchor=\"middle\" x=\"31.62\" y=\"-342.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-003</text>\n",
"<text text-anchor=\"middle\" x=\"31.62\" y=\"-330\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">random </text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;005 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>array-005</title>\n",
"<g id=\"a_node3\"><a xlink:title=\"name: array-005\n",
"shape: (50, 1, 987, 1920)\n",
"chunks: (10, 1, 987, 1920)\n",
"dtype: float64\n",
"chunk memory: 151.6 MB\n",
"\n",
"projected memory: 816.4 MB\n",
"tasks: 5\n",
"calls: &lt;module&gt; -&gt; create_cubed_data -&gt; random -&gt; map_direct -&gt; map_blocks -&gt; _map_blocks -&gt; blockwise\n",
"line: 4 in create_cubed_data\">\n",
"<polygon fill=\"#dcbeff\" stroke=\"black\" points=\"147.75,-285.5 69.5,-285.5 69.5,-249.5 147.75,-249.5 147.75,-285.5\"/>\n",
"<text text-anchor=\"middle\" x=\"108.62\" y=\"-270.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-005</text>\n",
"<text text-anchor=\"middle\" x=\"108.62\" y=\"-258\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">random (bw)</text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;003&#45;&gt;array&#45;005 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>array-003-&gt;array-005</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M50.66,-321.2C60.01,-312.7 71.43,-302.31 81.66,-293.01\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"83.5,-296.16 88.54,-286.85 78.79,-290.98 83.5,-296.16\"/>\n",
"</g>\n",
"<!-- array&#45;004 -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>array-004</title>\n",
"<g id=\"a_node2\"><a xlink:title=\"name: array-004\n",
"shape: (5, 1, 1, 1)\n",
"chunks: (1, 1, 1, 1)\n",
"dtype: int32\n",
"chunk memory: 4 bytes\n",
"\n",
"calls: &lt;module&gt; -&gt; create_cubed_data -&gt; random -&gt; map_direct -&gt; map_blocks -&gt; offsets_array\n",
"line: 4 in create_cubed_data\">\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"144.25,-357.5 81,-357.5 81,-321.5 144.25,-321.5 144.25,-357.5\"/>\n",
"<text text-anchor=\"middle\" x=\"112.62\" y=\"-342.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-004</text>\n",
"<text text-anchor=\"middle\" x=\"112.62\" y=\"-330\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">random </text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;004&#45;&gt;array&#45;005 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>array-004-&gt;array-005</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M111.64,-321.2C111.21,-313.74 110.7,-304.82 110.22,-296.47\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"113.67,-296.39 109.6,-286.6 106.68,-296.79 113.67,-296.39\"/>\n",
"</g>\n",
"<!-- array&#45;042 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>array-042</title>\n",
"<g id=\"a_node7\"><a xlink:title=\"name: array-042\n",
"shape: (5, 1, 987, 1920)\n",
"chunks: (1, 1, 987, 1920)\n",
"dtype: [('n', '&lt;i8'), ('total', '&lt;f8')]\n",
"chunk memory: 30.3 MB\n",
"\n",
"projected memory: 1.1 GB\n",
"tasks: 5\n",
"calls: &lt;dictcomp&gt; -&gt; quadratic_means -&gt; mean -&gt; reduction -&gt; blockwise\n",
"line: 382 in f\">\n",
"<polygon fill=\"#dcbeff\" stroke=\"black\" points=\"190.88,-213.5 122.38,-213.5 122.38,-177.5 190.88,-177.5 190.88,-213.5\"/>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-198.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-042</text>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-186\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">mean (bw)</text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;005&#45;&gt;array&#45;042 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>array-005-&gt;array-042</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M120.49,-249.2C126.01,-241.14 132.7,-231.39 138.81,-222.48\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"142.13,-224.83 144.9,-214.6 136.35,-220.87 142.13,-224.83\"/>\n",
"</g>\n",
"<!-- array&#45;006 -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>array-006</title>\n",
"<g id=\"a_node4\"><a xlink:title=\"name: array-006\n",
"shape: (50, 1, 987, 1920)\n",
"chunks: (10, 1, 987, 1920)\n",
"dtype: float64\n",
"chunk memory: 151.6 MB\n",
"\n",
"calls: &lt;module&gt; -&gt; create_cubed_data -&gt; random -&gt; map_direct -&gt; empty -&gt; full\n",
"line: 5 in create_cubed_data\">\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"232.25,-357.5 169,-357.5 169,-321.5 232.25,-321.5 232.25,-357.5\"/>\n",
"<text text-anchor=\"middle\" x=\"200.62\" y=\"-342.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-006</text>\n",
"<text text-anchor=\"middle\" x=\"200.62\" y=\"-330\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">random </text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;008 -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>array-008</title>\n",
"<g id=\"a_node6\"><a xlink:title=\"name: array-008\n",
"shape: (50, 1, 987, 1920)\n",
"chunks: (10, 1, 987, 1920)\n",
"dtype: float64\n",
"chunk memory: 151.6 MB\n",
"\n",
"projected memory: 816.4 MB\n",
"tasks: 5\n",
"calls: &lt;module&gt; -&gt; create_cubed_data -&gt; random -&gt; map_direct -&gt; map_blocks -&gt; _map_blocks -&gt; blockwise\n",
"line: 5 in create_cubed_data\">\n",
"<polygon fill=\"#dcbeff\" stroke=\"black\" points=\"243.75,-285.5 165.5,-285.5 165.5,-249.5 243.75,-249.5 243.75,-285.5\"/>\n",
"<text text-anchor=\"middle\" x=\"204.62\" y=\"-270.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-008</text>\n",
"<text text-anchor=\"middle\" x=\"204.62\" y=\"-258\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">random (bw)</text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;006&#45;&gt;array&#45;008 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>array-006-&gt;array-008</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M201.61,-321.2C202.04,-313.74 202.55,-304.82 203.03,-296.47\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"206.57,-296.79 203.65,-286.6 199.58,-296.39 206.57,-296.79\"/>\n",
"</g>\n",
"<!-- array&#45;007 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>array-007</title>\n",
"<g id=\"a_node5\"><a xlink:title=\"name: array-007\n",
"shape: (5, 1, 1, 1)\n",
"chunks: (1, 1, 1, 1)\n",
"dtype: int32\n",
"chunk memory: 4 bytes\n",
"\n",
"calls: &lt;module&gt; -&gt; create_cubed_data -&gt; random -&gt; map_direct -&gt; map_blocks -&gt; offsets_array\n",
"line: 5 in create_cubed_data\">\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"313.25,-357.5 250,-357.5 250,-321.5 313.25,-321.5 313.25,-357.5\"/>\n",
"<text text-anchor=\"middle\" x=\"281.62\" y=\"-342.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-007</text>\n",
"<text text-anchor=\"middle\" x=\"281.62\" y=\"-330\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">random </text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;007&#45;&gt;array&#45;008 -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>array-007-&gt;array-008</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M262.59,-321.2C253.24,-312.7 241.82,-302.31 231.59,-293.01\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"234.46,-290.98 224.71,-286.85 229.75,-296.16 234.46,-290.98\"/>\n",
"</g>\n",
"<!-- array&#45;008&#45;&gt;array&#45;042 -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>array-008-&gt;array-042</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M192.76,-249.2C187.24,-241.14 180.55,-231.39 174.44,-222.48\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"176.9,-220.87 168.35,-214.6 171.12,-224.83 176.9,-220.87\"/>\n",
"</g>\n",
"<!-- array&#45;043 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>array-043</title>\n",
"<g id=\"a_node8\"><a xlink:title=\"name: array-043\n",
"shape: (5, 1, 987, 1920)\n",
"chunks: (5, 1, 987, 1920)\n",
"dtype: [('n', '&lt;i8'), ('total', '&lt;f8')]\n",
"chunk memory: 151.6 MB\n",
"\n",
"projected memory: 2.0 GB\n",
"tasks: 1\n",
"write chunks: (5, 1, 987, 1920)\n",
"calls: &lt;dictcomp&gt; -&gt; quadratic_means -&gt; mean -&gt; reduction -&gt; rechunk\n",
"line: 382 in f\">\n",
"<polygon fill=\"#aaffc3\" stroke=\"black\" points=\"188.25,-141.5 125,-141.5 125,-105.5 188.25,-105.5 188.25,-141.5\"/>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-126.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-043</text>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-114\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">mean (rc)</text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;042&#45;&gt;array&#45;043 -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>array-042-&gt;array-043</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M156.62,-177.2C156.62,-169.74 156.62,-160.82 156.62,-152.47\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"160.13,-152.6 156.62,-142.6 153.13,-152.6 160.13,-152.6\"/>\n",
"</g>\n",
"<!-- array&#45;046 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>array-046</title>\n",
"<g id=\"a_node9\"><a xlink:title=\"name: array-046\n",
"shape: (1, 987, 1920)\n",
"chunks: (1, 987, 1920)\n",
"dtype: float64\n",
"chunk memory: 15.2 MB\n",
"\n",
"projected memory: 573.8 MB\n",
"tasks: 1\n",
"calls: mean -&gt; reduction -&gt; squeeze -&gt; map_blocks -&gt; _map_blocks -&gt; blockwise\n",
"line: 382 in f\">\n",
"<polygon fill=\"#dcbeff\" stroke=\"black\" points=\"190.88,-69.5 122.38,-69.5 122.38,-33.5 190.88,-33.5 190.88,-69.5\"/>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-54.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-046</text>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-42\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">mean (bw)</text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;043&#45;&gt;array&#45;046 -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>array-043-&gt;array-046</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M156.62,-105.2C156.62,-97.74 156.62,-88.82 156.62,-80.47\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"160.13,-80.6 156.62,-70.6 153.13,-80.6 160.13,-80.6\"/>\n",
"</g>\n",
"<!-- create&#45;arrays -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>create-arrays</title>\n",
"<g id=\"a_node10\"><a xlink:title=\"projected memory: 210.0 MB\n",
"tasks: 7\">\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"412.25,-357.5 331,-357.5 331,-321.5 412.25,-321.5 412.25,-357.5\"/>\n",
"<text text-anchor=\"middle\" x=\"371.62\" y=\"-336.38\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">create-arrays</text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"</g>\n",
"</svg>"
],
"text/plain": [
"<IPython.core.display.SVG object>"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"workloads['1.5GB']['uv'].data.visualize()"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "1fc776b4-7ecc-4663-b721-6f7c4a94a875",
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"image/svg+xml": [
"<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"420pt\" height=\"798pt\" viewBox=\"0.00 0.00 420.25 797.50\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 793.5)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-793.5 416.25,-793.5 416.25,4 -4,4\"/>\n",
"<text text-anchor=\"middle\" x=\"86.75\" y=\"-20\" font-family=\"Times,serif\" font-size=\"10.00\">num tasks: 15787</text>\n",
"<text text-anchor=\"middle\" x=\"86.75\" y=\"-7.25\" font-family=\"Times,serif\" font-size=\"10.00\">max projected memory: 2.0 GB</text>\n",
"<!-- array&#45;021 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>array-021</title>\n",
"<g id=\"a_node1\"><a xlink:title=\"name: array-021\n",
"shape: (50000, 1, 987, 1920)\n",
"chunks: (10, 1, 987, 1920)\n",
"dtype: float64\n",
"chunk memory: 151.6 MB\n",
"\n",
"calls: &lt;module&gt; -&gt; create_cubed_data -&gt; random -&gt; map_direct -&gt; empty -&gt; full\n",
"line: 4 in create_cubed_data\">\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"63.25,-789.5 0,-789.5 0,-753.5 63.25,-753.5 63.25,-789.5\"/>\n",
"<text text-anchor=\"middle\" x=\"31.62\" y=\"-774.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-021</text>\n",
"<text text-anchor=\"middle\" x=\"31.62\" y=\"-762\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">random </text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;023 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>array-023</title>\n",
"<g id=\"a_node3\"><a xlink:title=\"name: array-023\n",
"shape: (50000, 1, 987, 1920)\n",
"chunks: (10, 1, 987, 1920)\n",
"dtype: float64\n",
"chunk memory: 151.6 MB\n",
"\n",
"projected memory: 816.4 MB\n",
"tasks: 5000\n",
"calls: &lt;module&gt; -&gt; create_cubed_data -&gt; random -&gt; map_direct -&gt; map_blocks -&gt; _map_blocks -&gt; blockwise\n",
"line: 4 in create_cubed_data\">\n",
"<polygon fill=\"#dcbeff\" stroke=\"black\" points=\"147.75,-717.5 69.5,-717.5 69.5,-681.5 147.75,-681.5 147.75,-717.5\"/>\n",
"<text text-anchor=\"middle\" x=\"108.62\" y=\"-702.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-023</text>\n",
"<text text-anchor=\"middle\" x=\"108.62\" y=\"-690\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">random (bw)</text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;021&#45;&gt;array&#45;023 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>array-021-&gt;array-023</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M50.66,-753.2C60.01,-744.7 71.43,-734.31 81.66,-725.01\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"83.5,-728.16 88.54,-718.85 78.79,-722.98 83.5,-728.16\"/>\n",
"</g>\n",
"<!-- array&#45;022 -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>array-022</title>\n",
"<g id=\"a_node2\"><a xlink:title=\"name: array-022\n",
"shape: (5000, 1, 1, 1)\n",
"chunks: (1, 1, 1, 1)\n",
"dtype: int32\n",
"chunk memory: 4 bytes\n",
"\n",
"calls: &lt;module&gt; -&gt; create_cubed_data -&gt; random -&gt; map_direct -&gt; map_blocks -&gt; offsets_array\n",
"line: 4 in create_cubed_data\">\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"144.25,-789.5 81,-789.5 81,-753.5 144.25,-753.5 144.25,-789.5\"/>\n",
"<text text-anchor=\"middle\" x=\"112.62\" y=\"-774.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-022</text>\n",
"<text text-anchor=\"middle\" x=\"112.62\" y=\"-762\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">random </text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;022&#45;&gt;array&#45;023 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>array-022-&gt;array-023</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M111.64,-753.2C111.21,-745.74 110.7,-736.82 110.22,-728.47\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"113.67,-728.39 109.6,-718.6 106.68,-728.79 113.67,-728.39\"/>\n",
"</g>\n",
"<!-- array&#45;132 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>array-132</title>\n",
"<g id=\"a_node7\"><a xlink:title=\"name: array-132\n",
"shape: (5000, 1, 987, 1920)\n",
"chunks: (1, 1, 987, 1920)\n",
"dtype: [('n', '&lt;i8'), ('total', '&lt;f8')]\n",
"chunk memory: 30.3 MB\n",
"\n",
"projected memory: 1.1 GB\n",
"tasks: 5000\n",
"calls: &lt;dictcomp&gt; -&gt; quadratic_means -&gt; mean -&gt; reduction -&gt; blockwise\n",
"line: 382 in f\">\n",
"<polygon fill=\"#dcbeff\" stroke=\"black\" points=\"190.88,-645.5 122.38,-645.5 122.38,-609.5 190.88,-609.5 190.88,-645.5\"/>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-630.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-132</text>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-618\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">mean (bw)</text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;023&#45;&gt;array&#45;132 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>array-023-&gt;array-132</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M120.49,-681.2C126.01,-673.14 132.7,-663.39 138.81,-654.48\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"142.13,-656.83 144.9,-646.6 136.35,-652.87 142.13,-656.83\"/>\n",
"</g>\n",
"<!-- array&#45;024 -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>array-024</title>\n",
"<g id=\"a_node4\"><a xlink:title=\"name: array-024\n",
"shape: (50000, 1, 987, 1920)\n",
"chunks: (10, 1, 987, 1920)\n",
"dtype: float64\n",
"chunk memory: 151.6 MB\n",
"\n",
"calls: &lt;module&gt; -&gt; create_cubed_data -&gt; random -&gt; map_direct -&gt; empty -&gt; full\n",
"line: 5 in create_cubed_data\">\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"232.25,-789.5 169,-789.5 169,-753.5 232.25,-753.5 232.25,-789.5\"/>\n",
"<text text-anchor=\"middle\" x=\"200.62\" y=\"-774.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-024</text>\n",
"<text text-anchor=\"middle\" x=\"200.62\" y=\"-762\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">random </text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;026 -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>array-026</title>\n",
"<g id=\"a_node6\"><a xlink:title=\"name: array-026\n",
"shape: (50000, 1, 987, 1920)\n",
"chunks: (10, 1, 987, 1920)\n",
"dtype: float64\n",
"chunk memory: 151.6 MB\n",
"\n",
"projected memory: 816.4 MB\n",
"tasks: 5000\n",
"calls: &lt;module&gt; -&gt; create_cubed_data -&gt; random -&gt; map_direct -&gt; map_blocks -&gt; _map_blocks -&gt; blockwise\n",
"line: 5 in create_cubed_data\">\n",
"<polygon fill=\"#dcbeff\" stroke=\"black\" points=\"243.75,-717.5 165.5,-717.5 165.5,-681.5 243.75,-681.5 243.75,-717.5\"/>\n",
"<text text-anchor=\"middle\" x=\"204.62\" y=\"-702.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-026</text>\n",
"<text text-anchor=\"middle\" x=\"204.62\" y=\"-690\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">random (bw)</text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;024&#45;&gt;array&#45;026 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>array-024-&gt;array-026</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M201.61,-753.2C202.04,-745.74 202.55,-736.82 203.03,-728.47\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"206.57,-728.79 203.65,-718.6 199.58,-728.39 206.57,-728.79\"/>\n",
"</g>\n",
"<!-- array&#45;025 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>array-025</title>\n",
"<g id=\"a_node5\"><a xlink:title=\"name: array-025\n",
"shape: (5000, 1, 1, 1)\n",
"chunks: (1, 1, 1, 1)\n",
"dtype: int32\n",
"chunk memory: 4 bytes\n",
"\n",
"calls: &lt;module&gt; -&gt; create_cubed_data -&gt; random -&gt; map_direct -&gt; map_blocks -&gt; offsets_array\n",
"line: 5 in create_cubed_data\">\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"313.25,-789.5 250,-789.5 250,-753.5 313.25,-753.5 313.25,-789.5\"/>\n",
"<text text-anchor=\"middle\" x=\"281.62\" y=\"-774.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-025</text>\n",
"<text text-anchor=\"middle\" x=\"281.62\" y=\"-762\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">random </text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;025&#45;&gt;array&#45;026 -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>array-025-&gt;array-026</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M262.59,-753.2C253.24,-744.7 241.82,-734.31 231.59,-725.01\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"234.46,-722.98 224.71,-718.85 229.75,-728.16 234.46,-722.98\"/>\n",
"</g>\n",
"<!-- array&#45;026&#45;&gt;array&#45;132 -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>array-026-&gt;array-132</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M192.76,-681.2C187.24,-673.14 180.55,-663.39 174.44,-654.48\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"176.9,-652.87 168.35,-646.6 171.12,-656.83 176.9,-652.87\"/>\n",
"</g>\n",
"<!-- array&#45;133 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>array-133</title>\n",
"<g id=\"a_node8\"><a xlink:title=\"name: array-133\n",
"shape: (5000, 1, 987, 1920)\n",
"chunks: (14, 1, 987, 1920)\n",
"dtype: [('n', '&lt;i8'), ('total', '&lt;f8')]\n",
"chunk memory: 424.5 MB\n",
"\n",
"projected memory: 2.0 GB\n",
"tasks: 358\n",
"write chunks: (14, 1, 987, 1920)\n",
"calls: &lt;dictcomp&gt; -&gt; quadratic_means -&gt; mean -&gt; reduction -&gt; rechunk\n",
"line: 382 in f\">\n",
"<polygon fill=\"#aaffc3\" stroke=\"black\" points=\"188.25,-573.5 125,-573.5 125,-537.5 188.25,-537.5 188.25,-573.5\"/>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-558.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-133</text>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-546\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">mean (rc)</text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;132&#45;&gt;array&#45;133 -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>array-132-&gt;array-133</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M156.62,-609.2C156.62,-601.74 156.62,-592.82 156.62,-584.47\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"160.13,-584.6 156.62,-574.6 153.13,-584.6 160.13,-584.6\"/>\n",
"</g>\n",
"<!-- array&#45;134 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>array-134</title>\n",
"<g id=\"a_node9\"><a xlink:title=\"name: array-134\n",
"shape: (358, 1, 987, 1920)\n",
"chunks: (1, 1, 987, 1920)\n",
"dtype: [('n', '&lt;i8'), ('total', '&lt;f8')]\n",
"chunk memory: 30.3 MB\n",
"\n",
"projected memory: 1.1 GB\n",
"tasks: 358\n",
"calls: &lt;dictcomp&gt; -&gt; quadratic_means -&gt; mean -&gt; reduction -&gt; blockwise\n",
"line: 382 in f\">\n",
"<polygon fill=\"#dcbeff\" stroke=\"black\" points=\"190.88,-501.5 122.38,-501.5 122.38,-465.5 190.88,-465.5 190.88,-501.5\"/>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-486.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-134</text>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-474\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">mean (bw)</text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;133&#45;&gt;array&#45;134 -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>array-133-&gt;array-134</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M156.62,-537.2C156.62,-529.74 156.62,-520.82 156.62,-512.47\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"160.13,-512.6 156.62,-502.6 153.13,-512.6 160.13,-512.6\"/>\n",
"</g>\n",
"<!-- array&#45;135 -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>array-135</title>\n",
"<g id=\"a_node10\"><a xlink:title=\"name: array-135\n",
"shape: (358, 1, 987, 1920)\n",
"chunks: (14, 1, 987, 1920)\n",
"dtype: [('n', '&lt;i8'), ('total', '&lt;f8')]\n",
"chunk memory: 424.5 MB\n",
"\n",
"projected memory: 2.0 GB\n",
"tasks: 26\n",
"write chunks: (14, 1, 987, 1920)\n",
"calls: &lt;dictcomp&gt; -&gt; quadratic_means -&gt; mean -&gt; reduction -&gt; rechunk\n",
"line: 382 in f\">\n",
"<polygon fill=\"#aaffc3\" stroke=\"black\" points=\"188.25,-429.5 125,-429.5 125,-393.5 188.25,-393.5 188.25,-429.5\"/>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-414.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-135</text>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-402\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">mean (rc)</text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;134&#45;&gt;array&#45;135 -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>array-134-&gt;array-135</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M156.62,-465.2C156.62,-457.74 156.62,-448.82 156.62,-440.47\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"160.13,-440.6 156.62,-430.6 153.13,-440.6 160.13,-440.6\"/>\n",
"</g>\n",
"<!-- array&#45;136 -->\n",
"<g id=\"node11\" class=\"node\">\n",
"<title>array-136</title>\n",
"<g id=\"a_node11\"><a xlink:title=\"name: array-136\n",
"shape: (26, 1, 987, 1920)\n",
"chunks: (1, 1, 987, 1920)\n",
"dtype: [('n', '&lt;i8'), ('total', '&lt;f8')]\n",
"chunk memory: 30.3 MB\n",
"\n",
"projected memory: 1.1 GB\n",
"tasks: 26\n",
"calls: &lt;dictcomp&gt; -&gt; quadratic_means -&gt; mean -&gt; reduction -&gt; blockwise\n",
"line: 382 in f\">\n",
"<polygon fill=\"#dcbeff\" stroke=\"black\" points=\"190.88,-357.5 122.38,-357.5 122.38,-321.5 190.88,-321.5 190.88,-357.5\"/>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-342.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-136</text>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-330\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">mean (bw)</text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;135&#45;&gt;array&#45;136 -->\n",
"<g id=\"edge10\" class=\"edge\">\n",
"<title>array-135-&gt;array-136</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M156.62,-393.2C156.62,-385.74 156.62,-376.82 156.62,-368.47\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"160.13,-368.6 156.62,-358.6 153.13,-368.6 160.13,-368.6\"/>\n",
"</g>\n",
"<!-- array&#45;137 -->\n",
"<g id=\"node12\" class=\"node\">\n",
"<title>array-137</title>\n",
"<g id=\"a_node12\"><a xlink:title=\"name: array-137\n",
"shape: (26, 1, 987, 1920)\n",
"chunks: (14, 1, 987, 1920)\n",
"dtype: [('n', '&lt;i8'), ('total', '&lt;f8')]\n",
"chunk memory: 424.5 MB\n",
"\n",
"projected memory: 2.0 GB\n",
"tasks: 2\n",
"write chunks: (14, 1, 987, 1920)\n",
"calls: &lt;dictcomp&gt; -&gt; quadratic_means -&gt; mean -&gt; reduction -&gt; rechunk\n",
"line: 382 in f\">\n",
"<polygon fill=\"#aaffc3\" stroke=\"black\" points=\"188.25,-285.5 125,-285.5 125,-249.5 188.25,-249.5 188.25,-285.5\"/>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-270.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-137</text>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-258\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">mean (rc)</text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;136&#45;&gt;array&#45;137 -->\n",
"<g id=\"edge11\" class=\"edge\">\n",
"<title>array-136-&gt;array-137</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M156.62,-321.2C156.62,-313.74 156.62,-304.82 156.62,-296.47\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"160.13,-296.6 156.62,-286.6 153.13,-296.6 160.13,-296.6\"/>\n",
"</g>\n",
"<!-- array&#45;138 -->\n",
"<g id=\"node13\" class=\"node\">\n",
"<title>array-138</title>\n",
"<g id=\"a_node13\"><a xlink:title=\"name: array-138\n",
"shape: (2, 1, 987, 1920)\n",
"chunks: (1, 1, 987, 1920)\n",
"dtype: [('n', '&lt;i8'), ('total', '&lt;f8')]\n",
"chunk memory: 30.3 MB\n",
"\n",
"projected memory: 1.1 GB\n",
"tasks: 2\n",
"calls: &lt;dictcomp&gt; -&gt; quadratic_means -&gt; mean -&gt; reduction -&gt; blockwise\n",
"line: 382 in f\">\n",
"<polygon fill=\"#dcbeff\" stroke=\"black\" points=\"190.88,-213.5 122.38,-213.5 122.38,-177.5 190.88,-177.5 190.88,-213.5\"/>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-198.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-138</text>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-186\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">mean (bw)</text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;137&#45;&gt;array&#45;138 -->\n",
"<g id=\"edge12\" class=\"edge\">\n",
"<title>array-137-&gt;array-138</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M156.62,-249.2C156.62,-241.74 156.62,-232.82 156.62,-224.47\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"160.13,-224.6 156.62,-214.6 153.13,-224.6 160.13,-224.6\"/>\n",
"</g>\n",
"<!-- array&#45;139 -->\n",
"<g id=\"node14\" class=\"node\">\n",
"<title>array-139</title>\n",
"<g id=\"a_node14\"><a xlink:title=\"name: array-139\n",
"shape: (2, 1, 987, 1920)\n",
"chunks: (2, 1, 987, 1920)\n",
"dtype: [('n', '&lt;i8'), ('total', '&lt;f8')]\n",
"chunk memory: 60.6 MB\n",
"\n",
"projected memory: 2.0 GB\n",
"tasks: 1\n",
"write chunks: (2, 1, 987, 1920)\n",
"calls: &lt;dictcomp&gt; -&gt; quadratic_means -&gt; mean -&gt; reduction -&gt; rechunk\n",
"line: 382 in f\">\n",
"<polygon fill=\"#aaffc3\" stroke=\"black\" points=\"188.25,-141.5 125,-141.5 125,-105.5 188.25,-105.5 188.25,-141.5\"/>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-126.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-139</text>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-114\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">mean (rc)</text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;138&#45;&gt;array&#45;139 -->\n",
"<g id=\"edge13\" class=\"edge\">\n",
"<title>array-138-&gt;array-139</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M156.62,-177.2C156.62,-169.74 156.62,-160.82 156.62,-152.47\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"160.13,-152.6 156.62,-142.6 153.13,-152.6 160.13,-152.6\"/>\n",
"</g>\n",
"<!-- array&#45;142 -->\n",
"<g id=\"node15\" class=\"node\">\n",
"<title>array-142</title>\n",
"<g id=\"a_node15\"><a xlink:title=\"name: array-142\n",
"shape: (1, 987, 1920)\n",
"chunks: (1, 987, 1920)\n",
"dtype: float64\n",
"chunk memory: 15.2 MB\n",
"\n",
"projected memory: 391.9 MB\n",
"tasks: 1\n",
"calls: mean -&gt; reduction -&gt; squeeze -&gt; map_blocks -&gt; _map_blocks -&gt; blockwise\n",
"line: 382 in f\">\n",
"<polygon fill=\"#dcbeff\" stroke=\"black\" points=\"190.88,-69.5 122.38,-69.5 122.38,-33.5 190.88,-33.5 190.88,-69.5\"/>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-54.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-142</text>\n",
"<text text-anchor=\"middle\" x=\"156.62\" y=\"-42\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">mean (bw)</text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"<!-- array&#45;139&#45;&gt;array&#45;142 -->\n",
"<g id=\"edge14\" class=\"edge\">\n",
"<title>array-139-&gt;array-142</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M156.62,-105.2C156.62,-97.74 156.62,-88.82 156.62,-80.47\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"160.13,-80.6 156.62,-70.6 153.13,-80.6 160.13,-80.6\"/>\n",
"</g>\n",
"<!-- create&#45;arrays -->\n",
"<g id=\"node16\" class=\"node\">\n",
"<title>create-arrays</title>\n",
"<g id=\"a_node16\"><a xlink:title=\"projected memory: 210.0 MB\n",
"tasks: 13\">\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"412.25,-789.5 331,-789.5 331,-753.5 412.25,-753.5 412.25,-789.5\"/>\n",
"<text text-anchor=\"middle\" x=\"371.62\" y=\"-768.38\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">create-arrays</text>\n",
"</a>\n",
"</g>\n",
"</g>\n",
"</g>\n",
"</svg>"
],
"text/plain": [
"<IPython.core.display.SVG object>"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"workloads['1.5TB']['uv'].data.visualize()"
]
},
{
"cell_type": "markdown",
"id": "2966413b-714a-4770-b68a-e5a1d5e169e5",
"metadata": {},
"source": [
"Notice that the 1.5TB plan only has a 6 more nodes in it, despite processing 1000x as much data. This is because though there are 1000x as many chunks, the plan (cubed's DAG) is represented at the array-level, abstracting away the exact number of chunks.\n",
"\n",
"The reason there are more nodes for the larger job is that the iterative blockwise reduction takes a number of steps that is limited by the `allowed_mem` specified, and as `allowed_mem` is constant between workloads it takes more steps to reduce a larger dataset without exceeding that limit."
]
},
{
"cell_type": "markdown",
"id": "0dc803a1-2510-470d-aebe-770d03e1040a",
"metadata": {},
"source": [
"## Run benchmark"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "29ca2dcb-45e0-41ce-82db-1ff1de48b400",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"from cubed.extensions.tqdm import TqdmProgressBar\n",
"from cubed.extensions.history import HistoryCallback\n",
"from cubed.extensions.timeline import TimelineVisualizationCallback"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "7f952c22-eb3b-4200-a4ce-c770a4bc3468",
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "88c1b3dd6eeb46e1ba4837b7f9500ab1",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"create-arrays: 0%| | 0/15 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "f4540b7bd3bb4b21809527df7c78b588",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-005: 0%| | 0/5 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "43c28f2f9ee144348a41870b751391e4",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-008: 0%| | 0/5 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "0e643d75792d4842aa95d4c9b1d2b26a",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-032: 0%| | 0/5 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "bc1b42b5ff4e47db826756921ddd1f03",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-037: 0%| | 0/5 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "0fa80c830dd040ce92c844d5ff2a2323",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-042: 0%| | 0/5 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "20f5beecdbd7446389c05aaef332b3d1",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-033: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "318b030d6a014a018cd0c8f991a5edfb",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-038: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a52276510de949c9aaeabe7ecfd2ef55",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-043: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "45c9b2409d804cc99fac0f5b0ef1fb16",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-036: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "9d11c0f50cd9406498abb86ea86cb731",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-041: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "3580ded2f5d941479ea8e5a911ad7e1e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-046: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "b6be58eebf614fdc8d1c4e37213b2aab",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:coiled:Resolving your local Python environment...\n",
"INFO:coiled.package_sync:Using wheel @ /tmp/tmpgfw1pup1/cubed/cubed-0.9.0-py3-none-any.whl\n",
"INFO:coiled.package_sync:Using wheel @ /tmp/tmp0iu327o_/cubed-xarray/cubed_xarray-0.0.4-py3-none-any.whl\n",
"INFO:coiled:Package - cubed, Wheel built from /home/tom/Documents/Work/Code/cubed is missing conftest.py, examples/dataflow/setup.py, setup.py, venv/bin/activate_this.py\n",
"INFO:coiled:Package - cubed-xarray, Wheel built from /home/tom/Documents/Work/Code/cubed-xarray is missing setup.py\n",
"INFO:coiled:Starting upload for cubed-0.9.0-py3-none-any.whl\n",
"INFO:coiled:Starting upload for cubed_xarray-0.0.4-py3-none-any.whl\n"
]
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">╭────────────────────────────────────────── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">Package Info</span> ──────────────────────────────────────────╮\n",
"│ ╷ │\n",
"│ <span style=\"font-weight: bold\"> Package </span>│<span style=\"font-weight: bold\"> Note </span> │\n",
"│ ╶──────────────┼───────────────────────────────────────────────────────────────────────────────╴ │\n",
"│ cubed │ Wheel built from /home/tom/Documents/Work/Code/cubed is missing conftest.py, │\n",
"│ │ examples/dataflow/setup.py, setup.py, venv/bin/activate_this.py │\n",
"│ cubed-xarray │ Wheel built from /home/tom/Documents/Work/Code/cubed-xarray is missing │\n",
"│ │ setup.py │\n",
"│ ╵ │\n",
"╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
"</pre>\n"
],
"text/plain": [
"╭────────────────────────────────────────── \u001b[1;32mPackage Info\u001b[0m ──────────────────────────────────────────╮\n",
"│ ╷ │\n",
"│ \u001b[1m \u001b[0m\u001b[1mPackage \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1mNote \u001b[0m\u001b[1m \u001b[0m │\n",
"│ ╶──────────────┼───────────────────────────────────────────────────────────────────────────────╴ │\n",
"│ cubed │ Wheel built from /home/tom/Documents/Work/Code/cubed is missing conftest.py, │\n",
"│ │ examples/dataflow/setup.py, setup.py, venv/bin/activate_this.py │\n",
"│ cubed-xarray │ Wheel built from /home/tom/Documents/Work/Code/cubed-xarray is missing │\n",
"│ │ setup.py │\n",
"│ ╵ │\n",
"╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:coiled:Creating Cluster (name: run-95af2078, https://cloud.coiled.io/clusters/244099?account=dask ). This usually takes 1-2 minutes...\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "c5ea884bcca44584beefbb3442b14f72",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials\n",
"INFO:distributed.deploy.adaptive:Adaptive scaling started: minimum=0 maximum=100\n",
"INFO:coiled:Adaptive scaling up to 1 workers.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 6.78 s, sys: 828 ms, total: 7.61 s\n",
"Wall time: 3min 12s\n"
]
}
],
"source": [
"%%time\n",
"with logging_redirect_tqdm():\n",
" progress = TqdmProgressBar()\n",
" hist = HistoryCallback()\n",
" timeline_viz = TimelineVisualizationCallback()\n",
" \n",
" workloads['1.5GB'].compute(\n",
" executor=coiledexecutor,\n",
" memory='2 GiB', # must be greater than allowed_mem, see cubed issue #220\n",
" #cpu=1, # only need one CPU to process one chunk\n",
" account='dask', # use dask maintainers account\n",
" compute_purchase_option='spot_with_fallback',\n",
" region='us-east-2', # presumably I should specify this to avoid egress charges\n",
" callbacks=[progress, hist, timeline_viz],\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "ea73182f-42a7-4892-a7ed-14357008d50b",
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "7387c263ebcd4f298fcce73c3e3caa15",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"create-arrays: 0%| | 0/21 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "4b49d21f38fc410984494979a3698e0b",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-011: 0%| | 0/50 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "b294608efde443908cd094c0e2ea3a07",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-014: 0%| | 0/50 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "9622373c703f4c5a8e09ea09b88e6ddc",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-052: 0%| | 0/50 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ff752442e69e4f989f4c62de15fce147",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-059: 0%| | 0/50 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "c456e490f3f04de7b7271b4dd06d00bb",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-066: 0%| | 0/50 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "fe4bbdf061c34975bac663e78abd253e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-053: 0%| | 0/4 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2413634d79444f5694789c6fa668f54c",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-060: 0%| | 0/4 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "70c6e390edc044bc8062e553e1ba02fd",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-067: 0%| | 0/4 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "b296c8c184af46059d5c5858d9d0a910",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-054: 0%| | 0/4 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "eaefa8b7791e4bcfb181e2bf656282ff",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-061: 0%| | 0/4 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "342d30e85dea4ad18220a5f23d197411",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-068: 0%| | 0/4 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "9b471e97efc94cf1b1bbaa47f0e7daff",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-055: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "958b3490c15249c9bb8a208d0a1d6ee5",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-062: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "7072ad81a18748878b9e02f7b53bdfc3",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-069: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "71e232d5aede43bcb5081c053bd20f7b",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-058: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "3307156146df46b28f44113fd417f2f6",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-065: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2bb3dfd928434cbf836493bcdaa39552",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-072: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:coiled:Adaptive scaling up to 2 workers.\n",
"INFO:coiled:Adaptive scaling up to 3 workers.\n",
"INFO:coiled:Adaptive scaling up to 4 workers.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 4.83 s, sys: 1.06 s, total: 5.89 s\n",
"Wall time: 3min 33s\n"
]
}
],
"source": [
"%%time\n",
"with logging_redirect_tqdm():\n",
" progress = TqdmProgressBar()\n",
" hist = HistoryCallback()\n",
" timeline_viz = TimelineVisualizationCallback()\n",
" \n",
" workloads['15GB'].compute(\n",
" executor=coiledexecutor,\n",
" memory='2 GiB', # must be greater than allowed_mem, see cubed issue #220\n",
" #cpu=1, # only need one CPU to process one chunk - would be cheaper to use this but I get an error from Coiled\n",
" account='dask', # use dask maintainers account\n",
" compute_purchase_option='spot_with_fallback',\n",
" region='us-east-2', # presumably I should specify this to avoid egress charges - I hope this is the same region that bucket is in!\n",
" callbacks=[progress, hist, timeline_viz], # not supported for coiled executor yet\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "119df0fa-99d8-40a3-87d6-b59e8b9cffba",
"metadata": {},
"source": [
"To do the larger problem then for full parallelism I would in theory want the Coiled cluster to adaptively scale up to the same number of workers as chunks, but I can't find how to set that for `coiled.run`."
]
},
{
"cell_type": "markdown",
"id": "c5a48701-69d1-45c3-9790-dc5eda60fc73",
"metadata": {},
"source": [
"(It seems as if the coiled cluster only scaled up to 4 workers though.)"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "295f972d-6eec-4cdf-b39f-8477a926b7f4",
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "36da0b930dcc4861b082fb2a6c2681b7",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"create-arrays: 0%| | 0/27 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "8985fd1c30a548f180be7e25de729c52",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-017: 0%| | 0/500 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2ad5af5f17cf4fd997423c318bc14862",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-020: 0%| | 0/500 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "d0373bb4e99643ad8073feda55353878",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-078: 0%| | 0/500 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "688638515abd466d9a1de9d8f7c816b6",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-087: 0%| | 0/500 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "d82f10112b64478692841a168845341f",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-096: 0%| | 0/500 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "3ebf28ea916a4b5ea569f4f33d4699a9",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-079: 0%| | 0/36 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "3ea3e710fdec4416b76c7cf311ca9df6",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-088: 0%| | 0/36 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "6cc49525d7734180a9e8c4ca32200c3d",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-097: 0%| | 0/36 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e68732e26da74db09ea24a71a1bba342",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-080: 0%| | 0/36 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e1b00ef1721b49119a822937778f5e60",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-089: 0%| | 0/36 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "87342373cdf046c2b8d5db9cb4ce90aa",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-098: 0%| | 0/36 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a0049a84fbfc4cb9b1c2f7a6a8c10603",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-081: 0%| | 0/3 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a496dcdbb53e43a6a378c0b34e9db706",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-090: 0%| | 0/3 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "01bb337f8a164f59b6315a5d968aadd9",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-099: 0%| | 0/3 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "78b887ea461140839705e5b893de68ab",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-082: 0%| | 0/3 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "5a8bccdafdbe496290727561b6c26c72",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-091: 0%| | 0/3 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "aaebf32b79ea4b1295fb126eec73505b",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-100: 0%| | 0/3 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e4a2728fa5e24bff91fe32144934ec36",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-083: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "d860ac49ff104ad694d185cdbd233594",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-092: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2fd382d7bb07412489ff39aaef6bcaff",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-101: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "7591f5bc9eee41a5b1f81c81a9ef9d13",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-086: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "85472dc595dd41c7bdad18ea939e9854",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-095: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "259d21b75bb64942b6268d698ba395e8",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"array-104: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:coiled:Adaptive scaling up to 1 workers.\n",
"INFO:coiled:Adaptive scaling up to 2 workers.\n",
"INFO:coiled:Adaptive is removing 1 workers: ['run-95af2078-worker-cb91925975'].\n",
"INFO:coiled:Adaptive scaling up to 2 workers.\n",
"INFO:coiled:Adaptive is removing 1 workers: ['run-95af2078-worker-ff8d456d1a'].\n",
"INFO:coiled:Adaptive scaling up to 2 workers.\n",
"INFO:coiled:Adaptive is removing 1 workers: ['run-95af2078-worker-38b09ad796'].\n",
"ERROR:distributed.client:Failed to reconnect to scheduler after 30.00 seconds, closing client\n",
"ERROR:distributed.deploy.adaptive_core:Adaptive stopping due to error\n",
"Traceback (most recent call last):\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/utils.py\", line 1919, in wait_for\n",
" return await fut\n",
" ^^^^^^^^^\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/comm/tcp.py\", line 491, in connect\n",
" stream = await self.client.connect(\n",
" ^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/tornado/tcpclient.py\", line 279, in connect\n",
" af, addr, stream = await connector.start(connect_timeout=timeout)\n",
" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
"asyncio.exceptions.CancelledError\n",
"\n",
"The above exception was the direct cause of the following exception:\n",
"\n",
"Traceback (most recent call last):\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/comm/core.py\", line 355, in connect\n",
" comm = await wait_for(\n",
" ^^^^^^^^^^^^^^^\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/utils.py\", line 1918, in wait_for\n",
" async with asyncio.timeout(timeout):\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/asyncio/timeouts.py\", line 111, in __aexit__\n",
" raise TimeoutError from exc_val\n",
"TimeoutError\n",
"\n",
"The above exception was the direct cause of the following exception:\n",
"\n",
"Traceback (most recent call last):\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/deploy/adaptive_core.py\", line 228, in adapt\n",
" target = await self.safe_target()\n",
" ^^^^^^^^^^^^^^^^^^^^^^^^\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/deploy/adaptive_core.py\", line 160, in safe_target\n",
" n = await self.target()\n",
" ^^^^^^^^^^^^^^^^^^^\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/deploy/adaptive.py\", line 151, in target\n",
" return await self.scheduler.adaptive_target(\n",
" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/core.py\", line 1294, in send_recv_from_rpc\n",
" comm = await self.live_comm()\n",
" ^^^^^^^^^^^^^^^^^^^^^^\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/core.py\", line 1253, in live_comm\n",
" comm = await connect(\n",
" ^^^^^^^^^^^^^^\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/comm/core.py\", line 381, in connect\n",
" raise OSError(\n",
"OSError: Timed out trying to connect to tls://18.118.196.101:8786 after 30 s\n",
"INFO:distributed.deploy.adaptive_core:Adaptive stop\n",
"WARNING:distributed.deploy.cluster:Failed to sync cluster info multiple times - perhaps there's a connection issue? Error:\n",
"Traceback (most recent call last):\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/utils.py\", line 1919, in wait_for\n",
" return await fut\n",
" ^^^^^^^^^\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/comm/tcp.py\", line 491, in connect\n",
" stream = await self.client.connect(\n",
" ^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/tornado/tcpclient.py\", line 279, in connect\n",
" af, addr, stream = await connector.start(connect_timeout=timeout)\n",
" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
"asyncio.exceptions.CancelledError\n",
"\n",
"The above exception was the direct cause of the following exception:\n",
"\n",
"Traceback (most recent call last):\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/comm/core.py\", line 355, in connect\n",
" comm = await wait_for(\n",
" ^^^^^^^^^^^^^^^\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/utils.py\", line 1918, in wait_for\n",
" async with asyncio.timeout(timeout):\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/asyncio/timeouts.py\", line 111, in __aexit__\n",
" raise TimeoutError from exc_val\n",
"TimeoutError\n",
"\n",
"The above exception was the direct cause of the following exception:\n",
"\n",
"Traceback (most recent call last):\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/deploy/cluster.py\", line 168, in _sync_cluster_info\n",
" await self.scheduler_comm.set_metadata(\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/core.py\", line 1294, in send_recv_from_rpc\n",
" comm = await self.live_comm()\n",
" ^^^^^^^^^^^^^^^^^^^^^^\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/core.py\", line 1253, in live_comm\n",
" comm = await connect(\n",
" ^^^^^^^^^^^^^^\n",
" File \"/home/tom/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/comm/core.py\", line 381, in connect\n",
" raise OSError(\n",
"OSError: Timed out trying to connect to tls://18.118.196.101:8786 after 30 s\n"
]
},
{
"ename": "KeyboardInterrupt",
"evalue": "",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)",
"File \u001b[0;32m<timed exec>:6\u001b[0m\n",
"File \u001b[0;32m~/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/xarray/core/dataset.py:958\u001b[0m, in \u001b[0;36mDataset.compute\u001b[0;34m(self, **kwargs)\u001b[0m\n\u001b[1;32m 939\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"Manually trigger loading and/or computation of this dataset's data\u001b[39;00m\n\u001b[1;32m 940\u001b[0m \u001b[38;5;124;03mfrom disk or a remote source into memory and return a new dataset.\u001b[39;00m\n\u001b[1;32m 941\u001b[0m \u001b[38;5;124;03mUnlike load, the original dataset is left unaltered.\u001b[39;00m\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 955\u001b[0m \u001b[38;5;124;03mdask.compute\u001b[39;00m\n\u001b[1;32m 956\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m 957\u001b[0m new \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcopy(deep\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mFalse\u001b[39;00m)\n\u001b[0;32m--> 958\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mnew\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mload\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
"File \u001b[0;32m~/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/xarray/core/dataset.py:792\u001b[0m, in \u001b[0;36mDataset.load\u001b[0;34m(self, **kwargs)\u001b[0m\n\u001b[1;32m 789\u001b[0m chunkmanager \u001b[38;5;241m=\u001b[39m get_chunked_array_type(\u001b[38;5;241m*\u001b[39mlazy_data\u001b[38;5;241m.\u001b[39mvalues())\n\u001b[1;32m 791\u001b[0m \u001b[38;5;66;03m# evaluate all the chunked arrays simultaneously\u001b[39;00m\n\u001b[0;32m--> 792\u001b[0m evaluated_data \u001b[38;5;241m=\u001b[39m \u001b[43mchunkmanager\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcompute\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mlazy_data\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mvalues\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 794\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m k, data \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mzip\u001b[39m(lazy_data, evaluated_data):\n\u001b[1;32m 795\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mvariables[k]\u001b[38;5;241m.\u001b[39mdata \u001b[38;5;241m=\u001b[39m data\n",
"File \u001b[0;32m~/Documents/Work/Code/cubed-xarray/cubed_xarray/cubedmanager.py:69\u001b[0m, in \u001b[0;36mCubedManager.compute\u001b[0;34m(self, *data, **kwargs)\u001b[0m\n\u001b[1;32m 66\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mcompute\u001b[39m(\u001b[38;5;28mself\u001b[39m, \u001b[38;5;241m*\u001b[39mdata: \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCubedArray\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m \u001b[38;5;28mtuple\u001b[39m[np\u001b[38;5;241m.\u001b[39mndarray, \u001b[38;5;241m.\u001b[39m\u001b[38;5;241m.\u001b[39m\u001b[38;5;241m.\u001b[39m]:\n\u001b[1;32m 67\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mcubed\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m compute\n\u001b[0;32m---> 69\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mcompute\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mdata\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
"File \u001b[0;32m~/Documents/Work/Code/cubed/cubed/core/array.py:410\u001b[0m, in \u001b[0;36mcompute\u001b[0;34m(executor, callbacks, optimize_graph, resume, *arrays, **kwargs)\u001b[0m\n\u001b[1;32m 407\u001b[0m executor \u001b[38;5;241m=\u001b[39m PythonDagExecutor()\n\u001b[1;32m 409\u001b[0m _return_in_memory_array \u001b[38;5;241m=\u001b[39m kwargs\u001b[38;5;241m.\u001b[39mpop(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m_return_in_memory_array\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;28;01mTrue\u001b[39;00m)\n\u001b[0;32m--> 410\u001b[0m \u001b[43mplan\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mexecute\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 411\u001b[0m \u001b[43m \u001b[49m\u001b[43mexecutor\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mexecutor\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 412\u001b[0m \u001b[43m \u001b[49m\u001b[43mcallbacks\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcallbacks\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 413\u001b[0m \u001b[43m \u001b[49m\u001b[43moptimize_graph\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43moptimize_graph\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 414\u001b[0m \u001b[43m \u001b[49m\u001b[43mresume\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mresume\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 415\u001b[0m \u001b[43m \u001b[49m\u001b[43marray_names\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m[\u001b[49m\u001b[43ma\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mname\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43;01mfor\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43ma\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;129;43;01min\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43marrays\u001b[49m\u001b[43m]\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 416\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 417\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 419\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m _return_in_memory_array:\n\u001b[1;32m 420\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mtuple\u001b[39m(a\u001b[38;5;241m.\u001b[39m_read_stored() \u001b[38;5;28;01mfor\u001b[39;00m a \u001b[38;5;129;01min\u001b[39;00m arrays)\n",
"File \u001b[0;32m~/Documents/Work/Code/cubed/cubed/core/plan.py:174\u001b[0m, in \u001b[0;36mPlan.execute\u001b[0;34m(self, executor, callbacks, optimize_graph, resume, array_names, **kwargs)\u001b[0m\n\u001b[1;32m 172\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m callbacks \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m 173\u001b[0m [callback\u001b[38;5;241m.\u001b[39mon_compute_start(dag, resume\u001b[38;5;241m=\u001b[39mresume) \u001b[38;5;28;01mfor\u001b[39;00m callback \u001b[38;5;129;01min\u001b[39;00m callbacks]\n\u001b[0;32m--> 174\u001b[0m \u001b[43mexecutor\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mexecute_dag\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 175\u001b[0m \u001b[43m \u001b[49m\u001b[43mdag\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 176\u001b[0m \u001b[43m \u001b[49m\u001b[43mcallbacks\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcallbacks\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 177\u001b[0m \u001b[43m \u001b[49m\u001b[43marray_names\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43marray_names\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 178\u001b[0m \u001b[43m \u001b[49m\u001b[43mresume\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mresume\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 179\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 180\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 181\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m callbacks \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m 182\u001b[0m [callback\u001b[38;5;241m.\u001b[39mon_compute_end(dag) \u001b[38;5;28;01mfor\u001b[39;00m callback \u001b[38;5;129;01min\u001b[39;00m callbacks]\n",
"File \u001b[0;32m~/Documents/Work/Code/cubed/cubed/runtime/executors/coiled.py:48\u001b[0m, in \u001b[0;36mCoiledFunctionsDagExecutor.execute_dag\u001b[0;34m(self, dag, callbacks, array_names, resume, **coiled_kwargs)\u001b[0m\n\u001b[1;32m 46\u001b[0m ac \u001b[38;5;241m=\u001b[39m as_completed(futures)\n\u001b[1;32m 47\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m callbacks \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m---> 48\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m future \u001b[38;5;129;01min\u001b[39;00m ac:\n\u001b[1;32m 49\u001b[0m result, stats \u001b[38;5;241m=\u001b[39m future\u001b[38;5;241m.\u001b[39mresult()\n\u001b[1;32m 50\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m name \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
"File \u001b[0;32m~/miniconda3/envs/cubed_coiled/lib/python3.11/site-packages/distributed/client.py:5441\u001b[0m, in \u001b[0;36mas_completed.__next__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 5439\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mStopIteration\u001b[39;00m()\n\u001b[1;32m 5440\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mthread_condition:\n\u001b[0;32m-> 5441\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mthread_condition\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mwait\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtimeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m0.100\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m 5442\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_get_and_raise()\n",
"File \u001b[0;32m~/miniconda3/envs/cubed_coiled/lib/python3.11/threading.py:324\u001b[0m, in \u001b[0;36mCondition.wait\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m 322\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 323\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m timeout \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m0\u001b[39m:\n\u001b[0;32m--> 324\u001b[0m gotit \u001b[38;5;241m=\u001b[39m \u001b[43mwaiter\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43macquire\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtimeout\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 325\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 326\u001b[0m gotit \u001b[38;5;241m=\u001b[39m waiter\u001b[38;5;241m.\u001b[39macquire(\u001b[38;5;28;01mFalse\u001b[39;00m)\n",
"\u001b[0;31mKeyboardInterrupt\u001b[0m: "
]
}
],
"source": [
"%%time\n",
"with logging_redirect_tqdm():\n",
" progress = TqdmProgressBar()\n",
" hist = HistoryCallback()\n",
" timeline_viz = TimelineVisualizationCallback()\n",
" \n",
" workloads['150GB'].compute(\n",
" executor=coiledexecutor,\n",
" memory='2 GiB', # must be greater than allowed_mem, see cubed issue #220\n",
" #cpu=1, # only need one CPU to process one chunk - would be cheaper to use this but I get an error from Coiled\n",
" account='dask', # use dask maintainers account\n",
" compute_purchase_option='spot_with_fallback',\n",
" region='us-east-2', # presumably I should specify this to avoid egress charges - I hope this is the same region that bucket is in!\n",
" callbacks=[progress, hist, timeline_viz], # not supported for coiled executor yet\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "43df3ea0-03d7-4271-bea7-45f50d4268c7",
"metadata": {},
"source": [
"I killed this because it was going really slowly and with only 1 worker. We have 100 chunks, so what I wanted was for `coiled.run` to adaptively scale up to 100 workers automatically."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e1fe64a3-f073-4cd4-9207-cfaeb2882e37",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"%%time\n",
"with logging_redirect_tqdm():\n",
" progress = TqdmProgressBar()\n",
" hist = HistoryCallback()\n",
" timeline_viz = TimelineVisualizationCallback()\n",
" \n",
" workloads['1.5TB'].compute(\n",
" executor=coiledexecutor,\n",
" memory='2 GiB', # must be greater than allowed_mem, see cubed issue #220\n",
" #cpu=1, # only need one CPU to process one chunk - would be cheaper to use this but I get an error from Coiled\n",
" account='dask', # use dask maintainers account\n",
" compute_purchase_option='spot_with_fallback',\n",
" region='us-east-2', # presumably I should specify this to avoid egress charges - I hope this is the same region that bucket is in!\n",
" #callbacks=[progress, hist, timeline_viz], # not supported for coiled executor yet\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5bd28aea-3473-45f0-a9d7-be6f8bcab787",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.4"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment