Skip to content

Instantly share code, notes, and snippets.

@jbusecke
Last active July 4, 2025 00:56
Show Gist options
  • Save jbusecke/2768bb4db3102190bbb961ea6bf8849a to your computer and use it in GitHub Desktop.
Save jbusecke/2768bb4db3102190bbb961ea6bf8849a to your computer and use it in GitHub Desktop.
Display the source blob
Display the rendered blob
Raw
{
"cells": [
{
"cell_type": "markdown",
"id": "02d66e85-41b5-47d6-8315-459fb12f0940",
"metadata": {},
"source": [
"# Testing Branch \"merging\" in icechunk\n",
"The idea here is to append new data along a dimension (time) on a separate branch, test the resulting array, and then add the snapshot containing the new data to the main branch. \n",
"In this simple case, this behaves like a git merge, but the whole topic is more complicated (maybe Sebastian can elaborate?). "
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "a740f8bd-84eb-4442-a9cc-3a5accf26a8f",
"metadata": {},
"outputs": [],
"source": [
"import xarray as xr\n",
"import icechunk as ic\n",
"import numpy as np"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "a6112041-3ae7-4661-abce-b6ddaece5cba",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div><svg style=\"position: absolute; width: 0; height: 0; overflow: hidden\">\n",
"<defs>\n",
"<symbol id=\"icon-database\" viewBox=\"0 0 32 32\">\n",
"<path d=\"M16 0c-8.837 0-16 2.239-16 5v4c0 2.761 7.163 5 16 5s16-2.239 16-5v-4c0-2.761-7.163-5-16-5z\"></path>\n",
"<path d=\"M16 17c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z\"></path>\n",
"<path d=\"M16 26c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z\"></path>\n",
"</symbol>\n",
"<symbol id=\"icon-file-text2\" viewBox=\"0 0 32 32\">\n",
"<path d=\"M28.681 7.159c-0.694-0.947-1.662-2.053-2.724-3.116s-2.169-2.030-3.116-2.724c-1.612-1.182-2.393-1.319-2.841-1.319h-15.5c-1.378 0-2.5 1.121-2.5 2.5v27c0 1.378 1.122 2.5 2.5 2.5h23c1.378 0 2.5-1.122 2.5-2.5v-19.5c0-0.448-0.137-1.23-1.319-2.841zM24.543 5.457c0.959 0.959 1.712 1.825 2.268 2.543h-4.811v-4.811c0.718 0.556 1.584 1.309 2.543 2.268zM28 29.5c0 0.271-0.229 0.5-0.5 0.5h-23c-0.271 0-0.5-0.229-0.5-0.5v-27c0-0.271 0.229-0.5 0.5-0.5 0 0 15.499-0 15.5 0v7c0 0.552 0.448 1 1 1h7v19.5z\"></path>\n",
"<path d=\"M23 26h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
"<path d=\"M23 22h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
"<path d=\"M23 18h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
"</symbol>\n",
"</defs>\n",
"</svg>\n",
"<style>/* CSS stylesheet for displaying xarray objects in jupyterlab.\n",
" *\n",
" */\n",
"\n",
":root {\n",
" --xr-font-color0: var(\n",
" --jp-content-font-color0,\n",
" var(--pst-color-text-base rgba(0, 0, 0, 1))\n",
" );\n",
" --xr-font-color2: var(\n",
" --jp-content-font-color2,\n",
" var(--pst-color-text-base, rgba(0, 0, 0, 0.54))\n",
" );\n",
" --xr-font-color3: var(\n",
" --jp-content-font-color3,\n",
" var(--pst-color-text-base, rgba(0, 0, 0, 0.38))\n",
" );\n",
" --xr-border-color: var(\n",
" --jp-border-color2,\n",
" hsl(from var(--pst-color-on-background, white) h s calc(l - 10))\n",
" );\n",
" --xr-disabled-color: var(\n",
" --jp-layout-color3,\n",
" hsl(from var(--pst-color-on-background, white) h s calc(l - 40))\n",
" );\n",
" --xr-background-color: var(\n",
" --jp-layout-color0,\n",
" var(--pst-color-on-background, white)\n",
" );\n",
" --xr-background-color-row-even: var(\n",
" --jp-layout-color1,\n",
" hsl(from var(--pst-color-on-background, white) h s calc(l - 5))\n",
" );\n",
" --xr-background-color-row-odd: var(\n",
" --jp-layout-color2,\n",
" hsl(from var(--pst-color-on-background, white) h s calc(l - 15))\n",
" );\n",
"}\n",
"\n",
"html[theme=\"dark\"],\n",
"html[data-theme=\"dark\"],\n",
"body[data-theme=\"dark\"],\n",
"body.vscode-dark {\n",
" --xr-font-color0: var(\n",
" --jp-content-font-color0,\n",
" var(--pst-color-text-base, rgba(255, 255, 255, 1))\n",
" );\n",
" --xr-font-color2: var(\n",
" --jp-content-font-color2,\n",
" var(--pst-color-text-base, rgba(255, 255, 255, 0.54))\n",
" );\n",
" --xr-font-color3: var(\n",
" --jp-content-font-color3,\n",
" var(--pst-color-text-base, rgba(255, 255, 255, 0.38))\n",
" );\n",
" --xr-border-color: var(\n",
" --jp-border-color2,\n",
" hsl(from var(--pst-color-on-background, #111111) h s calc(l + 10))\n",
" );\n",
" --xr-disabled-color: var(\n",
" --jp-layout-color3,\n",
" hsl(from var(--pst-color-on-background, #111111) h s calc(l + 40))\n",
" );\n",
" --xr-background-color: var(\n",
" --jp-layout-color0,\n",
" var(--pst-color-on-background, #111111)\n",
" );\n",
" --xr-background-color-row-even: var(\n",
" --jp-layout-color1,\n",
" hsl(from var(--pst-color-on-background, #111111) h s calc(l + 5))\n",
" );\n",
" --xr-background-color-row-odd: var(\n",
" --jp-layout-color2,\n",
" hsl(from var(--pst-color-on-background, #111111) h s calc(l + 15))\n",
" );\n",
"}\n",
"\n",
".xr-wrap {\n",
" display: block !important;\n",
" min-width: 300px;\n",
" max-width: 700px;\n",
"}\n",
"\n",
".xr-text-repr-fallback {\n",
" /* fallback to plain text repr when CSS is not injected (untrusted notebook) */\n",
" display: none;\n",
"}\n",
"\n",
".xr-header {\n",
" padding-top: 6px;\n",
" padding-bottom: 6px;\n",
" margin-bottom: 4px;\n",
" border-bottom: solid 1px var(--xr-border-color);\n",
"}\n",
"\n",
".xr-header > div,\n",
".xr-header > ul {\n",
" display: inline;\n",
" margin-top: 0;\n",
" margin-bottom: 0;\n",
"}\n",
"\n",
".xr-obj-type,\n",
".xr-array-name {\n",
" margin-left: 2px;\n",
" margin-right: 10px;\n",
"}\n",
"\n",
".xr-obj-type {\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-sections {\n",
" padding-left: 0 !important;\n",
" display: grid;\n",
" grid-template-columns: 150px auto auto 1fr 0 20px 0 20px;\n",
"}\n",
"\n",
".xr-section-item {\n",
" display: contents;\n",
"}\n",
"\n",
".xr-section-item input {\n",
" display: inline-block;\n",
" opacity: 0;\n",
" height: 0;\n",
"}\n",
"\n",
".xr-section-item input + label {\n",
" color: var(--xr-disabled-color);\n",
" border: 2px solid transparent !important;\n",
"}\n",
"\n",
".xr-section-item input:enabled + label {\n",
" cursor: pointer;\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-section-item input:focus + label {\n",
" border: 2px solid var(--xr-font-color0) !important;\n",
"}\n",
"\n",
".xr-section-item input:enabled + label:hover {\n",
" color: var(--xr-font-color0);\n",
"}\n",
"\n",
".xr-section-summary {\n",
" grid-column: 1;\n",
" color: var(--xr-font-color2);\n",
" font-weight: 500;\n",
"}\n",
"\n",
".xr-section-summary > span {\n",
" display: inline-block;\n",
" padding-left: 0.5em;\n",
"}\n",
"\n",
".xr-section-summary-in:disabled + label {\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-section-summary-in + label:before {\n",
" display: inline-block;\n",
" content: \"►\";\n",
" font-size: 11px;\n",
" width: 15px;\n",
" text-align: center;\n",
"}\n",
"\n",
".xr-section-summary-in:disabled + label:before {\n",
" color: var(--xr-disabled-color);\n",
"}\n",
"\n",
".xr-section-summary-in:checked + label:before {\n",
" content: \"▼\";\n",
"}\n",
"\n",
".xr-section-summary-in:checked + label > span {\n",
" display: none;\n",
"}\n",
"\n",
".xr-section-summary,\n",
".xr-section-inline-details {\n",
" padding-top: 4px;\n",
" padding-bottom: 4px;\n",
"}\n",
"\n",
".xr-section-inline-details {\n",
" grid-column: 2 / -1;\n",
"}\n",
"\n",
".xr-section-details {\n",
" display: none;\n",
" grid-column: 1 / -1;\n",
" margin-bottom: 5px;\n",
"}\n",
"\n",
".xr-section-summary-in:checked ~ .xr-section-details {\n",
" display: contents;\n",
"}\n",
"\n",
".xr-array-wrap {\n",
" grid-column: 1 / -1;\n",
" display: grid;\n",
" grid-template-columns: 20px auto;\n",
"}\n",
"\n",
".xr-array-wrap > label {\n",
" grid-column: 1;\n",
" vertical-align: top;\n",
"}\n",
"\n",
".xr-preview {\n",
" color: var(--xr-font-color3);\n",
"}\n",
"\n",
".xr-array-preview,\n",
".xr-array-data {\n",
" padding: 0 5px !important;\n",
" grid-column: 2;\n",
"}\n",
"\n",
".xr-array-data,\n",
".xr-array-in:checked ~ .xr-array-preview {\n",
" display: none;\n",
"}\n",
"\n",
".xr-array-in:checked ~ .xr-array-data,\n",
".xr-array-preview {\n",
" display: inline-block;\n",
"}\n",
"\n",
".xr-dim-list {\n",
" display: inline-block !important;\n",
" list-style: none;\n",
" padding: 0 !important;\n",
" margin: 0;\n",
"}\n",
"\n",
".xr-dim-list li {\n",
" display: inline-block;\n",
" padding: 0;\n",
" margin: 0;\n",
"}\n",
"\n",
".xr-dim-list:before {\n",
" content: \"(\";\n",
"}\n",
"\n",
".xr-dim-list:after {\n",
" content: \")\";\n",
"}\n",
"\n",
".xr-dim-list li:not(:last-child):after {\n",
" content: \",\";\n",
" padding-right: 5px;\n",
"}\n",
"\n",
".xr-has-index {\n",
" font-weight: bold;\n",
"}\n",
"\n",
".xr-var-list,\n",
".xr-var-item {\n",
" display: contents;\n",
"}\n",
"\n",
".xr-var-item > div,\n",
".xr-var-item label,\n",
".xr-var-item > .xr-var-name span {\n",
" background-color: var(--xr-background-color-row-even);\n",
" border-color: var(--xr-background-color-row-odd);\n",
" margin-bottom: 0;\n",
" padding-top: 2px;\n",
"}\n",
"\n",
".xr-var-item > .xr-var-name:hover span {\n",
" padding-right: 5px;\n",
"}\n",
"\n",
".xr-var-list > li:nth-child(odd) > div,\n",
".xr-var-list > li:nth-child(odd) > label,\n",
".xr-var-list > li:nth-child(odd) > .xr-var-name span {\n",
" background-color: var(--xr-background-color-row-odd);\n",
" border-color: var(--xr-background-color-row-even);\n",
"}\n",
"\n",
".xr-var-name {\n",
" grid-column: 1;\n",
"}\n",
"\n",
".xr-var-dims {\n",
" grid-column: 2;\n",
"}\n",
"\n",
".xr-var-dtype {\n",
" grid-column: 3;\n",
" text-align: right;\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-var-preview {\n",
" grid-column: 4;\n",
"}\n",
"\n",
".xr-index-preview {\n",
" grid-column: 2 / 5;\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-var-name,\n",
".xr-var-dims,\n",
".xr-var-dtype,\n",
".xr-preview,\n",
".xr-attrs dt {\n",
" white-space: nowrap;\n",
" overflow: hidden;\n",
" text-overflow: ellipsis;\n",
" padding-right: 10px;\n",
"}\n",
"\n",
".xr-var-name:hover,\n",
".xr-var-dims:hover,\n",
".xr-var-dtype:hover,\n",
".xr-attrs dt:hover {\n",
" overflow: visible;\n",
" width: auto;\n",
" z-index: 1;\n",
"}\n",
"\n",
".xr-var-attrs,\n",
".xr-var-data,\n",
".xr-index-data {\n",
" display: none;\n",
" border-top: 2px dotted var(--xr-background-color);\n",
" padding-bottom: 20px !important;\n",
" padding-top: 10px !important;\n",
"}\n",
"\n",
".xr-var-attrs-in + label,\n",
".xr-var-data-in + label,\n",
".xr-index-data-in + label {\n",
" padding: 0 1px;\n",
"}\n",
"\n",
".xr-var-attrs-in:checked ~ .xr-var-attrs,\n",
".xr-var-data-in:checked ~ .xr-var-data,\n",
".xr-index-data-in:checked ~ .xr-index-data {\n",
" display: block;\n",
"}\n",
"\n",
".xr-var-data > table {\n",
" float: right;\n",
"}\n",
"\n",
".xr-var-data > pre,\n",
".xr-index-data > pre,\n",
".xr-var-data > table > tbody > tr {\n",
" background-color: transparent !important;\n",
"}\n",
"\n",
".xr-var-name span,\n",
".xr-var-data,\n",
".xr-index-name div,\n",
".xr-index-data,\n",
".xr-attrs {\n",
" padding-left: 25px !important;\n",
"}\n",
"\n",
".xr-attrs,\n",
".xr-var-attrs,\n",
".xr-var-data,\n",
".xr-index-data {\n",
" grid-column: 1 / -1;\n",
"}\n",
"\n",
"dl.xr-attrs {\n",
" padding: 0;\n",
" margin: 0;\n",
" display: grid;\n",
" grid-template-columns: 125px auto;\n",
"}\n",
"\n",
".xr-attrs dt,\n",
".xr-attrs dd {\n",
" padding: 0;\n",
" margin: 0;\n",
" float: left;\n",
" padding-right: 10px;\n",
" width: auto;\n",
"}\n",
"\n",
".xr-attrs dt {\n",
" font-weight: normal;\n",
" grid-column: 1;\n",
"}\n",
"\n",
".xr-attrs dt:hover span {\n",
" display: inline-block;\n",
" background: var(--xr-background-color);\n",
" padding-right: 10px;\n",
"}\n",
"\n",
".xr-attrs dd {\n",
" grid-column: 2;\n",
" white-space: pre-wrap;\n",
" word-break: break-all;\n",
"}\n",
"\n",
".xr-icon-database,\n",
".xr-icon-file-text2,\n",
".xr-no-icon {\n",
" display: inline-block;\n",
" vertical-align: middle;\n",
" width: 1em;\n",
" height: 1.5em !important;\n",
" stroke-width: 0;\n",
" stroke: currentColor;\n",
" fill: currentColor;\n",
"}\n",
"\n",
".xr-var-attrs-in:checked + label > .xr-icon-file-text2,\n",
".xr-var-data-in:checked + label > .xr-icon-database,\n",
".xr-index-data-in:checked + label > .xr-icon-database {\n",
" color: var(--xr-font-color0);\n",
" filter: drop-shadow(1px 1px 5px var(--xr-font-color2));\n",
" stroke-width: 0.8px;\n",
"}\n",
"</style><pre class='xr-text-repr-fallback'>&lt;xarray.Dataset&gt; Size: 1kB\n",
"Dimensions: (x: 3, y: 4, time: 7)\n",
"Dimensions without coordinates: x, y, time\n",
"Data variables:\n",
" a (x, y, time) float64 672B 0.2686 0.9632 0.08406 ... 0.6148 0.9757\n",
" b (x, y, time) float64 672B 0.8701 0.2753 0.1478 ... 0.5651 0.6018</pre><div class='xr-wrap' style='display:none'><div class='xr-header'><div class='xr-obj-type'>xarray.Dataset</div></div><ul class='xr-sections'><li class='xr-section-item'><input id='section-9b65efc4-3e64-46c2-8d1d-b88bd54a7629' class='xr-section-summary-in' type='checkbox' disabled ><label for='section-9b65efc4-3e64-46c2-8d1d-b88bd54a7629' class='xr-section-summary' title='Expand/collapse section'>Dimensions:</label><div class='xr-section-inline-details'><ul class='xr-dim-list'><li><span>x</span>: 3</li><li><span>y</span>: 4</li><li><span>time</span>: 7</li></ul></div><div class='xr-section-details'></div></li><li class='xr-section-item'><input id='section-87f55634-42c9-44e6-95bb-16727b265a49' class='xr-section-summary-in' type='checkbox' disabled ><label for='section-87f55634-42c9-44e6-95bb-16727b265a49' class='xr-section-summary' title='Expand/collapse section'>Coordinates: <span>(0)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'></ul></div></li><li class='xr-section-item'><input id='section-ce8e0034-6ad5-4fc2-9676-be3d17cd2acb' class='xr-section-summary-in' type='checkbox' checked><label for='section-ce8e0034-6ad5-4fc2-9676-be3d17cd2acb' class='xr-section-summary' >Data variables: <span>(2)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'><li class='xr-var-item'><div class='xr-var-name'><span>a</span></div><div class='xr-var-dims'>(x, y, time)</div><div class='xr-var-dtype'>float64</div><div class='xr-var-preview xr-preview'>0.2686 0.9632 ... 0.6148 0.9757</div><input id='attrs-5f913a64-494f-4b9d-870b-f61c9e090157' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-5f913a64-494f-4b9d-870b-f61c9e090157' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-9ba22573-faa3-432b-8927-efb0c09c6258' class='xr-var-data-in' type='checkbox'><label for='data-9ba22573-faa3-432b-8927-efb0c09c6258' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><pre>array([[[2.68586428e-01, 9.63157058e-01, 8.40649688e-02, 9.50868102e-01,\n",
" 2.33930515e-02, 4.91602497e-01, 6.83543195e-01],\n",
" [3.86928142e-01, 4.21546159e-01, 4.37755663e-01, 9.25800549e-01,\n",
" 8.94267037e-01, 1.23820068e-01, 5.69886146e-01],\n",
" [9.15443078e-01, 1.17220873e-01, 9.79217082e-02, 7.52954079e-01,\n",
" 2.58211172e-01, 1.12440856e-01, 4.18064437e-01],\n",
" [6.76614301e-01, 9.43812170e-01, 8.11823021e-01, 2.52114866e-01,\n",
" 7.95267765e-01, 8.61103571e-01, 4.10160222e-01]],\n",
"\n",
" [[7.20263299e-01, 6.34449454e-01, 3.77923971e-01, 4.65284430e-01,\n",
" 3.24996068e-01, 9.60683708e-01, 7.85786568e-01],\n",
" [2.47444747e-01, 2.99795853e-01, 1.35412704e-01, 9.76618343e-01,\n",
" 5.16842170e-01, 6.14832320e-01, 2.72568754e-01],\n",
" [9.92407280e-02, 2.09274746e-01, 7.46718666e-01, 6.16382734e-01,\n",
" 4.26803607e-01, 9.14184714e-01, 4.42593530e-01],\n",
" [5.65025945e-01, 7.22983855e-01, 8.06406270e-01, 8.47309505e-01,\n",
" 2.35064082e-01, 9.69442900e-04, 5.44656415e-01]],\n",
"\n",
" [[4.96242121e-01, 4.39204557e-02, 2.54792330e-01, 4.65056094e-01,\n",
" 7.13078341e-02, 4.23680131e-01, 8.84805502e-01],\n",
" [2.12011830e-01, 8.10176773e-01, 6.67336918e-01, 6.05770391e-01,\n",
" 5.57787613e-01, 2.28278245e-01, 7.00191100e-02],\n",
" [8.29432161e-01, 5.52143789e-01, 9.62765610e-01, 2.04277228e-01,\n",
" 8.52342822e-01, 6.19853101e-01, 6.08592767e-01],\n",
" [9.69458328e-02, 4.35198875e-01, 8.66140199e-01, 2.08096677e-01,\n",
" 3.63552026e-01, 6.14823162e-01, 9.75701551e-01]]])</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span>b</span></div><div class='xr-var-dims'>(x, y, time)</div><div class='xr-var-dtype'>float64</div><div class='xr-var-preview xr-preview'>0.8701 0.2753 ... 0.5651 0.6018</div><input id='attrs-297400c4-74a3-483e-b93e-c3d8cad21610' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-297400c4-74a3-483e-b93e-c3d8cad21610' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-334ddc3d-e7b6-4aca-9eaf-a5ec95982f43' class='xr-var-data-in' type='checkbox'><label for='data-334ddc3d-e7b6-4aca-9eaf-a5ec95982f43' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><pre>array([[[8.70090132e-01, 2.75315733e-01, 1.47795888e-01, 3.49603201e-01,\n",
" 1.22715776e-01, 4.04748037e-01, 2.33000906e-01],\n",
" [9.60376491e-01, 9.54662574e-02, 9.54189672e-01, 9.14835312e-01,\n",
" 6.13313639e-01, 1.84613734e-01, 8.86246792e-01],\n",
" [9.34738271e-01, 2.22016510e-01, 3.76010564e-01, 7.09902186e-01,\n",
" 3.62708170e-01, 2.00265554e-01, 1.63515135e-01],\n",
" [6.45244424e-01, 3.86516305e-02, 2.17353852e-01, 6.52530139e-01,\n",
" 1.86364705e-01, 4.12928516e-02, 7.92798146e-01]],\n",
"\n",
" [[3.04858653e-02, 3.44851775e-01, 5.64994747e-01, 8.21515880e-01,\n",
" 5.68067756e-02, 2.70709785e-01, 9.35618161e-01],\n",
" [2.35187045e-01, 8.47561346e-01, 7.22433861e-01, 2.34494639e-02,\n",
" 1.08107183e-01, 5.51906824e-01, 2.87657937e-01],\n",
" [6.82146344e-01, 3.96038779e-01, 6.98605283e-01, 6.42493983e-01,\n",
" 6.76487632e-01, 4.67832527e-01, 2.94352854e-01],\n",
" [4.43438276e-01, 4.17779540e-01, 6.84639850e-01, 4.67472349e-01,\n",
" 7.88849104e-01, 8.47498922e-01, 4.65924446e-01]],\n",
"\n",
" [[9.34924575e-01, 6.83603838e-01, 1.63842554e-02, 9.49170849e-01,\n",
" 1.19023400e-01, 6.21186705e-01, 8.37506956e-01],\n",
" [1.99560418e-01, 6.99863540e-01, 8.22530894e-02, 9.66199256e-01,\n",
" 4.37798203e-01, 6.04663253e-01, 1.92841938e-01],\n",
" [2.08038365e-01, 2.03249548e-01, 5.05775978e-01, 8.68567850e-01,\n",
" 5.37181938e-02, 9.35054688e-03, 9.51485803e-01],\n",
" [4.78357297e-02, 2.13181572e-01, 8.15205605e-01, 3.92972766e-04,\n",
" 7.51922762e-02, 5.65093764e-01, 6.01814175e-01]]])</pre></div></li></ul></div></li><li class='xr-section-item'><input id='section-ca0a5f3f-5b9a-467a-84fa-3cc1e47e8289' class='xr-section-summary-in' type='checkbox' disabled ><label for='section-ca0a5f3f-5b9a-467a-84fa-3cc1e47e8289' class='xr-section-summary' title='Expand/collapse section'>Indexes: <span>(0)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'></ul></div></li><li class='xr-section-item'><input id='section-98558840-e5f2-4946-9ea8-2934e523c330' class='xr-section-summary-in' type='checkbox' disabled ><label for='section-98558840-e5f2-4946-9ea8-2934e523c330' class='xr-section-summary' title='Expand/collapse section'>Attributes: <span>(0)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><dl class='xr-attrs'></dl></div></li></ul></div></div>"
],
"text/plain": [
"<xarray.Dataset> Size: 1kB\n",
"Dimensions: (x: 3, y: 4, time: 7)\n",
"Dimensions without coordinates: x, y, time\n",
"Data variables:\n",
" a (x, y, time) float64 672B 0.2686 0.9632 0.08406 ... 0.6148 0.9757\n",
" b (x, y, time) float64 672B 0.8701 0.2753 0.1478 ... 0.5651 0.6018"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# create test xarray dataset\n",
"nx,ny,nt = (3,4,7)\n",
"ds = xr.Dataset(\n",
" {\n",
" 'a':xr.DataArray(np.random.rand(nx, ny, nt), dims=['x','y', 'time']),\n",
" 'b':xr.DataArray(np.random.rand(nx, ny, nt), dims=['x','y', 'time']),\n",
" }\n",
")\n",
"ds"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "83a39c9d-8a8d-4ef3-bb77-95e95dcfd802",
"metadata": {},
"outputs": [],
"source": [
"repo = ic.Repository.create(ic.in_memory_storage())"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "c19420df-c610-45f2-bb71-728f26e44e33",
"metadata": {},
"outputs": [],
"source": [
"session = repo.writable_session(\"main\")"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "642d297b-c1a8-4992-aee0-8a8220b80ea6",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'1PX30836HX6E6WM0DXWG'"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# write only a part of the data to the store first\n",
"ds.isel(time=slice(0,5)).to_zarr(session.store, consolidated=False, mode='w')\n",
"session.commit(message='initial data')"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "00faf74f-5b17-4199-8efa-39defc6adb90",
"metadata": {},
"outputs": [],
"source": [
"# add a new branch and commit data to the branch\n",
"repo.create_branch(\n",
" 'add_time', \n",
" snapshot_id=repo.lookup_branch(\"main\") #branches of the lates commit to main!\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "6b3b1fb2-1402-4c32-b8da-1738fcb7db10",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'4Y57BXSFS6CBG6XE37HG'"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"session = repo.writable_session('add_time')\n",
"ds.isel(time=slice(5,None)).to_zarr(session.store, mode='a-', append_dim='time', consolidated=False)\n",
"session.commit('added new time steps')"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "4122733f-7b45-41b5-a0b5-f7ab53fb8e02",
"metadata": {},
"outputs": [],
"source": [
"# now open the store on the add_time branch and compare to full dataset\n",
"ds_branch = xr.open_zarr(session.store, consolidated=False)\n",
"xr.testing.assert_allclose(ds_branch, ds)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "b822d209-cc41-4cf0-a9cb-bbb9943f8707",
"metadata": {},
"outputs": [
{
"ename": "AssertionError",
"evalue": "Left and right Dataset objects are not close\nDiffering dimensions:\n (x: 3, y: 4, time: 5) != (x: 3, y: 4, time: 7)\nDiffering data variables:\nL a (x, y, time) float64 480B dask.array<chunksize=(3, 4, 5), meta=np.ndarray>\nR a (x, y, time) float64 672B 0.2686 0.9632 0.08406 ... 0.6148 0.9757\nL b (x, y, time) float64 480B dask.array<chunksize=(3, 4, 5), meta=np.ndarray>\nR b (x, y, time) float64 672B 0.8701 0.2753 0.1478 ... 0.5651 0.6018",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mAssertionError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[9], line 6\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[38;5;66;03m# now open the store on the add_time branch and compare to full dataset\u001b[39;00m\n\u001b[1;32m 5\u001b[0m ds_branch \u001b[38;5;241m=\u001b[39m xr\u001b[38;5;241m.\u001b[39mopen_zarr(session\u001b[38;5;241m.\u001b[39mstore, consolidated\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mFalse\u001b[39;00m)\n\u001b[0;32m----> 6\u001b[0m \u001b[43mxr\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtesting\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43massert_allclose\u001b[49m\u001b[43m(\u001b[49m\u001b[43mds_branch\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mds\u001b[49m\u001b[43m)\u001b[49m\n",
" \u001b[0;31m[... skipping hidden 1 frame]\u001b[0m\n",
"File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.12/site-packages/xarray/testing/assertions.py:242\u001b[0m, in \u001b[0;36massert_allclose\u001b[0;34m(a, b, rtol, atol, decode_bytes, check_dim_order)\u001b[0m\n\u001b[1;32m 238\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(a, Dataset):\n\u001b[1;32m 239\u001b[0m allclose \u001b[38;5;241m=\u001b[39m a\u001b[38;5;241m.\u001b[39m_coord_names \u001b[38;5;241m==\u001b[39m b\u001b[38;5;241m.\u001b[39m_coord_names \u001b[38;5;129;01mand\u001b[39;00m utils\u001b[38;5;241m.\u001b[39mdict_equiv(\n\u001b[1;32m 240\u001b[0m a\u001b[38;5;241m.\u001b[39mvariables, b\u001b[38;5;241m.\u001b[39mvariables, compat\u001b[38;5;241m=\u001b[39mcompat_variable\n\u001b[1;32m 241\u001b[0m )\n\u001b[0;32m--> 242\u001b[0m \u001b[38;5;28;01massert\u001b[39;00m allclose, formatting\u001b[38;5;241m.\u001b[39mdiff_dataset_repr(a, b, compat\u001b[38;5;241m=\u001b[39mequiv)\n\u001b[1;32m 243\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(a, Coordinates):\n\u001b[1;32m 244\u001b[0m allclose \u001b[38;5;241m=\u001b[39m utils\u001b[38;5;241m.\u001b[39mdict_equiv(a\u001b[38;5;241m.\u001b[39mvariables, b\u001b[38;5;241m.\u001b[39mvariables, compat\u001b[38;5;241m=\u001b[39mcompat_variable)\n",
"\u001b[0;31mAssertionError\u001b[0m: Left and right Dataset objects are not close\nDiffering dimensions:\n (x: 3, y: 4, time: 5) != (x: 3, y: 4, time: 7)\nDiffering data variables:\nL a (x, y, time) float64 480B dask.array<chunksize=(3, 4, 5), meta=np.ndarray>\nR a (x, y, time) float64 672B 0.2686 0.9632 0.08406 ... 0.6148 0.9757\nL b (x, y, time) float64 480B dask.array<chunksize=(3, 4, 5), meta=np.ndarray>\nR b (x, y, time) float64 672B 0.8701 0.2753 0.1478 ... 0.5651 0.6018"
]
}
],
"source": [
"# for comparison the data on the main branch does not pass the test yet! There are missing time steps (as expected). \n",
"session = repo.writable_session('main')\n",
"# now open the store on the add_time branch and compare to full dataset\n",
"ds_branch = xr.open_zarr(session.store, consolidated=False)\n",
"xr.testing.assert_allclose(ds_branch, ds)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "3e0189ae-da62-4a44-8675-e5fe3344e612",
"metadata": {},
"outputs": [],
"source": [
"# append branch commit to main branch and delete test branch\n",
"repo.reset_branch('main', repo.lookup_branch('add_time'))\n",
"repo.delete_branch('add_time')"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "baca932c-a33c-4df4-ba9c-7a574b3f4dd3",
"metadata": {},
"outputs": [],
"source": [
"# confirm that the main branch now has the full dataset!\n",
"session = repo.writable_session('main')\n",
"# now open the store on the add_time branch and compare to full dataset\n",
"ds_branch = xr.open_zarr(session.store, consolidated=False)\n",
"xr.testing.assert_allclose(ds_branch, ds)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "96c7712e-a34f-4a6c-8a2f-0986e60c1cb1",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'main'}"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"repo.list_branches()"
]
},
{
"cell_type": "markdown",
"id": "af94cdd9-f746-4e30-9bcc-fc6e0445855e",
"metadata": {},
"source": [
"!['jazz hands'](https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExcGFyZXAxZml2cW1odXBkdmd6YzA2enZ4M25zbWJleXRkbDEzaDJnNiZlcD12MV9naWZzX3NlYXJjaCZjdD1n/142UITjG5GjIRi/giphy.gif)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "331b29b1-efd1-424f-8fd8-d600d0429290",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.7"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment