Skip to content

Instantly share code, notes, and snippets.

@bennyistanto

bennyistanto/OSS_Phenology_ZonalStats_Categorical_and_Join_Tables_admin1.ipynb

Last active November 15, 2023 12:41
Show Gist options
  • Save bennyistanto/0d37731fdeb8118772227c8db514b3bd to your computer and use it in GitHub Desktop.
Save bennyistanto/0d37731fdeb8118772227c8db514b3bd to your computer and use it in GitHub Desktop.
Download ZIP
Zonal Statistics with Categorical
Display the source blob
Display the rendered blob
Raw
OSS_Phenology_ZonalStats_Categorical_and_Join_Tables_admin1.ipynb
{
"cells": [
{
"cell_type": "markdown",
"id": "c4c03770",
"metadata": {},
"source": [
" **Zonal Statistics based on raster value/class with categorical using admin boundaries**\n",
" "
]
},
{
"cell_type": "markdown",
"id": "f2d293c9",
"metadata": {},
"source": [
"**1. Zonal Statistics**\n",
"\n",
"This process required GeoTIFFs in a folder as input and shapefile as boundary zone. This utilize `rasterstats` library, and the process and output are similar to ArcPy Zonal Histogram (`categorical=True`), if the value is `False` then the result will produce simlar output like ArcPy Zonal Statistics."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "fcbc1a29",
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 1%|▋ | 1/84 [00:21<29:04, 21.01s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2003_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 2%|█▎ | 2/84 [00:34<22:23, 16.38s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2003_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 4%|█▉ | 3/84 [00:50<21:49, 16.16s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2003_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 5%|██▌ | 4/84 [01:03<20:01, 15.02s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2003_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 6%|███▏ | 5/84 [01:21<21:28, 16.31s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2004_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 7%|███▊ | 6/84 [01:35<19:49, 15.25s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2004_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 8%|████▍ | 7/84 [01:50<19:40, 15.34s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2004_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 10%|█████ | 8/84 [02:04<18:42, 14.76s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2004_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 11%|█████▋ | 9/84 [02:21<19:23, 15.51s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2005_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 12%|██████▏ | 10/84 [02:34<18:13, 14.77s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2005_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 13%|██████▊ | 11/84 [02:50<18:37, 15.31s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2005_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 14%|███████▍ | 12/84 [03:04<17:53, 14.91s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2005_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 15%|████████ | 13/84 [03:22<18:39, 15.77s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2006_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 17%|████████▋ | 14/84 [03:35<17:28, 14.98s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2006_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 18%|█████████▎ | 15/84 [03:52<17:46, 15.46s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2006_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 19%|█████████▉ | 16/84 [04:07<17:15, 15.22s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2006_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 20%|██████████▌ | 17/84 [04:25<18:06, 16.21s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2007_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 21%|███████████▏ | 18/84 [04:39<17:02, 15.50s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2007_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 23%|███████████▊ | 19/84 [04:56<17:15, 15.93s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2007_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 24%|████████████▍ | 20/84 [05:10<16:33, 15.52s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2007_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 25%|█████████████ | 21/84 [05:30<17:32, 16.70s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2008_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 26%|█████████████▌ | 22/84 [05:44<16:26, 15.90s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2008_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 27%|██████████████▏ | 23/84 [06:01<16:37, 16.36s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2008_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 29%|██████████████▊ | 24/84 [06:16<15:42, 15.72s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2008_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 30%|███████████████▍ | 25/84 [06:35<16:30, 16.78s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2009_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 31%|████████████████ | 26/84 [06:49<15:19, 15.85s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2009_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 32%|████████████████▋ | 27/84 [07:06<15:28, 16.30s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2009_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 33%|█████████████████▎ | 28/84 [07:21<14:48, 15.87s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2009_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 35%|█████████████████▉ | 29/84 [07:40<15:26, 16.85s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2010_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 36%|██████████████████▌ | 30/84 [07:53<14:13, 15.81s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2010_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 37%|███████████████████▏ | 31/84 [08:09<13:51, 15.69s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2010_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 38%|███████████████████▊ | 32/84 [08:23<13:12, 15.23s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2010_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 39%|████████████████████▍ | 33/84 [08:43<14:05, 16.57s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2011_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 40%|█████████████████████ | 34/84 [08:56<12:57, 15.55s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2011_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 42%|█████████████████████▋ | 35/84 [09:14<13:21, 16.36s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2011_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 43%|██████████████████████▎ | 36/84 [09:28<12:36, 15.75s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2011_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 44%|██████████████████████▉ | 37/84 [09:48<13:11, 16.83s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2012_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 45%|███████████████████████▌ | 38/84 [10:02<12:18, 16.06s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2012_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 46%|████████████████████████▏ | 39/84 [10:19<12:12, 16.28s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2012_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 48%|████████████████████████▊ | 40/84 [10:33<11:31, 15.71s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2012_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 49%|█████████████████████████▍ | 41/84 [10:52<11:53, 16.60s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2013_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 50%|██████████████████████████ | 42/84 [11:06<11:01, 15.75s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2013_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 51%|██████████████████████████▌ | 43/84 [11:22<10:57, 16.02s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2013_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 52%|███████████████████████████▏ | 44/84 [11:37<10:20, 15.52s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2013_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 54%|███████████████████████████▊ | 45/84 [11:55<10:43, 16.51s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2014_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 55%|████████████████████████████▍ | 46/84 [12:09<09:53, 15.61s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2014_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 56%|█████████████████████████████ | 47/84 [12:27<10:04, 16.33s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2014_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 57%|█████████████████████████████▋ | 48/84 [12:41<09:26, 15.73s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2014_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 58%|██████████████████████████████▎ | 49/84 [13:00<09:45, 16.73s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2015_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 60%|██████████████████████████████▉ | 50/84 [13:14<09:03, 15.98s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2015_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 61%|███████████████████████████████▌ | 51/84 [13:31<08:48, 16.00s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2015_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 62%|████████████████████████████████▏ | 52/84 [13:46<08:25, 15.80s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2015_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 63%|████████████████████████████████▊ | 53/84 [14:05<08:44, 16.92s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2016_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 64%|█████████████████████████████████▍ | 54/84 [14:20<08:05, 16.20s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2016_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 65%|██████████████████████████████████ | 55/84 [14:37<07:59, 16.52s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2016_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 67%|██████████████████████████████████▋ | 56/84 [14:52<07:28, 16.03s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2016_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 68%|███████████████████████████████████▎ | 57/84 [15:12<07:45, 17.22s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2017_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 69%|███████████████████████████████████▉ | 58/84 [15:27<07:07, 16.44s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2017_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 70%|████████████████████████████████████▌ | 59/84 [15:45<07:08, 17.13s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2017_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 71%|█████████████████████████████████████▏ | 60/84 [16:00<06:29, 16.24s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2017_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 73%|█████████████████████████████████████▊ | 61/84 [16:19<06:35, 17.21s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2018_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 74%|██████████████████████████████████████▍ | 62/84 [16:33<05:58, 16.29s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2018_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 75%|███████████████████████████████████████ | 63/84 [16:51<05:52, 16.80s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2018_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 76%|███████████████████████████████████████▌ | 64/84 [17:05<05:20, 16.04s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2018_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 77%|████████████████████████████████████████▏ | 65/84 [17:25<05:23, 17.02s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2019_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 79%|████████████████████████████████████████▊ | 66/84 [17:38<04:47, 16.00s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2019_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 80%|█████████████████████████████████████████▍ | 67/84 [17:55<04:33, 16.06s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2019_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 81%|██████████████████████████████████████████ | 68/84 [18:09<04:07, 15.47s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2019_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 82%|██████████████████████████████████████████▋ | 69/84 [18:28<04:07, 16.50s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2020_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 83%|███████████████████████████████████████████▎ | 70/84 [18:42<03:40, 15.74s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2020_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 85%|███████████████████████████████████████████▉ | 71/84 [18:59<03:30, 16.18s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2020_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 86%|████████████████████████████████████████████▌ | 72/84 [19:13<03:07, 15.61s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2020_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 87%|█████████████████████████████████████████████▏ | 73/84 [19:33<03:04, 16.77s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2021_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 88%|█████████████████████████████████████████████▊ | 74/84 [19:47<02:40, 16.09s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2021_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 89%|██████████████████████████████████████████████▍ | 75/84 [20:03<02:25, 16.19s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2021_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 90%|███████████████████████████████████████████████ | 76/84 [20:18<02:06, 15.84s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2021_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 92%|███████████████████████████████████████████████▋ | 77/84 [20:37<01:57, 16.76s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2022_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 93%|████████████████████████████████████████████████▎ | 78/84 [20:52<01:35, 15.98s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2022_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 94%|████████████████████████████████████████████████▉ | 79/84 [21:10<01:23, 16.62s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2022_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 95%|█████████████████████████████████████████████████▌ | 80/84 [21:24<01:03, 15.96s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2022_sos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 96%|██████████████████████████████████████████████████▏ | 81/84 [21:43<00:50, 16.96s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2023_eos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 98%|██████████████████████████████████████████████████▊ | 82/84 [21:57<00:31, 15.85s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2023_eos_season2.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Processing: 99%|███████████████████████████████████████████████████▍| 83/84 [22:12<00:15, 15.79s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2023_sos_season1.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 100%|████████████████████████████████████████████████████| 84/84 [22:27<00:00, 16.04s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2023_sos_season2.csv\n",
"Zonal Histogram processing complete!\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n"
]
}
],
"source": [
"import os\n",
"import geopandas as gpd\n",
"from rasterstats import zonal_stats\n",
"import pandas as pd\n",
"from tqdm import tqdm\n",
"\n",
"# Define input, zone, and output locations\n",
"input_folder = \"/mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step05_esa10m/final\"\n",
"zonefieldVector = \"/mnt/x/Temp/modis/lbn/bnd/lbn_bnd_adm1_edge.shp\"\n",
"output_folder = \"/mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1\"\n",
"selected_fields = ['adm1_src', 'adm1_name', 'adm0_src', 'adm0_name', 'geometry']\n",
"\n",
"# Ensure that the output folder exists\n",
"if not os.path.exists(output_folder):\n",
" os.makedirs(output_folder)\n",
"\n",
"# Read the vector data using Geopandas\n",
"vector_data_full = gpd.read_file(zonefieldVector)\n",
"# Select the desired fields and sort by 'adm3_src'\n",
"vector_data = vector_data_full[selected_fields].sort_values(by='adm1_src')\n",
"# Reset the index to ensure a proper join later on\n",
"vector_data = vector_data.reset_index(drop=True)\n",
"\n",
"# List all .tif files\n",
"tif_files = [f for f in os.listdir(input_folder) if f.endswith('.tif')]\n",
"\n",
"# Loop through each file in the input folder and compute Zonal Histogram\n",
"for filename in tqdm(tif_files, desc='Processing', unit='file'):\n",
" input_file_path = os.path.join(input_folder, filename)\n",
" output_file_path = os.path.join(output_folder, os.path.splitext(filename)[0] + \".csv\")\n",
"\n",
" # Compute Zonal Histogram using rasterstats\n",
" stats = zonal_stats(vector_data, input_file_path, categorical=True, all_touched=True)\n",
"\n",
" # Convert the stats to a pandas DataFrame and sort columns\n",
" stats_df = pd.DataFrame(stats).sort_index(axis=1)\n",
" stats_df = stats_df.reindex(sorted(stats_df.columns), axis=1) # Sort columns by header name\n",
"\n",
" # Merge the statistics back with the original geodataframe\n",
" vector_data_stats = vector_data.join(stats_df)\n",
"\n",
" # Add an 'ID' column based on the new sorted index\n",
" vector_data_stats.reset_index(inplace=True)\n",
" vector_data_stats.rename(columns={'index': 'ID'}, inplace=True)\n",
"\n",
" # Save to CSV\n",
" vector_data_stats.to_csv(output_file_path, index=False)\n",
" print(f\"Output file created: {output_file_path}\")\n",
"\n",
"print(\"Zonal Histogram processing complete!\")\n"
]
},
{
"cell_type": "markdown",
"id": "c9127a8a",
"metadata": {},
"source": [
"**2. Merge season1 and season2 into single data**\n",
"\n",
"We need to do SUM calculation for season1 and season2 in order to get total areas for each stages."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "b127fa84",
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2017_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2017_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2010_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2010_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2021_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2021_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2012_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2012_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2006_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2006_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2020_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2020_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2023_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2023_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2004_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2004_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2016_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2016_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2007_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2007_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2022_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2022_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2015_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2015_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2014_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2014_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2018_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2018_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2013_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2013_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2011_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2011_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2003_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2003_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2019_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2019_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2009_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2009_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2005_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2005_sos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2008_eos.csv\n",
"Merged file created: /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_2008_sos.csv\n",
"Merging and calculation complete!\n"
]
}
],
"source": [
"import pandas as pd\n",
"import os\n",
"from glob import glob\n",
"\n",
"# Define the output folder where the CSV files are stored\n",
"output_folder = \"/mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1\"\n",
"\n",
"# Get all the unique base filenames without season part\n",
"base_files = set(os.path.splitext(f)[0].rsplit('_', 2)[0] for f in os.listdir(output_folder) if f.endswith('.csv'))\n",
"\n",
"# Define the columns to sum and then multiply\n",
"sum_columns = [str(i) for i in range(1, 13)]\n",
"\n",
"# Process each base file\n",
"for base in base_files:\n",
" # Pattern to match season files for the same year and type (eos or sos)\n",
" for season_type in ['eos', 'sos']:\n",
" pattern = os.path.join(output_folder, f\"{base}_{season_type}_season*.csv\")\n",
" season_files = glob(pattern)\n",
"\n",
" # Skip if there aren't two seasons to process\n",
" if len(season_files) != 2:\n",
" print(f\"Skipping {base}_{season_type} as it does not have two seasons.\")\n",
" continue\n",
"\n",
" # Load the season data into DataFrames\n",
" season_dfs = [pd.read_csv(season_file) for season_file in season_files]\n",
"\n",
" # Ensure all expected columns are present in both DataFrames, fill missing with zeros\n",
" for df in season_dfs:\n",
" for col in sum_columns:\n",
" if col not in df.columns:\n",
" df[col] = 0\n",
"\n",
" # Merge the dataframes on the common fields\n",
" common_fields = ['ID', 'adm1_src', 'adm1_name', 'adm0_src', 'adm0_name']\n",
" merged_df = pd.merge(season_dfs[0], \n",
" season_dfs[1], \n",
" on=common_fields, \n",
" suffixes=('_season1', '_season2'))\n",
"\n",
" # Drop the geometry columns after merging\n",
" merged_df.drop(columns=['geometry_season1', 'geometry_season2'], inplace=True)\n",
"\n",
" # Sum the columns for each season and multiply by 0.01\n",
" for col in sum_columns:\n",
" merged_df[col] = (merged_df[f\"{col}_season1\"] + merged_df[f\"{col}_season2\"]) * 0.01\n",
"\n",
" # Drop the season-specific columns\n",
" merged_df.drop(columns=[f\"{col}_season1\" for col in sum_columns] + [f\"{col}_season2\" for col in sum_columns], inplace=True)\n",
"\n",
" # Save the merged DataFrame to a new CSV file, excluding the index\n",
" merged_filename = f\"{base}_{season_type}.csv\"\n",
" merged_df.to_csv(os.path.join(output_folder, merged_filename), index=False)\n",
" print(f\"Merged file created: {os.path.join(output_folder, merged_filename)}\")\n",
"\n",
"print(\"Merging and calculation complete!\")\n"
]
},
{
"cell_type": "markdown",
"id": "49aed0aa",
"metadata": {},
"source": [
"**3. Compile all data (each stages) into single csv**\n",
"\n",
"The input is each stages file (`eos` and `sos`), 1 file for 1 year data. Will compile it into single csv file for each stages."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "ecbcb8f5",
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Time series CSV files created for EOS and SOS.\n"
]
}
],
"source": [
"import pandas as pd\n",
"from pathlib import Path\n",
"\n",
"# Define the output folder where the merged CSV files are stored\n",
"output_folder = Path(\"/mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1\")\n",
"\n",
"# Patterns to match EOS and SOS files\n",
"eos_pattern = \"lbn_pheno_*_eos.csv\"\n",
"sos_pattern = \"lbn_pheno_*_sos.csv\"\n",
"\n",
"# Read all EOS and SOS files into DataFrames\n",
"eos_files = list(output_folder.glob(eos_pattern))\n",
"sos_files = list(output_folder.glob(sos_pattern))\n",
"\n",
"def process_files(files):\n",
" dfs = [] # list to hold all dataframes\n",
" for file in files:\n",
" df = pd.read_csv(file)\n",
" # Extract the year and month from the filename\n",
" year_month = file.stem.split('_')[2] # Assuming the format is lbn_pheno_YYYY_eos.csv or lbn_pheno_YYYY_sos.csv\n",
" year_month_formatted = f\"{year_month[:4]}{year_month[-2:]}\" if len(year_month) == 6 else year_month\n",
" # Create a column for each month by replicating the year_month value\n",
" for i in range(1, 13):\n",
" df[f'{year_month_formatted}{str(i).zfill(2)}'] = df[str(i)]\n",
" # Keep the location and new yyyymm columns\n",
" df = df[['ID', 'adm1_src', 'adm1_name', 'adm0_src', 'adm0_name'] + \n",
" [f'{year_month_formatted}{str(i).zfill(2)}' for i in range(1, 13)]]\n",
" dfs.append(df)\n",
" \n",
" # Concatenate all dataframes\n",
" final_df = pd.concat(dfs)\n",
" # Group by the location columns and sum the values across the files (rows) for each year-month\n",
" group_columns = ['ID', 'adm1_src', 'adm1_name', 'adm0_src', 'adm0_name']\n",
" final_df = final_df.groupby(group_columns, as_index=False).sum()\n",
" return final_df\n",
"\n",
"# Process EOS and SOS files\n",
"eos_df = process_files(eos_files)\n",
"sos_df = process_files(sos_files)\n",
"\n",
"# Save the final DataFrames to CSV\n",
"eos_df.to_csv(output_folder / \"lbn_pheno_eos_monthly_adm1.csv\", index=False)\n",
"sos_df.to_csv(output_folder / \"lbn_pheno_sos_monthly_adm1.csv\", index=False)\n",
"\n",
"print(\"Time series CSV files created for EOS and SOS.\")\n"
]
},
{
"cell_type": "markdown",
"id": "c00a9d39",
"metadata": {},
"source": [
"**4. Calculate annual from monthly timeseries**\n",
"\n",
"From the monthly timeseries data, will calculate the annual `SUM`, to get annual `sos` and `eos` value."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "10f83d8d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Annual values saved to /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_sos_annual_adm1.csv\n",
"Annual values saved to /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_eos_annual_adm1.csv\n"
]
}
],
"source": [
"import pandas as pd\n",
"from pathlib import Path\n",
"\n",
"# Define the output folder where the merged CSV files are stored\n",
"output_folder = Path(\"/mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1\")\n",
"\n",
"def calculate_annual_values(file_path):\n",
" # Read the CSV file\n",
" df = pd.read_csv(file_path)\n",
"\n",
" # Get all yyyymm columns for monthly data\n",
" monthly_columns = [col for col in df.columns if col.startswith('20')]\n",
"\n",
" # Calculate the annual sum for each year present in the columns\n",
" # Assuming the format is YYYYMM\n",
" years = sorted(set(col[:4] for col in monthly_columns))\n",
" for year in years:\n",
" # Sum all columns for the specific year\n",
" df[f'{year}_annual'] = df[[col for col in monthly_columns if col.startswith(year)]].sum(axis=1)\n",
"\n",
" # Keep the location, ID and annual columns\n",
" annual_columns = ['ID', 'adm1_src', 'adm1_name', 'adm0_src', 'adm0_name'] + \\\n",
" [f'{year}_annual' for year in years]\n",
" annual_df = df[annual_columns]\n",
"\n",
" # Construct the new filename\n",
" new_filename = file_path.stem.replace('monthly_adm1', 'annual_adm1') + '.csv'\n",
" annual_file_path = file_path.parent / new_filename\n",
" annual_df.to_csv(annual_file_path, index=False)\n",
"\n",
" print(f\"Annual values saved to {annual_file_path}\")\n",
"\n",
"# Call the function for both SOS and EOS CSV files\n",
"calculate_annual_values(output_folder / \"lbn_pheno_sos_monthly_adm1.csv\")\n",
"calculate_annual_values(output_folder / \"lbn_pheno_eos_monthly_adm1.csv\")\n"
]
},
{
"cell_type": "markdown",
"id": "8e7c7e40",
"metadata": {},
"source": [
"**5. Calculate monthly LTA and the timeseries anomaly**"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "534801c6",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Monthly anomalies saved to /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_sos__monthly_anomaly_with_LTA_2003_2022_adm1.csv\n",
"Monthly anomalies saved to /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_eos__monthly_anomaly_with_LTA_2003_2022_adm1.csv\n"
]
}
],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"from pathlib import Path\n",
"\n",
"# Define the output folder where the merged CSV files are stored\n",
"output_folder = Path(\"/mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1\")\n",
"\n",
"def calculate_monthly_lta_anomaly(file_path, start_year, end_year):\n",
" # Read the CSV file\n",
" df = pd.read_csv(file_path)\n",
"\n",
" # Get all yyyymm columns for monthly data\n",
" monthly_columns = [col for col in df.columns if col.startswith('20')]\n",
"\n",
" # Calculate the LTA for each month across the specified years\n",
" lta_columns = {}\n",
" for month in range(1, 13):\n",
" month_str = f'{month:02d}'\n",
" month_cols = [f'{year}{month_str}' for year in range(start_year, end_year + 1)]\n",
" month_cols = [col for col in month_cols if col in monthly_columns]\n",
" lta_columns[f'LTA_{month_str}'] = df[month_cols].mean(axis=1, skipna=True)\n",
"\n",
" # Add LTA columns to DataFrame\n",
" df = pd.concat([df, pd.DataFrame(lta_columns)], axis=1)\n",
"\n",
" # Calculate the anomaly for each month\n",
" anomaly_columns = {}\n",
" for col in monthly_columns:\n",
" year, month = col[:4], col[4:]\n",
" anomaly_col = f'{col}_anomaly'\n",
" lta_col = f'LTA_{month}'\n",
" conditions = [\n",
" (df[col].notna() & (df[col] != 0)),\n",
" (df[col] == 0)\n",
" ]\n",
" choices = [\n",
" (df[col] - df[lta_col]) / df[lta_col] * 100,\n",
" np.nan\n",
" ]\n",
" anomaly_columns[anomaly_col] = np.select(conditions, choices, default=np.nan)\n",
"\n",
" # Add anomaly columns to DataFrame\n",
" df = pd.concat([df, pd.DataFrame(anomaly_columns)], axis=1)\n",
"\n",
" # Construct the new filename by removing 'monthly_adm1_' and adding '_monthly_anomaly_with_LTA'\n",
" new_filename = file_path.stem.replace('monthly_adm1', '') + '_monthly_anomaly_with_LTA_' + \\\n",
" f'{start_year}_{end_year}_adm1.csv'\n",
" anomaly_file_path = file_path.parent / new_filename\n",
" df.to_csv(anomaly_file_path, index=False)\n",
"\n",
" print(f\"Monthly anomalies saved to {anomaly_file_path}\")\n",
"\n",
"# Define the Long Term Average period\n",
"start_year = 2003\n",
"end_year = 2022\n",
"\n",
"# Call the function for the monthly time series CSV file\n",
"calculate_monthly_lta_anomaly(output_folder / \"lbn_pheno_sos_monthly_adm1.csv\", start_year, end_year)\n",
"calculate_monthly_lta_anomaly(output_folder / \"lbn_pheno_eos_monthly_adm1.csv\", start_year, end_year)\n"
]
},
{
"cell_type": "markdown",
"id": "cf2eaeeb",
"metadata": {},
"source": [
"**6. Calculate annual LTA and the timeseries anomaly**"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "c85be44d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Annual ratio anomalies saved to /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_sos_annual_anomaly_with_LTA_2003_2022_adm1.csv\n",
"Annual ratio anomalies saved to /mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1/lbn_pheno_eos_annual_anomaly_with_LTA_2003_2022_adm1.csv\n"
]
}
],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"from pathlib import Path\n",
"\n",
"# Define the output folder where the merged CSV files are stored\n",
"output_folder = Path(\"/mnt/x/Temp/modis/lbn/gee/11_timesat_raw/geotiff/step06_zonalhist/adm1\")\n",
"\n",
"def calculate_ratio_anomaly(file_path, start_year, end_year):\n",
" # Read the CSV file with annual values\n",
" df = pd.read_csv(file_path)\n",
"\n",
" # Identify all the columns that represent years\n",
" annual_columns = [col for col in df.columns if '_annual' in col]\n",
"\n",
" # Calculate the LTA for each location, excluding NaN values\n",
" lta_columns = [f'{year}_annual' for year in range(start_year, end_year + 1)]\n",
" df['LTA'] = df[lta_columns].mean(axis=1, skipna=True)\n",
"\n",
" # Calculate the percent difference for each year compared to the LTA\n",
" for year_col in annual_columns: # This will cover all available years\n",
" anomaly_col = f'{year_col}_anomaly'\n",
" # Calculate anomaly only when annual data is not NaN or zero\n",
" conditions = [\n",
" (df[year_col].notna() & (df[year_col] != 0)),\n",
" (df[year_col] == 0)\n",
" ]\n",
" choices = [\n",
" (df[year_col] - df['LTA']) / df['LTA'] * 100, # normal anomaly calculation\n",
" np.nan # set anomaly as NaN if annual data is 0\n",
" ]\n",
" df[anomaly_col] = np.select(conditions, choices, default=np.nan)\n",
"\n",
" # Construct the new filename by removing 'monthly_adm1_' and adding '_monthly_anomaly_with_LTA'\n",
" new_filename = file_path.stem.replace('annual_adm1', '') + 'annual_anomaly_with_LTA_' + \\\n",
" f'{start_year}_{end_year}_adm1.csv'\n",
" anomaly_file_path = file_path.parent / new_filename\n",
" df.to_csv(anomaly_file_path, index=False)\n",
"\n",
" print(f\"Annual ratio anomalies saved to {anomaly_file_path}\")\n",
"\n",
"# Define the Long Term Average period\n",
"start_year = 2003\n",
"end_year = 2022\n",
"\n",
"# Call the function for both SOS and EOS annual data CSV files\n",
"calculate_ratio_anomaly(output_folder / \"lbn_pheno_sos_annual_adm1.csv\", start_year, end_year)\n",
"calculate_ratio_anomaly(output_folder / \"lbn_pheno_eos_annual_adm1.csv\", start_year, end_year)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fb8d4200",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.4"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Raw
ZonalStats_categorical_and_join_the_tables.md

This process required GeoTIFFs in a folder as input and shapefile as boundary zone. This utilize rasterstats library, and the process and output are similar to ArcPy Zonal Histogram (categorical=True), if the value is False then the result will produce simlar output like ArcPy Zonal Statistics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment