Skip to content

Instantly share code, notes, and snippets.

@bennyistanto

bennyistanto/zonal_statistics_categorical_landcover.ipynb

Created March 19, 2024 06:05
Show Gist options
  • Save bennyistanto/6907a3bdb9e8de25bafc2254fec7cd0d to your computer and use it in GitHub Desktop.
Save bennyistanto/6907a3bdb9e8de25bafc2254fec7cd0d to your computer and use it in GitHub Desktop.
Download ZIP
Zonal Statistics Categorical (Zonal Histogram) for ESA CCI Landcover
Display the source blob
Display the rendered blob
Raw
zonal_statistics_categorical_landcover.ipynb
{
"cells": [
{
"cell_type": "markdown",
"id": "185773ab",
"metadata": {},
"source": [
"# Zonal Statistics for Landcover"
]
},
{
"cell_type": "markdown",
"id": "1f66b5c2",
"metadata": {},
"source": [
"This process required GeoTIFFs in a folder as input and shapefile as boundary zone. This utilize `rasterstats` library. The script is designed to work using **ESA CCI Landcover** - https://cds.climate.copernicus.eu/cdsapp#!/dataset/satellite-land-cover?tab=overview\n",
"\n",
"-- <br>\n",
"Benny Istanto, Climate Geographer <br>\n",
"GOST/DECSC/DECDG, The World Bank <br>\n",
"\n",
"-----\n",
"In the script below, the use of the `zonal_stats` function from the `rasterstats` library with the `categorical=True` option performs a zonal histogram calculation. This calculates the count of pixels within each zone (defined by our vector data) that belong to each land cover class (categorical value) in the raster data.\n",
"\n",
"Since **ESA CCI Landcover** data has a spatial resolution of `300m x 300m`, each pixel represents an area of `90000 square meters` or `9 hectares`. Therefore, the count of pixels for each land cover class obtained from the zonal histogram reflects the number of `9-hectare` units of that class within the zone. Multiplying the pixel count by `9`, converts the pixel counts into hectares, providing a more interpretable measure of area for each land cover class within the specified zones."
]
},
{
"cell_type": "markdown",
"id": "26f9f683",
"metadata": {},
"source": [
"## 1. Categorical\n",
"\n",
"The process and output are similar to **ArcPy Zonal Histogram** (`categorical=True`), if the value is `False` then the result will produce simlar output like **ArcPy Zonal Statistics**."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "c91e3bb3",
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 0%| | 0/29 [00:00<?, ?file/s]/home/bennyistanto/anaconda3/envs/climate/lib/python3.12/site-packages/rasterstats/io.py:328: NodataWarning: Setting nodata to -999; specify nodata explicitly\n",
" warnings.warn(\n",
"Processing: 3%|██▍ | 1/29 [00:05<02:41, 5.75s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_1992.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 7%|████▊ | 2/29 [00:12<02:45, 6.12s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_1993.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 10%|███████▏ | 3/29 [00:18<02:37, 6.07s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_1994.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 14%|█████████▋ | 4/29 [00:24<02:34, 6.18s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_1995.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 17%|████████████ | 5/29 [00:30<02:27, 6.16s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_1996.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 21%|██████████████▍ | 6/29 [00:36<02:19, 6.09s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_1997.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 24%|████████████████▉ | 7/29 [00:42<02:13, 6.06s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_1998.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 28%|███████████████████▎ | 8/29 [00:48<02:07, 6.07s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_1999.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 31%|█████████████████████▋ | 9/29 [00:54<02:00, 6.03s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2000.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 34%|███████████████████████▊ | 10/29 [01:00<01:56, 6.12s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2001.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 38%|██████████████████████████▏ | 11/29 [01:06<01:49, 6.11s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2002.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 41%|████████████████████████████▌ | 12/29 [01:13<01:43, 6.11s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2003.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 45%|██████████████████████████████▉ | 13/29 [01:18<01:35, 5.96s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2004.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 48%|█████████████████████████████████▎ | 14/29 [01:24<01:27, 5.82s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2005.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 52%|███████████████████████████████████▋ | 15/29 [01:29<01:20, 5.78s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2006.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 55%|██████████████████████████████████████ | 16/29 [01:35<01:15, 5.79s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2007.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 59%|████████████████████████████████████████▍ | 17/29 [01:41<01:08, 5.70s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2008.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 62%|██████████████████████████████████████████▊ | 18/29 [01:46<01:02, 5.72s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2009.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 66%|█████████████████████████████████████████████▏ | 19/29 [01:52<00:56, 5.69s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2010.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 69%|███████████████████████████████████████████████▌ | 20/29 [01:58<00:50, 5.63s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2011.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 72%|█████████████████████████████████████████████████▉ | 21/29 [02:03<00:45, 5.64s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2012.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 76%|████████████████████████████████████████████████████▎ | 22/29 [02:09<00:39, 5.69s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2013.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 79%|██████████████████████████████████████████████████████▋ | 23/29 [02:15<00:34, 5.73s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2014.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 83%|█████████████████████████████████████████████████████████ | 24/29 [02:21<00:29, 5.93s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2015.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 86%|███████████████████████████████████████████████████████████▍ | 25/29 [02:27<00:23, 5.97s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2016.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 90%|█████████████████████████████████████████████████████████████▊ | 26/29 [02:33<00:17, 5.96s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2017.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 93%|████████████████████████████████████████████████████████████████▏ | 27/29 [02:40<00:12, 6.05s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2018.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 97%|██████████████████████████████████████████████████████████████████▌ | 28/29 [04:09<00:31, 31.21s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2019.csv\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing: 100%|█████████████████████████████████████████████████████████████████████| 29/29 [05:39<00:00, 11.72s/file]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output file created: /mnt/d/temp/esacci_landcover/zonal/moz/moz_phy_esacci_landcover_lccs_2020.csv\n",
"Zonal Histogram processing complete!\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n"
]
}
],
"source": [
"import os\n",
"import geopandas as gpd\n",
"from rasterstats import zonal_stats\n",
"import pandas as pd\n",
"from tqdm import tqdm\n",
"\n",
"# Set the country\n",
"iso3 = \"moz\" # lbn, syr, mmr, tur\n",
"\n",
"# Define input, zone, and output locations\n",
"input_folder = f\"/mnt/d/temp/esacci_landcover/lccs_class\"\n",
"zonefieldVector = f\"/mnt/d/temp/request/{iso3}/adm2/{iso3}_bnd_adm2_geoboundaries_a.shp\"\n",
"output_folder = f\"/mnt/d/temp/esacci_landcover/zonal/{iso3}\"\n",
"selected_fields = ['ADM2_PCODE', 'ADM2_PT', 'ADM1_PCODE', 'ADM1_PT', 'ADM0_PCODE', 'ADM0_PT', 'geometry']\n",
"\n",
"# Ensure that the output folder exists\n",
"if not os.path.exists(output_folder):\n",
" os.makedirs(output_folder)\n",
"\n",
"# Read the vector data using Geopandas\n",
"vector_data_full = gpd.read_file(zonefieldVector)\n",
"vector_data = vector_data_full[selected_fields].sort_values(by='ADM2_PCODE')\n",
"vector_data = vector_data.reset_index(drop=True)\n",
"\n",
"# List all .tif files\n",
"tif_files = [f for f in os.listdir(input_folder) if f.endswith('.tif')]\n",
"\n",
"# Loop through each file in the input folder and compute Zonal Histogram\n",
"for filename in tqdm(tif_files, desc='Processing', unit='file'): # tqdm is used here\n",
" input_file_path = os.path.join(input_folder, filename)\n",
"\n",
" # Modify the output filename to replace 'wld' with the iso3 code\n",
" output_filename = filename.replace(\"wld\", iso3).replace(\".tif\", \".csv\")\n",
" output_file_path = os.path.join(output_folder, output_filename)\n",
"\n",
" # Compute Zonal Histogram using rasterstats\n",
" stats = zonal_stats(vector_data, input_file_path, categorical=True, all_touched=True)\n",
"\n",
" # Convert the stats to a pandas DataFrame and sort columns\n",
" stats_df = pd.DataFrame(stats).sort_index(axis=1)\n",
" stats_df = stats_df.reindex(sorted(stats_df.columns), axis=1)\n",
"\n",
" # Merge the statistics back with the original geodataframe\n",
" vector_data_stats = vector_data.join(stats_df)\n",
"\n",
" # Add an 'ID' column based on the new sorted index\n",
" vector_data_stats.reset_index(inplace=True)\n",
" vector_data_stats.rename(columns={'index': 'ID'}, inplace=True)\n",
"\n",
" # After dropping 'geometry', ensure you're only selecting numeric columns for multiplication.\n",
" vector_data_stats.drop(columns=['geometry'], inplace=True)\n",
" \n",
" # Define a condition for selecting columns: numeric types excluding specific columns\n",
" is_numeric_and_not_id = lambda col: (vector_data_stats[col].dtype.kind in 'biufc' \n",
" and col not in ['ID', 'ADM2_PCODE', 'ADM2_PT', \n",
" 'ADM1_PCODE', 'ADM1_PT', \n",
" 'ADM0_PCODE', 'ADM0_PT'])\n",
" \n",
" # Use a list comprehension with the defined condition to select columns\n",
" value_columns = [col for col in vector_data_stats.columns if is_numeric_and_not_id(col)]\n",
"\n",
" # Now you can safely multiply without encountering the error.\n",
" vector_data_stats[value_columns] = vector_data_stats[value_columns] * 9\n",
" \n",
" # Continue with saving to CSV...\n",
" vector_data_stats.to_csv(output_file_path, index=False)\n",
"\n",
" print(f\"Output file created: {output_file_path}\")\n",
"\n",
"print(\"Zonal Histogram processing complete!\")\n"
]
},
{
"cell_type": "markdown",
"id": "59638bfd",
"metadata": {},
"source": [
"## Compile the result\n",
"\n",
"Compile and arrange each class into separate file"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "8e7ef617",
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Compiled data for column 10 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_10_adm2.csv\n",
"Compiled data for column 11 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_11_adm2.csv\n",
"Compiled data for column 12 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_12_adm2.csv\n",
"Compiled data for column 20 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_20_adm2.csv\n",
"Compiled data for column 30 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_30_adm2.csv\n",
"Compiled data for column 40 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_40_adm2.csv\n",
"Compiled data for column 50 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_50_adm2.csv\n",
"Compiled data for column 60 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_60_adm2.csv\n",
"Compiled data for column 61 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_61_adm2.csv\n",
"Compiled data for column 62 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_62_adm2.csv\n",
"Compiled data for column 70 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_70_adm2.csv\n",
"Compiled data for column 71 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_71_adm2.csv\n",
"Compiled data for column 72 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_72_adm2.csv\n",
"Compiled data for column 80 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_80_adm2.csv\n",
"Compiled data for column 81 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_81_adm2.csv\n",
"Compiled data for column 82 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_82_adm2.csv\n",
"Compiled data for column 90 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_90_adm2.csv\n",
"Compiled data for column 100 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_100_adm2.csv\n",
"Compiled data for column 110 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_110_adm2.csv\n",
"Compiled data for column 120 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_120_adm2.csv\n",
"Compiled data for column 121 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_121_adm2.csv\n",
"Compiled data for column 122 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_122_adm2.csv\n",
"Compiled data for column 130 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_130_adm2.csv\n",
"Compiled data for column 140 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_140_adm2.csv\n",
"Compiled data for column 150 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_150_adm2.csv\n",
"Compiled data for column 151 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_151_adm2.csv\n",
"Compiled data for column 152 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_152_adm2.csv\n",
"Compiled data for column 153 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_153_adm2.csv\n",
"Compiled data for column 160 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_160_adm2.csv\n",
"Compiled data for column 170 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_170_adm2.csv\n",
"Compiled data for column 180 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_180_adm2.csv\n",
"Compiled data for column 190 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_190_adm2.csv\n",
"Compiled data for column 200 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_200_adm2.csv\n",
"Compiled data for column 201 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_201_adm2.csv\n",
"Compiled data for column 202 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_202_adm2.csv\n",
"Compiled data for column 210 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_210_adm2.csv\n",
"Compiled data for column 220 saved to: /mnt/d/temp/esacci_landcover/zonal/moz_compile/moz_phy_esacci_landcover_lccs_220_adm2.csv\n",
"Data compilation complete!\n"
]
}
],
"source": [
"import os\n",
"import pandas as pd\n",
"\n",
"# Set the country\n",
"iso3 = \"moz\" # lbn, syr, mmr, tur\n",
"\n",
"def compile_zonal_stats(input_folder, output_folder, columns):\n",
" # Ensure that the output folder exists\n",
" if not os.path.exists(output_folder):\n",
" os.makedirs(output_folder)\n",
"\n",
" # Process each specified column\n",
" for column in columns:\n",
" compiled_data = pd.DataFrame()\n",
"\n",
" # List all CSV files in the input folder\n",
" csv_files = [f for f in os.listdir(input_folder) if f.endswith('.csv')]\n",
"\n",
" for csv_file in csv_files:\n",
" try:\n",
" year = csv_file.split('_')[-1].split('.')[0] # Extract year from filename\n",
" file_path = os.path.join(input_folder, csv_file)\n",
"\n",
" # Read the CSV file\n",
" df = pd.read_csv(file_path)\n",
"\n",
" # Check if the specified column exists\n",
" if str(column) in df.columns:\n",
" # Add the data for the specified column to the compiled DataFrame\n",
" df = df[['ID', 'ADM2_PCODE', 'ADM2_PT', 'ADM1_PCODE', 'ADM1_PT', 'ADM0_PCODE', 'ADM0_PT', str(column)]]\n",
" df.rename(columns={str(column): year}, inplace=True)\n",
"\n",
" if compiled_data.empty:\n",
" compiled_data = df\n",
" else:\n",
" # Merge with the compiled DataFrame\n",
" compiled_data = compiled_data.merge(df, on=['ID', 'ADM2_PCODE', 'ADM2_PT', 'ADM1_PCODE', 'ADM1_PT', 'ADM0_PCODE', 'ADM0_PT'], how='outer')\n",
" except Exception as e:\n",
" print(f\"Error processing file {csv_file}: {e}\")\n",
"\n",
" # Save the compiled data to a new CSV file\n",
" try:\n",
" output_file_path = os.path.join(output_folder, f\"{iso3}_phy_esacci_landcover_lccs_{column}_adm2.csv\")\n",
" compiled_data.to_csv(output_file_path, index=False)\n",
" print(f\"Compiled data for column {column} saved to: {output_file_path}\")\n",
" except Exception as e:\n",
" print(f\"Error saving compiled data for column {column}: {e}\")\n",
"\n",
"# Define input and output folder paths\n",
"input_folder = f\"/mnt/d/temp/esacci_landcover/zonal/{iso3}\" # Folder containing CSV files\n",
"output_folder = f\"/mnt/d/temp/esacci_landcover/zonal/{iso3}_compile\" # Folder to save compiled CSV files\n",
"\n",
"# Columns to compile\n",
"columns_to_compile = [10, 11, 12, 20, 30, 40, 50, 60, 61, 62, 70, 71, 72, 80, 81, 82, \n",
" 90, 100, 110, 120, 121, 122, 130, 140, 150, 151, 152, 153, 160,\n",
" 170, 180, 190, 200, 201, 202, 210, 220]\n",
"\n",
"# Compile data for each specified column\n",
"compile_zonal_stats(input_folder, output_folder, columns_to_compile)\n",
"\n",
"print(\"Data compilation complete!\")\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bc85482e",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.2"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment