Skip to content

Instantly share code, notes, and snippets.

View ThomasG77's full-sized avatar

Thomas Gratier ThomasG77

204 followers · 105 following
View GitHub Profile
@ThomasG77
ThomasG77 / README.md
Created July 21, 2026 16:07
Recipe to transform latest RNA data to parquet or download latest data

Recipe to transform latest RNA data to parquet or download data (for 2nd case, uncomment line starting with #)

Using Bash

You need to install curl, jq and duckdb

curl https://www.data.gouv.fr/api/1/datasets/58e53811c751df03df38f42d/| jq -r -c '[.resources[]|select(."type" == "main")]|sort_by(.created_at)|.[].url' | tail -n2 >| latest_urls.txt
# If you want to download, uncomment but here we take remote sources using Duckdb zipfs
# wget -i latest_urls.txt
@ThomasG77
ThomasG77 / get-pypi-package.py
Created July 17, 2026 18:36
Get remote URL of a compressed Pypi package when not available on Github
# Get Python library sources latest version url when not available on github
from bs4 import BeautifulSoup
import requests
packages = ['to-file-like-obj', 'stream-zip', 'stream-unzip']
for package in packages:
r = requests.get(f'https://pypi.org/simple/{package}/')
soup = BeautifulSoup(r.text, "html.parser")
last_version = soup.find('body').find_all('a')[-1]
last_version_name = last_version.text
@ThomasG77
ThomasG77 / export_dvf_to_geoparquet.py
Created May 18, 2026 15:47
Export DVF geolocalisées en Geoparquet
import gzip
import shutil
import logging
from urllib.request import urlretrieve
import duckdb
from bs4 import BeautifulSoup
import requests
geo_dvf_entry = 'https://files.data.gouv.fr/geo-dvf/latest/csv/'
@ThomasG77
ThomasG77 / README.md
Created March 24, 2026 10:22
Données archéologiques depuis https://archeologie.culture.gouv.fr

Depuis https://archeologie.culture.gouv.fr/fr

Données identifiées en passant par https://archeologie.culture.gouv.fr/fr

curl -s https://archeologie.culture.gouv.fr/france/fr/carte/data/5638 | jq -c -r '{"type": "FeatureCollection", "features": [.notices[] | {"type": "Feature", "properties": {"type": .type, "sur_titre": .sur_titre, "titre": .titre, "lien_fiche": .lien_fiche, "lien_site": .lien_site, "points_fort": .points_fort, "visuel_opacity": .visuel.opacity, "visuel_ancrage": .visuel.ancrage, "visuel_url": .visuel.url, "visuel_width": .visuel.width, "visuel_height": .visuel.height, "visuel_alt": .visuel.alt, "visuel_tag": .visuel.tag, "sous_titre": .sous_titre, "legende": .legende}, "geometry": {"coordinates": [(.lng|tonumber), (.lat|tonumber)], "type": "Point"}}]}' >| /tmp/archeologie-fr.geojson

curl -s https://archeologie.culture.gouv.fr/monde/fr/carte/data/3961 | jq -c -r '{"type": "FeatureCollection", "features": [.notices[] | {"type": "Feature", "properties": {"type": .type, "sur_titre": .sur
@ThomasG77
ThomasG77 / circo_2012.csv
Created February 4, 2026 16:24
We can't make this file beautiful and searchable because it's too large.
code_dept_canton,code_dept,nom_dept,code_canton,nom_canton,code_commune,nom_commune,circ_leg_1986,circ_leg_2012,code_insee
0101,01,AIN,1,AMBERIEU-EN-BUGEY,002,L'Abergement-de-Varey,2,5,01002
0101,01,AIN,1,AMBERIEU-EN-BUGEY,004,Ambérieu-en-Bugey,2,5,01004
0101,01,AIN,1,AMBERIEU-EN-BUGEY,007,Ambronay,2,5,01007
0101,01,AIN,1,AMBERIEU-EN-BUGEY,041,Bettant,2,5,01041
0101,01,AIN,1,AMBERIEU-EN-BUGEY,089,Château-Gaillard,2,5,01089
0101,01,AIN,1,AMBERIEU-EN-BUGEY,149,Douvres,2,5,01149
0101,01,AIN,1,AMBERIEU-EN-BUGEY,345,Saint-Denis-en-Bugey,2,5,01345
0101,01,AIN,1,AMBERIEU-EN-BUGEY,379,Saint-Maurice-de-Rémens,2,5,01379
0102,01,AIN,2,BÂGÉ-LE-CHÂTEL,023,Asnières-sur-Saône,2,4,01023
@ThomasG77
ThomasG77 / demo-wfs.py
Last active January 28, 2026 22:37
OGC WFS consumption URLs from a service
from owslib.wfs import WebFeatureService
url = 'https://ahocevar.com/geoserver/wfs'
wfs20 = WebFeatureService(url=url, version='2.0.0')
outputFormats = wfs20.getOperationByName('GetFeature').parameters.get('outputFormat').get('values')
for name, contentMetadata in wfs20.items():
for outputFormat in outputFormats:
print(wfs20.getGETGetFeatureRequest(name) + f'&OUTPUTFORMAT={quote(outputFormat)}', name, contentMetadata.title, contentMetadata.id, contentMetadata.abstract)
@ThomasG77
ThomasG77 / README.md
Last active December 3, 2025 13:52
https://gist.githack.com/ThomasG77/7b804ccec918a5e37f9f660fa392115d/raw/5af906e9b2fd87215ff7f2f67da09cfd1758bdf0/bookmarklets-minio.html

Bookmarklet pour avoir les urls réels sur du type https://console.object.files.data.gouv.fr/browser/meteofrance-drias/

Allez sur https://gist.githack.com/ThomasG77/7b804ccec918a5e37f9f660fa392115d/raw/db15d58773cacef1948fd13844369f5a24f1298f/bookmarklets-minio.html

Installer le bookmarklet/favori en faisant comme sur https://gist.github.com/ThomasG77/7b804ccec918a5e37f9f660fa392115d#file-install-et-demo-bookmarklet-gif

Ensuite, à chaque fois, il faudra dans le dossier où vous naviguer, si un fichier est présent, recliquer sur ce favori pour faire apparaître ce lien.

@ThomasG77
ThomasG77 / generate-qpv-parquet.sh
Created December 1, 2025 14:36
wget https://static.data.gouv.fr/resources/quartiers-prioritaires-de-la-politique-de-la-ville-qpv/20250206-161839/qp2024-france-hexagonale-outre-mer-wgs84-epsg4326.geojson
duckdb :memory: "INSTALL spatial;LOAD spatial;COPY (SELECT * FROM ST_Read('qp2024-france-hexagonale-outre-mer-wgs84-epsg4326.geojson')) TO 'qpv-2024.parquet' (FORMAT 'parquet', COMPRESSION 'zstd');"
rm qp2024-france-hexagonale-outre-mer-wgs84-epsg4326.geojson
gpio add bbox qpv-2024.parquet qpv-2024-with-bbox.parquet
rm qpv-2024.parquet
gpio sort hilbert qpv-2024-with-bbox.parquet qpv-2024.parquet
rm qpv-2024-with-bbox.parquet
gpio check all qpv-2024.parquet
@ThomasG77
ThomasG77 / README.md
Created November 19, 2025 12:47
Exemples de consommation Geo DVF Parquet via Duckdb

Exemples de consommation de DVF via parquet et Duckdb

# Une commune
duckdb :memory: "INSTALL spatial;LOAD spatial;COPY (SELECT id_mutation, date_mutation, numero_disposition, nature_mutation, valeur_fonciere::DOUBLE AS valeur_fonciere, adresse_numero, adresse_suffixe, adresse_nom_voie, adresse_code_voie, code_postal, code_commune, nom_commune, code_departement, ancien_code_commune, ancien_nom_commune, id_parcelle, ancien_id_parcelle, numero_volume, lot1_numero, lot1_surface_carrez::DOUBLE AS lot1_surface_carrez, lot2_numero, lot2_surface_carrez::DOUBLE AS lot2_surface_carrez, lot3_numero, lot3_surface_carrez::DOUBLE AS lot3_surface_carrez, lot4_numero, lot4_surface_carrez::DOUBLE AS lot4_surface_carrez, lot5_numero, lot5_surface_carrez::DOUBLE AS lot5_surface_carrez, nombre_lots, code_type_local, type_local, surface_reelle_bati::DOUBLE AS surface_reelle_bati, nombre_pieces_principales, code_nature_culture, nature_culture, code_nature_culture_speciale, nature_culture_speciale, surface_terrain::DOUBL
@ThomasG77
ThomasG77 / stats-surface-91114.sql
Created October 20, 2025 16:29
Get area stats for parcelles and batiments for each parcelle
INSTALL SPATIAL;
LOAD SPATIAL;
SET memory_limit = '16GB';
SET max_temp_directory_size = '125GB';
COPY (WITH batiments_91114 AS (
SELECT *
FROM
read_parquet('https://cadastre.data.gouv.fr/data/etalab-cadastre/2025-09-01/geoparquet/france/cadastre.parquet')
WHERE type_objet = 'batiments' AND departement = '91' AND commune = '91114' AND ST_GeometryType(geometry) IN ('POLYGON', 'MULTIPOLYGON')
),