Skip to content

Instantly share code, notes, and snippets.

View amotl's full-sized avatar

Andreas Motl amotl

  • $PYTHONPATH
View GitHub Profile
@amotl
amotl / cratedb-parallel.py
Last active November 20, 2020 15:14
Investigate connection pooling with "crate-python"
"""
About
-----
Investigate connection pooling with "crate-python" when
running within a multi-threaded environment.
This covers both cases using DBAPI and going through SQLAlchemy.
See also https://github.com/crate/crate-python/pull/373.
Setup
@amotl
amotl / duckdb_pandas_category.py
Last active November 28, 2020 19:49
Investigate DuckDB support for more Pandas data types
import duckdb
import pandas as pd
from pandas._testing import assert_frame_equal
def test_category():
"""
This test uses a Pandas category datatype making DuckDB croak.
"""
@amotl
amotl / Dockerfile
Last active October 10, 2020 22:35
mqttwarn amqp example
FROM rabbitmq:3
ADD rabbitmq.conf /etc/rabbitmq/
ADD definitions.json /etc/rabbitmq/
RUN chown rabbitmq:rabbitmq /etc/rabbitmq/rabbitmq.conf /etc/rabbitmq/definitions.json
@amotl
amotl / validate_site.py
Last active December 1, 2020 16:42
Quickly validate website leaf-node accessibility starting from a sitemap.xml
"""
Setup::
virtualenv .venv2 --python=python2
source .venv2/bin/activate
pip install requests
pip install git+https://github.com/KokocGroup/sitemap_parser
Synopsis::
@amotl
amotl / vale-summary.jq
Created December 9, 2020 19:03
Postprocessing JSON output of Vale using "jq"
# Summarize JSON output of Vale using jq.
# cat vale-report.json | jq --from-file vale-summary.jq > vale-summary.json
to_entries | map({
file: .key,
errors: .value | [ .[] | select(.Severity | contains("error")) ] | length,
warnings: .value | [ .[] | select(.Severity | contains("warning")) ] | length,
suggestions: .value | [ .[] | select(.Severity | contains("suggestion")) ] | length,
})
@amotl
amotl / opendata_dwd_de_fsspec.py
Created December 10, 2020 12:22
Demo for accessing https://opendata.dwd.de/ using fsspec, optionally using caching
"""
Demo for accessing https://opendata.dwd.de/ using fsspec, optionally using caching.
Setup
=====
::
pip install fsspec requests aiohttp
Resources
@amotl
amotl / simple_python_datasource.py
Created December 14, 2020 19:50 — forked from linar-jether/simple_python_datasource.py
Grafana python datasource - using pandas for timeseries and table data. inspired by and compatible with the simple json datasource
from flask import Flask, request, jsonify, json, abort
from flask_cors import CORS, cross_origin
import pandas as pd
app = Flask(__name__)
cors = CORS(app)
app.config['CORS_HEADERS'] = 'Content-Type'
@amotl
amotl / fsspec-dwd.py
Last active December 20, 2020 22:12
Investigate problems when scanning huge directory tree of DWD CDC HTTP server
"""
Investigate problems when scanning huge directory
tree of DWD CDC HTTP server.
This repro will reveal that fsspec seems to randomly
include a single folder within its results list::
python fsspec-dwd.py | grep -v zip
The result varies between single-line outputs of e.g.:
@amotl
amotl / grafana_api_dashboard_folder_flaw.py
Created January 5, 2021 08:38
Proof that grafana_api's `update_dashboard` will respawn the dashboard within the "General" folder
"""
About
=====
Proof that ``update_dashboard`` will respawn the dashboard within the "General" folder::
before: {'folderId': 1, 'folderTitle': 'testdrive', 'folderUrl': '/dashboards/f/RJS5vK-Mz/testdrive'}
after: {'folderId': 0, 'folderTitle': 'General', 'folderUrl': ''}
Synopsis