Skip to content

Instantly share code, notes, and snippets.

View mh0w's full-sized avatar
💭
🦆

Matthew Hawkes_ONS mh0w

💭
🦆
View GitHub Profile
@mh0w
mh0w / Supress specific warning messages.py
Created February 2, 2024 09:43
Supress specific warning messages
import requests
import warnings
# Suppress warnings containing the text "Unverified HTTPS request"
with warnings.catch_warnings():
warnings.filterwarnings("ignore", message="Unverified HTTPS request")
list_of_available_datasets = json.loads(
requests.get("https://api.beta.ons.gov.uk/v1/datasets", verify=False).text
)
@mh0w
mh0w / Helper script for using the ONS API.py
Last active April 25, 2024 14:11
Working with the Office for National Statistics API (developer.ons.gov.uk)
"""
Program for accessing the ONS API (see https://developer.ons.gov.uk).
The ONS API can be used to programmatically access certain data that ONS has published.
This program provides functions for (1) checking what datasets are available through the
ONS API and (2) accessing datasets through the ONS API. To use this program, run the
entire script; You will then be able to use the get_list_of_available_datasets() and
get_latest_data_from_ons_api() functions.
get_list_of_available_datasets()
@mh0w
mh0w / 1 - Get-EventLog in powershell.ps1
Last active June 18, 2024 16:39
Get-EventLog and working with evtx files
# Check available log names and meta data
Get-EventLog -list
# Remind yourself when you logged on on a particular day (for clocking hours)
# by checking the 'system' log, for days going back as far as 14 days ago
# printing only when instanceid is 7001 (user logon notification)
Get-EventLog system -after (get-date).AddDays(-14) | where {$_.InstanceId -eq 7001}
# Version that shows only the first entry for each day
Get-EventLog system -after (get-date).AddDays(-14) | where {$_.InstanceId -eq 7001} |
@mh0w
mh0w / replicate a dataframe n times (extending its length).py
Last active January 30, 2024 09:26
replicate a dataframe n times (extending its length)
import pandas as pd
# Create a toy dataframe for demonstration purposes
my_df = pd.DataFrame({"oa": ["a", "b", "c"]}) # 3 rows
"""
oa
0 a
1 b
2 c
@mh0w
mh0w / Common errors and possible solutions.txt
Last active November 22, 2024 11:41
Common errors and possible solutions
#### Issue: np.float not found
Explanation: the issue is that np.float is being called for, but np.float is deprecated in newer
versions of numpy.
Solution(s):
(i) use float instead
(ii) create np.float = float, (iii) update the package that calls for np.float
import numpy as np
np.float == float
@mh0w
mh0w / Check and (down) cast to different data types.py
Last active January 25, 2024 17:15
Check and cast to different data types
import pandas as pd # astype(), info(), memory_usage(), to_numeric()
import polars as pl # cast(), estimates_size()
import numpy as np
import pdcast as pdc # install via pip install pandas-downcast
pandas_df = pd.DataFrame(
{
"a": np.linspace(1, 10_000_000, 10_000_000),
"b": np.linspace(1.12, 10_000_000.12, 10_000_000),
"c": np.random.choice([1, 0], 10_000_000),
@mh0w
mh0w / Proxy test.py
Created December 18, 2023 13:39
Proxy test
import requests
def check_proxy(proxy: str):
proxies = {
"http": f"http://{proxy}",
"https": f"http://{proxy}"
}
try:
@mh0w
mh0w / intro_to_polars.py
Last active November 9, 2023 16:40
intro_to_polars
"""
Sources:
https://pola-rs.github.io/polars/getting-started/intro
https://kevinheavey.github.io/modern-polars/
.../DAP_CATS/intro_to_polars/-/tree/main?ref_type=heads
"""
import polars as pl
from datetime import date, datetime
import numpy as np
@mh0w
mh0w / Create animated population pyramids from age-sex-lad level population data.py
Last active November 3, 2023 14:41
Create animated population pyramids from age-sex-lad level population data
# -*- coding: utf-8 -*-
"""
Generate animated population pyramid gifs for all geographies for years 2011 to 2022.
I wrote this program quickly. It has not be quality assured.
@author: Matthew Hawkes
@date: 03-11-2023
Style: https://black.readthedocs.io
"""