Skip to content

Instantly share code, notes, and snippets.

View jbothma's full-sized avatar

JD Bothma jbothma

101 followers · 5 following
View GitHub Profile
@jbothma
jbothma / kzn_lawsoc_spider.py
Last active February 16, 2017 12:55
Scrapy Spider to scrape HTML table with identifiers in first column and related links on same and subsequent rows
"""
(KwaZulu-Natal Provincial Gazette v.1 no.1 published 2007-05-03)
Last gazette before it seems to be
"KwaZulu-Natal Provincial Provincial Gazette 6553 2007-04-26"
"""
from datetime import datetime
from gazettescrape.items import GazetteItem
from scrapy import Request
@jbothma
jbothma / flatten.py
Created March 29, 2017 13:08
Flatten/denormalise nested json objects in jsonlines input file to a flat CSV.
import fileinput
import json
import sys
import csv
from copy import copy
fieldnames = [
'regno',
'name',
'status',
@jbothma
jbothma / import2016mapit.sql
Last active April 26, 2017 08:59
Wazimap ZA 2016 demarcation names import from Mapit (code4sa/OpenUp)
BEGIN;
CREATE TEMPORARY TABLE geos2016
(
geo_level TEXT,
geo_code TEXT,
name TEXT,
parent_code TEXT,
parent_level TEXT
) ON COMMIT DROP;
@jbothma
jbothma / municipal-elections.sql
Last active May 4, 2017 13:51
Import party votes per geographic area for 2016 municipal elections to wazimap ZA
-- Remove BOM for psql \copy
-- sed -i '/^\s*$/d' *
-- Remove blank lines for psql \copy
-- sed -i '1s/^\xEF\xBB\xBF//' *
BEGIN;
CREATE TEMPORARY TABLE municipal_election
(
province TEXT,
@jbothma
jbothma / voter turnout 2016.sql
Created May 9, 2017 11:49
BAD voter turnout 2016
BEGIN;
CREATE TEMPORARY TABLE municipal_election
(
province TEXT,
muni TEXT,
ward TEXT,
votingdistrict TEXT,
votingstationname TEXT,
registeredvoters TEXT,
@jbothma
jbothma / import voter turnout.sql
Created May 9, 2017 12:45
Import wazimap voter turnout for 2016 municipal elections from all-ward-turnout dump obtained directly from IEC tech department
CREATE TEMPORARY TABLE voter_turnout
(
province TEXT,
muni TEXT,
ward TEXT,
registeredvoters TEXT,
mec7votes TEXT,
voterturnout TEXT,
percentvoterturnout TEXT
@jbothma
jbothma / pdf-ocr.sh
Last active October 3, 2020 10:26
Adding OCR text to a PDF (even a noisy one)
# usage: ocr-pdf scanned.pdf
set -euf -o pipefail -o xtrace
INFILE=$1
BASENAME=$(basename "$1" .pdf)
TIFFFILE=$BASENAME.tiff
OCRDPDFNOEXT=$BASENAME-OCRd-big
OCRDPDF=$OCRDPDFNOEXT.pdf
SMALLEROCRDPDF=$BASENAME-OCRd.pdf
@jbothma
jbothma / keybase.md
Created August 14, 2017 09:07
keybase.md

Keybase proof

I hereby claim:

  • I am jbothma on github.
  • I am jbothma (https://keybase.io/jbothma) on keybase.
  • I have a public key whose fingerprint is 1016 1320 E0BC FDD7 5F5F 48C2 B410 8474 DEE2 7D69

To claim this, I am signing this object:

@jbothma
jbothma / loop.sh
Created September 30, 2017 10:24
save and change IFS to loop over filenames with spaces
SAVEIFS=$IFS; IFS=$(echo -en "\n\b"); for f in `ls */*/*.xlsm`; do echo soffice --headless --convert-to xlsx \"$f\" --outdir \"$(dirname $f)\"; done; IFS=$SAVEIFS
@jbothma
jbothma / getting started programming.md
Last active November 9, 2017 10:39
Getting started with programming and web development for someone in a place like khayelitsha

This is the app to learn programming on your phone. I recommend you start with HTML, CSS and Javascript https://play.google.com/store/apps/details?id=com.sololearn

it teaches you stuff, then you have quizes where you can test what you've learned, and you can actually write some code and test it see it working

python is also a good one to learn after javascript

this can teach you basics of becoming a freelance website developer https://www.smashingmagazine.com/2009/07/the-roadmap-to-becoming-a-professional-freelance-web-designer/

some of it might not perfectly fit south africa or khayelitsha but you can imagine how it connects to the businesses around you