jennyd’s gists

jennyd / gist:8844665

Created February 6, 2014 13:57

Timing hits import using INSERT IGNORE vs. ON DUPLICATE KEY UPDATE

	time bundle exec rake import:all:hits

	With all hits imported already and no updates to counts to make:
	- master:
	real 0m43.212s
	user 0m8.517s
	sys 0m1.288s
	- hits_import_update_count:
	real 0m47.302s
	user 0m8.029s

jennyd / exclude_lines_containing_invalid_utf8_bytes.awk

Created September 11, 2014 10:36

Remove lines containing invalid UTF8 byte sequences from files

	#!/opt/mawk/bin/mawk -f

	##
	# Skip lines with invalid UTF8 byte sequences.
	# Adapted from http://unix.stackexchange.com/questions/6516/filtering-invalid-utf8 by @rgarner
	#
	$0 !~ /^(([\x00-\x7F])\|([\xC2-\xDF][\x80-\xBF])\|((([\xE0][\xA0-\xBF])\|([\xED][\x80-\x9F])\|([\xE1-\xEC\xEE-\xEF][\x80-\xBF]))([\x80-\xBF]))\|((([\xF0][\x90-\xBF])\|([\xF1-\xF3][\x80-\xBF])\|([\xF4][\x80-\x8F]))([\x80-\xBF][\x80-\xBF])))*$/ {
	next
	}

jennyd / falcon-logging-middleware.py

Last active February 11, 2024 22:09

Falcon logging middleware

	import falcon
	import logging

	logger = logging.getLogger(__name__)
	logger.addHandler(logging.FileHandler('test.log'))
	logger.setLevel(logging.INFO)

	class ResponseLoggerMiddleware(object):
	def process_response(self, req, resp):
	logger.info('{0} {1} {2}'.format(req.method, req.relative_uri, resp.status[:3]))

jennyd / example.py

Last active August 29, 2015 14:19

requests 2.6.1 errors on decoding JSON in PP response

	# with performanceplatform-client==0.8.2
	from performanceplatform.client import DataSet


	dataset = DataSet.from_group_and_type('https://www.performance.service.gov.uk/data',
	'govuk-info',
	'page-contacts')

	query_parameters = {
	'start_at': '2015-03-12T00:00:00Z',

jennyd / keybase.md

Created June 6, 2015 18:53

keybase.io verification

I hereby claim:

I am jennyd on github.
I am jennyd (https://keybase.io/jennyd) on keybase.
I have a public key whose fingerprint is DF1A 9E71 8F42 0F6B 64F2 B011 302C 8726 3CBA 8CAD

To claim this, I am signing this object:

jennyd / get-govuk-pageviews-from-pp.py

Created July 20, 2016 16:24

Getting pageview data from performance platform for a given GOV.UK path and time period

	from performanceplatform.client import DataSet


	query_parameters = {
	'group_by': 'pagePath',
	'period': 'day',
	'start_at': '2016-07-18T00:00:00Z',
	'end_at': '2016-07-20T00:00:00Z',
	'collect': 'uniquePageviews:sum',
	'filter_by': 'pagePath:/pay-council-tax/lambeth'

jennyd / analyse-topics.py

Created January 6, 2017 18:35

Analyse govuk-delivery subscription URLs and their query params

	#!/usr/bin/env python

	import csv
	import urlparse
	from collections import Counter


	'''Analyse data from govuk-delivery's database to find out which URLs exist as
	subscription topics and which query params they use. This will help us work out
	whether all Whitehall content is already tagged to enough things in the links

jennyd / keybase.md

Created October 14, 2018 17:29

keybase.io verification signed by new GPG key (978F C7D5 F0DC 7732 E660 9433 AAD7 2766 0A53 519B)

I hereby claim:

To claim this, I am signing this object:

jennyd / running-govuk-transition-locally.sh

Last active March 6, 2019 14:28

Getting started with local development on Transition (without the GOV.UK dev VM)

	# Prerequisites:
	# - ruby 2.6.1 (or another relatively recent version is probably fine too) and bundler installed
	# - postgres 9.3+ running (I've used 9.6 and 10.7 and they were both fine)
	# - if you're creating more than 100 mappings at once, the app processes that in the background using Sidekiq with Redis, but the app runs fine without Redis otherwise

	git clone [email protected]:alphagov/transition.git
	cd transition

	bundle install
	bundle exec rake db:setup

Jenny Duckett jennyd