simonw / index_django_docs_to_elasticsearch.py

Last active June 19, 2018 18:41

Example code for converting Django docs into a format suitable for import into Elasticsearch - for a talk I'm giving.

	#!/usr/bin/env python

	import re
	import os
	import urlparse
	import hashlib
	import json
	import sys

	title_re = re.compile(r'([=*]{3,})\n([^\n]+)\n\1\n')

simonw / policy.json

Created October 10, 2016 17:53

Amazon S3 policy document for allowing a specific user read-only access to a specific S3 bucket

	{
	"Version":"2012-10-17",
	"Statement":[
	{
	"Effect":"Allow",
	"Action":[
	"s3:ListBucket",
	"s3:GetBucketLocation"
	],
	"Resource":"arn:aws:s3:::specific-s3-bucket"

simonw / download_elasticsearch_index.py

Last active November 18, 2016 18:36

Download and dump contents of a remote Elasticsearch index to a local newline-separated JSON file using Python

	from elasticsearch import Elasticsearch
	from elasticsearch.helpers import scan
	import json

	REMOTE_URL = 'https://username:password@elasticsearch.example.com:9200/'
	LOCAL_FILE = '/tmp/elasticsearch-dump.txt'
	INDEX = 'my-index'
	DOC_TYPE = 'document'

	client = Elasticsearch([REMOTE_URL])

simonw / elasticsearch_dsl_facet_override.py

Created November 13, 2016 20:19

Over-riding elasticsearch-dsl FacetedSearch to use .filter, not .post_filter for selected facets - see https://twitter.com/simonw/status/797894804700966912

	class EmailSearch(FacetedSearch):
	doc_types = [Email]
	# fields that should be searched
	fields = ['subject', 'body']

	facets = {
	'inbox': TermsFacet(field='inbox'),
	}

	def filter(self, search):

simonw / rasterize.js

Last active December 5, 2016 21:40 — forked from FiloSottile/rasterize.js

PhantomJS rasterize.js with "Retina" output

	var system = require('system');
	var page = require('webpage').create(),
	address, output, size;

	if (system.args.length < 2 \|\| system.args.length > 3) {
	console.log('Usage: rasterize.js URL filename');
	phantom.exit();
	} else {
	console.log(system.args);
	address = system.args[1];

simonw / wget.md

Created December 9, 2016 06:38

Recursive wget ignoring robots

$ wget -e robots=off -r -np 'http://example.com/folder/'

-e robots=off causes it to ignore robots.txt for that domain
-r makes it recursive
-np = no parents, so it doesn't follow links up to the parent folder

simonw / simple-proxy.php

Last active March 12, 2017 04:08

Fix for simple-proxy.php WordPress plugin to get it working with Varnish (addresses a gzip headers error)

	<?php
	/*
	Plugin Name: Simple Proxy
	Description: A very simple proxy. Useful when you're moving from one server to another.
	Author: Greg Priday
	Author URI: http://siteorigin.com/
	Version: 1.0
	*/

	function simple_proxy_admin_menu(){

simonw / example.csv

Created December 11, 2016 22:13

How to read Excel files with quoted values that themselves contain newlines and commas using Python

We can make this file beautiful and searchable if this error is corrected: Any value after quoted field isn't allowed in line 2.

	post_id,post_name,post_type,post_date,post_title,post_content,post_status
	"10001","blah-blah-blah-blah","post","2011-01-16 18:49:52.857717","Post title here!","<h2>Post title!</h2>
	<p>Look,</p>
	<p>This contains <span class=\"caps\">HTML content</span>, and even contains commas</p>
	","publish"
	"10002","blah-blah-blah-blah2","post","2011-01-16 19:49:52.857717","Post title two!","<h2>Post title two!</h2>
	<p>More stuff</p>
	","publish"

simonw / 301-redirects.php

Created December 12, 2016 02:40

A version of the 301-redirects.php WordPress plugin that has been horribly hacked to understand regular expressions

	<?php
	/*

	Plugin Name: 301 Redirects
	Plugin URI: http://tonyspiro.com
	Description: A plugin that helps you add 301 redirects to your site.
	Version: 0.3
	Author: Tony Spiro
	Author URI: http://tonyspiro.com
	License: GPL2

simonw / recover_source_code.md

Last active September 9, 2026 17:52

How to recover lost Python source code if it's still resident in-memory

I screwed up using git ("git checkout --" on the wrong file) and managed to delete the code I had just written... but it was still running in a process in a docker container. Here's how I got it back, using https://pypi.python.org/pypi/pyrasite/ and https://pypi.python.org/pypi/uncompyle6

Attach a shell to the docker container

Install GDB (needed by pyrasite)

apt-get update && apt-get install gdb

Simon Willison simonw

How to recover lost Python source code if it's still resident in-memory

Attach a shell to the docker container

Install GDB (needed by pyrasite)