Skip to content

Instantly share code, notes, and snippets.

View simonw's full-sized avatar

Simon Willison simonw

View GitHub Profile
{
page(url:"http://www.eventbrite.com/") {
title: text(selector:"title")
cards: query(selector:"div.poster-card") {
name: text(selector:"[itemprop=name]")
url: attr(selector:"meta[itemprop=url]", name:"content")
image_src: attr(selector:".js-poster-image", name:"src")
}
}
}
@simonw
simonw / how-to-use-aws-elasticsearch.md
Last active November 28, 2023 20:01
How to use Amazon AWS Elasticsearch

How to use Amazon AWS Elasticsearch

The good news: you can get it running on the free tier (with a tiny instance).

The bad news: it's stuck on Elasticsearch 1.5.2 and dynamic scripting (Groovy) is disabled.

http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html

Authentication: the safest option is to create a brand new IAM user (using the tool at https://console.aws.amazon.com/iam/home?region=us-east-1 ) with its own access key and secret key. Then when you create the Elasticsearch instance you can paste in the following IAM string:

@simonw
simonw / parameterized.py
Created March 27, 2016 23:00
Parameterized unit tests for Python - syntactic sugar inspired by https://chris-lamb.co.uk/posts/generating-dynamic-python-tests-using-metaclasses
class ParameterizedTestsMeta(type):
def __new__(cls, name, bases, attrs):
skip_these_names = []
i = 1
for name, fn in attrs.items():
if name.startswith('test_') and callable(fn) and getattr(fn, '_parameterized_params', None):
skip_these_names.append(name)
for params in fn._parameterized_params:
attrs['%s_%d' % (name, i)] = cls.make_parameterized_test(fn, params)
i += 1
@simonw
simonw / how-to.md
Last active December 19, 2025 10:46
How to create a tarball of a git repository using "git archive"
@simonw
simonw / gist:e6e2b495e020413566e4a81ba9f11679
Created April 13, 2016 21:24
How to install MySQLdb on OS X as of April 13th 2016 using homebrew but without installing mysql-server from homebrew (say for example you installed MySQL using the official OS X installer package)
brew install mysql-connector-c
pip install mysql-python
From http://stackoverflow.com/questions/16182294/installing-mysql-python-on-mac-os-x
@simonw
simonw / strip-xssi.js
Created April 14, 2016 18:55
How to strip a known XSSI protecting prefix from JSON using jQuery ajaxBefore - prefix is "for (;;);"
jQuery.ajaxSetup({
dataFilter: function(string) {
return string.replace(/^for \(;;\);/, '')
}
});
@simonw
simonw / re_enable_lookups_allowed.py
Created May 4, 2016 22:34
How to re-enable arbitrary querystring lookups for Django objects in the admin (deliberately re-opening this old security hole so only do this if you 100% trust everyone with access to your Django admin panel: https://www.djangoproject.com/weblog/2010/dec/22/security/ )
admin.site.register(MyModel,
lookup_allowed=lambda *args, **kwargs: True,
)
@simonw
simonw / ssl_or_not.py
Created May 26, 2016 14:33
How to tell if a Python MySQLdb connection is using SSL or not
from django.db import connection
c = connection.cursor()
c.execute('SHOW STATUS like "Ssl_cipher"')
c.fetchall()
# Output of ((u'Ssl_cipher', u''),) means no SSL
# Output of ((u'Ssl_cipher', u'AES256-SHA'),) means using SSL
@simonw
simonw / sql_replace_in_django_orm.py
Last active January 22, 2025 18:12
How to use the SQL replace function in a Django ORM query
from django.db.models import F, Func, Value
from myapp.models import MyModel
# Annotation
MyModel.objects.filter(description__icontains='\r\n').annotate(
fixed_description=Func(
F('description'),
Value('\r\n'), Value('\n'),
function='replace',
)
@simonw
simonw / example-output.html
Created July 6, 2016 02:53
Generates a table showing what time every first-come-first-served campsite at Yellowstone National Park filled up for the past year, pulling data from the undocumented National Parks cartodb API. Example output rendered here: https://rawgit.com/simonw/d496b7400442a452a7f4f86af260543b/raw/7778ade95863c3f3a1f50f155a641c25159bfa07/example-output.html
<table border=1>
<tr>
<th>&nbsp;</th>
<th colspan="21">July 2015</th><th colspan="31">August 2015</th><th colspan="30">September 2015</th><th colspan="31">October 2015</th><th colspan="30">November 2015</th><th colspan="31">December 2015</th><th colspan="31">January 2016</th><th colspan="29">February 2016</th><th colspan="31">March 2016</th><th colspan="30">April 2016</th><th colspan="31">May 2016</th><th colspan="30">June 2016</th><th colspan="5">July 2016</th></tr>
<tr>
<th>&nbsp;</th>
<th>S</th>
<th>S</th>
<th>M</th>
<th>T</th>