Skip to content

Instantly share code, notes, and snippets.

@duydo
duydo / elasticsearch_best_practices.txt
Last active June 20, 2024 09:59
Elasticsearch - Index best practices from Shay Banon
If you want, I can try and help with pointers as to how to improve the indexing speed you get. Its quite easy to really increase it by using some simple guidelines, for example:
- Use create in the index API (assuming you can).
- Relax the real time aspect from 1 second to something a bit higher (index.engine.robin.refresh_interval).
- Increase the indexing buffer size (indices.memory.index_buffer_size), it defaults to the value 10% which is 10% of the heap.
- Increase the number of dirty operations that trigger automatic flush (so the translog won't get really big, even though its FS based) by setting index.translog.flush_threshold (defaults to 5000).
- Increase the memory allocated to elasticsearch node. By default its 1g.
- Start with a lower replica count (even 0), and then once the bulk loading is done, increate it to the value you want it to be using the update_settings API. This will improve things as possibly less shards will be allocated to each machine.
- Increase the number of machines you have so
@sgillies
sgillies / geo_interface.rst
Last active March 4, 2025 10:09
A Python Protocol for Geospatial Data

Author: Sean Gillies Version: 1.0

Abstract

This document describes a GeoJSON-like protocol for geo-spatial (GIS) vector data.

Introduction

Proposal for Improving Mass Assignment

For a while, I have felt that the following is the correct way to improve the mass assignment problem without increasing the burden on new users. Now that the problem with the Rails default has been brought up again, it's a good time to revisit it.

Sign Allowed Fields

When creating a form with form_for, include a signed token including all of the fields that were created at form creation time. Only these fields are allowed.

To allow new known fields to be added via JS, we could add:

@anemitz
anemitz / slaballoc_design
Created March 1, 2012 04:01
Slab Alloc
Design:
Note: The smallest block that can be allocated is 8 bytes due to the following structure.
Byte Layout:
0 3 7
----------------------------
| size of | offset to |
| data | next free |
----------------------------
@michalmarczyk
michalmarczyk / new-trace-ns.clj
Created February 17, 2012 00:46
Mass tracing utilities for Clojure
;;; See the inspirational SO question: http://stackoverflow.com/questions/3346382
;;; My old Gist with my first take on this: https://gist.github.com/492764
;;; Don Jackson's Gist based on the above: https://gist.github.com/1846993
;;; Licence: EPLv1, the same as Clojure
(in-ns 'clojure.tools.trace)
(defn trace-var
"If the specified Var holds an IFn and is not marked as a macro, its
@pfig
pfig / mkfavicon.sh
Created February 12, 2012 12:01
Make a multi-resolution favicon.ico from a source image, using ImageMagick
#!/bin/bash
# from
# http://bergamini.org/computers/creating-favicon.ico-icon-files-with-imagemagick-convert.html
convert source-WxW.png -resize 256x256 -transparent white favicon-256.png
convert favicon-256.png -resize 16x16 favicon-16.png
convert favicon-256.png -resize 32x32 favicon-32.png
convert favicon-256.png -resize 64x64 favicon-64.png
convert favicon-256.png -resize 128x128 favicon-128.png
@david-mcneil
david-mcneil / custom-clojure-map.clj
Created January 26, 2012 20:43
Creating a custom Clojure map type
(ns people
(:use [clojure.string :only (join)]
[clojure.pprint :only (pprint simple-dispatch)]))
;; we can make maps using the special literal form:
{:a 100
:b 200}
(class {:a 100 :b 200})
@Pet3ris
Pet3ris / fsm.clj
Created January 25, 2012 13:27
Finite State Machine in Clojure core.logic
(ns fsm
(:refer-clojure :exclude [==])
(:use [clojure.core.logic]))
;; Encoding a Finite State Machine and recognizing strings in its language in Clojure core.logic
;; We will encode the following FSM:
;;
;; (ok) --+---b---> (fail)
;; ^ |
@entaroadun
entaroadun / gist:1653794
Created January 21, 2012 20:10
Recommendation and Ratings Public Data Sets For Machine Learning

Movies Recommendation:

Music Recommendation:

@lucasrizoli
lucasrizoli / gist:1603274
Created January 12, 2012 21:33
70 Unique Ways to Encode <
<
%3C
&lt
&lt;
&LT
&LT;
&#60
&#060
&#0060
&#00060