Chris Trinh chtrinh

I'm hunting for the best solution on how to handle keeping large sets of DB records "sorted" in a performant manner.

Problem Description

Most of us have work on projects at some point where we have needed to have ordered lists of objects. Whether it be a to-do list sorted by priority, or a list of documents that a user can sort in whatever order they want.

A traditional approach for this on a Rails project is to use something like the acts_as_list gem, or something similar. These systems typically add some sort of "postion" or "sort order" column to each record, which is then used when querying out the records in a traditional order by position SQL query.

This approach seems to work fine for smaller datasets, but can be hard to manage on large data sets with hundreds (or thousands) of records needing to be sorted. Changing the sort position of even a single object will require updating every single record in the database that is in the same sort group. This requires potentially thousands of wri

This article has been given a more permanent home on my blog. Also, since it was first written, the development of the Promises/A+ specification has made the original emphasis on Promises/A seem somewhat outdated.

You're Missing the Point of Promises

Promises are a software abstraction that makes working with asynchronous operations much more pleasant. In the most basic definition, your code will move from continuation-passing style:

getTweetsFor("domenic", function (err, results) {
 // the rest of your code goes here.

A Bloom Filter in Redis

This is a really simple implementation of a bloom filter using Redis 2.6's lua scripting facility.

Though it "works", it's just a proof of concept. The choice and implementation of hashing functions leave something to be desired (I'm sure it's a fine implementation of CRC32 that I borrowed, but it's in pure lua, and, well, CRC32 is not the best function to use for a bloom filter).

Caveat Emptor, no refunds, etc. MIT License.

Changes:

this version includes backport of Greg Price's patch for speedup startup http://bugs.ruby-lang.org/issues/7158 .

ruby-core prefers his way to do thing, so that I abandon cached-lp and sorted-lf patches of mine.
this version integrates 'array as queue' patch, which improves performance when push/shift pattern is heavily used on Array.

This patch is accepted into trunk for Ruby 2.0 and last possible bug is found by Yui Naruse. It is used in production* for a couple of months without issues even with this bug.

This is a proof-of-concept of a couple of concurrent data structures written in Ruby.

The implementations are heavily commented for those interested. There are benchmarks (with results) included below. The results are interesting, but, as always, take with a grain of salt.

Data structures

AtomicLinkedQueue is a lock-free queue, built on atomic CAS operations.

	ActiveSupport.on_load(:active_record) do
	module Arel::NullsLastPredications
	def nulls_last
	Arel::Nodes::NullsLast.new self
	end
	end

	module Arel::Nodes
	class NullsLast < Unary
	def gsub *args

	# Here's a contrived example of a LEFT JOIN using ARel. This is an example of
	# the mechanics, not a real-world use case.

	# NOTE: In the gist comments, @ozydingo linked their general-purpose ActiveRecord
	# extension that works for any named association. That's what I really wanted!
	# Go use that! Go: https://gist.github.com/ozydingo/70de96ad57ab69003446

	# == DEFINITIONS
	# - A Taxi is a car for hire. A taxi has_many :passengers.
	# - A Passenger records one person riding in one taxi one time. It belongs_to :taxi.

	# Gemfile
	gem "puma"

	# Procfile
	web: bundle exec puma -p $PORT -e $RACK_ENV -C config/puma.rb

	# add to config block config/environments/production.rb
	config.threadsafe!

	# get rid of NewRelic after_fork code, if you were doing this:

	select fund_sums.total, fund_products.name
	from (
	select sum(allocation_amount) total, fund_product_id
	from transactions
	where transactions.investor_id = 490
	group by fund_product_id
	) fund_sums
	left join fund_products on fund_sums.fund_product_id = fund_products.id

	#!/usr/bin/env ruby
	#
	# Proof-of-Concept exploit for Rails Remote Code Execution (CVE-2013-0333)
	#
	# ## Advisory
	#
	# https://groups.google.com/forum/?fromgroups=#!topic/rubyonrails-security/1h2DR63ViGo
	#
	# ## Caveats
	#