Cursor based pagination for Rails models.

README.md

Example Usage

Users

id	name
1	Jane
2	Max
3	John
4	Scott
5	Mark

result = User.after(3) 
=>
[
  { id: 4, name: "Scott" },
  { id: 5, name: "Mark" }
]

result.has_more?
=> false

result.total_size
=> 2

result.total_count
=> 6

result = User.before(3) 
=>
[
  { id: 2, name: "Max" },
  { id: 1, name: "Jane" }
]

result.has_more?
=> false

result.total_size
=> 2

result.total_count
=> 6

result = User.after(1).limit(1)
=>
[
  { id: 2, name: "Max" }
]

result.has_more?
=> true

result.total_size
=> 5

result.total_count
=> 6

result = User.after(1).before(3)
=>
[
  { id: 2, name: "Max" }
]

result.has_more?
=> false

result.total_size
=> 1

result.total_count
=> 6

paginatable.rb

module Arel
  module Nodes
    module Cursor
      class LessThan < Arel::Nodes::LessThan; end
      class GreaterThan < Arel::Nodes::GreaterThan; end
    end
  end

  module Predications
    def before right
      Nodes::Cursor::LessThan.new self, quoted_node(right)
    end

    def after right
      Nodes::Cursor::GreaterThan.new self, quoted_node(right)
    end
  end
end

module Paginatable
  extend ActiveSupport::Concern

  included do
    def self.all(*args)
      super.extending(ActiveRecordRelationMethods)
    end

    def self.after(cursor)
      reorder(primary_key => :asc).where(
        arel_table[primary_key].after(cursor)
      ).extending(ActiveRecordRelationMethods)
    end

    def self.before(cursor)
      reorder(primary_key => :desc).where(
        arel_table[primary_key].before(cursor)
      ).extending(ActiveRecordRelationMethods)
    end
  end

  module ActiveRecordRelationMethods
    def has_more?
      @has_more ||= begin
        # Cache #size otherwise multiple calls to the database will occur.
        (results_size = size) > 0 && results_size < total_size
      end
    end

    # Returns number of records that exist in scope of the current cursor
    def total_size(column_name = :all) #:nodoc:
      # #count overrides the #select which could include generated columns
      # referenced in #order, so skip #order here, where it's irrelevant to the
      # result anyway.
      @total_size ||= begin
        context = except(:offset, :limit, :order)

        # Remove includes only if they are irrelevant
        context = context.except(:includes) unless references_eager_loaded_tables?

        args = [column_name]

        # .group returns an OrderedHash that responds to #count
        context = context.count(*args)

        if context.is_a?(Hash) || context.is_a?(ActiveSupport::OrderedHash)
          context.count
        else
          context.respond_to?(:count) ? context.count(*args) : context
        end
      end
    end
  
    # Returns number of records that exist without :offset, :limit, :order, :before or :after 
    def total_count(column_name = :all) #:nodoc:
      # #count overrides the #select which could include generated columns
      # referenced in #order, so skip #order here, where it's irrelevant to the
      # result anyway.
      @total_count ||= begin
        context = except(:offset, :limit, :order)

        context.where_values = where_values.reject do |value|
          value.is_a?(Arel::Nodes::Cursor::GreaterThan) ||
          value.is_a?(Arel::Nodes::Cursor::LessThan)
        end

        # Remove includes only if they are irrelevant
        context = context.except(:includes) unless references_eager_loaded_tables?

        args = [column_name]

        # .group returns an OrderedHash that responds to #count
        context = context.count(*args)

        if context.is_a?(Hash) || context.is_a?(ActiveSupport::OrderedHash)
          context.count
        else
          context.respond_to?(:count) ? context.count(*args) : context
        end
      end
    end
  end
end

Where do we put this file paginatable.rb?

@overdrivemachines that file goes to the directory app/models/concerns

I'd like to point out, that line 81 seems not work with Rails 5 due to some renamings.

This seems to do the trick:

[...]
        context.where_clause = where_values_hash.reject do |value|
[...]

That's a bad implementation of cursor based pagination. You break the sorting. What if my collection sorted by name? But I want to get new collection skipping objects up to some id, but remain the same sorting options. I.e we have this:

id	name
1	Jane
2	Max
3	John
4	Scott
5	Mark

User.order(:name) will return me full collection:

id	name
1	Jane
3	John
5	Mark
2	Max
4	Scott

Of course I want to paginate, so I limit my collection with User.order(:name).limit(2) (in real life it could be not 2 but 20):

id	name
1	Jane
3	John

And what if I want the second page? But I don't want to use offset, if you've got here to see "cursor pagination" instead of a simple pagination, then you probably know why you need this. So what?

I use User.order(:name).after(3).limit(2) and should get this:

id	name
5	Mark
2	Max

But your solution will reorder my collection by id, returning some nonsense

@jerrygreen

That's a bad implementation of cursor based pagination. You break the sorting.

It's not a bad implementation or nonsense. It served my use case just fine. Just because it doesn't fit yours doesn't make it bad. I did not intend for it to account for sorting on anything other than the primary key. This is how most cursor based apis work. It doesn't break your sorting, it intentionally removes it. You can't sort by arbitrary columns. The field you compare the cursor to has to be unique, sequential and immutable.

It's been several years since I posted this. I think the solution I would implement now is to generate an opaque token for the cursor that contains the necessary information. One cursor for the next page and one for the previous would be sent back to the client.

Making it opaque to the client means you can change the implementation server side, which is a win.

Slack has a decent write up on this: https://slack.engineering/evolving-api-pagination-at-slack-1c1f644f8e12

@jerrygreen you can always sort by name, id.