Skip to content

Instantly share code, notes, and snippets.

@tjwallace
Last active January 25, 2024 11:19
Show Gist options
  • Save tjwallace/f2fdadd656c74cdbfcc3218e6188d466 to your computer and use it in GitHub Desktop.
Save tjwallace/f2fdadd656c74cdbfcc3218e6188d466 to your computer and use it in GitHub Desktop.

Output on Ruby 2.6, MacBook Pro (13-inch, 2017, 3.5 GHz Intel Core i7)

Note: I manually removed some BigDecimal deprecation warnings

Rehearsal ------------------------------------------------
as_json        0.122644   0.000508   0.123152 (  0.123823)
fast_jsonapi   0.184139   0.000829   0.184968 (  0.186195)
grape_entity   0.709880   0.003083   0.712963 (  0.714929)
blueprinter    0.220149   0.004481   0.224630 (  0.225596)
ams            0.880767   0.027975   0.908742 (  0.912454)
roar           0.733756   0.006674   0.740430 (  0.743307)
panko          0.142707   0.000671   0.143378 (  0.145400)
--------------------------------------- total: 3.038263sec

                   user     system      total        real
as_json        0.120163   0.000531   0.120694 (  0.121254)
fast_jsonapi   0.178100   0.001448   0.179548 (  0.182374)
grape_entity   0.714065   0.001980   0.716045 (  0.718069)
blueprinter    0.178831   0.001490   0.180321 (  0.181874)
ams            0.885708   0.007647   0.893355 (  0.901771)
roar           0.658007   0.004593   0.662600 (  0.666706)
panko          0.125161   0.001391   0.126552 (  0.127679)
Warming up --------------------------------------
             as_json     1.000  i/100ms
        fast_jsonapi     1.000  i/100ms
        grape_entity     1.000  i/100ms
         blueprinter     1.000  i/100ms
                 ams     1.000  i/100ms
                roar     1.000  i/100ms
               panko     1.000  i/100ms
Calculating -------------------------------------
             as_json      6.697  (± 2.7%) i/s -     67.000  in  10.154134s
        fast_jsonapi      4.492  (± 6.8%) i/s -     42.000  in  10.202085s
        grape_entity      1.421  (± 3.0%) i/s -     15.000  in  10.595573s
         blueprinter      5.239  (± 1.8%) i/s -     53.000  in  10.162482s
                 ams      1.152  (± 2.3%) i/s -     12.000  in  10.448728s
                roar      1.428  (± 2.4%) i/s -     15.000  in  10.554087s
               panko      7.278  (± 2.5%) i/s -     72.000  in  10.016180s
                   with 95.0% confidence

Comparison:
               panko:        7.3 i/s
             as_json:        6.7 i/s - 1.09x  (± 0.04) slower
         blueprinter:        5.2 i/s - 1.39x  (± 0.04) slower
        fast_jsonapi:        4.5 i/s - 1.62x  (± 0.12) slower
                roar:        1.4 i/s - 5.10x  (± 0.19) slower
        grape_entity:        1.4 i/s - 5.12x  (± 0.20) slower
                 ams:        1.2 i/s - 6.32x  (± 0.22) slower
                   with 95.0% confidence

Calculating -------------------------------------
             as_json    55.358M memsize (     0.000  retained)
                       839.569k objects (     0.000  retained)
                         8.000  strings (     0.000  retained)
        fast_jsonapi    37.181M memsize (     0.000  retained)
                       489.944k objects (     0.000  retained)
                        50.000  strings (     0.000  retained)
        grape_entity   113.684M memsize (     0.000  retained)
                       939.629k objects (     0.000  retained)
                         4.000  strings (     0.000  retained)
         blueprinter    36.730M memsize (     0.000  retained)
                       319.867k objects (     0.000  retained)
                         0.000  strings (     0.000  retained)
                 ams   167.435M memsize (     0.000  retained)
                         1.649M objects (     0.000  retained)
                         6.000  strings (     0.000  retained)
                roar   130.565M memsize (     0.000  retained)
                       859.561k objects (     0.000  retained)
                         1.000  strings (     0.000  retained)
               panko    55.365M memsize (     0.000  retained)
                       839.642k objects (     0.000  retained)
                        13.000  strings (     0.000  retained)

Comparison:
         blueprinter:   36729880 allocated
        fast_jsonapi:   37180800 allocated - 1.01x more
             as_json:   55357768 allocated - 1.51x more
               panko:   55364912 allocated - 1.51x more
        grape_entity:  113683920 allocated - 3.10x more
                roar:  130565336 allocated - 3.55x more
                 ams:  167434632 allocated - 4.56x more
require 'bundler/inline'
gemfile do
source 'https://rubygems.org'
gem 'oj'
gem 'benchmark-ips', require: 'benchmark/ips'
gem 'kalibera'
gem 'benchmark-memory', require: 'benchmark/memory'
gem 'activesupport'
# https://github.com/Netflix/fast_jsonapi
gem 'fast_jsonapi'
# https://github.com/ruby-grape/grape-entity
gem 'grape-entity'
# https://github.com/procore/blueprinter
gem 'blueprinter'
# https://github.com/rails-api/active_model_serializers/tree/0-10-stable
gem 'active_model_serializers', '~> 0.10.0'
# https://github.com/trailblazer/roar
# https://github.com/trailblazer/representable
gem 'roar'
gem 'multi_json'
# https://github.com/yosiat/panko_serializer
gem 'panko_serializer'
end
require 'active_support'
require 'active_support/core_ext/object' # For Hash#deep_dup
# Define models
Issue = Struct.new(:id, :number, :title, :user, :labels) do
alias_method :read_attribute_for_serialization, :send
def label_ids
labels.map(&:id)
end
def user_id
user.id
end
end
User = Struct.new(:id, :login) do
alias_method :read_attribute_for_serialization, :send
end
Label = Struct.new(:id, :name, :color) do
alias_method :read_attribute_for_serialization, :send
end
# Define serializers
module FastJsonApi
class IssueSerializer
include FastJsonapi::ObjectSerializer
attributes :number, :title
has_many :labels
belongs_to :user
end
class LabelSerializer
include FastJsonapi::ObjectSerializer
attributes :name, :color
end
class UserSerializer
include FastJsonapi::ObjectSerializer
attributes :login
end
end
module GrapeEntity
class Label < Grape::Entity
expose :id
expose :name
expose :color
end
class User < Grape::Entity
expose :id
expose :login
end
class Issue < Grape::Entity
expose :id
expose :number
expose :title
expose :labels, using: Label
expose :user, using: User
end
end
Blueprinter.configure do |config|
config.generator = Oj
config.sort_fields_by = :definition
end
module BluePrint
class Label < Blueprinter::Base
identifier :id
fields :name, :color
end
class User < Blueprinter::Base
identifier :id
field :login
end
class Issue < Blueprinter::Base
identifier :id
fields :number, :title
association :labels, blueprint: Label
association :user, blueprint: User
end
end
ActiveModelSerializers.logger = nil
module Ams
class Label < ActiveModel::Serializer
attributes :id, :name, :color
end
class User < ActiveModel::Serializer
attributes :id, :login
end
class Issue < ActiveModel::Serializer
attributes :id, :number, :title
has_many :labels, serializer: Label
belongs_to :user, serializer: User
end
end
require 'roar/decorator'
require 'roar/json'
module ROAR
class IssueRepresenter < Roar::Decorator
include Roar::JSON
property :id
property :number
property :title
collection :labels do
property :id
property :name
property :color
end
property :user do
property :id
property :login
end
end
end
module PANKO
class LabelSerializer < Panko::Serializer
attributes :id, :name, :color
end
class UserSerializer < Panko::Serializer
attributes :id, :login
end
class IssueSerializer < Panko::Serializer
attributes :id, :number, :title
has_many :labels, serializer: LabelSerializer
has_one :user, serializer: UserSerializer
end
end
# Generate data
users = Array.new(10) { |i| User.new(i, "User #{i}") }
labels = Array.new(4) { |i| Label.new(i, "Label #{i}", 'ffffff') }
issues = Array.new(10_000) { |i| Issue.new(i, i, "Issue #{i}", users.sample, labels.sample(rand(2..4))) }
serializers = [
{
name: :as_json,
serializer: -> { issues.as_json },
output_inspector: ->(output) { output.first }
},
{
name: :fast_jsonapi,
serializer: -> { FastJsonApi::IssueSerializer.new(issues, include: [:labels, :user]).serializable_hash },
output_inspector: ->(output) { output[:data].first }
},
{
name: :grape_entity,
serializer: -> { GrapeEntity::Issue.represent(issues).as_json },
output_inspector: ->(output) { output.first }
},
{
name: :blueprinter,
serializer: -> { BluePrint::Issue.render_as_hash(issues) },
output_inspector: ->(output) { output.first }
},
{
name: :ams,
serializer: -> { ActiveModelSerializers::SerializableResource.new(issues, each_serializer: Ams::Issue).as_json },
output_inspector: ->(output) { output.first }
},
{
name: :roar,
serializer: -> { ROAR::IssueRepresenter.for_collection.new(issues).as_json },
output_inspector: ->(output) { output.first }
},
{
name: :panko,
serializer: -> { Panko::ArraySerializer.new(issues, each_serializer: PANKO::IssueSerializer).as_json },
output_inspector: ->(output) { output['subjects'].first }
}
]
# Display output
serializers.each do |name:, serializer:, output_inspector:|
puts "\n#{name}:\n"
puts output_inspector.call(serializer.call).inspect
puts
end
# Run benchmarks
require 'benchmark'
Benchmark.bmbm do |b|
serializers.each do |name:, serializer:, **_other|
b.report(name, &serializer)
end
end
%i[ips memory].each do |bench|
Benchmark.send(bench) do |b|
b.config(time: 10, warmup: 5, stats: :bootstrap, confidence: 95) if b.respond_to?(:config)
serializers.each do |name:, serializer:, **_other|
b.report(name, &serializer)
end
b.compare!
end
end
@pcreux
Copy link

pcreux commented Jan 16, 2019

So... as_json, panko or fast_jsonapi. Which one would you go with?

@hotrungnhan
Copy link

hotrungnhan commented Jan 25, 2024

2024 Results

Machine : Macbook air M1 ( battery)
Ruby Versions : ruby 3.2.2 (2023-03-30 revision e51014f9c0) [arm64-darwin23]
sadly, i can't make json-api work.

grape-entity (1.0.0)
blueprinter (1.0.1)
roar (1.2.0)
panko_serializer (0.8.1)
oj (3.16.3)
active_model_serializers (0.10.14)

#
Rehearsal ------------------------------------------------
as_json        0.043739   0.000073   0.043812 (  0.044145)
grape_entity   0.526970   0.004110   0.531080 (  0.533710)
blueprinter    0.263931   0.002690   0.266621 (  0.273178)
ams            1.470586   0.003469   1.474055 (  1.491836)
roar           0.533820   0.005840   0.539660 (  0.593379)
panko          0.067007   0.000695   0.067702 (  0.068115)
--------------------------------------- total: 2.922930sec

                   user     system      total        real
as_json        0.042489   0.000067   0.042556 (  0.042811)
grape_entity   0.517752   0.003749   0.521501 (  0.522805)
blueprinter    0.226224   0.002563   0.228787 (  0.230431)
ams            1.470077   0.002183   1.472260 (  1.476308)
roar           0.462844   0.001211   0.464055 (  0.465213)
panko          0.040305   0.000002   0.040307 (  0.040432)
Warming up --------------------------------------
             as_json     2.000 i/100ms
        grape_entity     1.000 i/100ms
         blueprinter     1.000 i/100ms
                 ams     1.000 i/100ms
                roar     1.000 i/100ms
               panko     1.000 i/100ms
Calculating -------------------------------------
             as_json     18.610 (± 2.2%) i/s -    184.000 in  10.065971s
        grape_entity      1.826 (± 7.6%) i/s -     18.000 in  10.272393s
         blueprinter      4.140 (± 2.6%) i/s -     42.000 in  10.248080s
                 ams      0.626 (± 4.6%) i/s -      7.000 in  11.237138s
                roar      2.025 (± 2.9%) i/s -     21.000 in  10.433426s
               panko     18.021 (± 1.4%) i/s -    177.000 in  10.007943s
                   with 95.0% confidence

Comparison:
             as_json:       18.6 i/s
               panko:       18.0 i/s - same-ish: difference falls within error
         blueprinter:        4.1 i/s - 4.50x  (± 0.15) slower
                roar:        2.0 i/s - 9.19x  (± 0.33) slower
        grape_entity:        1.8 i/s - 10.20x  (± 0.81) slower
                 ams:        0.6 i/s - 29.71x  (± 1.56) slower
                   with 95.0% confidence

Calculating -------------------------------------
             as_json    23.777M memsize (   416.000  retained)
                       269.656k objects (     4.000  retained)
                         8.000  strings (     2.000  retained)
        grape_entity    82.731M memsize (     4.128k retained)
                         1.059M objects (    53.000  retained)
                         2.000  strings (     1.000  retained)
         blueprinter    59.098M memsize (   368.000  retained)
                       908.898k objects (     6.000  retained)
                         0.000  strings (     0.000  retained)
                 ams   138.688M memsize (     7.684M retained)
                         1.808M objects (    60.039k retained)
                         8.000  strings (     0.000  retained)
                roar   136.043M memsize (     1.288k retained)
                       798.916k objects (    11.000  retained)
                         1.000  strings (     0.000  retained)
               panko    23.781M memsize (     2.216k retained)
                       269.706k objects (    32.000  retained)
                        12.000  strings (     5.000  retained)

Comparison:
             as_json:   23777336 allocated
               panko:   23780560 allocated - 1.00x more
         blueprinter:   59098184 allocated - 2.49x more
        grape_entity:   82730568 allocated - 3.48x more
                roar:  136043320 allocated - 5.72x more
                 ams:  138687872 allocated - 5.83x more
# frozen_string_literal: true

require 'bundler/inline'

gemfile do
  source 'https://rubygems.org'
  gem 'oj'
  gem 'benchmark-ips', require: 'benchmark/ips'
  gem 'kalibera'
  gem 'benchmark-memory', require: 'benchmark/memory'

  # gem 'activesupport'

  # https://github.com/Netflix/fast_jsonapi
  # gem 'fast_jsonapi'

  # https://github.com/ruby-grape/grape-entity
  gem 'grape-entity'

  # https://github.com/procore/blueprinter
  gem 'blueprinter'

  # https://github.com/rails-api/active_model_serializers/tree/0-10-stable
  gem 'active_model_serializers'

  # https://github.com/trailblazer/roar
  # https://github.com/trailblazer/representable
  gem 'roar'
  gem 'multi_json'

  # https://github.com/yosiat/panko_serializer
  gem 'panko_serializer'
end

# require 'active_support'
# require 'active_support/core_ext/object' # For Hash#deep_dup

# Define models
Issue = Struct.new(:id, :number, :title, :user, :labels) do
  alias_method :read_attribute_for_serialization, :send

  def label_ids
    labels.map(&:id)
  end

  delegate :id, to: :user, prefix: true
end
User = Struct.new(:id, :login) do
  alias_method :read_attribute_for_serialization, :send
end
Label = Struct.new(:id, :name, :color) do
  alias_method :read_attribute_for_serialization, :send
end

# Define serializers
# module FastJsonApi
#   class IssueSerializer
#     include FastJsonapi::ObjectSerializer

#     attributes :number, :title
#     has_many :labels
#     belongs_to :user
#   end

#   class LabelSerializer
#     include FastJsonapi::ObjectSerializer

#     attributes :name, :color
#   end

#   class UserSerializer
#     include FastJsonapi::ObjectSerializer

#     attributes :login
#   end
# end

module GrapeEntity
  class Label < Grape::Entity
    expose :id
    expose :name
    expose :color
  end

  class User < Grape::Entity
    expose :id
    expose :login
  end

  class Issue < Grape::Entity
    expose :id
    expose :number
    expose :title
    expose :labels, using: Label
    expose :user, using: User
  end
end

Blueprinter.configure do |config|
  config.generator = Oj
  config.sort_fields_by = :definition
end

module BluePrint
  class Label < Blueprinter::Base
    identifier :id
    fields :name, :color
  end

  class User < Blueprinter::Base
    identifier :id
    field :login
  end

  class Issue < Blueprinter::Base
    identifier :id
    fields :number, :title
    association :labels, blueprint: Label
    association :user, blueprint: User
  end
end

ActiveModelSerializers.logger = nil
module Ams
  class Label < ActiveModel::Serializer
    attributes :id, :name, :color
  end

  class User < ActiveModel::Serializer
    attributes :id, :login
  end

  class Issue < ActiveModel::Serializer
    attributes :id, :number, :title
    has_many :labels, serializer: Label
    belongs_to :user, serializer: User
  end
end

require 'roar/decorator'
require 'roar/json'
module ROAR
  class IssueRepresenter < Roar::Decorator
    include Roar::JSON

    property :id
    property :number
    property :title

    collection :labels do
      property :id
      property :name
      property :color
    end

    property :user do
      property :id
      property :login
    end
  end
end

module PANKO
  class LabelSerializer < Panko::Serializer
    attributes :id, :name, :color
  end

  class UserSerializer < Panko::Serializer
    attributes :id, :login
  end

  class IssueSerializer < Panko::Serializer
    attributes :id, :number, :title
    has_many :labels, serializer: LabelSerializer
    has_one :user, serializer: UserSerializer
  end
end

# Generate data
users = Array.new(10) { |i| User.new(i, "User #{i}") }
labels = Array.new(4) { |i| Label.new(i, "Label #{i}", 'ffffff') }
issues = Array.new(10_000) { |i| Issue.new(i, i, "Issue #{i}", users.sample, labels.sample(rand(2..4))) }

serializers = [
  {
    name: :as_json,
    serializer: -> { issues.as_json },
    output_inspector: ->(output) { output.first }
  },
  # {
  #   name: :fast_jsonapi,
  #   serializer: -> { FastJsonApi::IssueSerializer.new(issues, include: %i[labels user]).serializable_hash },
  #   output_inspector: ->(output) { output[:data].first }
  # },
  {
    name: :grape_entity,
    serializer: -> { GrapeEntity::Issue.represent(issues).as_json },
    output_inspector: ->(output) { output.first }
  },
  {
    name: :blueprinter,
    serializer: -> { BluePrint::Issue.render_as_hash(issues) },
    output_inspector: ->(output) { output.first }
  },
  {
    name: :ams,
    serializer: lambda {
                  ActiveModelSerializers::SerializableResource.new(issues, each_serializer: Ams::Issue,
                                                                           root: :key).as_json
                },
    output_inspector: ->(output) { output.first }
  },
  {
    name: :roar,
    serializer: -> { ROAR::IssueRepresenter.for_collection.new(issues).as_json },
    output_inspector: ->(output) { output.first }
  },
  {
    name: :panko,
    serializer: -> { Panko::ArraySerializer.new(issues, each_serializer: PANKO::IssueSerializer).as_json },
    output_inspector: ->(output) { output['subjects'].first }
  }
]

# Display output
serializers.each do |obj|
  name, serializer, output_inspector = obj.values_at(:name, :serializer, :output_inspector)
  puts "\n#{name}:\n"
  puts output_inspector.call(serializer.call).inspect
  puts
end

# Run benchmarks
require 'benchmark'
Benchmark.bmbm do |b|
  serializers.each do |obj|
    name, serializer = obj.values_at(:name, :serializer)
    b.report(name, &serializer)
  end
end

%i[ips memory].each do |bench|
  Benchmark.send(bench) do |b|
    b.config(time: 10, warmup: 5, stats: :bootstrap, confidence: 95) if b.respond_to?(:config)

    serializers.each do |obj|
      name, serializer = obj.values_at(:name, :serializer)
      b.report(name, &serializer)
    end

    b.compare!
  end
end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment