MatthewRDodds

Elasticsearch indexing speed tests

Trying to find the fastest way to index all the things

Iterating over the entire collection and serializing and indexing for each item:

    user     system      total        real
10.630000   0.400000  11.030000 ( 24.820179)

MatthewRDodds

Sidekiq::RetrySet.new.find { |job| job.queue == 'queue_name' }

MatthewRDodds

Error when starting vagrant

The HGFS kernel module was not found on the running virtual machine. This must be installed for shared folders to work properly. Please install the VMware tools within the guest and try again. Note that the VMware tools installation will succeed even if HGFS fails to properly install. Carefully read the output of the VMware tools installation to verify the HGFS kernel modules were installed properly.

Solution

MatthewRDodds

==> dev: Attempting graceful shutdown of VM...
    dev:
    dev: Vagrant insecure key detected. Vagrant will automatically replace
    dev: this with a newly generated keypair for better security.
    dev:
    dev: Inserting generated public key within guest...
    dev: Removing insecure key from the guest if its present...
    dev: Key inserted! Disconnecting and reconnecting using new SSH key...
==> dev: Stopping the VMware VM...

MatthewRDodds

require 'sidekiq'

Sidekiq.configure_server do |config|
  config.redis = { url: ENV['REDIS_URL'] }
end

Sidekiq.redis do |conn|
  conn.smembers('queues').each do |q|
    # Sidekiq::Queue.new(q).clear
    puts q

MatthewRDodds

With PostgresExt

       user     system      total        real
   0.340000   0.020000   0.360000 (  0.742707)
   0.350000   0.030000   0.380000 (  0.751849)
   0.360000   0.010000   0.370000 (  0.798465)
   0.330000   0.020000   0.350000 (  0.723302)
   0.360000   0.020000   0.380000 (  0.750776)
 0.370000 0.010000 0.380000 ( 0.748968)

MatthewRDodds

Sidekiq.redis{ |conn| conn.smembers('queues').map{|q| Sidekiq::Queue.new(q).clear} }; Sidekiq::RetrySet.new.clear

MatthewRDodds

sudo /usr/share/elasticsearch/bin/plugin -install mobz/elasticsearch-head