Configuring Tire to work with Bonsai

configuring tire for bonsai.md

1. Configure Tire to use the Bonsai ElasticSearch Heroku add-on

gem 'tire'

config/initializers/bonsai.rb

ENV['ELASTICSEARCH_URL'] = ENV['BONSAI_URL']

app/models/article.rb

class Article
  include Tire::Model::Search
  include Tire::Model::Callbacks
end

2. Create the index and import your documents

rake environment tire:import CLASS=Article FORCE=true

---

Known issues

There are no "known issues" with Tire and Bonsai at the moment. Having trouble with something in particular? Drop us a line at [email protected].

• Custom index analyzers must be set at index creation time. Ability to dynamically create, modify and destroy indexes.
• Bulk import uses cluster-level /_bulk handler rather than the index-level _bulk handler, causing bulk imports to fail. Issue 327
• Multi-model search is not scoped within the index. Issue 322
• Undefined method [] for nil:NilClass on index.settings. Bonsai shared cluster indices are mapped to a random identifier, whereas Tire expects the logical index name in the _index response. To be fixed within Bonsai. Email [email protected] if this is affecting you. Issue 386
• ES Alias API not fully supported in Bonsai. Email [email protected] if you would like to beta test that.

there is a problem with this technique

if your setting the index_name in the class you will run into the problem that the Callbacks are not working since the instance.index != Class.index

the work around for this is to create your own observer for after_save and after_destroy

def after_save(record)
   Class.index.store(record)
end

def after_destroy(record)
   Class.index.remove(record)
end

since I only have 1 model i am using this works for me but it can be pretty tedious if you have multiple model maybe you can use active record observer that is not inferred

read more about it here. http://api.rubyonrails.org/classes/ActiveRecord/Observer.html

so, I followed instructions in the gist, adding this to my initializers/bonsai.rb

if ENV['BONSAI_INDEX_URL'] Tire.configure do url "http://index.bonsai.io" end BONSAI_INDEX_NAME = ENV['BONSAI_INDEX_URL'][/[^\/]+$/] else app_name = Rails.application.class.parent_name.underscore.dasherize app_env = Rails.env BONSAI_INDEX_NAME = "#{app_name}-#{app_env}" end

Also added this to the model

index_name BONSAI_INDEX_NAME

Now, when I run my app locally in development I get the following error

[REQUEST FAILED] curl -X GET "http://localhost:9200/dynamite-urbanite-development/city/_search?load=true&pretty=true" -d '{"query":{"query_string":{"query":"texas","default_operator":"AND"}}}'

Started GET "/cities?utf8=%E2%9C%93&query=texas" for 127.0.0.1 at 2012-07-03 22:12:02 -0400
Connecting to database specified by database.yml
Processing by CitiesController#index as HTML
Parameters: {"utf8"=>"✓", "query"=>"texas"}
Completed 500 Internal Server Error in 3ms

Tire::Search::SearchRequestFailed (404 : {"error":"IndexMissingException[[dynamite-urbanite-development] missing]","status":404}):
app/models/city.rb:20:in search' app/controllers/cities_controller.rb:3:inindex'

So, I fixed the above problem by re - indexing my City model. My app works in development, but when I pushed the changes to Heroku, the application now crashes. Any ideas? The error message from Heroku logs are not help enough, all they tell me is that my app did indeed crash.

Have setup the initializer as above and when I curl the URL for BONSAI_INDEX_URL found in heroku config the test response is successful but when interacting with a model I get the infamous connection refused error. Is there a bug with Tire 0.4.2 or something?

bump

For debugging:

Trying to curl locally works, but trying to curl from heroku returns the following:

heroku run curl -XPOST http://index.bonsai.io/e2ljko3c8qhc53zmdf0/test/hello -d '{"title":"Hello world"}' gives this...

Runningcurl -XPOST http://index.bonsai.io/e2ljko3c8qhc53zmdf0/test/hello -d {"title":"Hello world"}attached to terminal... up, run.1 {"error":"RemoteTransportException[[Polaris][inet[/10.31.156.223:9300]][index]]; nested: MapperParsingException[Failed to parse]; nested: JsonParseException[Unexpected character ('H' (code 72)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n at [Source: [B@1f8fbddc; line: 1, column: 117]]; ","status":400}

Using Tire 0.4.3, I'm receiving the following when trying to bulk import records:

<Tire::Index:0x00000008d5a770 @name="myindexname", @response=502 :

502 Bad Gateway

For those interested, I just soft deployed a big change to how provisioning works, which should be much more compatible with how Tire operates.

New addons will get a different environment variable: BONSAI_URL, which can be set directly for the Tire URL. You can now create indexes at this URL using the usual ElasticSearch Index APIs. Specifying the index name is now optional, though recommended, since I still think that applications should only need a single index per environment, as with your database.

I'll announce more about that on Monday, in particular about number of indexes and shards allowed within our production plans. In the mean time, check out the revised gist, and let me know if you have any questions. You can always email me directly at [email protected] as well.

Cheers, folks, and thanks for all the great feedback throughout the beta! Looking forward to launching this service very soon :-)

/cc @bryanmtl @pawel2105 @verdi327 @zacksiri @rahilsondhi @ernesto-jimenez @jayinteractive @kikouli

Can't wait to see updated notes :)

@nz, btw do you use latest Tire version from github or just latest stable release?

@nz is it possible to set the default analyzer on the _all field? I have tried this with the following settings in an ActiveModel

settings index: { analysis: { analyzer: { default: { type: 'snowball', language: 'english' } } } }

...but it looks like Bonsai is silently rejecting them...

curl -XGET 'http://juniper-1234.us-east-1.bonsai.io/artworks_staging/_settings'                             
{"xxxx1234":{"settings":{"index.number_of_replicas":"1","index.number_of_shards":"1","index.version.created":"190999"}}}

I need to find a way to have the _all field analyzed with snowball. Do you/anyone know a solution to this?

The instructions on

https://devcenter.heroku.com/articles/bonsai#using-tire-with-rails-3x 

does not mention setting up a custom index 'index_name INDEX_NAME' for every model. Are the instructions in this gist still necessary for the current version of Bonsai Search?

@digitalplaywright: the "shared index" approach — specifying the same index_name per model — is now optional. You may now create multiple indexes, subject to the total shards allowed per plan. (Currently set to two during beta.)

@pietia any relatively recent version of Tire should be fine, I'm working to support what's popular

@zefer can you drop me an email ([email protected]) with the full index URL, and perhaps a curl command that's trying to POST or PUT the settings? We do filter that settings payload to track/enforce shards and replicas, but analyzers should pass through as provided.

Adding the following to an initializer will set the index prefix for all models:

Tire::Model::Search.index_prefix "#{Rails.application.class.parent_name.downcase}_#{Rails.env.to_s.downcase}"

Nice tip, @evanwhalen. I wonder if something like that could be suggested for a contribution upstream to Tire proper. Scoping by application and environment in some way or another seems like a no-brainer for local development and testing with multiple applications.

I still have trouble with search with multiple indexes.

> Tire.search("profiles,companies") { query { string "*world*"} }

[REQUEST FAILED] curl -X GET 'http://[email protected]/profiles,companies/_search?pretty' -d '{"query":{"query_string":{"query":"*world*"}}}'
Tire::Search::SearchRequestFailed: 404 : {"error": "Index not found", "status": 404}

Is it possible to use Bonsai with multiple indexes search?

miry, I have the same problem as well. I'm looking into creating an index that the merged version of the index I'd like to search.

@miry and @Will-Sommers, this issue is on my end. I still have some work to do for supporting multi-index search syntax in Elasticsearch. I'll take a stab at that this week.

@miry @Will-Sommers — in the mean time, merging into a combined index (as per the "optional" in the instructions) should work fine for your purposes. You can then execute searches with:

Tire.search(INDEX_NAME) { query { string 'hello world' } }

Just to note, I've had to write a shell script to create my index with the proper analyzers and scoring. When importing multiple types into a single index only the first model takes settings in my experience.

Can anyone point me in the direction of an explanation / more information regarding this please? -

# Optional, but recommended: use a single index per application per environment.
# Caveat: This convention not be entirely supported throughout Tire's API.

Is the default to have an index for each model?

What's the advantages/disadvantages of a single index for the app?

How many indexes do Bonsai allow?

Thanks,

Ian

This was a helpful gist, thanks for posting it up.

@ichilton—

Is the default to have an index for each model?

This is Tire's default, yes.

What's the advantages/disadvantages of a single index for the app?

Each index carries its own overhead, and multiple indices can be wasteful for small applications. It is negligible, but in a shared cluster environment, that adds up. Creating an index per model can help as an additional level of natural partitioning, but in most cases that is premature optimization, and you will be better served by consolidating into a single index per environment.

The main disadvantage for using a single index with Tire is that it doesn't always fit within Tire's conventions. If you ever have trouble related to index naming conventions, you should just stick with Tire's defaults.

How many indexes do Bonsai allow?

The total primary shards varies per plan. All plans should be able to comfortable index 2–3 models. We're still working on exactly where to set those numbers, so email me at [email protected] if you ever bump into those limits.

@aarongray—

Thanks! Happy to help.

I have the same problem as @pawel2105 had and wonder what is the solution? Via curl locally all works fine. But I get this error when I try to import.

→ heroku run rake environment tire:import CLASS=Initiative FORCE=true
Running rake environment tire:import CLASS=Initiative FORCE=true attached to terminal... up, run.5891
[IMPORT] Deleting index 'stadt-gestalten-production'
rake aborted!
Server broke connection
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.6.7/lib/restclient/request.rb:182:in rescue in transmit' /app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.6.7/lib/restclient/request.rb:140:intransmit'
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in execute' /app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.6.7/lib/restclient/request.rb:33:inexecute'
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.6.7/lib/restclient.rb:88:in head' /app/vendor/bundle/ruby/2.0.0/gems/tire-0.6.0/lib/tire/http/client.rb:43:inhead'
/app/vendor/bundle/ruby/2.0.0/gems/tire-0.6.0/lib/tire/index.rb:18:in exists?' /app/vendor/bundle/ruby/2.0.0/gems/tire-0.6.0/lib/tire/tasks.rb:14:increate_index'
/app/vendor/bundle/ruby/2.0.0/gems/tire-0.6.0/lib/tire/tasks.rb:110:in `block (3 levels) in <top (required)>'
Tasks: TOP => tire:import => tire:import:model

Hi @criscrossed, you should email [email protected] with your account information so I can check that out.

Updated the setup instructions to remove the "optional" combined index strategy. Our plans offer sufficient shards these days to support most common index-per-model sharding strategies.

@nz - have you switched over to elasticsearch-rails gem? i'm trying to install it and it's giving me problems with Faraday.