Upgrading to the Bonsai ElasticSearch final architecture

For our initial beta launch, we designed our Heroku addon integration to automatically create one index per Heroku app. However, feedback from users such as yourself showed us that there are some ElasticSearch use cases where it is necessary to have the ability to create and destroy your own indexes. Furthermore, some applications or ElasticSearch clients required more than one index per application.

Rather than pre-create one index per application, our new architecture provides a URL that behaves like an ElasticSearch cluster endpoint, where you may create, update and destroy indexes within your account. We find this is a lot more useful in a handful of situations, and along the way we've been able to overhaul and improve our architecture to accommodate this design.

Upgrading instructions

To upgrade to the latest architecture, you will need to remove the addon, and add it again. You will also need to create a new index, either with curl, or as per your ElasticSearch client's recommendations, using your new BONSAI_URL environment variable.

This upgrade process will require that you create a new index and reimport your data. This may not be a problem for beta or staging applications; however, others may want to prepare their application with a suitable search maintenance page. Alternatively, see also our thoughts toward the end of this documentation on preserving your existing index.

Replace the addon

Removing and re-adding the addon is simple. Do note that these commands may restart your application without a search index.

heroku addons:remove bonsai
heroku addons:add bonsai

The full output of this process should look something like the following:

$ heroku addons:remove bonsai

 !    WARNING: Destructive Action
 !    This command will affect the app: sharp-mountain-4005
 !    To proceed, type "sharp-mountain-4005" or re-run this command with --confirm sharp-mountain-4005

> sharp-mountain-4005
Removing bonsai on sharp-mountain-4005... done, v160 (free)

$ heroku addons:add bonsai
Adding bonsai on sharp-mountain-4005... done, v161 (free)
Use `heroku addons:docs bonsai` to view documentation.

Create a new index

If you use Tire, you should refer to our updated Bonsai integration instructions for Tire. Other clients may also have their own scripts and conventions for creating an index, and you should review their documentation.

For other client integrations which expect to be configured with an index URL, you may create an index using your new BONSAI_URL config variable.

$ BONSAI_URL=`heroku config | grep BONSAI_URL | awk '{print $2}'`
$ curl -X POST $BONSAI_URL/acme-staging
{"ok":true,"acknowledged":true}

At this point you should end up with a single index, analogous to your previous pre-created index. You should reconfigure your application and reimport your data accordingly.

If you have any questions, just let us know via email to [email protected].

Comparing and contrasting the old and new architectures

How do you know if you're running on our old architecture, versus our new? How do the two line up conceptually?

Old URL

Our old architecture created a randomly-named index within our shared cluster.

http://index.bonsai.io/u04o0bx11pgjy0d3uuy2
http://us-east-1.bonsai.io/u04o0bx11pgjy0d3uuy2

In these examples, the cluster endpoint is at index.bonsai.io or us-east-1.bonsai.io. The index name is u04o0bx11pgjy0d3uuy2. Your hostname may be either index.bonsai.io or us-east-1.bonsai.io, both are synonymous for these indexes.

New URL

Our new architecture creates a unique cluster endpoint, where you may create your own index(es).

http://zk2t130t:[email protected]/

In this example, the API username is zk2t130t, the API password is xtfgj64zstexzocc. These credentials are required to create, modify or delete an index. In the future, we may enforce these, or another set of credentials, for document creation and modification as well. The hostname elm-6795212.us-east-1.bonsai.io identifies your unique shared cluster endpoint. There is no index name in this URL.

Preserving your existing index

We will not delete your index right away, however your Heroku app may have its BONSAI_INDEX_URL environment variable removed during this upgrade process. You may hard-code the URL of your existing index where needed to preserve your search functionality while creating a replacement index.

nz/Upgrading Bonsai beta to release architecture.md