Elasticsearch migration guide

Contents:

Introduction
Differences
API Mapping
Migration script
Front-end components
Conclusion

If you are currently using Elasticsearch and plan to migrate to Meilisearch for your application, follow this guide to help you with the transition.

Differences

Before migrating, you may want to understand better what differentiates Elasticsearch from Meilisearch.

Elasticsearch is a distributed search engine that uses clusters and nodes to optimize performance, whereas Meilisearch is a single instance/node with customizable startup options. Meilisearch is still evolving, and many functionalities are not yet supported. You can think of Meilsearch as similar to the Elasticsearch component in the ELK Stack, but without the rest of the components like Kibana, Logstash and Beats.

You may see what new upgrades our developers are working on by visiting the public roadmap.

APIs Mapping

This section lists the commonly used Elasticsearch REST APIs and their Meilisearch API mappings.

Topic	Elasticsearch	Meilisearch
Create an Index	curl -X PUT "localhost:9200/index-name”	curl \
-X POST 'localhost:7700/indexes' \
-H 'Content-Type: application/json' \
--data-binary '{
"uid": "index-name"
}’
Get an index	curl -X GET "localhost:9200/index-name”	curl \
-X GET 'localhost:7700/indexes/index-name’
Delete an index	curl -X DELETE "localhost:9200/index-name”	curl \
-X DELETE 'localhost:7700/indexes/index-name’
Update an index
List all indices	curl -X GET "localhost:9200/*”	curl \
-X GET 'localhost:7700/indexes’
Index stats	curl -X GET "localhost:9200/index-name/_stats”	curl \
-X GET 'localhost:7700/indexes/index-name/stats’
Add a document	curl \
-X POST 'localhost:9200/index-name/_create/_id' \
-H 'Content-Type: application/json' \
-d'{"key" : "value"}'	curl \
-X POST 'localhost:7700/indexes/index-name/documents' \
-H 'Content-Type: application/json' \
--data-binary '[{
"id":_id,
"key":"value"
}]’
Get a document by id	curl -X GET 'localhost:9200/index-name/_doc/_id'	curl \
-X GET 'localhost:7700/indexes/index-name/documents/_id'
Update a document	curl \
-X POST 'localhost:9200/index-name/_update/_id' \
-H 'Content-Type: application/json' \
-d'{
"doc":{
"key" : "new-value"
}
}'	curl \
-X PUT 'localhost:7700/indexes/index-name/documents' \
-H 'Content-Type: application/json' \
--data-binary '[{
"id": _id,
"key": "new-value"
}]’
Delete a document	curl -X DELETE 'localhost:9200/index-name/_doc/_id'	curl \
-X DELETE 'localhost:7700/indexes/index-name/documents/_id'
search	curl -X GET 'localhost:9200/index-name/_search/?q=field:value'	curl \
-X POST 'localhost:7700/indexes/index-name/search' \
-H 'Content-Type: application/json' \
--data-binary '{ "q": "" }'
Cluster/Instance Health	curl -X GET 'localhost:9200/_cluster/health'	curl \
-X GET 'localhost:7700/health'
more APIs here

You can read more about the Elasticsearch APIs here and Meilisearch APIs here.

Migration Script

This guide will show you how to migrate data from Elasticsearch to Meilisearch using NodeJs and it has no impact on the programming language that will be used with Meilisearch in the future.

Initialize project

Start by creating a directory elastic-meilisearch-migration and initialize it as an npm project:

mkdir elastic-meilisearch-migration
cd elastic-meilisearch-migration
npm init -y

Next, create a script.js file:

touch script.js

This file will contain our migration code.

Install required packages

To get started, you'll need two npm packages. The first is @elastic/elasticsearch, the JavaScript client for the ElasticSearch API, and the second is meilisearch, the JavaScript client for the Meilisearch API. To install them, run the below command:

npm install --save @elastic/elasticsearch meilisearch

Create client objects

We need client instances of both Elasticsearch and Meilisearch to access their respective API. Paste the below code in script.js:

const { Client: ElasticSearch } = require('@elastic/elasticsearch')
const { MeiliSearch } = require('meilisearch')

const esClient = new ElasticSearch({
  cloud: {
    id: 'ES_CLOUD_ID',
  },
  auth: {
    username: 'ES_USER_NAME',
    password: 'ES_PASSWORD',
  },
})

const meiliClient = new MeiliSearch({
  host: 'MEILI_HOST',
  apiKey: 'MEILI_API_KEY',
})

First, we create an Elasticsearch client esClient using elastic cloud credentials. You can also try other authentication methods. Replace the original values for ES_CLOUD_ID,ES_USER_NAME and ES_PASSWORD.

Then, we create a Meilisearch client meiliClient by providing the host url and API key. Also, replace MEILI_HOST and MEILI_API_KEY with their respective values.

Fetch data from Elasticsearch

This could be a challenging step because we will iterate the index until the entire index is not searched. In each iteration, we search for the subsequent 10,000 documents, format them, and store them. The iteration will end when we get an empty result. To achieve such a paginated search result, we will use the Search After method.

This logic has been reduced to the below code.

async function getElasticSearchDocuments() {
  let isFetchingComplete = false
  let search_after = null
  let documents = []

  const searchObject = {
    index: 'ES_INDEX',
    size: 10000,
    body: {
      query: {
        match_all: {},
      },
    },
    sort: {
      'id.keyword': 'asc',
    },
  }

  do {
    if (search_after) searchObject.search_after = search_after

    const result = await esClient.search(searchObject)
    const hits = result.hits.hits

    isFetchingComplete = hits.length == 0
    search_after = hits[hits.length - 1]?.sort || null
    documents = documents.concat(hits.map(hit => hit._source))
  } while (isFetchingComplete == false)

  return documents
}

Replace ES_INDEX with the index name you want to export.

Upload data to Meilisearch

Add the following code:

  const documents = await getElasticSearchDocuments()
  const meiliIndex = meiliClient.index('MEILI_INDEX')
  await meiliIndex.addDocuments(documents)

The variable documents contain an array of documents ready to be uploaded to Meilisearch.

After that, we get the index instance and upload all the documents with the addDocuments method. Meilisearch will create the index if it doesn't already exist.

Note: The addDocuments method is asynchronous, which means the request is not handled as soon as it is received, but it is put into a queue and processed.

Replace MEILI_INDEX with the index name where you would like documents to be added.

That's all! When you're ready to run the script, enter the below command:

node script.js

Finished Script

Here's the complete script code.

const { Client: ElasticSearch } = require('@elastic/elasticsearch')
const { MeiliSearch } = require('meilisearch')

const esClient = new ElasticSearch({
  cloud: {
    id: 'ES_CLOUD_ID',
  },
  auth: {
    username: 'ES_USER_NAME',
    password: 'ES_PASSWORD',
  },
})

const meiliClient = new MeiliSearch({
  host: 'MEILI_HOST',
  apiKey: 'MEILI_API_KEY',
})

async function getElasticSearchDocuments() {
  let isFetchingComplete = false
  let search_after = null
  let documents = []

  const searchObject = {
    index: 'ES_INDEX',
    size: 10000,
    body: {
      query: {
        match_all: {},
      },
    },
    sort: {
      'id.keyword': 'asc',
    },
  }

  do {
    if (search_after) searchObject.search_after = search_after

    const result = await esClient.search(searchObject)
    const hits = result.hits.hits

    isFetchingComplete = hits.length == 0
    search_after = hits[hits.length - 1]?.sort || null
    documents = documents.concat(hits.map(hit => hit._source))
  } while (isFetchingComplete == false)

  return documents
}

;(async () => {
  const documents = await getElasticSearchDocuments()
  const meiliIndex = meiliClient.index('MEILI_INDEX')
  await meiliIndex.addDocuments(documents)
})()

Front-end components

Instant Meilisearch is a plugin connecting your Meilisearch instance with InstantSearch, giving you access to many (but not all) of the same front-end components as Algolia users. Here is an up-to-date list of components compatible with Instant Meilisearch.

shivaylamba/elasticsearchmigration.md