Python signed requests to AWS Elasticsearch

README.md

Python Signed Requests to AWS Elasticsearch

AWS provides a basic example for making signed requests to managed Elasticsearch clusters, but the example doesn't cover how to deal with EC2 instance profiles, which will require passing the session token.

Here's a simple example, which also handles authenticating when using an instance profile instead of a generated user level access key and secret, leverages the elastic bulk upload API to add multiple records at the same time, and threads the request so it can be made in a non-blocking fashion.

example.py

from elasticsearch import Elasticsearch, RequestsHttpConnection, helpers
from requests_aws4auth import AWS4Auth
import boto3
import json
import uuid
import threading
import datetime

def upload(records):

  credentials = boto3.Session().get_credentials()

  if credentials.token != None:
    awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, 'us-east-1', 'es', session_token=credentials.token)
  else:
    awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, 'us-east-1', 'es')

  es = Elasticsearch(
    hosts = [{'host': 'es-asdf1234.us-east-1.es.amazonaws.com', 'port': 443}],
    http_auth = awsauth,
    use_ssl = True,
    verify_certs = True,
    connection_class = RequestsHttpConnection
  )

  body = []

  for r in records:
    i = {}
    i["_id"] = str(uuid.uuid4())
    i["_index"] = 'Elastic-Index-' + str(datetime.datetime.now().date())
    i["_type"] = 'Elastic-Type'
    i["_source"] = json.dumps(r)
    body.append(i)

  t = threading.Thread(target=helpers.bulk, args=(es, body,))

  t.start()
  t.join()