Making a signed HTTP request to AWS Elasticsearch in Python

aws elasticsearch python
python elasticsearch
aws elasticsearch authentication
aws4auth
aws elasticsearch c# example
aws elasticsearch endpoint
aws elasticsearch query examples
http-aws-es

I'm trying to make a simple Python Lambda that makes snapshots of our Elasticsearch database. This is done through Elasticsearch's REST API using simple HTTP requests.

However, for AWS, I have to sign these requests. I have a feeling it can be achieved through boto3's low-level clients probably with generate_presigned_url, but I cannot for the life of me figure out how to invoke this function correctly. For example, what are the valid ClientMethods? I've tried ESHttpGet but to no avail.

Can anyone point me in the right direction?

There are several Python extensions to the requests library that will perform the SigV4 signing for you. I have used this one and it works well.

Get started with Amazon Elasticsearch Service: an easy way to send , There are several Python extensions to the requests library that will perform the SigV4 signing for you. I have used this one and it works well. From the terminal, run the following commands: pip install boto3 pip install elasticsearch pip install requests pip install requests-aws4auth. The following sample code establishes a secure connection to the specified Amazon ES domain and indexes a single document. You must provide values for region and host .

I struggled for a while to do a similar thing. Currently the boto3 library doesn't support making signed es requests, though since I raised an issue with them it's become a feature request.

Here's what I've done in the meantime using DavidMuller's library mentioned above and boto3 to get my STS session credentials:

import boto3
from aws_requests_auth.aws_auth import AWSRequestsAuth
from elasticsearch import Elasticsearch, RequestsHttpConnection

session = boto3.session.Session()
credentials = session.get_credentials().get_frozen_credentials()

es_host = 'search-my-es-domain.eu-west-1.es.amazonaws.com'
awsauth = AWSRequestsAuth(
    aws_access_key=credentials.access_key,
    aws_secret_access_key=credentials.secret_key,
    aws_token=credentials.token,
    aws_host=es_host,
    aws_region=session.region_name,
    aws_service='es'
)

# use the requests connection_class and pass in our custom auth class
es = Elasticsearch(
    hosts=[{'host': es_host, 'port': 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)

print(es.info())

Hope this saves somebody some time.

Making a signed HTTP request to AWS Elasticsearch in Python , AWS signature version 4 signing process for the python requests module working together to host and review code, manage projects, and build software together. import AWSRequestsAuth # let's talk to our AWS Elasticsearch cluster auth requests.get('http://search-service-foobar.us-east-1.es.amazonaws.com',  The following are code examples for showing how to use elasticsearch.RequestsHttpConnection().They are from open source Python projects. You can vote up the examples you like or vote down the ones you don't like.

While other answers are perfectly fine, I wanted to eliminate the use of external packages. Obviously, botocore itself has all the required functionality to sign requests it was just a matter of looking at the source code. This is what I ended up with for sending AWS API requests directly (things are hardcoded for the demonstration purposes):

      import boto3
      import botocore.credentials
      from botocore.awsrequest import AWSRequest
      from botocore.endpoint import URLLib3Session
      from botocore.auth import SigV4Auth

      params = '{"name": "hello"}'
      headers = {
        'Host': 'ram.ap-southeast-2.amazonaws.com',
      }
      request = AWSRequest(method="POST", url="https://ram.ap-southeast-2.amazonaws.com/createresourceshare", data=params, headers=headers)
      SigV4Auth(boto3.Session().get_credentials(), "ram", "ap-southeast-2").add_auth(request)    


      session = URLLib3Session()
      r = session.send(request.prepare())

DavidMuller/aws-requests-auth: AWS signature version 4 , Hi, I'm writing a python lambda function to process incoming ELB logs but I'm struggling to find out how to either make my bulk index post through the cl. aws_service='es' ) # use the requests connection_class and pass in  (Conceivably, the authentication class is flexible enough to be used with any AWS service, but it was initially created to interface with AWS Elasticsearch instances.) Installation pip install aws-requests-auth Usage

I recently published requests-aws-sign, which provides AWS V4 request signing for the Python requests library.

If you look at this code you will see how you can use Botocore to generate the V4 request signing.

Making signed requests to Elasticsearch · Issue #853 · boto/boto3 , It will be placed on hold for dead_timeout seconds and the request will be retried on another node. If a connection fails multiple times in a row the timeout will get  This chapter presents a solution: use Amazon API Gateway to restrict users to a subset of the Elasticsearch APIs and AWS Lambda to sign requests from API Gateway to Amazon ES. Note Standard API Gateway and Lambda pricing applies, but within the limited usage of this tutorial, costs should be negligible.

why not just use requests?

import requests
headers = {'Content-Type': 'application/json',}
data = '{"director": "Burton, Tim", "genre": ["Comedy","Sci-Fi","R-rated"],"profit" : 98 , "year": 1996, "actor": ["Jack Nicholson","PierceBrosnan","Sarah Jessica Parker"], "title": "Mars Attacks!"}'
response = requests.post('https://search-your-awsendpoint.us-west-2.es.amazonaws.com/yourindex/_yourdoc/', headers=headers, data=data)

this worked for me

Python Elasticsearch Client, This API Gateway documentation might help. https://docs.aws.amazon.com/​apigateway/api-reference/signing-requests/. This script provided  Browse other questions tagged python amazon-web-services elasticsearch elasticsearch-dsl-py or ask your own question. The Overflow Blog Podcast 235: An emotional week, and the way forward

Connect to elasticsearch in AWS using key credentials, One of my obstacles was to get an AWS Elasticsearch JavaScript client was to use a HTTP client to make requests to the Elasticsearch HTTP API of my domain. It failed misearably, AWS requires that HTTP requests are signed with JavaScript · Kotlin · Linux · Machine Learning · Other · python · React  Elasticsearch:-Elasticsearch is a real-time distributed search and analytics engine. It allows you to explore your data at a speed and at a scale never before possible.

AWS Elasticsearch JavaScript Client, The AWS public facing documentation provides some python examples Query AWS ES cluster by signing http requests with AWS IAM roles (python) Posted in Elasticsearch So make sure you are using unexpired token)  Sample Python Client Save the following sample Python code as a Python file, such as register-repo.py. The client requires the AWS SDK for Python (Boto 3), requests and requests-aws4auth packages. The client contains commented-out examples for other snapshot operations.

AWS Blog » Query AWS ES cluster by signing http requests with , While making an http call to elasticsearch service from AWS Lambda function, we need to sign the the http request with Signature Version 4. I'm writing a python lambda function to process incoming ELB logs and put them into an Elasticsearch domain. So far the boto3 library has been great, but I'm struggling to find out how to either make my bulk index post through the client or how to sign a direct rest api call.

Comments
  • Ok did some digging. Don't think generate_presigned_url is the way to go. I think the only operations available to me are high-level ES operations listed in the docs: boto3.readthedocs.io/en/latest/reference/services/…
  • Yeah, I'd been hoping to keep it to just boto3 and botocore, but I've wound up using the AWSAuthConnection in boto - seems to do the trick.
  • BotocoreHTTPSession causes an ImportError on recent versions of boto3 (1.9.42 at time of writing). Changing that line to the following resolves the error for me: from botocore.httpsession import URLLib3Session
  • This implies your endpoint allows public access or (hopefully) is hidden behind a VPC. The docs say "If your domain access policy includes IAM users or roles, you must sign requests to the Elasticsearch APIs."
  • i am calling this using lambda and yes its a public facing. if you want it inside a vpc ,you can create lambda inside vpc and attach iam role to the lambda. that was my use case.
  • so it's public-facing and you allow anyone to put data in it?
  • docs.aws.amazon.com/elasticsearch-service/latest/developerguide/… please follow that document. eventhough it is public facing i can restrict the requests based on either iam permissions or ip address . thanks
  • There are still cases where you will need signed requests regardless of resource policies, vpc security groups or iam permissions, such as manual snapshotting.