How do I create a (dockerized) Elasticsearch index using a python script running in a docker container?

docker elasticsearch
elasticsearch dockerfile
elasticsearch python
elasticsearch docker volume
elasticsearch docker tutorial
kibana docker
elasticsearch docker-compose
build a search engine with python + elasticsearch

I'm trying to index a containerized Elasticsearch db using the Python client https://github.com/elastic/elasticsearch-py called from a script (running in a container too). By looking at existing pieces of code, it seems that docker-compose is a useful tool to use for my purpose. My dir structure is

  • docker-compose.yml
  • indexer
    • Dockerfile
    • indexer.py
    • requirements.txt
  • elasticsearch
    • Dockerfile

My docker-compose.yml reads

version: '3'

services:
  elasticsearch:
    build: elasticsearch/
    ports: 
      - 9200:9200
    networks:
      - deploy_network
    container_name: elasticsearch

  indexer:
    build: indexer/
    depends_on:
      - elasticsearch
    networks:
      - deploy_network
    container_name: indexer

networks:
  deploy_network:
    driver: bridge

indexer.py reads

from elasticsearch import Elasticsearch
from elasticsearch.helpers import bulk

es = Elasticsearch(hosts=[{"host":'elasticsearch'}]) # what should I put here?

actions = [
    {
    '_index' : 'test',
    '_type' : 'content',
    '_id' : str(item['id']),
    '_source' : item,
    }
for item in [{'id': 1, 'foo': 'bar'}, {'id': 2, 'foo': 'spam'}]
]

# create index
print("Indexing Elasticsearch db... (please hold on)")
bulk(es, actions)
print("...done indexing :-)")

The Dockerfile for the elasticsearch service is

FROM docker.elastic.co/elasticsearch/elasticsearch-oss:6.1.3
EXPOSE 9200
EXPOSE 9300

and that for the indexer is

FROM python:3.6-slim
WORKDIR /app
ADD . /app
RUN pip install -r requirements.txt
ENTRYPOINT [ "python" ]
CMD [ "indexer.py" ]

with requirements.txt containing only elasticsearch to be downloaded with pip.

Running with docker-compose run indexer gives me the error message at https://pastebin.com/6U8maxGX (ConnectionRefusedError: [Errno 111] Connection refused). elasticsearch is up as far as I can see with curl -XGET 'http://localhost:9200/'or by running docker ps -a.

How can I modify my docker-compose.yml or indexer.py to solve the problem?

The issue is a synchronisation bug: elasticsearch hasn't fully started when indexer tries to connect to it. You'll have to add some retry logic which makes sure that elasticsearch is up and running before you try to run queries against it. Something like running es.ping() in a loop until it succeeds with an exponential backoff should do the trick.

UPDATE: The Docker HEALTHCHECK instruction can be used to achieve a similar result (i.e. make sure that elasticsearch is up and running before trying to run queries against it).

Install Elasticsearch with Docker, To get a three-node Elasticsearch cluster up and running in Docker, you can use Docker Compose: Create a docker-compose.yml file: version: '2.2' services:� If this is the case then logically speaking a post build script looks good to me. As definitely docker container needs to be running before we do something on it. Regarding folder, you can use it in conjunction with startup shell script using bash profile. That said, post run scripts looks cleaner to me. – Anuj Yadav Feb 22 '16 at 3:40

Making more explicit @Mihai_Todor update, we could use HEALTHCHECK (docker 1.12+), for instance with a command like:

curl -fsSL "http://$(hostname --ip-address):9200/_cat/health?h=status" | grep -E '^green'

To answer this question using using HEALTHCHECK:

FROM python:3.6-slim

WORKDIR /app
ADD . /app
RUN pip install -r requirements.txt

HEALTHCHECK CMD curl -fsSL "http://$(hostname --ip-address):9200/_cat/health?h=status" | grep -E '^green'

ENTRYPOINT [ "python" ]
CMD [ "indexer.py" ]

Building a Full-Text Search App Using Docker and Elasticsearch, Wikipedia uses Elasticsearch to provide full-text search with An "index" is a data structure to allow for ultra-fast data query and Docker is a containerization engine that allows applications to be run in isolated Next, we'll create a docker- compose.yml file to define each container in our application stack. docker stop daemon docker rm <your first container name> docker rm daemon To remove all containers, we can use the following command: docker rm -f $(docker ps -aq) docker rm is the command to remove the container.-f flag (for rm) stops the container if it’s running (i.e., force deletion).-q flag (for ps) is to print only container IDs.

I use retry to make sure Elasticsearch is ready to accept connections:

from retrying import retry

client = Elasticsearch()


class IndexerService:

    @staticmethod
    @retry(wait_exponential_multiplier=500, wait_exponential_max=100000)
    def init():
        MyDocumentIndex.init()

# Here we will wait until ES is ready, or 100 sec passed.
IndexerService.init()

It tries in 500 ms, 1 sec, 2 sec, 4 sec until 100 sec.

Reference: https://github.com/rholder/retrying

Dockerizing your first web app with python and flask - DEV, First, we have to build a docker application with three containers: Ports: configures docker port with host machine port where docker runs. link with our python script that extract and store the data in an Elasticsearch cluster, Of course, we can see if our Elasticsearch index and Kibana dashboard have� When running an initialization script to create a new index on startup, authentication fails with default username and password (elastic:changeme). Waiting a few seconds then manually running the exact same command works perfectly. My current workaround is to call the health status endpoint in a loop until the request doesn't fail from an authentication error, then I run my index init script

What is Docker and How to Use it With Python (Tutorial), We decided to share their passion and knowledge with all of you and created this Along with Python, we are going to run Nginx and Redis containers. like PostgreSQL, Redis, Elasticsearch on the system – you can run it in containers. Place the file name Dockerfile in examples/curl directory with the following contents: Dismiss Join GitHub today. GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.

Using Docker with Elasticsearch, Logstash, and Kibana (ELK , Learn how to Dockerize an application built using Python/Django and the ELK stack. UPDATE: The docker-compose file has been updated to allow django "For searching and indexing data" image: elasticsearch:5.4 container_name: / app/ WORKDIR /app # create user and add to docker group RUN� This sample Docker Compose file brings up a three-node Elasticsearch cluster. Node es01 listens on localhost:9200 and es02 and es03 talk to es01 over a Docker network.. Please note that this configuration exposes port 9200 on all network interfaces, and given how Docker manipulates iptables on Linux, this means that your Elasticsearch cluster is publically accessible, potentially ignoring any

A Docker Tutorial for Beginners, The industry standard today is to use Virtual Machines (VMs) to run software Until a few releases ago, running Docker on OSX and Windows was quite a hassle. To start, create a new blank file in our favorite text-editor and save it in the The app's backend is written in Python (Flask) and for search it uses Elasticsearch� The Docker Quickstart training module teaches you how to: Set up your Docker environment (on this page) Build and run your image. Share images on Docker Hub. Docker concepts. Docker is a platform for developers and sysadmins to build, run, and share applications with containers.

Comments