Setup ELK Stack & Apache Kafka from Docker Compose YAML File

This tutorial is to setup ELK stack and Apache Kafka environment using one docker-compose YAML file. Please note that for following tutorial you should have a docker agent and docker compose installed on your host machine. All the images are present on docker hub and only two commands are used to get the environment up and running. Following are the images deployed by this script :

1) Default official ElasticSearch Image from docker hub at port 9200 and the line 9200:9200 binds the port of docker image of ElasticSearch to the machine IP at 9200 port.

2)Default official Kibana Image from docker hub at port 5601 and the line 5601:5601 binds the port of docker image of Kibana to the machine IP at 5601 port. and links it to elasticsearch by using the line elasticsearch:elasticsearch.

3) Default official Zookeeper Image from docker hub at port 2181 and the line 2181:2181 binds the port of docker image of Zookeeper to the machine IP at 2181 port.

4)Kafka Image is ches/kafka image from docker hub. The docker image deployed at port 9092 and the line 9092:9092 binds the port of docker image of ches/kafka to the machine IP at 9092 port. Creates a default topic topictest and connects to zookeeper on 2181 port.

5) Default official Logstash Image from docker hub at port 12201 and the line 12201:12201 binds the port of docker image of ElasticSearch to the machine IP at 12201 port. It takes 2 inputs one from kafka topic (topictest)  and one from tcp on port. And output of logstash is sent to elasticsearch on port 9200.

Below is the code for docker-compose.yml file. In the file replace this text : ||Enter Your IP for machine here e.g 12.12.12.12|| with the IP for your machine or localhost if you want to run it on your local machine.

 elasticsearch:

  image: elasticsearch

  ports:

  - 9200:9200

kibana:

  image: kibana

  links:

  - elasticsearch:elasticsearch

  ports:

  - 5601:5601

  environment:

  - ELASTICSEARCH_URL=http://elasticsearch:9200

zookeeper:

    image: zookeeper

    ports:

      - "2181:2181"

kafka:

    image: ches/kafka

    links:

      - zookeeper:zookeeper

    ports:

      - "9092:9092"

    environment:

      KAFKA_ADVERTISED_HOST_NAME: "||Enter Your IP for machine here e.g 12.12.12.12||"

      KAFKA_ADVERTISED_PORT: "9092"

      KAFKA_CREATE_TOPICS: "topictest:1:1"

      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

logstash:

  image: logstash:latest

  links:

  - elasticsearch:elasticsearch

  ports:

  - 12201:12201

  command: logstash -e "input { kafka { bootstrap_servers => '||Enter Your IP for machine here e.g 12.12.12.12||:9092' client_id => 'logstash' topics => 'topictest' } tcp { port => 12602 codec => json } }  output { elasticsearch { hosts => '||Enter Your IP for machine here e.g 12.12.12.12||:9200' } }"

Then go to terminal for the root and run the following two commands :

sudo sysctl -w vm.max_map_count=262144

sudo docker-compose up

When the terminal shoes the following line logstash is up  :

Now open a new terminal and write sudo docker ps and following images should be visible :

Now in browser write ur hostname:9200 following page should show up. This means elasticsearch is up and running

Now in browser write ur hostname:5601 Kibana homepage will show up :

Please ask me any questions in the comments section below.

p.s Sometimes the kafka server doesnot start up on first time so run docker compose command again and you are good to go.