Streaming Data Pipeline – Kafka + ELK Stack

Streaming weather data using Apache Kafka and Elastic Stack.

Data source: https://openweathermap.org/api

Objectives: Develop a streaming data pipeline to extract weather data from OpenWeather API using Apache Kafka, Logstash, Elasticserach and Kibana (Kafka + ELK Stack).

Screen Shot 2021-11-11 at 17 46 57

To summarize, Python was used to develop a Kakfa producer that requests weather data from OpenWeather API every minute and sends it as a message to Apache Kafka. Logstash, as a Kafka consumer, consumes the data and stores it into Elasticsearch. Kibana uses the data from Elasticsearch to display the dashboard.

Kibana Weather Dashboard

Screen Shot 2021-11-11 at 17 46 57

Steps:

  • bash elk/start_elastic_docker.sh
  • bash kafka/start_kafka_docker.sh
  • Create a topic using kafka manager: localhost:9000

Logstash installed locally*

  • $LOGSTASH_HOME/bin/logstash -f $LOGSTASH_HOME/config/pipeline.conf

Before running Kafka Producer, is needed to set the API key inside the weather_api_key.ini file*

  • python3 weather_kfk_producer.py
  • Access Kibana: localhost:5601
  • Create an index pattern: must match with your index name inside pipeline.conf
  • Develop your dashboard.

GitHub

View Github