Streaming Data Pipeline – Kafka + ELK Stack
Streaming weather data using Apache Kafka and Elastic Stack.
Data source: https://openweathermap.org/api
Objectives: Develop a streaming data pipeline to extract weather data from OpenWeather API using Apache Kafka, Logstash, Elasticserach and Kibana (Kafka + ELK Stack).
To summarize, Python was used to develop a Kakfa producer that requests weather data from OpenWeather API every minute and sends it as a message to Apache Kafka. Logstash, as a Kafka consumer, consumes the data and stores it into Elasticsearch. Kibana uses the data from Elasticsearch to display the dashboard.
Kibana Weather Dashboard
- Create a topic using kafka manager:
Logstash installed locally*
$LOGSTASH_HOME/bin/logstash -f $LOGSTASH_HOME/config/pipeline.conf
Before running Kafka Producer, is needed to set the API key inside the
- Access Kibana:
- Create an index pattern: must match with your index name inside
- Develop your dashboard.