Needs
Simple Kafka in Docker set up for development. Single broker in Kafka, and Zookeeper should be usable for other than Kafka.
Tried various methods following several blogs/tutorials but none of them worked or suitable for my needs.
Prerequisite
Docker is installed & running, docker-compose is installed.
My Environment
Ubuntu 20.04
Steps
- Save above as docker-compose.yml
- $ sudo docker-compose up --build -d
Test
- Download Kafka to use Kafka commands
https://kafka.apache.org/downloads - Create topic
$ ./kafka-topics.sh -zookeeper localhost:2181 --create --topic test --partitions 3 --replication-factor 1 - List topics
$ ./kafka-topics.sh --zookeeper localhost:2181 --list - Describe topic
$ ./kafka-topics.sh --zookeeper localhost:2181 --describe --topic test
Test Python code
$ pip install kafa-python
producer.py
from json import dumps
from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers=['localhost:9092'],
value_serializer=lambda x:
dumps(x).encode('utf-8'))
for e in range(10):
data = {'number' : e}
producer.send('test', value=data)
consumer.py
from kafka import KafkaConsumer
from json import loads
consumer = KafkaConsumer(
'test',
bootstrap_servers=['localhost:9092'],
auto_offset_reset='earliest',
enable_auto_commit=True,
group_id='my-group',
value_deserializer=lambda x: loads(x.decode('utf-8')))
for message in consumer:
message = message.value
print(message)
Run publisher first, and then consumer.
REFERENCE
- Kafka/Zookeeper Docker Images: https://hub.docker.com/r/wurstmeister/kafka/
- Docker kafrka, code, https://github.com/wurstmeister/kafka-docker
- https://www.bennettnotes.com/post/setup-kafka-zookeeper-in-docker/
- https://analyticshut.com/manage-apache-kafka-topics/
READINGS
- Kafka, https://aws.amazon.com/msk/what-is-kafka/
- Zookepper, https://data-flair.training/blogs/zookeeper-tutorial/
- Docker-app, https://towardsdatascience.com/an-end-to-end-machine-learning-project-with-python-pandas-keras-flask-docker-and-heroku-c987018c42c7
- python-kafka
No comments:
Post a Comment