December 14, 2020

[Note] Kafka in Docker

Needs

Simple Kafka in Docker set up for development.  Single broker in Kafka, and Zookeeper should be usable for other than Kafka.

Tried various methods following several blogs/tutorials but none of them worked or suitable for my needs.

 

Prerequisite

Docker is installed & running, docker-compose is installed.

My Environment

Ubuntu 20.04


Steps



  1. Save above as docker-compose.yml
  2. $ sudo docker-compose up --build -d

Test

  1. Download Kafka to use Kafka commands
    https://kafka.apache.org/downloads 
  2. Create topic
    $ ./kafka-topics.sh -zookeeper localhost:2181 --create --topic test --partitions 3 --replication-factor 1
  3. List topics
    $ ./kafka-topics.sh --zookeeper localhost:2181 --list 
  4. Describe topic
    $ ./kafka-topics.sh --zookeeper localhost:2181 --describe --topic test

Test Python code

$ pip install kafa-python


producer.py

from json import dumps
from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers=['localhost:9092'],
value_serializer=lambda x:
dumps(x).encode('utf-8'))

for e in range(10):
data = {'number' : e}
producer.send('test', value=data)

 

consumer.py

from kafka import KafkaConsumer
from json import loads

consumer = KafkaConsumer(
'test',
bootstrap_servers=['localhost:9092'],
auto_offset_reset='earliest',
enable_auto_commit=True,
group_id='my-group',
value_deserializer=lambda x: loads(x.decode('utf-8')))

for message in consumer:
message = message.value
print(message)


Run publisher first, and then consumer.



REFERENCE


READINGS

 

Labels: , , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)