- Create A Topic
- Delete a Topic
- Add a Partition
- Describe a Topic
- Reassign partitions
- Checking Lag
- How to Set Up librdkafka with Pykafka
- Kafkacat
Create A Topic
1 2 |
./kafka-topics.sh --create --zookeeper myzookeeper:2181 --replication-factor 3 --partitions 3 --topic mytopic |
Delete a Topic
1 2 |
./kafka-topics.sh --zookeeper myzookeeper:2181 --delete --topic mytopic |
Add a Partition
1 2 |
./kafka-topics.sh --zookeeper myzookeeper:2181 --alter --topic mytopic --partitions 2 |
Describe a Topic
Note this describes all the topics, regardless of what topic is supplied.
1 2 |
./kafka-topics.sh --describe mytopic |
Reassign Partitions
Create a file reassign.json
:
1 2 3 |
{ "version":1, "partitions": [{"topic":"my-topic", "partition":0, "replicas":[0,1,2] }] } |
Then execute:
1 2 |
./kafka-reassign-partitions.sh --zookeeper myzookeeper:2181 --reassignment-json-file reassign.json --execute |
Checking Lag
A simple solution for quickly checking lag is as follows. However, a longer term solution is to set up a Grafana instance and report lag to this using statsd.
1 2 |
./kafka-consumer-offset-checker.sh --zookeeper myzookeeper:2181 --group my_group_id --topic mytopic |
How to Set Up librdkafka with Pykafka
Follow this reference.
In summary:
1 2 3 4 5 6 |
git clone https://github.com/edenhill/librdkafka.git cd librdkafka/ ./configure make sudo make install |
Then:
1 2 3 4 |
git clone https://github.com/Parsely/pykafka.git cd pykafka python setup.py develop |
The last part, calling setup.py develop
to put pykafka in develop mode, is necessary for creating the link between the two.
Kafkacat
kafkacat is a useful tool, here is a kafkacat quick reference.