Kafka Useful Commands

First time trying with Kafka

Install Kafka on Mac locally

brew install --cask java
brew install kafka

Run Zookeeper and Kafka

brew services start zookeeper
brew service start kafka

Create topic

kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test-sk

Run producer console

kafka-console-producer --broker-list localhost:9092 --topic test-sk

Write each message into the topic separate by pressing ENTER; then press CTR+C to exit the console.

Run consumer console

kafka-console-consumer --bootstrap-server localhost:9092 --topic test-sk --from-beginning

List Kafka topics

kafka-topics --list --zookeeper localhost:2181

Describe a topic

kafka-topics --describe --zookeeper localhost:2181 --topic test-sk

To update the configuration for a Kafka topic

For example, retention.ms

kafka-topics --alter --zookeeper localhost:2181 --topic test-sk --config retention.ms=1000

To delete configuration in a topic

For example, retention.ms

kafka-topics --alter --zookeeper localhost:2181 --topic test-sk --delete-config retention.ms

To check the beginning and ending offset

kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic test-sk --time -1

Kafka delete records in partitions from earliest offset to specific offset

kafka-delete-records --bootstrap-server localhost:9092 --offset-json-file offset-json.json

offset-json.json:

{
  "partitions": [
    {
      "topic": "test-sk",
      "partition": 0,
      "offset": -1
    }
  ],
  "version": 1
}

-1 is the last offset.