服务端有几个脚本是挺有用的,后面我将挑几个脚本来学习一下
脚本名称 | 脚本用途 |
---|---|
kafka-topics.sh | topic管理脚本 |
connect-distributed.sh | 连接分布式模式脚本 |
connect-standalone.sh | 连接单机模式脚本 |
kafka-topics.sh
--partitions
创建或修改主题的分区数
--replication-factor
副本因子,副本数量
--replica-assignment
手动指定分区副本分配方案,使用该参数,不用指定--partitions 和 --replication-factor
--topic
主题名称
--zookeeper
连接kafka zk地址
--alter
修改分区,副本,配置
--bootstrap-server
kafka服务器地址
--create
创建主题
--delete
删除主题
--list
列出所有的可用主题
1[root@10 kafka_2]# bin/kafka-topics.sh --zookeeper 10.211.55.3:2181 --list 2__consumer_offsets 3first 4test 5topic-3 6topic-4 7topic-5 8topic-6 9topic-admin10topic-create-diff11topic-two
--describe
列出主题的详细信息
--exclude-internal
使用--list --describe 命令时是否列出内部主题,默认列出内部主题
--command-config
以配置文件的形式修改Admin Client的配置,支持的配置见org.apache.kafka.clients.admin.AdminClientConfig
1//me.properties2request.timeout.ms=20000034//5bin/kafka-topics.sh --bootstrap-server 10.211.55.3:9092 --topic topic-two --list --command-config config/me.properties
--config
在创建/修改主题的时候可以对主题默认参数进行覆盖,具体支持的参数见http://kafka.apachecn.org/documentation.html#topicconfigs
该参数将在以后废弃,请使用kafka-configs.sh1[root@10 kafka_2.11-2.2.0]# bin/kafka-topics.sh --bootstrap-server 10.211.55.3:9092 --topic topic-two --describe 2Topic:topic-two PartitionCount:1 ReplicationFactor:1 Configs:segment.bytes=1073741824,retention.bytes=1073741824 3Topic: topic-two Partition: 0 Leader: 0 Replicas: 0 Isr: 0 4 5[root@10 kafka_2.11-2.2.0]# bin/kafka-topics.sh --zookeeper 10.211.55.3:2181 --alter --topic topic-two --config segment.bytes=1048577 6WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases. 7 Going forward, please use kafka-configs.sh for this functionality 8Updated config for topic topic-two. 910[root@10 kafka_2.11-2.2.0]# bin/kafka-topics.sh --zookeeper 10.211.55.3:2181 --describe --topic topic-two11Topic:topic-two PartitionCount:1 ReplicationFactor:1 Configs:segment.bytes=104857712Topic: topic-two Partition: 0 Leader: 0 Replicas: 0 Isr: 0
----delete-config
删除一个配置项
1[root@10 kafka_2.11-2.2.0]# bin/kafka-topics.sh --zookeeper 10.211.55.3:2181 --topic topic-two --alter --delete-config segment.bytes 2WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.3 Going forward, please use kafka-configs.sh for this functionality4Updated config for topic topic-two.56[root@10 kafka_2.11-2.2.0]# bin/kafka-topics.sh --zookeeper 10.211.55.3:2181 --topic topic-two --describe7Topic:topic-two PartitionCount:1 ReplicationFactor:1 Configs:8 Topic: topic-two Partition: 0 Leader: 0 Replicas: 0 Isr: 0
--disable-rack-aware
忽略机架信息
有两个broker,一个配了机架信息,另一个没配,在创建topic的时候就会报错1[root@10 kafka_2.11-2.2.0]# bin/kafka-topics.sh --zookeeper 10.211.55.3:2181 --create --topic topic-6 --replication-factor 1 --partitions 2 2Error while executing topic command : Not all brokers have rack information. Add --disable-rack-aware in command line to make replica assignment without rack information. 3[2018-12-27 05:22:40,834] ERROR kafka.admin.AdminOperationException: Not all brokers have rack information. Add --disable-rack-aware in command line to make replica assignment without rack information. 4 at kafka.zk.AdminZkClient.getBrokerMetadatas(AdminZkClient.scala:71) 5 at kafka.zk.AdminZkClient.createTopic(AdminZkClient.scala:54) 6 at kafka.admin.TopicCommand$ZookeeperTopicService.createTopic(TopicCommand.scala:274) 7 at kafka.admin.TopicCommand$TopicService$class.createTopic(TopicCommand.scala:134) 8 at kafka.admin.TopicCommand$ZookeeperTopicService.createTopic(TopicCommand.scala:266) 9 at kafka.admin.TopicCommand$.main(TopicCommand.scala:60)10 at kafka.admin.TopicCommand.main(TopicCommand.scala)11 (kafka.admin.TopicCommand$)1213[root@10 kafka_2.11-2.2.0]# bin/kafka-topics.sh --zookeeper 10.211.55.3:2181 --create --topic topic-6 --replication-factor 1 --partitions 2 --disable-rack-aware14Created topic topic-6.
--if-exists
只有当主题存在时,相关命令才会执行,不会显示错误
1[root@10 kafka_2]# bin/kafka-topics.sh --zookeeper 10.211.55.3:2181 --topic topic-7 --alter --config segment.bytes=104857 --if-exists 2 3[root@10 kafka_2]# bin/kafka-topics.sh --zookeeper 10.211.55.3:2181 --topic topic-7 --alter --config segment.bytes=104857 4Error while executing topic command : Topics in [] does not exist 5[2018-12-27 06:01:25,638] ERROR java.lang.IllegalArgumentException: Topics in [] does not exist 6 at kafka.admin.TopicCommand$.kafka$admin$TopicCommand$$ensureTopicExists(TopicCommand.scala:416) 7 at kafka.admin.TopicCommand$ZookeeperTopicService.alterTopic(TopicCommand.scala:294) 8 at kafka.admin.TopicCommand$.main(TopicCommand.scala:62) 9 at kafka.admin.TopicCommand.main(TopicCommand.scala)10 (kafka.admin.TopicCommand$)
--if-not-exists
创建主题的时候,只有当主题不存在时,命令才执行,存在时不会报错
1[root@10 kafka_2]# bin/kafka-topics.sh --zookeeper 10.211.55.3:2181 --topic topic-6 --create --partitions 1 --replication-factor 1 --if-not-exists23[root@10 kafka_2]# bin/kafka-topics.sh --zookeeper 10.211.55.3:2181 --topic topic-6 --create --partitions 1 --replication-factor 1 4Error while executing topic command : Topic 'topic-6' already exists.5[2018-12-27 06:07:54,185] ERROR org.apache.kafka.common.errors.TopicExistsException: Topic 'topic-6' already exists.6 (kafka.admin.TopicCommand$)
--topics-with-overrides
显示覆盖过配置的主题
--unavailable-partitions
查看没有leader副本的分区
1[root@10 kafka_2]# bin/kafka-topics.sh --zookeeper 10.211.55.3:2181 --topic topic-6 --describe --unavailable-partitions2 Topic: topic-6 Partition: 0 Leader: -1 Replicas: 1 Isr: 1
--under-replicated-partitions
查看所有包含失效副本的分区
connect-distributed.sh & connect-standalone.sh
Kafka Connect 是一款可扩展并且可靠的在 Apache Kafka 和其他系统之间进行数据传输的工具。
1bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties23bin/connect-distributed.sh config/connect-distributed.properties
参考:
[1] Kafka之实战指南-朱小厮 [2] 阿飞的博客 [3] Apache Kafka