一、broker的迁移
创新互联公司主打移动网站、
成都做网站、网站制作、网站改版、网络推广、网站维护、国际域名空间、等互联网信息服务，为各行业提供服务。在技术实力的保障下，我们为客户承诺稳定，放心的服务，根据网站的内容与功能再决定采用什么样的设计。最后，要实现符合网站需求的内容、功能与设计，我们还会规划稳定安全的技术方案做保障。
1.查看zookeeper和kafka启动情况
[root@slave1 ~]# pssh -h hostlist -i 'jps'
[1] 22:08:11 [SUCCESS] 20.0.5.12
3492 QuorumPeerMain
8140 Jps
6414 Kafka
[2] 22:08:11 [SUCCESS] 20.0.5.13
3490 QuorumPeerMain
4972 Kafka
22972 Jps
[3] 22:08:11 [SUCCESS] 20.0.5.11
7369 QuorumPeerMain
23754 Jps
11534 Kafka
[4] 22:08:11 [SUCCESS] 20.0.5.14
21263 Jps
[5] 22:08:11 [SUCCESS] 20.0.5.15
20818 Jps
2.创建一个topic
[root@slave1 ~]# /opt/kafka/bin/kafka-topics.sh --create --zookeeper 20.0.5.11:2181 --replication-factor 3 --partitions 3 --topic topic1
Created topic "topic1".
[root@slave1 ~]# /opt/kafka/bin/kafka-topics.sh --describe --zookeeper 20.0.5.11:2181 --topic topic1
Topic:topic1 PartitionCount:3 ReplicationFactor:3 Configs:
Topic: topic1 Partition: 0 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2
Topic: topic1 Partition: 1 Leader: 2 Replicas: 2,1,3 Isr: 2,1,3
Topic: topic1 Partition: 2 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1
3.在要迁移的节点上启动kafka进程
[root@slave1 ~]# pssh -h hostlist '/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties > /root/kafka.log 2>1&'
[1] 22:17:21 [SUCCESS] 20.0.5.12
[2] 22:17:21 [SUCCESS] 20.0.5.11
[3] 22:17:21 [SUCCESS] 20.0.5.13
[4] 22:17:21 [SUCCESS] 20.0.5.14
[5] 22:17:21 [SUCCESS] 20.0.5.15
[root@slave1 ~]# pssh -h hostlist -i 'jps'
[1] 22:17:29 [SUCCESS] 20.0.5.12
3492 QuorumPeerMain
8461 Jps
6414 Kafka
[2] 22:17:29 [SUCCESS] 20.0.5.14
21298 Kafka
21609 Jps
[3] 22:17:29 [SUCCESS] 20.0.5.11
7369 QuorumPeerMain
24844 Jps
11534 Kafka
[4] 22:17:29 [SUCCESS] 20.0.5.13
3490 QuorumPeerMain
4972 Kafka
23292 Jps
[5] 22:17:29 [SUCCESS] 20.0.5.15
20854 Kafka
21165 Jps
4.创建需要迁移的topic的json文件
[root@slave1 ~]# cat topic_move.json
{"topics": [{"topic": "topic1"}],
"version":1
}5.生成迁移规则
[root@slave1 ~]# /opt/kafka/bin/kafka-reassign-partitions.sh --zookeeper 20.0.5.11:2181 --topics-to-move-json-file topic_move.json --broker-list "3,4,5" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"topic1","partition":1,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"topic1","partition":0,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"topic1","partition":2,"replicas":[3,2,1],"log_dirs":["any","any","any"]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"topic1","partition":1,"replicas":[5,3,4],"log_dirs":["any","any","any"]},{"topic":"topic1","partition":0,"replicas":[4,5,3],"log_dirs":["any","any","any"]},{"topic":"topic1","partition":2,"replicas":[3,4,5],"log_dirs":["any","any","any"]}]}6.将生成的数据写入新的json文件
[root@slave1 ~]# cat re_node.json
{"version":1,"partitions":[
{"topic":"topic1","partition":1,"replicas":[5,3,4],"log_dirs":["any","any","any"]},
{"topic":"topic1","partition":0,"replicas":[4,5,3],"log_dirs":["any","any","any"]},
{"topic":"topic1","partition":2,"replicas":[3,4,5],"log_dirs":["any","any","any"]}]
}7.开始迁移
[root@slave1 ~]# /opt/kafka/bin/kafka-reassign-partitions.sh --zookeeper 20.0.5.11:2181 --reassignment-json-file re_node.json --execute
Current partition replica assignment
{"version":1,"partitions":[{"topic":"topic1","partition":1,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"topic1","partition":0,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"topic1","partition":2,"replicas":[3,2,1],"log_dirs":["any","any","any"]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.
[root@slave1 ~]# /opt/kafka/bin/kafka-reassign-partitions.sh --zookeeper 20.0.5.11:2181 --reassignment-json-file re_node.json --verify
Status of partition reassignment:
Reassignment of partition topic1-1 completed successfully
Reassignment of partition topic1-0 completed successfully
Reassignment of partition topic1-2 completed successfully8.查看新的分区
[root@slave1 ~]# /opt/kafka/bin/kafka-topics.sh --describe --zookeeper 20.0.5.11:2181 --topic topic1
Topic:topic1 PartitionCount:3 ReplicationFactor:3 Configs:
Topic: topic1 Partition: 0 Leader: 4 Replicas: 4,5,3 Isr: 5,3,4
Topic: topic1 Partition: 1 Leader: 5 Replicas: 5,3,4 Isr: 5,3,4
Topic: topic1 Partition: 2 Leader: 3 Replicas: 3,4,5 Isr: 5,3,4
#迁移完后修改好producer和consumer指向新的连接后再停止旧的服务
另外有需要云服务器可以了解下创新互联scvps.cn，海内外云服务器15元起步，三天无理由+7*72小时售后在线，公司持有idc许可证，提供“云服务器、裸金属服务器、高防服务器、香港服务器、美国服务器、虚拟主机、免备案服务器”等云主机租用服务以及企业上云的综合解决方案，具有“安全稳定、简单易用、服务可用性高、性价比高”等特点与优势，专为企业上云打造定制，能够满足用户丰富、多元化的应用场景需求。
本文题目：kafka集群安装及管理（二）-创新互联
网站路径：
http://cdkjz.cn/article/djehji.html
