Ansible + Docker 部署 Apache Kafka 3.9 集群
1. 准备工作
1.2 主机列表
| IP | 主机名 | 内存（GB） | CPU核数 | 磁盘 | 操作系统 | CPU 架构 |
|---|---|---|---|---|---|---|
| 10.0.0.13 | arc-pro-dc01 my.registry.com | 16 | 1 | 500GB | CentOS 7.9.2009 | x86_64 |
| 10.0.0.14 | arc-pro-dc02 | 16 | 1 | 500GB | CentOS 7.9.2009 | x86_64 |
| 10.0.0.15 | arc-pro-dc03 | 16 | 1 | 500GB | CentOS 7.9.2009 | x86_64 |
1.3 已安装服务
| 版本 | arc-pro-dc01 | arc-pro-dc02 | arc-pro-dc03 | |
|---|---|---|---|---|
| Ansible | 2.9.27 | ✅ | ||
| Harbor | v2.13.2 | ✅ | ||
| Docker | 28.1.1 | ✅ | ✅ | ✅ |
| Docker Compose | v2.39.2 | ✅ | ✅ | ✅ |
说明：
- 每个服务器的 IP 均是静态的
- 每个服务器的防火墙都已关闭
- 每个服务器的 SELINUX 已经禁用
- 每个服务器均存在一个管理员用户 admin，该用户可以免密码执行 sudo 命令；
- 在 arc-pro-dc01 机器上，可以使用 admin 用户免密码 ssh 到其他机器；
- 服务器之间的时间同步；
- 所有操作均使用 admin 用户完成，Kafka 集群的所属用户为 admin；
- 私有镜像仓库地址：https://my.registry.com:10443。
为使集群满足以上要求，参考下列文章进行配置：
- 使用 VMware Workstation 安装 CentOS-7 虚拟机
- 用 Ansible 批量完成 CentOS 7 操作系统基础配置
- 使用 Ansible 批量安装 Docker
- Docker 私有镜像仓库 Harbor 安装部署带签名认证
1.3 Kafka 集群规划
| 版本 | arc-pro-dc01 | arc-pro-dc02 | arc-pro-dc03 | |
|---|---|---|---|---|
| Kafka | 3.9.1 | Kafka Broker | Kafka Broker | Kafka Broker |
1.4 镜像准备
找一个可以连接互联网的、已经安装了 docker 的服务器，下载镜像
docker pull apache/kafka:3.9.1
docker image save apache/kafka:3.9.1 -o kafka.3.9.1.tar.gz
将 kafka.3.9.1.tar.gz 上传到本集群任意一台服务器，导入镜像：
docker load -i kafka.3.9.1.tar.gz
docker tag apache/kafka:3.9.1 my.registry.com:10443/library/apache/kafka:3.9.1
# 上传到私服
docker push my.registry.com:10443/library/apache/kafka:3.9.1
2. Ansible 文件
2.1 Ansible 目录结构
说明：在 arc-pro-dc01 机器上，执行 ansible 命令的基础目录为 /home/admin/ansible
$ tree /home/admin/ansible/
/home/admin/ansible/
├── ansible.cfg
├── hosts
└── kafka├── docker-compose.yml.j2└── start-kafka-container.yml
2.2 ansible.cfg
[defaults]
inventory=./hosts
host_key_checking=False
2.3 hosts
[kafka]
arc-pro-dc01 broker_id=1
arc-pro-dc02 broker_id=2
arc-pro-dc03 broker_id=3
2.4 docker-compose.yml.j2
services:kafka:image: my.registry.com:10443/library/apache/kafka:3.9.1restart: unless-stoppedcontainer_name: kafkahostname: {{ inventory_hostname }}network_mode: hostenvironment:KAFKA_NODE_ID: {{ broker_id }}KAFKA_PROCESS_ROLES: broker,controllerKAFKA_LISTENERS: PLAINTEXT://{{ inventory_hostname }}:9092,CONTROLLER://{{ inventory_hostname }}:9093KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://{{ inventory_hostname }}:9092,CONTROLLER://{{ inventory_hostname }}:9093KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXTKAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLERKAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXTKAFKA_CONTROLLER_QUORUM_VOTERS: >-{% for host in groups['kafka'] -%}{{ hostvars[host]['broker_id'] }}@{{ host }}:9093{% if not loop.last %},{% endif %}{%- endfor %}KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 3000KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2KAFKA_NUM_PARTITIONS: 3KAFKA_LOG_DIRS: /var/lib/kafka/datavolumes:- {{ kafka_data_dir }}:/var/lib/kafka/data- {{ kafka_log_dir }}:/opt/kafka/logs- {{ kafka_conf_dir }}:/opt/kafka/config
2.4 start-kafka-container.yml
---
- name: Start Kafka Containerhosts: kafkabecome: truegather_facts: falsevars:kafka_owner: adminkafka_group: admincompose_file_dir: /opt/app/kafkakafka_conf_dir: /etc/kafkakafka_log_dir: /data/kafka/logskafka_data_dir: /data/kafka/datatasks:- name: Remove old kafka container if existscommand: docker rm -f kafkaignore_errors: true- name: Remove kafka directories if they existfile:path: "{{ item }}"state: absentloop:- "{{ compose_file_dir }}"- "{{ kafka_conf_dir }}"- "{{ kafka_log_dir }}"- "{{ kafka_data_dir }}"- name: Create kafka directoriesfile:path: "{{ item }}"state: directoryowner: "{{ kafka_owner }}"group: "{{ kafka_group }}"mode: '0755'loop:- "{{ compose_file_dir }}"- "{{ kafka_conf_dir }}"- "{{ kafka_log_dir }}"- "{{ kafka_data_dir }}"- name: Deploy docker-compose.ymltemplate:src: docker-compose.yml.j2dest: "{{ compose_file_dir }}/docker-compose.yml"owner: "{{ kafka_owner }}"group: "{{ kafka_group }}"mode: '0644'- name: Start kafka containercommand: docker-compose -f {{ compose_file_dir }}/docker-compose.yml up -d
3. 部署
在 Ansible （arc-pro-dc01） 管理机执行
$ pwd
/home/admin/ansible$ ansible-playbook kafka/start-kafka-container.yml
4. 测试
[admin@arc-pro-dc01 ~]$ docker exec kafka \/opt/kafka/bin/kafka-topics.sh \--bootstrap-server arc-pro-dc01:9092,arc-pro-dc02:9092,arc-pro-dc03:9092 \--topic test-topic \--partitions 3 \--replication-factor 2 \--create
Created topic test-topic.# 查看每台服务器的数据目录
[admin@arc-pro-dc01 ~]$ ll /data/kafka/data | grep test-topic
drwxr-xr-x 2 admin admin 167 Sep 25 18:05 test-topic-0
drwxr-xr-x 2 admin admin 167 Sep 25 18:05 test-topic-2
[admin@arc-pro-dc02 ~]$ ll /data/kafka/data | grep test-topic
drwxr-xr-x 2 admin admin 167 Sep 25 18:05 test-topic-0
drwxr-xr-x 2 admin admin 167 Sep 25 18:05 test-topic-1
[admin@arc-pro-dc03 ~]$ ll /data/kafka/data | grep test-topic
drwxr-xr-x 2 admin admin 167 Sep 25 18:05 test-topic-1
drwxr-xr-x 2 admin admin 167 Sep 25 18:05 test-topic-2[admin@arc-pro-dc02 ~]$ docker exec kafka \/opt/kafka/bin/kafka-console-consumer.sh \--bootstrap-server arc-pro-dc01:9092,arc-pro-dc02:9092,arc-pro-dc03:9092 \--topic test-topic \--from-beginning[admin@arc-pro-dc03 ~]$ docker exec --workdir /opt/kafka/bin/ -it kafka sh
/opt/kafka/bin $ ./kafka-console-producer.sh \--bootstrap-server arc-pro-dc01:9092,arc-pro-dc02:9092,arc-pro-dc03:9092 \--topic test-topic>hello# arc-pro-dc02 的终端输出 hello