22. Kafka. Установка и настройка

В этой части описывается настройка Kafka.
Tags:

2021-07-23

Добавление сервиса Kafka

  1. В консоли Cloudera Manager в меню выбираем ‘Add Service’:
  2. Выбираем Kafka.
  3. Выбираем зависимости:
  4. Распределяем роли с некоторыми условиями:
    • Из-за особенностей работы, которая приводит к высокой утилизации ресурсов системы, под Kafka Brokers лучше выделить отдельные хосты. Чрезвычайно не рекомендуется размещать Kafka на ZooKeeper-узлах.
  5. Изменяем настройки:
#PropertyValueDescription
1. ZooKeeper Root
zookeeper.chroot 		/kafka The znode in ZooKeeper used as a root for this Kafka cluster.
2. Enable Kerberos Authentication
kerberos.auth.enable 		☑ Enables Kerberos authentication for this Kafka service.
3. Data Directories
log.dirs 		/data/kafka/data A list of one or more directories in which Kafka data is stored. Each new partition created is placed in the directory that currently has the least amount of partitions. Each directory should be on its own separate drive.
4. Enable TLS/SSL for Kafka Broker
ssl_enabled 		☑ Encrypt communication between clients and Kafka Broker using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).
5. Kafka Broker TLS/SSL Server JKS Keystore File Location
ssl.keystore.location 		/opt/cloudera/security/pki/server.jks The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Kafka Broker is acting as a TLS/SSL server. The keystore must be in JKS format.
6. Kafka Broker TLS/SSL Server JKS Keystore File Password
ssl.keystore.password.generator 		По умолчанию: changeit. The password for the Kafka Broker JKS keystore file.
7. Kafka Broker TLS/SSL Server JKS Keystore Key Password
ssl.key.password.generator 		По умолчанию: changeit. The password that protects the private key contained in the JKS keystore used when Kafka Broker is acting as a TLS/SSL server.
8. Kafka Broker TLS/SSL Client Trust Store File
ssl.truststore.location 		/usr/java/jdk1.8.0_181-cloudera/jre/lib/security/jssecacerts The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Kafka Broker might connect to. This is used when Kafka Broker is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.
9. Kafka Broker TLS/SSL Client Trust Store Password
ssl.truststore.password.generator 		По умолчанию: changeit. The password for the Kafka Broker TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.
10. Enable TLS/SSL for Kafka MirrorMaker
ssl_enabled 		☑ Encrypt communication between clients and Kafka MirrorMaker using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).
11. Kafka MirrorMaker TLS/SSL Server JKS Keystore File Location
ssl.keystore.location 		/opt/cloudera/security/pki/server.jks The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Kafka MirrorMaker is acting as a TLS/SSL server. The keystore must be in JKS format.
12. Kafka MirrorMaker TLS/SSL Server JKS Keystore File Password
ssl.keystore.password.generator 		По умолчанию: changeit. The password for the Kafka MirrorMaker JKS keystore file.
13. Kafka MirrorMaker TLS/SSL Server JKS Keystore Key Password
ssl.key.password.generator 		По умолчанию: changeit. The password that protects the private key contained in the JKS keystore used when Kafka MirrorMaker is acting as a TLS/SSL server.
14. Kafka MirrorMaker TLS/SSL Client Trust Store File
ssl.truststore.location 		/usr/java/jdk1.8.0_181-cloudera/jre/lib/security/jssecacerts The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Kafka MirrorMaker might connect to. This is used when Kafka MirrorMaker is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.
15. Kafka MirrorMaker TLS/SSL Client Trust Store Password
ssl.truststore.password.generator 		По умолчанию: changeit. The password for the Kafka MirrorMaker TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.
  1. Наблюдаем запуск ролей.
  2. Визард успешно закончен.

3. Перенастройка размещения log’ов

  1. В настройках Kafka, используя категорию ‘Logs’, изменяем следующие параметры, добавляя ‘/data’ вместо ‘/var’:
#PropertyValueDescription
1. Kafka Broker Log Directory
kafka.log4j.dir 		/data/log/kafka The log directory for log files of the role Kafka Broker.
2. Kafka MirrorMaker Log Directory
kafka_mirrormaker.log4j.dir 		/data/log/kafka The log directory for log files of the role Kafka MirrorMaker.
  1. Нажимаем Save Changes.

4. Дополнительные настройки

Так как сервера стоят в отдельном VLAN и перехват трафика маловероятен, то выбираем протокол безопасности без шифрования трафика SASL_PLAINTEXT, так как TLS-шифрование является ресурсоёмкой операцией. При существования риска перехвата трафика необходимо выбрать протокол SASL_SSL.

  1. В настройках Kafka следующие параметры приводим к указанному виду:
#PropertyValueDescription
1. Sentry Service ◉ Sentry
○ none		 Name of the Sentry service that this Kafka service instance depends on
2. Enable Kerberos Authentication
kerberos.auth.enable 		☑ Enables Kerberos authentication for this Kafka service.
3. Enable Sentry Privileges Caching
sentry.kafka.caching.enable 		☑ Enables Sentry privilege caching. Only applicable if authorization via Sentry is enabled.
4. Enable Authenticated Communication with the JMX Agent
jmx.auth.enabled 		☑ Enables Authenticated Communication with the JMX Agent.
5. Name of User with read-only access to the JMX Agent
jmx.monitor.user 		Default value: monitorRole Specifies the name of the user that has read-only privileges when using password file-based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.
6. Password of User with read-only Access to the JMX agent
jmx.monitor.user.passwd 		************ Specifies the password of the user that has read-only privileges when using password file-based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.
7. Name of User with Read-Write Access to the JMX Agent
jmx.control.user 		Default value: controlRole Specifies the name of the user that has read-write privileges when using password file-based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.
8. Password of user with read-write access to the JMX agent
jmx.control.user.passwd 		************ Specifies the password of the user that has read-write privileges when using password file-based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.
9. Enable TLS/SSL for Kafka JMX
jmx.ssl.enabled 		☑ Encrypt communication between clients and Kafka JMX using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).
10. Enable TLS client authentication for JMX port
jmx.ssl.client.auth.enabled 		☑ If enabled, a valid client certificate must be presented by the JMX client in order to connect to the JMX port. Ensure that the trusted CA certificates are present in either the Kafka JMX TLS/SSL Server Trust Store file or the global trust store.
11. SSL Client Authentication
ssl.client.auth

Kafka Broker Default Group

◉ none
○ required
○ requested 		Client authentication mode for SSL connections. This configuration has three valid values, "required", "requested" and "none". If set to "required", client authentication is required. If set to "requested", client authentication is requested and clients without certificates can still connect. If set to "none", which is the default value, no client authentication is required.
12. Inter Broker Protocol
security.inter.broker.protocol

Kafka Broker Default Group

○ INFERRED
○ PLAINTEXT
○ SSL
◉ SASL_PLAINTEXT
○ SASL_SSL 		Protocol to be used for inter-broker communication. INFERRED uses the same protocol that is configured for external clients.
13. Authenticate Zookeeper Connection
authenticate.zookeeper.connection

Kafka Broker Default Group

☑		 Enables authentication of SASL connections with zookeeper, if Kerberos authentication is enabled. It also allows a broker to set SASL ACL on zookeeper nodes, which locks these nodes down so that only a Kafka broker can modify it.
14. Super users
super.users

Kafka Broker Default Group

kafka
ken
atlas (на время попытки внедрения Apache Atlas в кластер)		 Users who are allowed to perform any action on the Kafka cluster.
15. Source Kafka Cluster's Security Protocol
source.security.protocol

Kafka Broker Default Group

○ INFERRED
○ PLAINTEXT
○ SSL
◉ SASL_PLAINTEXT
○ SASL_SSL 		Protocol used for communication with source Kafka cluster.
16. Destination Kafka Cluster's Security Protocol
destination.security.protocol

Kafka Broker Default Group

○ INFERRED
○ PLAINTEXT
○ SSL
◉ SASL_PLAINTEXT
○ SASL_SSL 		Protocol used for communication with destination Kafka cluster.
17. Source Kafka Cluster's Client Auth
source.ssl.client.auth 		☑ Only required if the source Kafka cluster requires client authentication.
18. Destination Kafka Cluster's Client Auth
destination.ssl.client.auth 		☑ Only required if destination Kafka cluster requires client authentication.
  1. Нажимаем Save Changes.
  2. Перезапускаем все зависимые сервисы по приглашению Cloudera Manager Console.