Kafka SASL SSL authentication like in Confluent

Confluent changed pricing policy which forced us to move all dev environments down to the ground

But configuring Kafka is not so easy as it seems, especially when we are talking about authentication

In this note I'm going to recreate step by step actions required to get SASL_SSL authentication working same way as in Concluent cloud

TLDR: Jump to "Kafka SASL_SSL + LetsEncrypt" for final configuration

Confluent Kafka

To simplify things suppose we have dedicated server with a public ip address

We wont do any fancy containers/kubernetes for simplicity

At the very end we want our client to connect to Kafka with config like:

bootstrap.servers=xxx-yyyyy.europe-west3.gcp.confluent.cloud:9092
ssl.endpoint.identification.algorithm=https
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="mac" password="123";

which was taken from confluent cloud

Prerequisites

Before anything else we need to perform some house keeping

echo $USER ALL=NOPASSWD: ALL | sudo tee /etc/sudoers.d/$USER
sudo apt-get update && sudo apt-get upgrade -y && sudo apt-get dist-upgrade -y && sudo reboot
sudo apt autoremove -y && sudo apt autoclean -y
sudo timedatectl set-timezone Europe/Kiev

Plain - aka anonymous http

Before doing any authorization lets get up and running as is

# java is required
sudo apt install -y default-jre

# download kafka
wget https://archive.apache.org/dist/kafka/2.6.1/kafka_2.12-2.6.1.tgz
tar -xzf kafka_2.12-2.6.1.tgz
cd kafka_2.12-2.6.1

# start kafka and zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties

# demo
bin/kafka-topics.sh --create --topic demo --bootstrap-server localhost:9092
bin/kafka-topics.sh --describe --topic demo --bootstrap-server localhost:9092
bin/kafka-console-producer.sh --topic demo --bootstrap-server localhost:9092
bin/kafka-console-consumer.sh --topic demo --from-beginning --bootstrap-server localhost:9092
bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

# cleanup
rm -rf /tmp/kafka-logs /tmp/zookeeper

Because of how Kafka configured by default you wont be able to connect to it from outside

Try this on a client machine

wget https://archive.apache.org/dist/kafka/2.6.1/kafka_2.12-2.6.1.tgz
tar -xzf kafka_2.12-2.6.1.tgz
cd kafka_2.12-2.6.1
bin/kafka-topics.sh --bootstrap-server 178.20.154.77:9092 --list

You will receive an error complaining:

[2021-11-14 22:41:35,906] WARN [AdminClient clientId=adminclient-1] Error connecting to node kafka:9092 (id: 0 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException: kafka

By default Kafka listens on all interfaces port 9092, but whenever client is connected Kafka asks him to send requests to its hostname, in my case it is kafka and it is not resolvable from outside and as a result not reachable (think of it like if you was trying to ping kafka, what is kafka where to find its ip address)

Easy fix is to add kafka public ip address to your hosts like this:

echo '178.20.154.77 kafka' | sudo tee -a /etc/hosts

After this everything will work as expected

Notes:

  • do not forget to remove this hosts record we wont need it
  • if your virtual machine has public ip address never ever leave kafka as is - technically it is publicly accessible and has no authentication at all

Proper, but still insecure way will be to set hostname to something we can resolve:

sudo hostnamectl set-hostname kafka.marchenko.net.ua

For this to work - you need to restart kafka, it reads hostname on start only, also be sure to cleanup everything, kafka and zookeeper will rememeber previous setup, actually cleanup on each and every step

So now we can try:

bin/kafka-topics.sh --bootstrap-server kafka.marchenko.net.ua:9092 --list
bin/kafka-topics.sh --bootstrap-server 178.20.154.77:9092 --list

Both should work

Links:

we will need stop and cleanup kafka often so here is little stop.sh script

#!/usr/bin/env bash
for pid in $(ps a | grep java | grep -v grep | awk '{print $1}')
do
	kill -9 $pid
	echo $pid
done
rm -rf /tmp/kafka-logs /tmp/zookeeper

Listeners & Advertised Listeners

Before moving forward we need to figure out what exactly is listeners and advertised.listeners, why the heek we need second if we are not building cluster and just trying to build single node Kafka

Both listeners and advertised.listeners is a comma separated lists of PROTOCOL:IP:PORT

listeners are describing interfeces to which Kafka will bind on start (e.g. like server.listen('0.0.0.0:9092'))

advertised.listeners are instructions to clients to where send requests

Even if you are running single node Kafka cluster it still speaks to itself via advertised.listeners rather than listeners, also it always prevers PLAINTEXT

Deep inside Kafka consists of series components, most important for us are server and controller

Can viaualize it like

G cluster_0 Broker Server Server Controller Controller Server->Controller Zookeeper Zookeeper Controller->Zookeeper

If we have multiple nodes

G cluster_0 Broker 1 cluster_1 Broker 2 Server1 Server1 Controller1 Controller1 Server1->Controller1 Controller1->Server1 Server2 Server2 Controller1->Server2 Zookeeper Zookeeper Controller1->Zookeeper Server2->Controller1 Controller2 Controller2

In cluster only one controller is active master and others are chilling and is here only for failover

Thats the reason why even in single node Kafka still goes via advertised listeners

So in default config we have:

# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092

# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092

Note that both listeners and advertised.listeners are configured with PLAINTEXT procol which actually means allow anonymous access

Also listeners does not specify ip address which means that we gonna listen all interfaces (aka 0.0.0.0) and our kafka will be open to the whole world IF advertised.listeners is not set and hostname can be resolved to reachable ip address

So for example if we have public ip address 178.20.154.77 and a domain name pointing to it we can configure kafka like

listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://178.20.154.77:9092

Will allow anonymous connections from inside and outside

And following config

listeners=PLAINTEXT://127.0.0.1:9092
advertised.listeners=PLAINTEXT://127.0.0.1:9092

Will allow only local connection

The reason for all this madness will be solved later with authentication, because each option can have comma separated list and brokers tent to use plaintext communications idea here is to have plaintext over private local network for inter broker communications and sasl ssl over public one for clients

Kafka Authentication and Encryption

Before jumping into both authentication and encryption we need to define some kind of terms

Encryption

Whenever we are talking aboun encryption in Kafka it is literally the same as talking about HTTP (PLAINTEXT) and HTTPS (SSL)

Authentication

Under the hood Kafka uses JAAS (Java Authentication and Authorization Service)

JAAS has set of so called mechanisms - ways to verify credentials and prpotocols - thats story about http vs https

There are numerous of available mechnisms like kerberos and whatsoever, but we are looking for a plain one because it is used by confluent cloud and we wish to have exactly the same

Kafka Security HTTP Analogy

In simple words imagine you have web site

http, anonymous

listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://kafka.marchenko.net.ua:9092

https, anonymous

listeners=SSL://0.0.0.0:9092
advertised.listeners=SSL://kafka.marchenko.net.ua:9092

http, basic auth

listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://kafka.marchenko.net.ua:9092

security.protocol=SASL_PLAINTEXT
sasl.enabled.mechanisms=PLAIN

https, basic auth

listeners=SSL://0.0.0.0:9092
advertised.listeners=SSL://kafka.marchenko.net.ua:9092

security.protocol=SASL_SSL
sasl.enabled.mechanisms=PLAIN

Plus specific options, step by step we are going to configure this options till we get desired SASL_SSL

SASL_PLAINTEXT - aka HTTP Basic Auth

The easiest possible way to start with authentication is SASL_PLAINTEXT for it to work we are using following config on a server:

listeners=PLAINTEXT://0.0.0.0:9093,SASL_PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://127.0.0.1:9093,SASL_PLAINTEXT://kafka.marchenko.net.ua:9092

security.protocol=SASL_PLAINTEXT
sasl.enabled.mechanisms=PLAIN

listener.name.sasl_plaintext.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="admin" \
  password="hello" \
  user_admin="hello" \
  user_mac="123";

Notes:

  • take a look how we are preventing PLAINTEXT access from outside (it will be used by kafka itself)
  • take attention to this huge JAAS config, its key has sasl_plaintext inside, it is important and should match chosen security.protocol (e.g. when you will try SASL_SSL do not forget to change it also)
  • value of JAAS config has username and password which are kind of root user and will be used by kafka itself if there is no PLAINTEXT available, all other stings are describing available users in form user_[username]="[password]"
  • be sure to not have any spaces after \
  • if you put everything into single line be sure to escape equal signs
  • take a note on ports, because we can not bind multiple times we are using non default 9093 port for plain text

And now on a client side we need config file:

client.properties

sasl.mechanism=PLAIN
security.protocol=SASL_PLAINTEXT
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username\="mac" password\="123";

And try to connect:

bin/kafka-topics.sh --bootstrap-server kafka.marchenko.net.ua:9092 --command-config client.properties --list

Everything should work fine, but if you will try to connect without providing password it should fail with timeout - here you have your at least very very basic auth

bin/kafka-topics.sh --bootstrap-server kafka.marchenko.net.ua:9093 --list

Note that still from a server we can connect without any passwords because of plaintext, we can remove it, so even local connections will go with authentication

SASL_PLAINTEXT interbroker

To require authentication even between brokers we need following config:

listeners=SASL_PLAINTEXT://0.0.0.0:9092
advertised.listeners=SASL_PLAINTEXT://kafka.marchenko.net.ua:9092

security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN

listener.name.sasl_plaintext.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="admin" \
  password="hello" \
  user_admin="hello" \
  user_mac="123";

Same way as for client we are defining mechanism and protocol to use

IMPORTANT: make sure that your username ini jaas config appears two times, like admin in our example, otherwise Kafka wont start complaining that it can not connect to itsefl because of invalid username or password

From now on you gonna need client.properties on both server and client

It is the same as configuring basic auth for a http server (still without https)

SSL

In this part we are going to leave authentication for a moment and configure SSL (HTTPS)

Self Signed Certificates

For our "https" to work we gonna need certificates, you can create self signed certificates like so:

openssl req -subj "/CN=kafka.marchenko.net.ua/" -newkey rsa:2048 -nodes -keyout kafka.marchenko.net.ua.key -x509 -days 365 -out kafka.marchenko.net.ua.pem

This command will create kafka.marchenko.net.ua.key which is our private key and should be kept in secret and kafka.marchenko.net.ua.pem which is our certificate and can be sent to anyone

For Kafka to work with this certificates we need to pack them into so called pk12 keychain

openssl pkcs12 -export -out server.keystore.p12 -in kafka.marchenko.net.ua.pem -inkey kafka.marchenko.net.ua.key

This one will generate server.keystore.p12 which will be used by kafka server and inside contains both private key and certificate created before

But also we need one more so called truststore jks for clients (think of it as adding self signed certificates to trusted root)

keytool -keystore client.truststore.jks -import -file kafka.marchenko.net.ua.pem

Kafka SSL

As with basic auth lets start with both plaintext and ssl in our server properties

listeners=PLAINTEXT://localhost:9093,SSL://:9092
advertised.listeners=PLAINTEXT://localhost:9093,SSL://kafka.marchenko.net.ua:9092

ssl.keystore.location=/home/mac/kafka_2.12-2.6.1/server.keystore.p12
ssl.keystore.password=123456

Now for the client we must grab trust store

scp kafka:kafka_2.12-2.6.1/client.truststore.jks .

And following ssl.properties

security.protocol=SSL

ssl.truststore.location=/Users/mac/Downloads/kafka_2.12-2.6.1/client.truststore.jks
ssl.truststore.password=123456

Now if we will try to connect from client

bin/kafka-topics.sh --bootstrap-server kafka.marchenko.net.ua:9092 --command-config ssl.properties --list

Letsencrypt

As you can guess it is not actually the same as in confluent cloud, for us to be the same we gonna need valid certificate so it will work without trustore

First of all install certbot

sudo apt install certbot

And start certificate creation wizerd

sudo certbot certonly --standalone

Note that with method expects that port 80 is free and will be used for a challanged, certbot will bring its own web service for this

After certificates are created you should see congratulations message with following info:

  • certificate: /etc/letsencrypt/live/kafka.marchenko.net.ua/fullchain.pem
  • private key: /etc/letsencrypt/live/kafka.marchenko.net.ua/privkey.pem
  • secrets for upgrade: /etc/letsencrypt
  • recommendation to backup this folder
  • command to renew certificate: certbot renew

Kafka wants p12 keystore so once again lets create it:

# just cleaning up everything
rm kafka.marchenko.net.ua.* client.truststore.jks server.keystore.p12
sudo cp /etc/letsencrypt/live/kafka.marchenko.net.ua/fullchain.pem .
sudo cp /etc/letsencrypt/live/kafka.marchenko.net.ua/privkey.pem .
sudo chown mac:mac fullchain.pem
sudo chown mac:mac privkey.pem

# create jks keychain for java/kafka, password 123456
openssl pkcs12 -export -out server.keystore.p12 -in fullchain.pem -inkey privkey.pem

No need to change anything in server properties, file path and password left the same, but now our client properties might be as simple as le.properties

security.protocol=SSL

And everything should work

bin/kafka-topics.sh --bootstrap-server kafka.marchenko.net.ua:9092 --command-config le.properties --list

One final touch, lets configure our server to work only via ssl

listeners=SSL://:9092
advertised.listeners=SSL://kafka.marchenko.net.ua:9092
security.inter.broker.protocol=SSL

ssl.keystore.location=/home/mac/kafka_2.12-2.6.1/server.keystore.p12
ssl.keystore.password=123456

Note that from now on even local connections shold be made to fqdn, localhost does not work anymore because certificate was signed for our domain

Kafka SASL_SSL + LetsEncrypt

Now it is time to combine pieces together - our SSL (HTTPS) and SASL_PLAINTEXT (Basic Auth)

Once again starting with a PLAINTEXT for simplicity

Our server properties now will look like

listeners=PLAINTEXT://:9093,SASL_SSL://:9092
advertised.listeners=PLAINTEXT://127.0.0.1:9093,SASL_SSL://kafka.marchenko.net.ua:9092

security.protocol=SASL_SSL
sasl.enabled.mechanisms=PLAIN

ssl.keystore.location=/home/mac/kafka_2.12-2.6.1/server.keystore.p12
ssl.keystore.password=123456

# do not forget to replace sasl_plaintext to sasl_ssl
listener.name.sasl_ssl.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="admin" \
  password="hello" \
  user_admin="hello" \
  user_mac="123";

And once again our final touch to allow only secure and basic auth so need to configure server

So here is our final setup

server.properties

listeners=SASL_SSL://:9092
advertised.listeners=SASL_SSL://kafka.marchenko.net.ua:9092

security.inter.broker.protocol=SASL_SSL
sasl.mechanism.inter.broker.protocol=PLAIN

security.protocol=SASL_SSL
sasl.enabled.mechanisms=PLAIN

ssl.keystore.location=/home/mac/kafka_2.12-2.6.1/server.keystore.p12
ssl.keystore.password=123456

listener.name.sasl_ssl.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="admin" \
  password="hello" \
  user_admin="hello" \
  user_mac="123";

client.properties

ssl.endpoint.identification.algorithm=https
sasl.mechanism=PLAIN
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required serviceName\="Kafka" username\="mac" password\="123";

TODO

  • Schema Registry
  • Rest Proxy
  • Connect