Kafka mTLS using PEM certs. Loading pvt key from file

Ops
1

Erik Godding Boye
Hi! I am trying to set up Kafka mTLS using PEM certificates. Is it really true that there is no way to load the private key from file? I can only make it work if the key is specified as a literal (property or env) to my app. :thinking_face:

Erik Godding Boye
Is seems like some clients (at least librdkafka) supports some natural properties like ssl.key.location , ssl.certificate.location and ssl.ca.location , but the Java client support for PEM certificates seems “bolted” on… :unamused:

Neil Buesing @nbuesing
I haven’t used PEM yet myself, but I thought it was just another type supported and you would use the same property that would be a JKS file and it would not be the .pem file instead.

security.protocol=SSL
ssl.keystore.type=PEM
ssl.keystore.location=.../chain.pem (would need intermediate if one was used, I believe)
ssl.key.password=
ssl.truststore.type=PEM
ssl.truststore.location=.../cert.pem

Erik Godding Boye
Well, it is supposed to work that way. Documentation is not very good in this area, but I found this blog post. There seems to be two related issues that I dont seem to get around:

  1. I have to deliver the private key in the same file as the signed certificates (and any intermediates), and this is a very uncommon format (I have never ever seen it before). The signed certificate and private key should be kept far apart IMO. We currently use cert-manager to deliver the certificates, and are trying to get away from the pesky keystores/truststores, and cert-manager delivers the certificate and private key in separate locations. I tried to cat the files, but then I get into the other issue…

  2. It seems like the private key must be “protected” by a passphrase when delivered with the signed certificate. And the private key delivered to me is not protected by any passphrase. Which is very common in cloud applications… And AFAIK the security provided by keystores and/or private key password is not sufficient nowadays anyway. The crypto is just not good enough, and you need to protect your private key by other means anyway.

How did Kafka/Java end up in this “dead end”? Any chance to get proper support for PEM into Kafka/Java, or has that train already left the station? :thinking_face:

Jordan Moore @OneCricketeer
The “solution” that my previous team came up with was to wrap the Kafka client creation with an intermediate service registry that returns the keys from a KMS application

Related - Kafka: Schema Registry PEM authentication | by Elliot West | Expedia Group Technology | Medium

Jordan Moore
e.g. STREAMP-5897 by andykruth-ii · Pull Request #295 · ExpediaGroup/stream-registry · GitHub

Erik Godding Boye

Thanks @OneCricketeer , interesting!