I needed data from a company for a private project and a kafka topic was created where the data is sent. After watching countless videos in the meantime, I thought I could do the integration. My way to do this is as follows:
Kafka topic (ext. company) > Kafka connect (standalone on my side) > mysql sink
Now I was told to connect to the topic:
topicname
servicename.pem (incl. pkey)
servicename.jks (i think this is keystore)
server.cer.pem (i generated truststore.jks from this one)
bootstrap.server urls
username and password
First problem, if I have a keystore the authentication is actually pure SSL and I don’t need username and password and if it is SASL_SSL I don’t need a keystore. I am a bit confused and have tried all possible combinations. Of course I have also tried purely as a consumer because I think I’m still missing the schema-registry url. Or, before I ask the company and uncovered myself as a complete newbie, could someone here make the connection with the information given to me? Would this be possible at all? Thanks for the help.
if the Authentication method is SASL_SSL you still might have a key and truststore. (In the scenarios where I saw SASL_PLAIN in the past it was common that you still have keystore/truststore but not sure if it is mandatory - I guess not if your os provides proper default cas)
The difference to real TLS Authentication is that the authentication itself will be done with SASL but still there is TLS for encryption of the data on transport - just no two-way TLS authentication is done. And that TLS Connection needs to be trustful.
SSL can be configured for encryption or authentication. You may configure just SSL encryption (by default SSL encryption includes certificate authentication of the server) and independently choose a separate mechanism for client authentication, e.g. SSL, SASL, etc. Note that SSL encryption, technically speaking, already enables 1-way authentication in which the client authenticates the server certificate. So when referring to SSL authentication, it is really referring to 2-way authentication in which the broker also authenticates the client certificate.
Also according to that page you only should need a private-key/cert pair on the client for 2-way authentication. BUT you need a proper CA on the client also without 2-way authentication.
=> So I assume that the keystore might not be mandatory in your case
SSL (two-way TLS) authentication is only given if the SECURITY_PROTOCOL_CONFIG is “SSL” and in that case → yes there is no need for user and password.
If it the listener is SASL_PLAIN on the broker side it should work with:
Well, really apreciate your support. And yeah i think it was a part of the key.
[2021-12-17 23:43:09,448] INFO Successfully logged in. (org.apache.kafka.common.security.authenticator.AbstractLogin:61)
I think thats good, or
Unfortunately i got a lot of warnings now in a loop. And no datastream. But i will try to find this out. Maybe you or someone else have an idea during i will search for it:
[AdminClient clientId=adminclient-1] Failed to create channel due to (org.apache.kafka.common.network.SaslChannelBuilder:239)
org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
Caused by: org.apache.kafka.common.KafkaException: Principal could not be determined from Subject, this may be a transient failure due to Kerberos re-login
and
WARN [AdminClient clientId=adminclient-1] Error connecting to node kafka1.URL:9093 (id: -1 rack: null) (org.apache.kafka.clients.NetworkClient:982)
java.io.IOException: Channel could not be created for socket java.nio.channels.SocketChannel[closed]
Connection is now working. It was pure SSL and i had to download public certificate from bootstrap server to create a new truststore file. The given caroot cert was the issue.
