Accessing kafka python client

Clients Non-Java Clients
1

i’m trying to access Kafka on GCP, and that is working fine with the console producer

Command used -
$CONFLUENT_HOME/bin/kafka-console-producer --broker-list 146.148.110.164:9094 --topic my-topic1 --producer.config /Users/karanalang/Documents/Technology/gcp_certs_nov28/strimzi-client-ssl-user2.properties

Contents of the file → strimzi-client-ssl-user2.properties

bootstrap.servers=my-cluster-lb-ssl-cert-kafka-bootstrap:9093
security.protocol=SSL
ssl.truststore.location=/Users/karanalang/Documents/Technology/gcp_certs_nov28/ca.p12
ssl.truststore.password=1234567
ssl.keystore.location=/Users/karanalang/Documents/Technology/gcp_certs_nov28/user2.p12
ssl.keystore.password=1234567

However, when i try to access using python Confluent Kafka client, it is giving me error -

Traceback (most recent call last):
  File "/Users/karanalang/PycharmProjects/Kafka/KafkaPython_SSL.py", line 43, in <module>
    producer = Producer(conf)
cimpl.KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Failed to create producer: ssl.key.location failed: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch"}

here is the code for the Kafka producer → 

from confluent_kafka import Producer

kafkaBrokers='146.148.110.164:9094'
caRootLocation='/Users/karanalang/Documents/Technology/gcp-certs-dec3/caroot.pem'
certLocation='/Users/karanalang/Documents/Technology/gcp-certs-dec3/ca1.crt.pem'
keyLocation='/Users/karanalang/Documents/Technology/gcp-certs-dec3/user1.pem'
password='1234567'

conf = {'bootstrap.servers': kafkaBrokers,
        'security.protocol': 'SSL',
        'ssl.ca.location':certLocation,
        'ssl.certificate.location': certLocation,
        'ssl.key.location':keyLocation,
        'ssl.key.password' : password
}
topic = 'my-topic'

producer = Producer(conf)

for n in range(100):
        producer.produce(topic, key=str(n), value="val -> "+str(n))

producer.flush()

Any ideas on whet needs to be done to fix this issue ?

I’ve tried couple of ways to get the user private key from GCP

  1. openssl pkcs12 -in user2.p12 -nodes -nocerts -out onlykey.pem -passin pass:NjFYcmHYNEcf

  2. directly export the user.key from the GCO secret.
    kc get secret my-bridge -n kafka -o jsonpath='{.data.user\.key}' | base64 -d > user.key
    (& then saving it as user.pem)

but that is not fixing the issue.
Any inputs on this is really appreciated !

Also, wrt the CARoot.pem and certificate.pem,
i’m using the following command.

keytool -exportcert \
 -rfc \
 -alias ca.crt \
 -file CARoot.pem \
 -keystore ca.p12 \
 -storepass qwV1avSoUJw8 \
 -storetype PKCS12 \
 -v

keytool -exportcert \
 -rfc \
 -alias ca.crt \
 -file certificate.pem \
 -keystore ca.p12 \
 -storepass qwV1avSoUJw8 \
 -storetype PKCS12 \
 -v

both the files → certificate.pem & CARoot.pem are essentially the same
and assigned to ssl.ca.location, ssl.certificate.location in the Producer conf.
Is that correct ?