I want to configure MongoSinkConnector to read the data from Kafka topics and load the same to MongoDB Collection. I have TLS enabled on my Mongo DB (AWS Document DB) cluster. I have ca-bundle.pem file for accessing the Mongo Cluster. I have created a new keystore on my local machine, added the ca-bundle.pem file to the newly created keystore. Mounted the keystore to the kafka-connect docker container by specifying the keystore binding in kafka-connect volumes in docker-compose file.
volumes:
- ./plugins:/tmp/connect-plugins:Z
- ./mykeystore.jks:/home/appuser/mykeystore.jks
I have also specified few env variables in kafka-connect docker container env variables as shown below:
CONNECT_SSL_TRUSTSTORE_TYPE: ssl.truststore.type=JKS
CONNECT_SSL_TRUSTSTORE_LOCATION: /home/appuser/mykeystore.jks
CONNECT_SSL_TRUSTSTORE_PASSWORD: mystorepass
CONNECT_KAFKA_OPTS: “-Djavax.net.ssl.trustStore=/home/appuser/mykeystore.jks
-Djavax.net.ssl.trustStorePassword=mystorepass
-Djavax.net.ssl.keyStore=/home/appuser/mykeystore.jks
-Djavax.net.ssl.keyStorePassword=mystorepass”
CONNECT_KAFKA_JMX_OPTS: “-Djavax.net.ssl.trustStore=/home/appuser/mykeystore.jks
-Djavax.net.ssl.trustStorePassword=mystorepass”
CONNECT_JAVA_OPTS: “-Djavax.net.debug=ssl:handshake”
I am then using cURL command to configure the MongoSinkConnector with Kafka-connect as shown below:
curl -X POST
-H “Content-Type: application/json”
–data ‘{“name”: “mongo-sink”,
“config”: {
“connector.class”:“com.mongodb.kafka.connect.MongoSinkConnector”,
“topics”:“KAFKA_TEST.wireless.wholesale.whapi.activateSubscription”, “connection.uri”:“mongodb://username:pwd@aws-docdb-cluster-hostname:5000/?tls=true&replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false”,
“key.converter”: “org.apache.kafka.connect.storage.StringConverter”,
“value.converter”: “org.apache.kafka.connect.json.JsonConverter”,
“value.converter.schemas.enable”: false,
“database”:“KAFKA_TEST”,
“collection”:“wireless.activateSubscription”
}
}’
http://localhost:8083/connectors -w “\n”
@mmuehlbeyer @dtroiano Could anyone of you please help me with this issue, I got stuck with these from past 1 week. I need to resolve this asap as it is a blocker for all my work.
any errors?
what does
curl http://localhost:8083/connectors/mongo-sink/status
say?
Best,
Michael
@mmuehlbeyer I am getting below error while executing the curl command. I can see this error in my kafka-connect docker container logs.
INFO Exception in monitor thread while connecting to server docdb-2022-07-07-19-03-05.cluster-csac2qtnbivs.us-west-2.docdb.amazonaws.com:5000 (org.mongodb.driver.cluster)
com.mongodb.MongoSocketException: docdb-2022-07-07-19-03-05.cluster-csac2qtnbivs.us-west-2.docdb.amazonaws.com: Name or service not known
Here “docdb-2022-07-07-19-03-05.cluster-csac2qtnbivs.us-west-2.docdb.amazonaws.” is the Document DB (Mongo) host name
I am unable to create the connector.
where is your kafka cluster running?
aws? confluent cloud? onprem?
my kafka-cluster is running on my local system and Document DB (My mongo db) cluster is on AWS. I have ca-bundle.pem file which I usually use to access Document DB (My mongo db) from my local machine.
I see
seems network related
your kafka-connect can’t resolve the endpoint of your mongo db instance.
I guess there is there a proxy or firewall in place right?
I am on VPN. But from my local machine (where I have all of my docker containers running) am able to access that doc db cluster. Hence networking team confirmed that there is no blockers in accessing document db from local.
ok so it might be related to docker
is it possible to ping or curl the host from within the docker container?
@mmuehlbeyer
ping “mongodb (doc db) hostname” → This doesn’t work in my local machine and docker container. But I am able to list databases/collections available in MongoDB from my local machine.
Below image while pinging Document DB host from local machine
Below image while pinging Doc DB host from kafka-connect docker container
so I think it’s related to the names resolution from within the container.
from my running container I could at least resolve the name.
@mmuehlbeyer yes, I added an entry with in /etc/hosts file inside docker container. It is now able to resolve the host name,
But when I execute the curl command to create MongoSinkConnector am getting the below error.
ok so one step further 
does your mongodb cluster enforce tls connections?
Yes, MongoDB cluster has TLS enabled. I have the certificate (.pem file) required to access this cluster. Hence I have done the below steps:
- I have created a keystore on my host machine (mykeystore.jks)
- I have converted .pem to .crt file (sudo mycert.pem >> mycert.crt)
- Added this .crt file to mykeystore.jks
- In my docker compose file, Under the volumes of my “kafka-connect” docker container I have specified to mount this keystore file to a folder inside the kafka-connect docker container.
Even after following all these I still get the same error “Unable to find the valid certification path to the requested target”
ok I see.
though all of your provided configs are for the kafka tls part.
I can’t see any config related to mongodb tls/ssl config.
a monodb tls config should be like the following , right?
"connection.uri":"mongodb://hostname1.domain.name:27017,pre-hostname2.domain.name:27017/?ssl=true"
best,
michael
yes, I have my mongodb connection uri specified in the curl command.
curl -X POST
-H “Content-Type: application/json”
–data ‘{“name”: “mongo-sink”,
“config”: {
“connector.class”:“com.mongodb.kafka.connect.MongoSinkConnector”,
“topics”:“KAFKA_TEST.wireless.wholesale.whapi.activateSubscription”, “connection.uri”:“mongodb://username:pwd@aws-docdb-cluster-hostname:5000/?tls=true&replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false”,
“key.converter”: “org.apache.kafka.connect.storage.StringConverter”,
“value.converter”: “org.apache.kafka.connect.json.JsonConverter”,
“value.converter.schemas.enable”: false,
“database”:“KAFKA_TEST”,
“collection”:“wireless.activateSubscription”
}
}’
http://localhost:8083/connectors -w “\n”
I see though there is no config related to ssl/tls
Could you please elaborate on what is “config related to ssl/tls” ?
How to identify which path does the kafka-connect refer to w.r.t certification path as error in the logs says “Unable to identify the valid certification path to the requested target”
I think you have to tell the mongodb connector
to use ssl for connecting to your mongo endpoint
need to check the docs by myself
How about this query ? Any clue how to find this out. I have enabled ssl debugging but unable to find anything in kafka-connect container logs.
