Hi all,
I am doing POC on kafka Connect using SFTP JSON source connector.
I have SFTP running on ubuntu server. WinSCP from local machine to SFTP machine works fine i.e. i am able to FTP JSON files from local machine to SFTP machine.
Confluent kafka is running on AWS EC2 ( say machine 1)
SFTP is running on another machine EC2 ( say machine2)
When using SFTP source connector on Confluent Platform Kafka, getting following error in connect.log
[2021-10-28 20:45:38,015] INFO SftpJsonSourceConnectorConfig values:
batch.size = 1000
behavior.on.error = FAIL
cleanup.policy = NONE
empty.poll.wait.ms = 250
error.path = /home/sftpfossuser/sftpdir/error
file.minimum.age.ms = 0
finished.path = /home/sftpfossuser/sftpdir/finished
input.file.pattern = *.json
input.path = /home/sftpfossuser/sftpdir/input
kafka.topic = SFTPTopic
kerberos.keytab.path =
kerberos.user.principal =
key.schema =
parser.timestamp.date.formats = [yyyy-MM-dd’T’HH:mm:ss, yyyy-MM-dd’ 'HH:mm:ss]
parser.timestamp.timezone = UTC
processing.file.extension = .PROCESSING
proxy.password = [hidden]
proxy.username =
schema.generation.enabled = false
schema.generation.key.fields =
schema.generation.key.name = defaultkeyschemaname
schema.generation.value.name = defaultvalueschemaname
sftp.host = aa.bb.cc.dd
sftp.password = [hidden]
sftp.port = 22
sftp.proxy.url =
sftp.username = sftpfossuser
timestamp.field =
timestamp.mode = PROCESS_TIME
tls.passphrase = [hidden]
tls.pemfile =
tls.private.key = [hidden]
tls.public.key = [hidden]
value.schema =
(io.confluent.connect.sftp.source.SftpJsonSourceConnectorConfig:372)
[2021-10-28 20:45:38,073] ERROR WorkerConnector{id=SftpJsonSourceConnectorConnector_Kshitij} Error while starting connector (org.apache.kafka.connect.runtime.WorkerConnector:193)
org.apache.kafka.connect.errors.ConnectException: Failed to establish connection with SFTP.
at io.confluent.connect.sftp.connection.SftpConnection.(SftpConnection.java:90)
at io.confluent.connect.sftp.source.AbstractSftpSourceConnectorConfig.(AbstractSftpSourceConnectorConfig.java:168)
at io.confluent.connect.sftp.source.SftpSourceConnectorConfig.(SftpSourceConnectorConfig.java:101)
at io.confluent.connect.sftp.source.SftpJsonSourceConnectorConfig.(SftpJsonSourceConnectorConfig.java:23)
at io.confluent.connect.sftp.SftpJsonSourceConnector.start(SftpJsonSourceConnector.java:45)
at org.apache.kafka.connect.runtime.WorkerConnector.doStart(WorkerConnector.java:185)
at org.apache.kafka.connect.runtime.WorkerConnector.start(WorkerConnector.java:210)
at org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:349)
at org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:332)
at org.apache.kafka.connect.runtime.WorkerConnector.doRun(WorkerConnector.java:141)
at org.apache.kafka.connect.runtime.WorkerConnector.run(WorkerConnector.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:519)
at com.jcraft.jsch.Session.connect(Session.java:183)
at io.confluent.connect.sftp.connection.SftpConnection.init(SftpConnection.java:159)
at io.confluent.connect.sftp.connection.SftpConnection.(SftpConnection.java:88)
… 15 more
Can someone help as to what i am missing.
Any help in resolution is highly appreciated.
Thanks in advance.