Passing data from kafka cluster to timescaledb

Confluent Cloud
21

not yet
still in progress

22

short update

tried several options locally
stack starting but getting an error as well :smiley:
what I’ve tried to far

table.name.format=mytmp
connection.password=postgres
tasks.max=1
topics=temperature
value.converter.schema.registry.url=http://schema-registry:8081
key.converter.schemas.enable=false
auto.evolve=true
connection.user=postgres
value.converter.schemas.enable=false
auto.create=true
connection.url=jdbc:postgresql://postgres:5432/
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schema.registry.url=http://schema-registry:8081
key.converter=org.apache.kafka.connect.json.JsonConverter
pk.mode=none
23

You’re getting the same error ?
I feel like there are two possibilities here :

  • the connector configuration file and the json schema configuration file aren’t well done and they are the cause of the error.
  • The postgres sink connector isn’t compatible with TimescaleDB database.

i don’t know if anybody managed to send data to a timescaleDB database before.
let me know if you manage to find more info or fix the problem and I’ll tell you as well if i do.

24

ah no a more generic error :expressionless:

I’ve managed it once to send data to timescale db with jdbc sink but seems as the config does not work as expected need to check

best,
michael

25

so you think the problem is coming from the connector’s config file ? we had the same problem with the jdbc and the postgres sink connectors.

26

I’m not 100% sure

according to your error it seems to be related to data conversion errors

see also Kafka Connect JDBC Sink deep-dive: Working with Primary Keys

27

Ok, I’ll try to find a solution through this article. and if i can’t find any, what do you think the best solution or alternative here ?
Thank you.

28

I think there hase to be a proper solution :slight_smile:

going to restart my example from a year ago and keep you posted about the outcome

29

Thank you so much for your time, I would really appreciate it if you help me out with this one.

I’ll be waiting for your response.

30

Hello @mmuehlbeyer,
Any updates on this problem ?
should i try another solution ?
Thank you.

31

not yet unfortunately
other things going on at the moment

32

No problem. Just let me know if you find anythin please.

Thank you.

33

Hello again @mmuehlbeyer,
did you manage to find a solution in your local system ?
I’m trying to find out whether it is possible to make this work or maybe i should look for another way.
Thank you

34

hi @Oussgasmi
unfortunately not as I was on holidays :wink:

best,
michael