Passing data from kafka cluster to timescaledb

Confluent Cloud
21

not yet
still in progress

1 Like
22

short update

tried several options locally
stack starting but getting an error as well :smiley:
what I’ve tried to far

table.name.format=mytmp
connection.password=postgres
tasks.max=1
topics=temperature
value.converter.schema.registry.url=http://schema-registry:8081
key.converter.schemas.enable=false
auto.evolve=true
connection.user=postgres
value.converter.schemas.enable=false
auto.create=true
connection.url=jdbc:postgresql://postgres:5432/
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schema.registry.url=http://schema-registry:8081
key.converter=org.apache.kafka.connect.json.JsonConverter
pk.mode=none
23

You’re getting the same error ?
I feel like there are two possibilities here :

  • the connector configuration file and the json schema configuration file aren’t well done and they are the cause of the error.
  • The postgres sink connector isn’t compatible with TimescaleDB database.

i don’t know if anybody managed to send data to a timescaleDB database before.
let me know if you manage to find more info or fix the problem and I’ll tell you as well if i do.

24

ah no a more generic error :expressionless:

I’ve managed it once to send data to timescale db with jdbc sink but seems as the config does not work as expected need to check

best,
michael

1 Like
25

so you think the problem is coming from the connector’s config file ? we had the same problem with the jdbc and the postgres sink connectors.

26

I’m not 100% sure

according to your error it seems to be related to data conversion errors

see also Kafka Connect JDBC Sink deep-dive: Working with Primary Keys

2 Likes
27

Ok, I’ll try to find a solution through this article. and if i can’t find any, what do you think the best solution or alternative here ?
Thank you.

28

I think there hase to be a proper solution :slight_smile:

going to restart my example from a year ago and keep you posted about the outcome

1 Like
29

Thank you so much for your time, I would really appreciate it if you help me out with this one.

I’ll be waiting for your response.

30

Hello @mmuehlbeyer,
Any updates on this problem ?
should i try another solution ?
Thank you.

31

not yet unfortunately
other things going on at the moment

1 Like
32

No problem. Just let me know if you find anythin please.

Thank you.

1 Like
33

Hello again @mmuehlbeyer,
did you manage to find a solution in your local system ?
I’m trying to find out whether it is possible to make this work or maybe i should look for another way.
Thank you

34

hi @Oussgasmi
unfortunately not as I was on holidays :wink:

best,
michael