# Kafka JSON Source Connector

**URL:** <https://forum.confluent.io/t/kafka-json-source-connector/2072>\
**Category:** Kafka Connect\
**Created:** [21 June 2021 10:16 UTC](https://forum.confluent.io/t/kafka-json-source-connector/2072 "2021-06-21T10:16:28Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![DestroyerAlpha](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/destroyeralpha/32/883_2.png) [@DestroyerAlpha](https://forum.confluent.io/u/DestroyerAlpha)\
**Post date:** [21 June 2021 10:16 UTC](https://forum.confluent.io/t/kafka-json-source-connector/2072/1 "2021-06-21T10:16:29Z")

</div>

I have a bunch of log files, which are basically just JSON arrays. I have to load them in a Kafka Topic and then onto a PostgreSQL database. How I am doing it right now is, using a Producer written in Python to process the JSON and serialize the data as AVRO and send it to Kafka. After that I am using JDBC sink connector to load it onto PostgreSQL. Is there a better way to do it? My main concern is that because I am writing my own producer code, I would have to end up maintaining it going forward. Hence, I was trying to find some Open Source Connectors that do the job but sadly none of them work (tried filepulse and spooldir). Is there a better way to do it? The log files being generated are from a proprietary data agent deployed on a system.

---

<div class="post-metadata">

**Author:** ![rmoff](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/rmoff/32/3_2.png) [@rmoff](https://forum.confluent.io/u/rmoff)\
**Post date:** [21 June 2021 10:22 UTC](https://forum.confluent.io/t/kafka-json-source-connector/2072/2 "2021-06-21T10:22:55Z")

</div>

Welcome to the forum 🙂

Kafka Connect is indeed generally a great way to do this kind of integration. Can you expand a bit on “none of them work” in terms of the filepulse and spooldir connectors?

---

<div class="post-metadata">

**Author:** ![DestroyerAlpha](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/destroyeralpha/32/883_2.png) [@DestroyerAlpha](https://forum.confluent.io/u/DestroyerAlpha)\
**Post date:** [21 June 2021 11:32 UTC](https://forum.confluent.io/t/kafka-json-source-connector/2072/4 "2021-06-21T11:32:44Z")

</div>

Hey, thank you for your response. Your tutorials really helped to set everything up.  
So, I need each JSON array element to represent a row in the database. The explode array configuration was throwing me an error with filepulse. I raised an [issue on Github](https://github.com/streamthoughts/kafka-connect-file-pulse/issues/144) but did not get a response yet. I tried Spooldir but it just stored the array as a single event in the topic. So I ended up creating my own producer.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/1X/c49438c90c9df282e9996fdf6971be890c71b65a.svg) [@system](https://forum.confluent.io/u/system)\
**Post date:** [21 July 2021 11:33 UTC](https://forum.confluent.io/t/kafka-json-source-connector/2072/5 "2021-07-21T11:33:00Z")

</div>

Closed on 21 July 2021 11:33 UTC

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.