# NEWBE Question about generic connector for API source **URL:** \ **Category:** Self-Managed Connectors\ **Created:** [30 January 2025 09:53 UTC](https://forum.confluent.io/t/newbe-question-about-generic-connector-for-api-source/36883 "2025-01-30T09:53:17Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![EMPIRE-STATE-HUMAN](https://avatars.discourse-cdn.com/v4/letter/e/b3f665/32.png) [@EMPIRE-STATE-HUMAN](https://forum.confluent.io/u/EMPIRE-STATE-HUMAN)\ **Post date:** [30 January 2025 09:53 UTC](https://forum.confluent.io/t/newbe-question-about-generic-connector-for-api-source/36883/1 "2025-01-30T09:53:17Z")
Hi I am completely new to world of Kafka. I understand the basic principals and I can clearly see the potential of this platform. However, I am not a developer and know nothing about about Java. I explore this field using Docker and ChatGPT/Copilot (yes… I am one of “them”). What I want to do is to explore this using Docker Desktop on my own computer to start with. As a start I thought I would try to implement a connector that pulls data from an public API source with frequently updated data, such as a weather or finance API. But right out the gate I am completely stuck. When I go to the Confluent Hub then it seems as if the HTTP source connector (if that is what I am supposed to work with?) is not available. Copilot doesn’t seem to understand that so it keeps on adding it to the Dockerfile and then failing. After a while it tries other solutions on GitHub instead but fails there as well as the ZIP file or the JAR files are no longer available OR it just points to a repo and then fail. Long story short… is there an “easy” way to get a connector or Docker image for this purpose which is free? I seem to be running around in circles here… Cheers ---
**Author:** ![dtroiano](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/dtroiano/32/1961_2.png) [@dtroiano](https://forum.confluent.io/u/dtroiano)\ **Post date:** [30 January 2025 22:38 UTC](https://forum.confluent.io/t/newbe-question-about-generic-connector-for-api-source/36883/2 "2025-01-30T22:38:55Z")
To get started with an HTTP source in Docker, I would recommend checking out [this](https://docs.confluent.io/platform/current/get-started/platform-quickstart.html#step-1-download-and-start-cp) Confluent Platform quick start (just Step 1). From there, if you have an HTTP API to pull data from, install the HTTP Source Connector: From your host terminal: ```auto % docker exec -it connect /bin/bash ``` In the container: ```auto $ confluent-hub install --no-prompt confluentinc/kafka-connect-http-source:latest ``` From your host terminal: ```auto % docker compose restart connect ``` When the connect cluster comes back up [here](http://localhost:9021/clusters/MkU3OEVBNTcwNTJENDM2Qk/management/connect), you’ll see the HTTP Source connector, and from there you can fill out the form to pull data from an API: ![image](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/2X/d/d8004b534a4715aacf9c054ff9407f3caf2fcf74.png) Have you picked an API to use? _Disclaimer: The linked `cp-all-in-one` demo runs Confluent Platform’s Enterprise licensed platform product. There is a “forever” [Developer license](https://docs.confluent.io/platform/current/installation/license.html#developer-license) for running one (non-production) broker, which is what this demo does._ ---
**Author:** ![EMPIRE-STATE-HUMAN](https://avatars.discourse-cdn.com/v4/letter/e/b3f665/32.png) [@EMPIRE-STATE-HUMAN](https://forum.confluent.io/u/EMPIRE-STATE-HUMAN)\ **Post date:** [31 January 2025 17:20 UTC](https://forum.confluent.io/t/newbe-question-about-generic-connector-for-api-source/36883/3 "2025-01-31T17:20:40Z")
Hi Many thanks for your response! That was exactly what I was looking for. 🙏 👏 I am very greatful for that. I was thinking to connect to the Nasdaq API and get the share prices as that should produce a lot of new data on a regular basis. But there was quite a lot to figure out for the HTTP connector and not super intuitive what to put in where. The URL is: [https://data.nasdaq.com/api/v3/datasets/WIKI/PRICES.json](https://data.nasdaq.com/api/v3/datasets/WIKI/PRICES.json) I do have an API key and Copilot can give me the content of the configuration file. However, when I click my way in to the GUI, I cannot see anywhere to paste in the full configuration. Instead each setting seems to have its own field and it is not obvious what value in the configuration maps to which field in the GUI. 🤔 ---
**Author:** ![dtroiano](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/dtroiano/32/1961_2.png) [@dtroiano](https://forum.confluent.io/u/dtroiano)\ **Post date:** [31 January 2025 19:20 UTC](https://forum.confluent.io/t/newbe-question-about-generic-connector-for-api-source/36883/4 "2025-01-31T19:20:57Z")
From [this page](http://localhost:9021/clusters/MkU3OEVBNTcwNTJENDM2Qk/management/connect/connect-default/connectors/browse) you can click the button to upload the full JSON config: ![image](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/2X/a/ae37e87b411acfd928ce3609e06324484f1a98a6.png) The configuration should resemble this: ```json { "name": "my-http-source-connector-name", "config": { "connector.class": "io.confluent.connect.http.HttpSourceConnector", "topic.name.pattern": "my-topic", "url": "https://YOUR_API_ENDPOINT", "http.request.method": "get", // other params } } ``` ---
**Author:** ![EMPIRE-STATE-HUMAN](https://avatars.discourse-cdn.com/v4/letter/e/b3f665/32.png) [@EMPIRE-STATE-HUMAN](https://forum.confluent.io/u/EMPIRE-STATE-HUMAN)\ **Post date:** [3 February 2025 13:22 UTC](https://forum.confluent.io/t/newbe-question-about-generic-connector-for-api-source/36883/5 "2025-02-03T13:22:49Z")
Got it. Thank you. The more I poke around the more questions pops up… Can parts of the request URL be dynamically set? For instance, assume that I have a message in a topic for each symbol on Nasdaq, and I need to append that to the request URL for API calls on each specific company. I guess I could create a static HTTP connector for each company, but that seems like a very ineffective way to do it… ---
**Author:** ![dtroiano](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/dtroiano/32/1961_2.png) [@dtroiano](https://forum.confluent.io/u/dtroiano)\ **Post date:** [3 February 2025 18:14 UTC](https://forum.confluent.io/t/newbe-question-about-generic-connector-for-api-source/36883/6 "2025-02-03T18:14:36Z")
> [@EMPIRE-STATE-HUMAN](#): > > Can parts of the request URL be dynamically set? Are you using [this](https://docs.confluent.io/cloud/current/connectors/cc-http-source.html) connector and not he `v2` one? The `entity.names` parameter with values that then get templated into `${entityName}` might work for some number of companies. I doubt that it will scale to all NASDAQ listings, though. And this would require tickers in the connector config, not coming from a topic. If the pattern of consuming from a topic and using data in the events to call out to another API is required, then I don’t think that the HTTP source connector is the way to go. The NASDAQ API (including docs) is pretty closed down without an account so I haven’t been able to dig into the API to see how well it fits with the HTTP Source connector. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/1X/c49438c90c9df282e9996fdf6971be890c71b65a.svg) [@system](https://forum.confluent.io/u/system)\ **Post date:** [10 February 2025 18:14 UTC](https://forum.confluent.io/t/newbe-question-about-generic-connector-for-api-source/36883/7 "2025-02-10T18:14:47Z")
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.