HTTP Source Connector: Support for String offsets

Kafka Connect
1

Hi,

Author of https://www.http-feeds.org/ here.

I want to set up an HTTP Source Connector, that connects to an HTTP Feeds.

I use this configuration:
{
“http.request.method”: “GET”,
“http.request.parameters”: “lastEventId=${offset}”,
“http.offset.mode”: “CHAINING”,
“http.offset.json.pointer”: “/id”,
“output.data.format”: “JSON”
}

The response from the HTTP API is something like:

HTTP/1.1 200 OK
Content-Type: application/cloudevents-batch+json

[{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "1c6b8c6e-d8d0-4a91-b51c-1f56bd04c758",
  "time" : "2021-01-01T00:00:01Z",
  "subject" : "9521234567899",
  "data" : {
    "sku": "9521234567899",
    "updated": "2022-01-01T00:00:01Z",
    "quantity": 5
  }
},{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "292042fb-ab04-4653-af90-19a24032bffe",
  "time" : "2021-12-01T00:00:15Z",
  "subject" : "9521234512349",
  "data" : {
    "sku": "9521234512349",
    "updated": "2022-01-01T00:00:12Z",
    "quantity": 0
  }
},{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "fa3e2a22-398c-4d02-ad08-9415e43178e6",
  "time" : "2021-01-01T00:00:22Z",
  "subject" : "9521234567899",
  "data" : {
    "sku": "9521234567899",
    "updated": "2022-01-01T00:00:21Z",
    "quantity": 4
  }
}]

The only problem is that the next chained request is:

GET https://example.http-feeds.org/inventory?lastEventId="fa3e2a22-398c-4d02-ad08-9415e43178e6"

The problems are the quotes (") in the offset query parameter.

Any idea, how those can be removed?

2

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.