Kafka sink connector for mongo returns “collection already exists for … that is not a timeseries collection” even though it is

Kafka Connect
1

Hi,
I have been working for a little with a kafka sink connector to fill up my timeseries collections.
the config is :

{
  "name": "mongo-events-sink",
  "config" : {
    "connector.class": "com.mongodb.kafka.connect.MongoSinkConnector",
    "tasks.max": "2",
    "topics": "mydb.events",
    "connection.uri": "mongodb://writer:<passwd>@rs-1-1:27017/?directConnection=true&appName=mongosh+2.2.5",
    "database": "mydb",
    "collection": "events",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter.schemas.enable": "false",
    "document.id.strategy": "com.mongodb.kafka.connect.sink.processor.id.strategy.BsonOidStrategy",
    "document.id.strategy.overwrite.existing": "true",
    "writemodel.strategy": "com.mongodb.kafka.connect.sink.writemodel.strategy.InsertOneDefaultStrategy",
    "delete.on.null.values": "false",
    "timeseries.timefield": "_timestamp",
    "timeseries.metafield": "_metadata",
    "timeseries.timefield.auto.convert": true
  }
}

my collection is created with :

db = db.getSiblingDB("mydb")

db.createCollection(
    "mydb.events",
    {
        timeseries: {
            timeField: "_timestamp",
            metaField: "_metadata",
            granularity: "seconds"
        },
    }

)

db.events.createIndex( { "_metadata.eventId": 1, "_timestamp": 1} )

db = db.getSiblingDB("admin")
db.grantRolesToUser("reader", [{"role":"read", "db": "mydb"}])
db.grantRolesToUser("writer", [{"role":"readWrite", "db": "mydb"}])

This configuration has been working lately but i had to restart my mongo RS and kafka instance to integrate new config and things have not been working since then (even though it feels like nothing has changed on the config side).

When i run:
curl -X POST -H "Content-Type: application/json" --data @config/mongo-mydb-events-sink-config.json http://localhost:8083/connectors -w "\n",
i get :
{"error_code":400,"message":"Connector configuration is invalid and contains the following 1 error(s):\nA collection already exists for: mydb.eventsthat is not a timeseries collection.\nYou can also find the above list of errors at the endpoint /connector-plugins/{connectorType}/config/validate"}.

2

I suspect that the exception is coming from here. I’d recommend writing a short client program to perform the same check that the connector is performing, to see what the collStats command returns.

I also see this exception message here, but even there you need to get past the collStats timeseries key check right above it, so either way I suspect that the key won’t be present.

3

‘db.runCommand({“collStats”:“events”})’ result does not contains any key called ‘timeseries’, contrary to what the connector code requires.
‘db.getCollectionInfos()’ returns :
‘’’
{
name: ‘mydb.events’,
type: ‘timeseries’,
options: {
timeseries: {
timeField: ‘_timestamp’,
metaField: ‘_metadata’,
granularity: ‘seconds’,
bucketMaxSpanSeconds: 3600
}
},
info: { readOnly: false }
}
‘’’
which says the collection is in fact a timeseries collection.
I also have to says that this configuration was working a few weeks ago, and that this error poped up after a restart of my mongoDB.

4

Hm, this is a tricky one :slight_smile:

Given that it worked a few weeks ago, you might want to dig into any version changes, either of Mongo server or the connector itself. Have you been on the same version of the connector the whole time?

You might also find something out if you start from a completely clean slate. i.e., if you were going to open a JIRA in Mongo’s KAFKA project, you’d want to give clean slate repro steps.

Finally, if any of the previous steps don’t yield a solution, I would recommend opening a bug. If you can point to a minimal repro where timeseries is the type value and a key in options and point to connector source code looking in the wrong place, that feels like a connector bug to raise in Mongo’s JIRA.

5

@lpnc just curious what is nindexes in the returned collStats? I’m wondering if this is actually the source of the error.

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.