KafkaJsonSchemaSerializer - ClassNotFoundException

Schema Registry
1

I’m trying to do simple kotlin producer which will validate my message.

@RestController("/api/")
 class Controller {

  @GetMapping("/")
  fun send() {
      val props: Properties = Properties()
      props[ProducerConfig.BOOTSTRAP_SERVERS_CONFIG] = "localhost:9092"
      props[ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG] = "org.apache.kafka.common.serialization.StringSerializer"

      props["schema.registry.url"] = "http://127.0.0.1:8982"
//        props[ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG] = "io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializer"
      props[ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG] = KafkaJsonSchemaSerializer::class.java

//        props[KafkaJsonSchemaSerializerConfig.FAIL_INVALID_SCHEMA] = true
//        props[KafkaJsonSchemaSerializerConfig.AUTO_REGISTER_SCHEMAS] = true
      println("Send message")
      val producer = KafkaProducer<String, Message>(props)
      val message = Message(
          firstName = "John",
          lastName = "Doe",
          dateBorn = Instant.now(),
      )
      //message
      val producerRecord = ProducerRecord<String, Message>("message", "someKey", message)
      producer.send(producerRecord).get()
      producer.close()
  }
}

And when i try to send message i get this:

java.lang.ClassNotFoundException: scala.Serializable
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) ~[na:na]
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) ~[na:na]

I try to use two ways to chose correct class but all not work

2

Are you able to share the entire project, either directly here or on GitHub?

This will likely have more to do with your Gradle / Maven dependencies (e.g., conflicts), or compile vs runtime version differences, or classpath, than with the source code itself, so a complete runnable project that others can attempt repro and tweak would be best to work through those possibilities.

3

My repo for test - GitHub - mankass/schemaRegistryProducer

4

Thanks for sharing. This does look to be a transitive dependency version conflict with Scala dependencies. To find this out, I first checked which version of Scala has that class and found it’s in 2.12 and not in 2.13:

% jar tvf scala-library-2.12.10.jar| grep Serial
  ...
   377 Tue Sep 10 21:11:10 EDT 2019 scala/Serializable.class

whereas with 2.13 I don’t see that class output.

Then I ran gradle dependencies in the project to see if there are dependency conflicts. It has a lot of output but the fishy line is this:

+--- com.kjetland:mbknor-jackson-jsonschema_2.12:1.0.39
|    +--- org.scala-lang:scala-library:2.12.10 -> 2.13.14

Gradle is going with 2.13.14 instead of 2.12.10 because other dependencies have the same (transitive) scala-library dependency and they conflict so it throws up its hands and hopes for the best with the higher version.

There are two dependencies pulling in the scala-library transitive dependency: org.springframework.kafka:spring-kafka-test and io.confluent:kafka-json-schema-serializer… I deleted the former out of laziness, and for the latter added an exclude to ignore the module that pulls it in:

    implementation("io.confluent:kafka-json-schema-serializer:7.8.0") {
        exclude(group = "com.kjetland", module="mbknor-jackson-jsonschema_2.13")
    }

With those two changes, I was able to see the expected correct gradle dependencies output (this is the litmus test if you try other tweaks):

+--- com.kjetland:mbknor-jackson-jsonschema_2.12:1.0.39
|    +--- org.scala-lang:scala-library:2.12.10

And the app ran without hitting that exception. Make sure that you refresh Gradle dependencies if running from within an IDE since they can get cached and still be on the classpath despite the exclude.

A few final tweaks / ideas:

  • There’s a typo spring.kafka.producer.ackss
  • I’d recommend getting on a more recent version of Kafka, as preventative medicine. Confluent Platform 5.5.3 is pretty old. I ran against 7.8.0 using this Docker Compose and it worked - just be sure to update the Schema Registry URL to http://localhost:8081 in the couple of places where it’s configured to hit port 8982.
  • In the Schema Registry codebase I see a io.yokota:mbknor-jackson-jsonschema-java8 dependency – it feels like this exists due to the problem you ran into… haven’t chased down where that lives but I see forks like this (" NOTE: This is the Java rewrite of the original project, without Scala dependencies.") so I would bet that others have hit pain like this before. Possibly io.confluent:kafka-json-schema-serializer should look into that.
  • An entirely different approach that you could investigate is to use a shadow plugin that rewrites packages. Basically, you should be able to relocate packages so that multiple versions of a dependency can be used rather than having to settle on one.