Loading Confluent Libraries in HDInsight Spark - Jupyter

Schema Registry
1

I have to thank you on the excellent article on how to consume Confluent Kafka in to Databricks.

Using this article, I now have a working setup for our data. So thank you. :smiley:

But now I have been asked if the same can be done with HDInsight Spark and Jupyter Notebooks.
It looks like I should be able to, but the part I am struggling with is the loading of the specific libraries needed (confluent-kafka[avro,json,protobuf]>=1.4.2).

Do you have any tips on how this can be done with HDInsight Spark Jupyter Notebooks?

Thank you,
Raoul