AWS Confluent based layer for Lambda usage

Clients Non-Java Clients
1

hi hi
Does anyone have a zip file for a python: confluent client / import that they know works…
Python 3.9
I’ve followed multiple examples and keep ending with the same result:
“errorMessage”: “Unable to import module ‘lambda_function’: No module named ‘confluent_kafka.cimpl’”,
or
“errorMessage”: “Unable to import module ‘lambda_function’: No module named ‘confluent_kafka’”,
CONFLUENT: would really be great if this was available as a useable ARN based layer.
G

2

Hi guys…

Got the python import eventually to work…

Resolution was to build a EC2 instance, install python 3.9.16, create virtual env, install libraries =2.1.1 , zip libraries… upload as a layer.

Initially used a EC2 instance I had, build on Ubuntu, did not work, had to spin a new EC2 using amazon Linux (as that is what Lambda’s use). Worked.

Now to dig a hole through our firewalls to get outside onto the cluster.

If anyone needs these libraries, happy to share.
G

3

Hi georgelza,

I’m also received the same error while executing the AWS lambda function. It will be help full if u share the libraries. Can you please share the libraries to this mail id aajayksk12@gmail.com

Thanks in advance

4

I am also trying to import confluent_kafka and followed the steps below:

python3.9 -m venv aws_layer_creation
source aws_layer_creation/bin/activate
mkdir -p streamer_lib/python/lib/python3.9/site-packages
pip install confluent_kafka -t streamer_lib/python/lib/python3.9/site-packages/
deactivate
sudo rm -rf aws_layer_creation
cd streamer_lib/
zip -r9 ../streamer_lib.zip python

Created Layer using the above created zip file.
Getting the below error:

Unable to import module 'lambda_function': /lib64/libpthread.so.0: version `GLIBC_2.28' not found (required by /opt/python/lib/python3.9/site-packages/confluent_kafka/../confluent_kafka.libs/librdkafka-36eea7b5.so.1)

Can you please share the zip file with me?
suryadeepanlm@gmail.com

5

did you come right, got all this working my side in the end, for Python.
G

6

Hi Georgelza,

I received the following error
Error: Lambda function (create-pretrade-topics-lambda) returned error: ({“errorMessage”: “Unable to import module ‘pretrade-topics-lambda’: /lib64/libpthread.so.0: version `GLIBC_2.28’ not found (required by /opt/python/confluent_kafka/…/confluent_kafka.libs/librdkafka-bc407dcd.so.1)”, “errorType”: “Runtime.ImportModuleError”, “requestId”: “”, “stackTrace”: })

Can you please share the libraries to my email sangam.sridhar@gmail.com
Thanks