Busy trying to put together a Lambda deployed client.
there is a couple of ways to do it… but as soon as you introduce external C libraries (as used by confluent_kakfa when client is Golang based) then the options get less quickly.
The current attempt is following a instruction that copies the golang binary into a AWS provided imaged (specific for lambda), but it now seem to be missing the gcc version and lib low level libraries.
Has anyone deployed their kafka client onto AWS Lambda,
<to make it fun for myself, I’m working on a M1 MAC so to make it simpler I’m using a AWS EC2 instance based on the AmazonLinux OS, the same ami that is used for Lambda
Need some assistance please.
when client is Golang based
Have you tried using sarama or franz-go instead of a librdkafka based one?
I’ve been able to use kafka-python and Java producers in a lambda
mind sharing your code.
please include the Dockerfile
The Lambda deployment seems to add very specific bits…
Happy to share my code… GitHub - georgelza/GoLambdaLoader
I’m building my docker image on a EC2 instance build using an Amazon Linux AMI, where I installed docker.:
I do a
when I try and run a test… I get the below in the logs.
/main: /lib64/libc.so.6: version `GLIBC_2.32’ not found (required by /main)
Why did you comment out the CGO_ENABLED=0 line of your Dockerfile?
note, see Dockerfile in repo, with it added back in I err out. it now seems to not pull in the dependencies.
NOTE: If I change the Dockerfile to build from fedora, where I then install Golang and do a yum install of gcc then it works… but as we know, for Lambda I need this image as it has the other additional AWS hooks to be used at the time of a S3 event call.
Step 7/12 : ENV CGO_ENABLED=0
—> Running in ffe4438e5317
Removing intermediate container ffe4438e5317
Step 8/12 : RUN go build -trimpath -ldflags=“-s -w” -o main main.go
—> Running in da72994df359
./main.go:58:18: undefined: kafka.Producer
./main.go:171:14: undefined: kafka.ConfigMap
./main.go:183:25: undefined: kafka.NewProducer
./main.go:273:23: undefined: kafka.Message
./main.go:274:28: undefined: kafka.TopicPartition
./main.go:274:86: undefined: kafka.PartitionAny
The command ‘/bin/sh -c go build -trimpath -ldflags=“-s -w” -o main main.go’ returned a non-zero code: 1
make: *** [Makefile:20: dbuild] Error 1
… it would be bl00dy nice if someone build a docker image that originates out of the amazonlinux 2 root with the required bits added that we can use on Lambda for Confluent Kakfa usage.
above my skill set.
Got it working using the Fedora base docker image.
with the building being done on the EC2,
interesting, it’s working at 5120recs/sec, where as the python version is doing north of 7000.
Solution: GitHub - georgelza/GoLambdaLoader
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.