Hi all
Busy trying to put together a Lambda deployed client.
there is a couple of ways to do it… but as soon as you introduce external C libraries (as used by confluent_kakfa when client is Golang based) then the options get less quickly.
The current attempt is following a instruction that copies the golang binary into a AWS provided imaged (specific for lambda), but it now seem to be missing the gcc version and lib low level libraries.
Has anyone deployed their kafka client onto AWS Lambda,
<to make it fun for myself, I’m working on a M1 MAC so to make it simpler I’m using a AWS EC2 instance based on the AmazonLinux OS, the same ami that is used for Lambda
Need some assistance please.
G
when client is Golang based
Have you tried using sarama or franz-go instead of a librdkafka based one?
I’ve been able to use kafka-python and Java producers in a lambda
mind sharing your code.
please include the Dockerfile
The Lambda deployment seems to add very specific bits…
Happy to share my code… GitHub - georgelza/GoLambdaLoader
I’m building my docker image on a EC2 instance build using an Amazon Linux AMI, where I installed docker.:
I do a
make dbuild
make push
when I try and run a test… I get the below in the logs.
/main: /lib64/libc.so.6: version `GLIBC_2.32’ not found (required by /main)
Please help,
G
Why did you comment out the CGO_ENABLED=0 line of your Dockerfile?
note, see Dockerfile in repo, with it added back in I err out. it now seems to not pull in the dependencies.
NOTE: If I change the Dockerfile to build from fedora, where I then install Golang and do a yum install of gcc then it works… but as we know, for Lambda I need this image as it has the other additional AWS hooks to be used at the time of a S3 event call.
G
Step 7/12 : ENV CGO_ENABLED=0
—> Running in ffe4438e5317
Removing intermediate container ffe4438e5317
—> 784adc163d99
Step 8/12 : RUN go build -trimpath -ldflags=“-s -w” -o main main.go
—> Running in da72994df359
command-line-arguments
./main.go:58:18: undefined: kafka.Producer
./main.go:171:14: undefined: kafka.ConfigMap
./main.go:183:25: undefined: kafka.NewProducer
./main.go:273:23: undefined: kafka.Message
./main.go:274:28: undefined: kafka.TopicPartition
./main.go:274:86: undefined: kafka.PartitionAny
The command ‘/bin/sh -c go build -trimpath -ldflags=“-s -w” -o main main.go’ returned a non-zero code: 1
make: *** [Makefile:20: dbuild] Error 1
… it would be bl00dy nice if someone build a docker image that originates out of the amazonlinux 2 root with the required bits added that we can use on Lambda for Confluent Kakfa usage.
above my skill set.
G
Got it working using the Fedora base docker image.
with the building being done on the EC2,
interesting, it’s working at 5120recs/sec, where as the python version is doing north of 7000.
unexpected…
Solution: GitHub - georgelza/GoLambdaLoader
G