Client in docker deployed on AWS Lambda

Ops Containers
1

Hi all

Busy trying to put together a Lambda deployed client.
there is a couple of ways to do it… but as soon as you introduce external C libraries (as used by confluent_kakfa when client is Golang based) then the options get less quickly.

The current attempt is following a instruction that copies the golang binary into a AWS provided imaged (specific for lambda), but it now seem to be missing the gcc version and lib low level libraries.

Has anyone deployed their kafka client onto AWS Lambda,
<to make it fun for myself, I’m working on a M1 MAC so to make it simpler I’m using a AWS EC2 instance based on the AmazonLinux OS, the same ami that is used for Lambda

Need some assistance please.

G

2

when client is Golang based

Have you tried using sarama or franz-go instead of a librdkafka based one?

I’ve been able to use kafka-python and Java producers in a lambda

3

mind sharing your code.
please include the Dockerfile
The Lambda deployment seems to add very specific bits…
Happy to share my code… GitHub - georgelza/GoLambdaLoader

I’m building my docker image on a EC2 instance build using an Amazon Linux AMI, where I installed docker.:

I do a
make dbuild
make push

when I try and run a test… I get the below in the logs.

/main: /lib64/libc.so.6: version `GLIBC_2.32’ not found (required by /main)

Please help,

G

4

Why did you comment out the CGO_ENABLED=0 line of your Dockerfile?

5

note, see Dockerfile in repo, with it added back in I err out. it now seems to not pull in the dependencies.

NOTE: If I change the Dockerfile to build from fedora, where I then install Golang and do a yum install of gcc then it works… but as we know, for Lambda I need this image as it has the other additional AWS hooks to be used at the time of a S3 event call.

G

Step 7/12 : ENV CGO_ENABLED=0
—> Running in ffe4438e5317
Removing intermediate container ffe4438e5317
—> 784adc163d99
Step 8/12 : RUN go build -trimpath -ldflags=“-s -w” -o main main.go
—> Running in da72994df359

command-line-arguments

./main.go:58:18: undefined: kafka.Producer
./main.go:171:14: undefined: kafka.ConfigMap
./main.go:183:25: undefined: kafka.NewProducer
./main.go:273:23: undefined: kafka.Message
./main.go:274:28: undefined: kafka.TopicPartition
./main.go:274:86: undefined: kafka.PartitionAny
The command ‘/bin/sh -c go build -trimpath -ldflags=“-s -w” -o main main.go’ returned a non-zero code: 1
make: *** [Makefile:20: dbuild] Error 1

6

… it would be bl00dy nice if someone build a docker image that originates out of the amazonlinux 2 root with the required bits added that we can use on Lambda for Confluent Kakfa usage.

above my skill set.

G

7

Got it working using the Fedora base docker image.
with the building being done on the EC2,
interesting, it’s working at 5120recs/sec, where as the python version is doing north of 7000.
unexpected…
Solution: GitHub - georgelza/GoLambdaLoader
G

8

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.