Connecting local docker ksqldb cli to ConfluentCloud hosted cluster

Confluent Cloud
1

Hi guys

So I’ve done the first, got ccloud working, I’ve got Robin’s ais app into a AWS hosted EC2 pod hosting a docker environment, it’s consuming and pushing successfully to my Confluent Cloud cluster/topic.

Now want to get ksqldb cli working on my MAC… hitting wall.

Can someone maybe share what needs to be set, how.

G

2

Hi,

a good point to start:

https://docs.confluent.io/cloud/current/cp-component/ksql-cloud-config.html
https://docs.confluent.io/cloud/current/cp-component/ksqldb-ccloud-cli.html

full cloud etl example
https://docs.confluent.io/platform/current/tutorials/examples/cloud-etl/docs/index.html#

HTH,
M

3

must be missing something, the first url seem to tell me how to configure ksqldb to talk to my Confluent cluster … - this i was doing via the web interface.
The second url seems to configure ccloud how to connect to the ksqldb cluster, to manage it…

neither seem to enable/configure ksqldb cli, where I can run the relevant sql.

G

4

I think there is a solution

@rmoff blogged about it some months ago

basically it seems to work with

docker run --interactive --tty --rm \
  confluentinc/ksqldb-server:0.15.0 \
  ksql -u $CCLOUD_KSQL_API_KEY \
       -p $CCLOUD_KSQL_API_SECRET \
          $CCLOUD_KSQL_ENDPOINT

code snippet taken from above blog post (credits to Robin)

Michael

1 Like
5

thanks, will go have a read and report back.

G

1 Like
6

trying to wok through the how to…
but it talks of a $0.88 cost.hour, adding up to $640/month…

well that does not fit well into the voucher amount…

hows everyone else doing these demo’s at these rates.

<really need a demo type cluster that say only allows 1 week’s retention, max 10GB, all features enabled, zero cost, never production>

G

7

anyone have clarity on this?
G

8

you’re talking about demos clusters?

9

yes, where there is a zero cost.

Question, when we currently create a confluent cluster, using the vouchers, how do we shut the cluster down.

G

10

and then back to cost management.

if we go and create that small standard cluster, and allocate to it the voucher to not pay for it, we now need a ksqldb, and I noticed it’s a $0.88 cost/hour, which rolls up to $688/month.
Not fitting into the voucher. so how do we realistically manage this.

Or MY suggest Confluent might rather create a Demo/education cluster, where nothing is charged, with small enough limits to prohibit production usage.

G

11

afaik there is no possibility to shutdown the cluster
only delete is possible

1 Like
12

I think it depends a little on your use and test case

is it necessary to have the cluster running for a whole month for test/demo cases?

if not you might create some deployment scripts to easily recreate kafka topics and so on and delete the cluster after every test

13

… this is all learning/teach myself Kafka, KSQLDB etc.

via personal Python and Golang programs and then following Robin’s Blogs.

G

14

Are you aware of a screen where your invoice/billing is visible, where you could see what the current usage/$$$ of the voucher is sitting at?

G

15

ignore, found it, Billing and payment

G

16

ok I see
but what about a small local demo env based on docker?

17

that is a option, can can investigate… just thinking, everyone is trying to learn the stack, Confluent gives us these vouchers, but then we sitting with situation if we enable something like ksqldb it will eat about 3x the voucher price.

most people doing these demo/blog examples I can promise you is not thining/noticing that $0.88/hour amount, rolling up to $688/month.

They would all come in, enable the bits and do the demo, done with first, by that time Robin prob release another great demo that has to be tried,

and come end of the month, whammy… knockout, as that $$$ goes of your personal CC.

G

18

… my local band width is also not that great, but might be the best/cost effective/risk free option.

G

19

back to the local cluster idea…

One of the attractive reasons to use the cloud based confluent platform is the pre configured/availability of the connectors to sites like Elastic…

G

20

understand and agree
so maybe a split dev environment might be a good way to go

for example:

part 1 local dev env
based on confluent quickstart

part 2 cloud dev env
cloud environment based on confluent cloud
only started when needed, shutdown/deleted after usage
prepare deploy scripts to easy built-up a new cluster

just my thoughts to omit huge costst

M