Confluent for Kubernetes Configure Initial Queries

ksqlDB
1

Hi all,

I’m looking for some advice on how to configure queries in ksqlDB when deploying with Confluent for Kubernetes operator. I know there is a headless mode that accepts a --queries-file. Is there a way to use that with the kubernetes operator?

The confluent helm charts supported headless mode. Is there an alternative method for confluent for kubernetes operator deployments?

Paul

2

Posting a response to my own question. I figured out a way to do this.

To figure this out I started by inspecting the KSQL CRD using
kubectl explain KsqlDB.spec
to show the docs for spec that shows there is a section in the CRD called mounted volumes

   mountedVolumes       <Object>
     mountedVolumes list the custom volumes that need to be mounted into the
     underlying statefulset. A change to this setting will roll the cluster.

Further inspecting that section by using
kubectl explain KsqlDB.spec.mountedVolumes

Shows the details of the mountedVolumes section

KIND:     KsqlDB
VERSION:  platform.confluent.io/v1beta1

RESOURCE: mountedVolumes <Object>

DESCRIPTION:
     mountedVolumes list the custom volumes that need to be mounted into the
     underlying statefulset. A change to this setting will roll the cluster.

FIELDS:
   volumeMounts <[]Object> -required-
     volumeMounts specify the list of volume mounts for the pods in the
     statefulset.

   volumes      <[]Object> -required-
     volumes specify the list of volumes that can be mounted into the pods of
     statefulset.

This means we can use mounted volume to mount the SQL Queries file. So I create a config map for the sql file by adding a config map to my helm chart that’s deploying confluent for kubernetes components.

apiVersion: v1
kind: ConfigMap
metadata:
  name: ksqldb-ksql-queries-configmap
  namespace: {{ .Values.namespace }}
data:
  {{- $files := .Files }}
  {{- range tuple "queries.sql" }}
  {{ . }}: |-
{{ $files.Get . | indent 4 }}
  {{- end }}

And then I can use the mounted volumes section to mount the config map in the container.

apiVersion: platform.confluent.io/v1beta1
kind: KsqlDB
metadata:
  name: ksqldb
  namespace: {{ .Values.namespace }}
spec:
  replicas: {{ .Values.ksqldb.replicas }}
  image:
    application: confluentinc/cp-ksqldb-server:{{ .Values.version }}
    init: confluentinc/confluent-init-container:2.3.0
  configOverrides:
    server:
      - access.control.allow.origin=*
      - access.control.allow.methods=GET,POST,HEAD
      - access.control.allow.headers=X-Requested-With,Content-Type,Accept,Origin,Authorization
      - ksql.queries.file=/etc/ksql/queries/queries.sql
  dataVolumeCapacity: 10Gi
  mountedVolumes:
    volumeMounts:
      - name: ksql-queries
        mountPath: /etc/ksql/queries
    volumes:
      - name: ksql-queries
        configMap:
          name: ksqldb-ksql-queries-configmap

Note setting the ksql queries file server config override to complete
- ksql.queries.file=/etc/ksql/queries/queries.sql

3

Here’s how I did this for the sake of sharing answering my own post.

I started by inspecting the CRD using kubectl explain

Explain ksqldb.spec and ksql.spec.mountedVolumes

ksqldb crd has a section in the spec for mountedVolumes that will allow us to mount our sql file.

kubectl explain KsqlDB.spec

KIND:     KsqlDB
VERSION:  platform.confluent.io/v1beta1

RESOURCE: spec <Object>

DESCRIPTION:
     spec defines the desired state of the ksqlDB cluster.

FIELDS:
...
   mountedVolumes       <Object>
     mountedVolumes list the custom volumes that need to be mounted into the
     underlying statefulset. A change to this setting will roll the cluster.
...

Next take a look at the mounted volumes section
kubectl explain KsqlDB.spec.mountedVolumes

KIND:     KsqlDB
VERSION:  platform.confluent.io/v1beta1

RESOURCE: mountedVolumes <Object>

DESCRIPTION:
     mountedVolumes list the custom volumes that need to be mounted into the
     underlying statefulset. A change to this setting will roll the cluster.

FIELDS:
   volumeMounts <[]Object> -required-
     volumeMounts specify the list of volume mounts for the pods in the
     statefulset.

   volumes      <[]Object> -required-
     volumes specify the list of volumes that can be mounted into the pods of
     statefulset.

We have the option to mount a volume in the ksqldb pod. So we can create a config map from our sql file and mount it to the sqldb pod. I’m building a helm chart to deploy but it would work as yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: ksqldb-ksql-queries-configmap
  namespace: {{ .Values.namespace }}
data:
  {{- $files := .Files }}
  {{- range tuple "queries.sql" }}
  {{ . }}: |-
{{ $files.Get . | indent 4 }}
  {{- end }}

Now I can configure my ksql template to include the config map

apiVersion: platform.confluent.io/v1beta1
kind: KsqlDB
metadata:
  name: ksqldb
  namespace: {{ .Values.namespace }}
spec:
  replicas: {{ .Values.ksqldb.replicas }}
  image:
    application: confluentinc/cp-ksqldb-server:{{ .Values.version }}
    init: confluentinc/confluent-init-container:2.3.0
  configOverrides:
    server:
      - access.control.allow.origin=*
      - access.control.allow.methods=GET,POST,HEAD
      - access.control.allow.headers=X-Requested-With,Content-Type,Accept,Origin,Authorization
      - ksql.queries.file=/etc/ksql/queries/queries.sql
  dataVolumeCapacity: 10Gi
  mountedVolumes:
    volumeMounts:
      - name: ksql-queries
        mountPath: /etc/ksql/queries
    volumes:
      - name: ksql-queries
        configMap:
          name: ksqldb-ksql-queries-configmap

Note the config override - ksql.queries.file=/etc/ksql/queries/queries.sql that tells ksqldb to use the file provided in the volume to set initial queries.

4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.