Connector def for Beats source logs from Kafka to Elastic

Kafka Connect
1

Can’t imagine no one has gotten this working.

I want to publish system logs (they all have the same standard structure, so once one is working, it’s just configuring topics and indeces), published onto a Kafka topic by Beats Framework to ElasticSearch.

Anyone have the kSQLDB call or RestAPI call to handle these streams ?

G

2

Hi @georgelza

I’m not totally clear on your question. Are you looking to process system logs that are already published on a Kafka topic, or are you looking to ingest data from Filebeats?

If you are looking to ingest, then Filebeats can send data directly to Kafka (Configure the Kafka output | Filebeat Reference [8.4] | Elastic)

If you are looking to process existing data on Kafka, could you be more specific on your question as what you are looking to accomplish?

Rick

3

Hi Rick

I’m looking for the kSQL call to configure/create a sink connector to push data into ElasticSearch.
for logs like Syslog logs, they like error.log, login.log, https.log all use the same structure.

As it stands now, I’m able to push the data (syslog data) onto kafka topic using filebeat module.

I"ll have a look at the link provided, looked, was using it already.

@rmoff had a syslog block/example, which stopped on the topic, with some ksql processing : We ❤️ syslogs: Real-time syslog Processing with Apache Kafka and KSQL—Part 1: Filtering

This blog … is the source to the sink connector configuration that I’m after.

G

4

You can find the general Elasticsearch Sink Connector here: https://docs.confluent.io/kafka-connect-elasticsearch/current/configuration_options.html. Also, the cp-demo example from confluent runs an elasticsearch connector if you’d like to find a full working example with Confluent Platform: Confluent Platform Demo (cp-demo) | Confluent Documentation.

Another example that might be simpler is here: Clickstream Data Analysis Pipeline Using ksqlDB | Confluent Documentation

Once you have the proper configuration to deploy the connector via the REST API, you can use ksqlDB to manage the connector, here is some docs on that: Connectors - ksqlDB Documentation

1 Like
5

… seen it all, worked though most of this by now.

but thanks.

G

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.