Pull Query failing from partitioned Materialised table

Hi,

We are currently running an evaluation against the Confluent Community Platform and have run in to a snag when trying to run a pull query against a partitioned materialised table, it is failing to return results. However even this seems inconsistent though, as it will return if you have a paritioned materialised table that is using a single partitioned stream to provide the data. Here should be some steps that will allow the issue to be replicated;

kafka-topics --create --topic source-topic-1 --bootstrap-server localhost:9092 --config cleanup.policy=compact --config segment.ms=86400000 --partitions 1 --replication-factor 3
kafka-topics --create --topic source-topic-6 --bootstrap-server localhost:9092 --config cleanup.policy=compact --config segment.ms=86400000 --partitions 6 --replication-factor 3
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"type\":\"record\",\"name\":\"sourceConfig\",\"namespace\":\"example\",\"fields\":[{\"name\":\"value\",\"type\":\"string\"}]}"}' http://localhost:8081/subjects/source-topic-1-value/versions
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"type\":\"record\",\"name\":\"sourceConfig\",\"namespace\":\"example\",\"fields\":[{\"name\":\"value\",\"type\":\"string\"}]}"}' http://localhost:8081/subjects/source-topic-6-value/versions
 CREATE STREAM `source_topic_1_stream`(
>    `key` string KEY,
>    `value` string
>) WITH (
>    kafka_topic='source-topic-1',
>    value_format='avro'
>);

 CREATE STREAM `source_topic_6_stream`(
>    `key` string KEY,
>    `value` string
>) WITH (
>    kafka_topic='source-topic-6',
>    value_format='avro'
>);

CREATE TABLE `source_pull_topic_1_from_1` WITH (KAFKA_TOPIC='source-pull-topic-1', PARTITIONS=1, REPLICAS=3, VALUE_FORMAT='avro') AS SELECT  `key`,   LATEST_BY_OFFSET(`value`) `value` FROM `source_topic_1_stream` GROUP BY `key` EMIT CHANGES;

CREATE TABLE `source_pull_topic_6_from_1` WITH (KAFKA_TOPIC='source-pull-topic-1', PARTITIONS=6, REPLICAS=3, VALUE_FORMAT='avro') AS SELECT  `key`,   LATEST_BY_OFFSET(`value`) `value` FROM `source_topic_1_stream` GROUP BY `key` EMIT CHANGES;

CREATE TABLE `source_pull_topic_1_from_6` WITH (KAFKA_TOPIC='source-pull-topic-6', PARTITIONS=1, REPLICAS=3, VALUE_FORMAT='avro') AS SELECT  `key`,   LATEST_BY_OFFSET(`value`) `value` FROM `source_topic_6_stream` GROUP BY `key` EMIT CHANGES;

CREATE TABLE `source_pull_topic_6_from_6` WITH (KAFKA_TOPIC='source-pull-topic-6', PARTITIONS=6, REPLICAS=3, VALUE_FORMAT='avro') AS SELECT  `key`,   LATEST_BY_OFFSET(`value`) `value` FROM `source_topic_6_stream` GROUP BY `key` EMIT CHANGES;

As best as I can tell there is no CLI producer to insert in to an AVRO topic, however I suspect that this can just be done in KSQL we have a python script where we are doing it, which I am not in a position to share. However I would recommend inserting at least 6 records in to each source topic.

When selecting a push query from any of the materialised tables, all data is returned as expected. When select a pull query however, either in KSQL or over the KSQL API, selecting the data from either of the materialised tables that user source-topic-1 as the object they are materialising from, again everything works as expected. However when attempting to do a pull query against the materialised tables that have been built on top of source-topic-6 this is where the pull query stops working.

If you try and select a key that is stored on a partition that is not on the same node as KSQL, you just get an empty response, where it is failing to find the record. If you query for a key that is on the same node as KSQL, you get the following message in KSQL;

 ERROR Exhausted standby hosts to try. (io.confluent.ksql.cli.console.Console:345)
Exhausted standby hosts to try.
Query terminated

Which maybe missleading as the following exception is recorded in the KSQL log which will follow in the comments as I have exceeded the char count.

We are running a three node cluster, the current (7.0.0) version of the confluent community platform. The VM’s that are running the cluster have been hardened so we have set various ENV variables to repoint the /tmp directories for the likes of snappy etc.
Any pointers as to what could be causing the problem would be really helpfull in moving us forwards, unless we have hit an actual bug with the platform? If anything is unclear please let me know and I will try and clarify.

Thanks in advance,

Leander

Nov 15 12:27:15 boxname ksql-server-start[service-id]: [2021-11-15 12:27:15,772] ERROR Error while handling chunk (io.confluent.ksql.rest.client.KsqlTarget)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: io.confluent.ksql.rest.client.KsqlRestClientException: Failed to deserialise object
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlClientUtil.deserialize(KsqlClientUtil.java:53)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.toRows(KsqlTarget.java:454)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.lambda$postQueryRequest$3(KsqlTarget.java:197)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.lambda$null$11(KsqlTarget.java:310)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.HttpClientResponseImpl.handleChunk(HttpClientResponseImpl.java:232)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$beginResponse$0(Http1xClientConnection.java:486)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:237)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:127)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleChunk(Http1xClientConnection.java:322)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.access$1900(Http1xClientConnection.java:242)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection.handleResponseChunk(Http1xClientConnection.java:631)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:601)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:577)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:43)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:229)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:164)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.lang.Thread.run(Thread.java:834)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: Caused by: com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input: was expecting closing '"' for name
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at [Source: (byte[])"{"header":{""; line: 1, column: 13] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["header"])
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:390)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:349)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1822)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:567)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:449)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:362)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:195)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4593)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3609)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlClientUtil.deserialize(KsqlClientUtil.java:51)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: ... 39 more
Nov 15 12:27:15 boxname ksql-server-start[service-id]: Caused by: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: was expecting closing '"' for name
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at [Source: (byte[])"{"header":{""; line: 1, column: 13]
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:662)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.slowParseName(UTF8StreamJsonParser.java:1918)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1709)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:766)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:191)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.std.ReferenceTypeDeserializer.deserialize(ReferenceTypeDeserializer.java:197)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:542)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:565)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: ... 47 more
Nov 15 12:27:15 boxname ksql-server-start[service-id]: [2021-11-15 12:27:15,776] ERROR Error while handling chunk (io.confluent.ksql.rest.client.KsqlTarget)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: io.confluent.ksql.rest.client.KsqlRestClientException: Failed to deserialise object
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlClientUtil.deserialize(KsqlClientUtil.java:53)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.toRows(KsqlTarget.java:454)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.lambda$postQueryRequest$3(KsqlTarget.java:197)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.lambda$null$11(KsqlTarget.java:310)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.HttpClientResponseImpl.handleChunk(HttpClientResponseImpl.java:232)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$beginResponse$0(Http1xClientConnection.java:486)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:237)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:127)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleChunk(Http1xClientConnection.java:322)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.access$1900(Http1xClientConnection.java:242)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection.handleResponseChunk(Http1xClientConnection.java:631)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:601)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:577)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:43)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:229)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:164)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:432)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.lang.Thread.run(Thread.java:834)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'queryId': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at [Source: (byte[])"queryId":"query_1636979235722","schema":"`key` STRING KEY, `value` STRING"}}"; line: 1, column: 9]
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2337)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:720)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3593)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2688)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:870)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:762)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4684)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4586)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3609)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlClientUtil.deserialize(KsqlClientUtil.java:51)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: ... 41 more
Nov 15 12:27:15 boxname ksql-server-start[service-id]: [2021-11-15 12:27:15,776] WARN Error forwarding query to node http://apl00342i2.il2management.dev:8088/. Falling back to standby state which may return stale results (io.confluent.ksql.physical.pull.HARouting)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: io.confluent.ksql.rest.client.KsqlRestClientException: Error issuing POST to KSQL server. path:/query
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.executeSync(KsqlTarget.java:388)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.executeRequestSync(KsqlTarget.java:305)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.post(KsqlTarget.java:271)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.postQueryRequest(KsqlTarget.java:192)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.server.services.DefaultKsqlClient.makeQueryRequest(DefaultKsqlClient.java:129)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.physical.pull.HARouting.forwardTo(HARouting.java:349)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.physical.pull.HARouting.executeOrRouteQuery(HARouting.java:308)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.physical.pull.HARouting.lambda$executeRounds$5(HARouting.java:191)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.lang.Thread.run(Thread.java:834)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: Caused by: java.util.concurrent.ExecutionException: io.confluent.ksql.rest.client.KsqlRestClientException: Failed to deserialise object
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.executeSync(KsqlTarget.java:386)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: ... 11 more
Nov 15 12:27:15 boxname ksql-server-start[service-id]: Caused by: io.confluent.ksql.rest.client.KsqlRestClientException: Failed to deserialise object
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlClientUtil.deserialize(KsqlClientUtil.java:53)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.toRows(KsqlTarget.java:454)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.lambda$postQueryRequest$3(KsqlTarget.java:197)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.lambda$null$11(KsqlTarget.java:310)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.HttpClientResponseImpl.handleChunk(HttpClientResponseImpl.java:232)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$beginResponse$0(Http1xClientConnection.java:486)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:237)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:127)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleChunk(Http1xClientConnection.java:322)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.access$1900(Http1xClientConnection.java:242)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection.handleResponseChunk(Http1xClientConnection.java:631)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:601)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:577)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:43)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:229)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:164)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: ... 1 more
Nov 15 12:27:15 boxname ksql-server-start[service-id]: Caused by: com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input: was expecting closing '"' for name
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at [Source: (byte[])"{"header":{""; line: 1, column: 13] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["header"])
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:390)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:349)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1822)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:567)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:449)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:362)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:195)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4593)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3609)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlClientUtil.deserialize(KsqlClientUtil.java:51)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: ... 39 more
Nov 15 12:27:15 boxname ksql-server-start[service-id]: Caused by: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: was expecting closing '"' for name
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at [Source: (byte[])"{"header":{""; line: 1, column: 13]
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:662)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.slowParseName(UTF8StreamJsonParser.java:1918)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1709)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:766)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:191)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.std.ReferenceTypeDeserializer.deserialize(ReferenceTypeDeserializer.java:197)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:542)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:565)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: ... 47 more
Nov 15 12:27:15 boxname ksql-server-start[service-id]: [2021-11-15 12:27:15,823] ERROR Error while handling chunk (io.confluent.ksql.rest.client.KsqlTarget)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: io.confluent.ksql.rest.client.KsqlRestClientException: Failed to deserialise object
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlClientUtil.deserialize(KsqlClientUtil.java:53)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.toRows(KsqlTarget.java:454)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.lambda$postQueryRequest$3(KsqlTarget.java:197)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.lambda$null$11(KsqlTarget.java:310)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.HttpClientResponseImpl.handleChunk(HttpClientResponseImpl.java:232)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$beginResponse$0(Http1xClientConnection.java:486)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:237)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:127)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleChunk(Http1xClientConnection.java:322)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.access$1900(Http1xClientConnection.java:242)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection.handleResponseChunk(Http1xClientConnection.java:631)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:601)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:577)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:43)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:229)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:164)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.lang.Thread.run(Thread.java:834)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: Caused by: com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input: was expecting closing '"' for name
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at [Source: (byte[])"{"header":{""; line: 1, column: 13] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["header"])
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:390)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:349)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1822)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:567)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:449)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:362)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:195)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4593)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3609)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlClientUtil.deserialize(KsqlClientUtil.java:51)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: ... 39 more
Nov 15 12:27:15 boxname ksql-server-start[service-id]: Caused by: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: was expecting closing '"' for name
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at [Source: (byte[])"{"header":{""; line: 1, column: 13]
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:662)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.slowParseName(UTF8StreamJsonParser.java:1918)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1709)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:766)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:191)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.std.ReferenceTypeDeserializer.deserialize(ReferenceTypeDeserializer.java:197)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:542)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:565)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: ... 47 more
Nov 15 12:27:15 boxname ksql-server-start[service-id]: [2021-11-15 12:27:15,824] ERROR Error while handling chunk (io.confluent.ksql.rest.client.KsqlTarget)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: io.confluent.ksql.rest.client.KsqlRestClientException: Failed to deserialise object
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlClientUtil.deserialize(KsqlClientUtil.java:53)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.toRows(KsqlTarget.java:454)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.lambda$postQueryRequest$3(KsqlTarget.java:197)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.lambda$null$11(KsqlTarget.java:310)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.HttpClientResponseImpl.handleChunk(HttpClientResponseImpl.java:232)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$beginResponse$0(Http1xClientConnection.java:486)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:237)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:127)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleChunk(Http1xClientConnection.java:322)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.access$1900(Http1xClientConnection.java:242)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection.handleResponseChunk(Http1xClientConnection.java:631)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:601)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:577)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:43)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:229)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:164)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:432)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.lang.Thread.run(Thread.java:834)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'queryId': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at [Source: (byte[])"queryId":"query_1636979235844","schema":"`key` STRING KEY, `value` STRING"}}"; line: 1, column: 9]
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2337)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:720)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3593)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2688)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:870)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:762)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4684)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4586)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3609)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlClientUtil.deserialize(KsqlClientUtil.java:51)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: ... 41 more
Nov 15 12:27:15 boxname ksql-server-start[service-id]: [2021-11-15 12:27:15,824] WARN Error forwarding query to node http://apl00343i2.il2management.dev:8088/. Falling back to standby state which may return stale results (io.confluent.ksql.physical.pull.HARouting)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: io.confluent.ksql.rest.client.KsqlRestClientException: Error issuing POST to KSQL server. path:/query
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.executeSync(KsqlTarget.java:388)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.executeRequestSync(KsqlTarget.java:305)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.post(KsqlTarget.java:271)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.postQueryRequest(KsqlTarget.java:192)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.server.services.DefaultKsqlClient.makeQueryRequest(DefaultKsqlClient.java:129)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.physical.pull.HARouting.forwardTo(HARouting.java:349)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.physical.pull.HARouting.executeOrRouteQuery(HARouting.java:308)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.physical.pull.HARouting.lambda$executeRounds$5(HARouting.java:191)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.lang.Thread.run(Thread.java:834)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: Caused by: java.util.concurrent.ExecutionException: io.confluent.ksql.rest.client.KsqlRestClientException: Failed to deserialise object
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.executeSync(KsqlTarget.java:386)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: ... 11 more
Nov 15 12:27:15 boxname ksql-server-start[service-id]: Caused by: io.confluent.ksql.rest.client.KsqlRestClientException: Failed to deserialise object
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlClientUtil.deserialize(KsqlClientUtil.java:53)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.toRows(KsqlTarget.java:454)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.lambda$postQueryRequest$3(KsqlTarget.java:197)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlTarget.lambda$null$11(KsqlTarget.java:310)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.HttpClientResponseImpl.handleChunk(HttpClientResponseImpl.java:232)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$beginResponse$0(Http1xClientConnection.java:486)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:237)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:127)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleChunk(Http1xClientConnection.java:322)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.access$1900(Http1xClientConnection.java:242)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection.handleResponseChunk(Http1xClientConnection.java:631)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:601)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:577)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:43)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:229)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:164)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: ... 1 more
Nov 15 12:27:15 boxname ksql-server-start[service-id]: Caused by: com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input: was expecting closing '"' for name
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at [Source: (byte[])"{"header":{""; line: 1, column: 13] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["header"])
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:390)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:349)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1822)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:567)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:449)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:362)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:195)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4593)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3609)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.rest.client.KsqlClientUtil.deserialize(KsqlClientUtil.java:51)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: ... 39 more
Nov 15 12:27:15 boxname ksql-server-start[service-id]: Caused by: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: was expecting closing '"' for name
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at [Source: (byte[])"{"header":{""; line: 1, column: 13]
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:662)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.slowParseName(UTF8StreamJsonParser.java:1918)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1709)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:766)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:191)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.std.ReferenceTypeDeserializer.deserialize(ReferenceTypeDeserializer.java:197)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:542)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:565)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: ... 47 more
Nov 15 12:27:15 boxname ksql-server-start[service-id]: [2021-11-15 12:27:15,833] ERROR Exception occurred while writing to connection stream:  (io.confluent.ksql.rest.server.resources.streaming.PullQueryStreamWriter)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: io.confluent.ksql.execution.streams.materialization.MaterializationException: Exhausted standby hosts to try.
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.physical.pull.HARouting.groupByHost(HARouting.java:243)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.physical.pull.HARouting.executeRounds(HARouting.java:183)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at io.confluent.ksql.physical.pull.HARouting.lambda$handlePullQuery$4(HARouting.java:146)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Nov 15 12:27:15 boxname ksql-server-start[service-id]: at java.base/java.lang.Thread.run(Thread.java:834)

How does our Python script partition the data? Note that different clients may use different default partitioning, however, ksqlDB expects data to be partitions using murmur2 hashing on the key (the official Java client partitions data this way).

Does you pull query return data if you do SELECT * FROM table ?

The python script shouldnt be partitioning the data, it uses the AVRO python library create an Avro Producer that sends the data. My understanding from what I read is that providing you don’t apecify any manual partitioning, Kafka will utilise the default strategy to partition the data, which I think from memory is based on a hash of the key. Not ruling out that the issue isnt the python kafka library as it wouldnt be the first problem that I have come across using it, but I would expect the issue to be more prevalent if that’s the case? I will have a look at the library though to see if I can work out what partitioning startegy its using.

In terms of the query outlined above, my understanding is that doesnt result in a pull query, a pull query needs to be restricted by passing in a where clause based on the timestamp or the key, so that query outlined would just return an error. With an “emit changes” on the end turning it in to a push query, returns the materialised data set i.e. with duplicate keyed records removed.

Thanks

Hi,

Just double checked and there are no partition parameters specified as part of the AvroProducer;

https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#avro-producer

I have also had a quick scan of the code and again there is no obvious overwriting of the partition strategy;

confluent-kafka-python/src/confluent_kafka/avro at a5663da7ea76e58d02b13e4e6703ea6a9c52ec11 · confluentinc/confluent-kafka-python · GitHubinit.py#L33

I would assume that this means that it will just be relying on the default partitioning strategy.

Interestingly enough there does now appear to be a SerialiserProducer that I hadnt spotted before that does have a partitioning parameter;

https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#serde-producer (cant add more than 2 links)

But that seems to be pass in the partition you want, or again just use the behind the scenes default. Also not entire sure whether or not this is live as the title seems to suggest that the API is subject to change.

Thanks,

Leander

Yes, but the Python client might to it differently then the Java client… The “default strategy” is not defined by “Kafka”, but each client can implement its own default strategy. And even if two clients pick key-based hash partitioning, they could use a different hash function and thus partition the data differently.

As you point to Confluent docs, ie, you are using the Confluent Python client: I know that it uses a different partitioning compared to the Java client. You need to tell it to use murmur2 (cf Kafka different partitions are selected python confluent_kafka library v/s apache kafka Java - Stack Overflow)

In terms of the query outlined above, my understanding is that doesnt result in a pull query, a pull query needs to be restricted by passing in a where clause based on the timestamp or the key,

Not in newer versions of ksqlDB.

Hi,

Thanks, having set the partitionor to murmu2_random has resolved the issue with the pull queries, both where the data is and isnt in a partition that is on the same node as KSQL.

I have also (within the KSQL session) set ksql.query.pull.table.scan.enabled = true, which would allow for a full table pull query. this seems to work ok, where I have very few events to be returned but if there are more than a handfull or events it will return “Exhausted standby hosts to try.” is this related to “max.task.idle.ms” setting you mentioned on my other post?

I did have a look at what this value is meant to do and I didnt understand the explanation. At present this value is set to 0 (the default) and I am not sure how I am meant to work out what I should set this to instead, is it just a case of incrementing it by trail and error?

I am planning on marking your previous post as the solution, but I wasnt sure if that was then going to lock this post and stop me from asking about the max.task.idle.ms value

Thanks for all your help so far,

Leander

No. The config has nothing to do with pull queries.

Having had a look through the log the issue is related to the chunking off the response from the primary KSQL node. New thread started here;

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.