Kafka Streams logging spurious error on GlobalStream thread shutdown in versions later than 2.5.1

Recently, inside Bloomberg, we were in process of upgrading KafkaStreams to 2.8.1 and discovered a logging message spouted by KafkaStreams at error level which started showing up in normal flow of shutting down our servers for maintenance every weekend. Generally our telemetry systems will flag such messages emitted at level of ERROR. It is bit of a pain for our team to start looking into such messages which are not actual errors.

The change since 2.5.1 that does this is in file KafkaStreams.java in method public synchronized void onChange(final Thread thread, final ThreadStateTransitionValidator abstractNewState, final ThreadStateTransitionValidator abstractOldState).

This code block in 2.8.1 and later:

else if (thread instanceof GlobalStreamThread) {
                    // global stream thread has different invariants
                    final GlobalStreamThread.State newState = (GlobalStreamThread.State) abstractNewState;
                    globalThreadState = newState;


                    if (newState == GlobalStreamThread.State.RUNNING) {
                        maybeSetRunning();
                    } else if (newState == GlobalStreamThread.State.DEAD) {
                        log.error("Global thread has died. The streams application or client will now close to ERROR.");
                        closeToError();
                    }
                }

was having a check before log.error("Global thread has died. The streams application or client will now close to ERROR.");
It used to be like this before:

if (setState(State.DEAD)) {
log.error("Global thread has died. The streams application or client will now close to ERROR.");

Under normal conditions of shutdown setState(State.DEAD) will return false and the error message would not be printed. This is no longer the case as we always see the error message spouted.

Please take a look at this and let us know of steps. I can also fix this by creating a PR to 2.8.1 and latest trunk.

3 Likes

There is already a Jira for it and we are working on a fix already: [KAFKA-13423] Error falsely reported on Kafka Streams app when GlobalKTables are used - ASF JIRA

2 Likes

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.