Recently, inside Bloomberg, we were in process of upgrading KafkaStreams to 2.8.1 and discovered a logging message spouted by KafkaStreams at error level which started showing up in normal flow of shutting down our servers for maintenance every weekend. Generally our telemetry systems will flag such messages emitted at level of ERROR. It is bit of a pain for our team to start looking into such messages which are not actual errors.
The change since 2.5.1 that does this is in file KafkaStreams.java
in method public synchronized void onChange(final Thread thread, final ThreadStateTransitionValidator abstractNewState, final ThreadStateTransitionValidator abstractOldState).
This code block in 2.8.1 and later:
else if (thread instanceof GlobalStreamThread) {
// global stream thread has different invariants
final GlobalStreamThread.State newState = (GlobalStreamThread.State) abstractNewState;
globalThreadState = newState;
if (newState == GlobalStreamThread.State.RUNNING) {
maybeSetRunning();
} else if (newState == GlobalStreamThread.State.DEAD) {
log.error("Global thread has died. The streams application or client will now close to ERROR.");
closeToError();
}
}
was having a check before log.error("Global thread has died. The streams application or client will now close to ERROR.");
It used to be like this before:
if (setState(State.DEAD)) {
log.error("Global thread has died. The streams application or client will now close to ERROR.");
Under normal conditions of shutdown setState(State.DEAD)
will return false and the error message would not be printed. This is no longer the case as we always see the error message spouted.
Please take a look at this and let us know of steps. I can also fix this by creating a PR to 2.8.1 and latest trunk.