Broker checkpoint .tmp files stuck

One of our Broker disks ran out of space and went offline. The broker is back online but it appears 2 of the checkpoint files got stuck while out of space, and could not rollover the tmp files:

kafka/kafka-logs
recovery-point-offset-checkpoint
log-start-offset-checkpoint
recovery-point-offset-checkpoint.tmp
log-start-offset-checkpoint.tmp

Should I manually rename the tmp files or is there another recommended action?

Thanks

Hi @b3cops

welcome :slight_smile:

did you get any errors during startup regarding the tmp files?
did you check the file size?

some insights regarding the files

hth.
michael

Thanks for the info.

I did miss a startup error for this disk:

Malformed line in checkpoint file (/hadoop/kafka/kafka-logs/log-start-offset-checkpoint)

The log-start-offset-checkpoint does look malformed with many “@@@@” and no text.

The log-start-offset-checkpoint.tmp looks valid.

hmm ok I see
without seeing everything I would try to rename the problematic file
and start over again.

best,
michael