Config setup issues

Ops
1

Hi,

We had issues on producer not sending messages and we tried a lot of solutions which did not work. Finally, we thought to install kafka on single node but the cluster settings is overriding the single node settings.

OS:- Centos 7
kafka version 2.12-3.0.0

cluster size 2
zookeeper runs on 3 nodes

we try to run the
~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties

In server.properties , we changed the
zookeeper.connect=localhost:2181

But we get SocketServer BrokerId Unexpected error from IP address, the broker is of the other node which should not be in contention while running in single node.

Also what is the command to uninstall kafka in linux.

2

hi @sajid

could eleaborate more?
my understanding so far:
you’re running 3 nodes:

  • 2 of them running kafka
  • 3 of the running zookeeper, right?

depends on how you’ve installed the stack.
did you use zip/tarball installation?

best,
michael

3

We have kafka running on 2 nodes. Kafka monitor running on 3 rd node.

Zookeeper is running in all the three systems.

Yes we used zip to install kafka, earlier I assume it was installed using ansible playbook. As we were not able to uninstall it, we deleted the kafka folder in server and reinstalled it using unzipping the downloaded file from apache website

4

ok I see

and how does you config look like?
server.properties and zookeeper.properties would good to see.

best,
michael

5

Kafka server properties of the current installation

broker.id=0

num.network.threads=3

num.io.threads=8

socket.send.buffer.bytes=102400

socket.receive.buffer.bytes=102400

socket.request.max.bytes=104857600

log.dirs=/tmp/kafka-logs

num.partitions=1

num.recovery.threads.per.data.dir=1

offsets.topic.replication.factor=1

transaction.state.log.replication.factor=1

transaction.state.log.min.isr=1

log.retention.hours=168

log.segment.bytes=1073741824

log.retention.check.interval.ms=300000

zookeeper.connect=localhost:2181

zookeeper.connection.timeout.ms=18000

group.initial.rebalance.delay.ms=0

Zookeeper properties

dataDir=/tmp/zookeeper

clientPort=2181

maxClientCnxns=0

admin.enableServer=false

6

if we run the zookeper and kafka services from

~/kafka/bin/zookeeper-server-start.sh ~/kafka/config/zookeeper.properties
~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties

The zookeeper runs perfectly whereas Kafka shows warning {socket Server BrokerID ] Unexpected error from I/p address.Closing connection

The ip address is not from the server on which kafka was installed.

7

if you’re running a multi node zookeeper setup you zk config should look more like:

tickTime=2000
dataDir=/var/lib/zookeeper/
clientPort=2181
initLimit=5
syncLimit=2
server.1=zoo1:2888:3888
server.2=zoo2:2888:3888
server.3=zoo3:2888:3888
autopurge.snapRetainCount=3
autopurge.purgeInterval=24

and zookeeper connect in server.properties should be like
zookeeper.connect=zoo1:2181,zoo2:2181,zoo3:2181

see

https://docs.confluent.io/platform/current/installation/installing_cp/deb-ubuntu.html#manual-install-using-systemd-on-ubuntu-and-debian
for a howto

best,
michael

8

Yeah Michael,

But we tried to do a standalone installation for a single node,by deleting the earlier configurations.

We did edited the hosts file and has only 1 ip for both kafka and zookeeper. However, the kafka is looking for other nodes and is not working as a single node. We are unsure how the properties are getting overidden, as similar process when ran on personal laptop, it did setup Kafka correctly and producer was able to transmit messages and consumer was able to receive as well

The same setting on server is not working correctly.

9

ok I see
my understanding was that you have a multi node setup in place
nevertheless:
did you check if there is a some config files in /etc/kafka/ ?

I think there is s.th left

best,
michael