Determine which port is used - `java.net.SocketException: Invalid argument`

Hello.

I use a confluent 5.5.0 and run all components on the same box. All components work fine, except kafka: it writes to log such messages:

ERROR Error while accepting connection (kafka.network.Acceptor)
java.net.SocketException: Invalid argument
at sun.nio.ch.Net.setIntOption0(Native Method)
at sun.nio.ch.Net.setSocketOption(Net.java:341)
at sun.nio.ch.SocketChannelImpl.setOption(SocketChannelImpl.java:190)
at sun.nio.ch.SocketAdaptor.setBooleanOption(SocketAdaptor.java:271)
at sun.nio.ch.SocketAdaptor.setTcpNoDelay(SocketAdaptor.java:306)
at kafka.network.Acceptor.accept(SocketServer.scala:654)
at kafka.network.Acceptor.run(SocketServer.scala:579)
at java.lang.Thread.run(Thread.java:748)

and tries to create TCP socket with port number “0”. In netstat is looks like this

. . kafka 63549 java 0 0 104256 0 IDLE

in more detail this looks likes this

817: S_IFSOCK mode:0666 dev:16377,0 ino:15910 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK
sockname: AF_INET 0.0.0.0 port: 0
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(256000),SO_RCVBUF(104256)
congestion control: newreno

and time by time counts of this face TCP sockets only increases. I have a limit for number for open files by process. And when kafka reaches this limit - its killed by operation system. Increasing limits it is not solution, because it reaches anyway, may be some time later.

Hi,

could you provide some details about your environment?
how did you start the stack?
which os?
what are your open file limit settings?

It starts in next sequence: zookeeper, kafka, ksql, kafka-rest, control-center, schema-registry
OS - Oracle Solaris 11.4
Limit 260000

ok should be enough for a demo env

what happens if you only start zookeeper and kafka?
same errors?

Yes, the same errors.
I think, it’s related to zookeeper, but i can’t understand, how? Some time ago, I tried to set variable JMX_PORT at bin/kafka-run-class and zookeeper started, but the kafka - didn’t. Error was little other, such as “can’t use port”. When i disabled JMX_PORT kafka started. So, zookeeper and kafka use the same environment and same predefined variables, like JMX_PORT.

zookeeper use only this ports (via netstat command): 2181, 49146. The first - is standard, the second - randomly setted. In my mind, kafka wants to run zookeeper by itself, not separated. I also tried not to run zookeeper, but in this case kafka couldn’t run, because it couldn’t connect to zookeeper.

hmm ok understand

so if my understanding is right the issue is related to JMX ports?

starting kafka and zookeeper without jmx works?

Not exactly.
JMX_PORT is an example (may be they use another variable at the same time), which shows, that zookeeper and kafka try to use the same port:

  • with JMX_PORT kafka cann’t run
  • without JMX_PORT (with default options in bin/kafka-run-class) kafka runs, but write to logs errors, that i described in first message.

Do you understand, what I mean?

ok it’s getting clearer :wink:

did you monitor your open files during the error?

Yes, but they are thousands of files. I don’t know, which of them to monitor closer.

sorry for the misunderstanding

was thinking about the open files of the kafka process
lsof|grep kafka|wc -l

Because, it’s Oracle Solaris, there is no command lsof. Instead I use pfiles, which show all files using by the process
Number of regular files (it’s not growing):

# pfiles 63549 | grep S_IFREG | wc -l
704

Number of fake sockets with port “0”:

# pfiles 63549|grep 'sockname: AF_INET 0.0.0.0  port: 0'|wc -l;sleep 30;pfiles 63549|grep 'sockname: AF_INET 0.0.0.0  port: 0'|wc -l
36002
36007

It increased on 5 new fake socket during 30 seconds.

hmm ok

which java version is in place?

and did you try with a newer Confluent/Kafka version?

1 Like