Determine which port is used - `java.net.SocketException: Invalid argument`

Ops
1

Hello.

I use a confluent 5.5.0 and run all components on the same box. All components work fine, except kafka: it writes to log such messages:

ERROR Error while accepting connection (kafka.network.Acceptor)
java.net.SocketException: Invalid argument
at sun.nio.ch.Net.setIntOption0(Native Method)
at sun.nio.ch.Net.setSocketOption(Net.java:341)
at sun.nio.ch.SocketChannelImpl.setOption(SocketChannelImpl.java:190)
at sun.nio.ch.SocketAdaptor.setBooleanOption(SocketAdaptor.java:271)
at sun.nio.ch.SocketAdaptor.setTcpNoDelay(SocketAdaptor.java:306)
at kafka.network.Acceptor.accept(SocketServer.scala:654)
at kafka.network.Acceptor.run(SocketServer.scala:579)
at java.lang.Thread.run(Thread.java:748)

and tries to create TCP socket with port number “0”. In netstat is looks like this

. . kafka 63549 java 0 0 104256 0 IDLE

in more detail this looks likes this

817: S_IFSOCK mode:0666 dev:16377,0 ino:15910 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK
sockname: AF_INET 0.0.0.0 port: 0
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(256000),SO_RCVBUF(104256)
congestion control: newreno

and time by time counts of this face TCP sockets only increases. I have a limit for number for open files by process. And when kafka reaches this limit - its killed by operation system. Increasing limits it is not solution, because it reaches anyway, may be some time later.

2

Hi,

could you provide some details about your environment?
how did you start the stack?
which os?
what are your open file limit settings?

3

It starts in next sequence: zookeeper, kafka, ksql, kafka-rest, control-center, schema-registry
OS - Oracle Solaris 11.4
Limit 260000

4

ok should be enough for a demo env

what happens if you only start zookeeper and kafka?
same errors?

5

Yes, the same errors.
I think, it’s related to zookeeper, but i can’t understand, how? Some time ago, I tried to set variable JMX_PORT at bin/kafka-run-class and zookeeper started, but the kafka - didn’t. Error was little other, such as “can’t use port”. When i disabled JMX_PORT kafka started. So, zookeeper and kafka use the same environment and same predefined variables, like JMX_PORT.

zookeeper use only this ports (via netstat command): 2181, 49146. The first - is standard, the second - randomly setted. In my mind, kafka wants to run zookeeper by itself, not separated. I also tried not to run zookeeper, but in this case kafka couldn’t run, because it couldn’t connect to zookeeper.

6

hmm ok understand

so if my understanding is right the issue is related to JMX ports?

starting kafka and zookeeper without jmx works?

7

Not exactly.
JMX_PORT is an example (may be they use another variable at the same time), which shows, that zookeeper and kafka try to use the same port:

  • with JMX_PORT kafka cann’t run
  • without JMX_PORT (with default options in bin/kafka-run-class) kafka runs, but write to logs errors, that i described in first message.

Do you understand, what I mean?

8

ok it’s getting clearer :wink:

did you monitor your open files during the error?

9

Yes, but they are thousands of files. I don’t know, which of them to monitor closer.

10

sorry for the misunderstanding

was thinking about the open files of the kafka process
lsof|grep kafka|wc -l

11

Because, it’s Oracle Solaris, there is no command lsof. Instead I use pfiles, which show all files using by the process
Number of regular files (it’s not growing):

# pfiles 63549 | grep S_IFREG | wc -l
704

Number of fake sockets with port “0”:

# pfiles 63549|grep 'sockname: AF_INET 0.0.0.0  port: 0'|wc -l;sleep 30;pfiles 63549|grep 'sockname: AF_INET 0.0.0.0  port: 0'|wc -l
36002
36007

It increased on 5 new fake socket during 30 seconds.

12

hmm ok

which java version is in place?

and did you try with a newer Confluent/Kafka version?

1 Like
13

Sorry for long reply. I found solution. It’s a bug, that related to setsockopt TCP_NODELAY (in java terms it’s a setTcpNoDelay), and java can’t set it.

https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6799574

After set xnet_skip_checks=1 everything became OK.
Please close the topic

1 Like
14

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.