Kafka CA certificate change issue

Ops
1

we need some help regarding ca certificate authority change in kafka . Currently we are connecting in kafka using ssl implementation.

kafka version used is 1.1.1

below is server.properties

listeners=INT://$PVT_HOST_NAME:9094,EXT://$PVT_HOST_NAME:9092
advertised.listeners=INT://$PVT_HOST_NAME:9094,EXT://$PUB_HOST_NAME:9092
ssl.keystore.location=$SSL_DIR/broker.keystore.jks
ssl.keystore.password=changeit
ssl.key.password=changeit
ssl.truststore.location=$SSL_DIR/broker.truststore.jks
SUPER_USERS_CONFIG=$SUPER_USERS_CONFIG"User:CN=br$c.broker.kafka-$CLUSTER_NAME-$ENV,OU=broker,O=server
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,EXT:SSL,INT:PLAINTEXT
inter.broker.listener.name=INT

options tried :-

  1. generating new certificates and updating into existing keystore and truststore (we are observing that client is able to connect using one ca is getting accepted old ca or new ca)
  2. only private keys in keystore and root certs in truststore (we are observing that client is able to connect using one ca is getting accepted old ca or new ca)
  3. muliple keystore and truststore files using comma separated format (client connect is not working at all)
    eg …
    ssl.keystore.location=$SSL_DIR/broker-oldca.keystore.jks,$SSL_DIR/broker-newca.keystore.jks
    ssl.truststore.location=$SSL_DIR/broker-oldca.truststore.jks,broker-newca.truststore.jks

can anyone please help us on this, as this change in authority will cause outage and connection issues with existing clients.

Current Result : only one certificate is working , eaither the old one or new one
Expected Result : both the certificates(keystore & trust store ) should work, old one & new one.

Validation process : After updating the broker certificates we are trying to connect to broker ( from kafka tool) by using consumer certificates.

Thanks.
Naresh

2

hey

what does

openssl s_client -debug -connect $yourhostname:9092 -tls1
say?

3 
 openssl s_client -debug -connect *****:9092 -tls1
 
CONNECTED(00000005)
write to 0x55e3505d4630 [0x55e3505e6bb0] (149 bytes => 149 (0x95))
0000 - 16 03 01 00 90 01 00 00-8c 03 01 f6 86 bb 53 1f   ..............S.
0010 - e4 64 ea e9 fb f3 45 7e-97 94 92 d1 4b 12 39 8d   .d....E~....K.9.
0020 - 07 50 c7 ee 77 6c 9b 1a-e8 a5 56 00 00 12 c0 0a   .P..wl....V.....
0030 - c0 14 00 39 c0 09 c0 13-00 33 00 35 00 2f 00 ff   ...9.....3.5./..
0040 - 01 00 00 51 00 00 00 29-00 27 00 01 24 64 66 2d   ...Q...).'..$***
0050 - 61 70 70 2d 6b 61 66 6b-61 2d 74 65 73 74 2e 6c   *************.co
0070 - 6d 00 0b 00 04 03 00 01-02 00 0a 00 0c 00 0a 00   m...............
0080 - 1d 00 17 00 1e 00 19 00-18 00 23 00 00 00 16 00   ..........#.....
0090 - 00 00 17 00 00                                    .....
read from 0x55e3505d4630 [0x55e3505dd8a3] (5 bytes => 5 (0x5))
0000 - 16 03 01 06 7b                                    ....{
read from 0x55e3505d4630 [0x55e3505dd8a8] (1659 bytes => 1659 (0x67B))
0000 - 02 00 00 51 03 01 61 63-cb 9e d8 d6 d9 a2 45 e4   ...Q..ac......E.
0010 - ee 5a 6e 13 9b 8e cd 97-45 be bc fd cc 52 a8 35   .Zn.....E....R.5
0020 - 1f f2 14 8f fd b5 20 61-63 cb 9e 17 5f 7b 16 e4   ...... ac..._{..
0030 - b7 71 ef 5a ad 94 c0 3f-8a a0 7f b9 e6 86 b0 c6   .q.Z...?........
0040 - 39 41 27 c9 c3 e8 ac c0-14 00 00 09 ff 01 00 01   9A'.............
0050 - 00 00 17 00 00 0b 00 04-a4 00 04 a1 00 04 9e 30   ...............0
0060 - 82 04 9a 30 82 03 82 a0-03 02 01 02 02 14 3c f7   ...0..........<.
0070 - da 9a 03 16 0d a7 67 08-fe 67 97 28 3b 09 82 01   ......g..g.$;...
0080 - aa aa 30 0d 06 09 2a 86-48 86 f7 0d 01 01 0b 05   ..0...*.H.......
0090 - 00 30 21 31 1f 30 1d 06-03 55 04 03 13 16 6c 63   .0!1.0...U....**
00a0 - 67 2d 64 61 74 61 66 61-62 72 69 63 2d 64 65 76   *-**********-dev
00b0 - 2d 70 6b 69 30 1e 17 0d-32 30 30 36 33 30 31 30   -pki0...20063010
00c0 - 32 38 31 35 5a 17 0d 32-33 30 36 32 37 30 32 32   2815Z..230627022
00d0 - 38 34 34 5a 30 48 31 13-30 11 06 03 55 04 0a 13   844Z0H1.0...U...
00e0 - 0a 64 61 74 61 66 61 62-72 69 63 31 0f 30 0d 06   .**********1.0..
00f0 - 03 55 04 0b 13 06 62 72-6f 6b 65 72 31 20 30 1e   .U....broker1 0.
0100 - 06 03 55 04 03 13 17 60-72 31 2e 62 72 6f 6b 65   ..U....br1.broke
0110 - 72 2e 64 66 2e 6b 61 66-6b 61 2d 64 65 76 30 82   r.**.kafka-dev0.
0120 - 01 22 30 0d 06 09 2a 86-48 86 f7 0d 01 01 01 05   ."0...*.H.......
0130 - 00 03 82 01 0f 00 30 82-01 0a 02 82 01 01 00 ab   ......0.........
0140 - 92 1f 67 a1 8a c8 96 6d-0e b9 2e 49 f7 12 63 8b   ..g....m...I..c.
0150 - 3c 8b 28 df ea 54 2f a2-16 e1 3b 58 3e 76 59 1a   <.(..T/...;X>vY.
0160 - 6b 62 5a 66 43 2b 6f fe-af c0 6d 0f ec 5d 6d 40   kbZfC+o...m..]m@
0170 - 6b b4 9b 51 a3 97 52 8b-e7 33 05 6e 87 93 13 8c   k..Q..R..3.n....
0180 - 49 55 bd c6 05 38 91 5a-0a 28 28 2e 86 03 96 a8   IU...8.Z.((.....
0190 - 8f 4a 2f 83 df ad e9 ce-24 83 ad df 64 29 f6 91   .J/.....$...d)..
01a0 - 21 ac 7a d6 a3 d7 64 a1-7e 6b 00 90 04 69 65 f2   !.z...d.~k...ie.
01b0 - c2 78 e4 4f 87 16 d5 bb-7b bf 3d 66 46 a9 4a 13   .x.O....{.=fF.J.
01c0 - de 2e 36 e0 45 be 33 27-b7 67 5e ea 8d 4b 0b 75   ..6.E.3'.g^..K.u
01d0 - 90 d4 3b 62 5e 7a 50 36-07 30 d5 c8 d3 33 c1 06   ..;b^zP6.0...3..
01e0 - 38 c4 f0 47 ad fc 67 3f-8b 8d 67 d9 40 48 d7 d7   8..G..g?..g.@H..
01f0 - 0a 25 22 48 14 00 1f a3-9b 70 00 cb 1c 76 08 74   .%"H.....r...v.t
0200 - c4 93 95 ff a5 58 1a 38-32 1c e6 79 58 6e 09 8f   .....X.82..yXn..
0210 - f5 ba 3f 84 31 cc df 74-b3 24 65 cf 28 46 9d 6d   ..?.1..t.$e.(F.m
0220 - 48 5f 1e 99 8b 2a b7 9b-99 31 7a 61 b5 4e 79 9f   H_...*...1za.Ny.
0230 - 93 6f 6d e1 67 03 e7 3a-45 b9 5e 24 5b f2 2f 02   .om.g..:E.^$[./.
0240 - 03 01 00 01 a3 82 01 a1-30 82 01 9d 30 0e 06 03   ........0...0...
0250 - 55 1d 0f 01 01 ff 04 04-03 02 03 a8 30 13 06 03   U...........0...
0260 - 55 1d 25 04 0c 30 0a 06-08 2b 06 01 05 05 07 03   U.%..0...+......
0270 - 01 30 1d 06 03 55 1d 0e-04 16 04 14 00 20 e6 11   .0...U....... ..
0280 - 31 89 96 4e 3b c2 d6 64-1f 57 79 6a 20 48 d8 43   1..N;..d.Wyj H.C
0290 - 30 1f 06 03 55 1d 23 04-18 30 16 80 14 03 4f 53   0...U.#..0....OS
02a0 - f4 fc 42 9e 46 8d 67 4e-f8 d7 13 b9 b6 1a 6c 2d   ..B.F.gN......l-
02b0 - 59 30 6a 06 08 2b 06 01-05 01 07 01 01 04 5e 30   Y0j..+........^0
02c0 - 5c 30 5a 06 08 2b 06 01-05 05 07 30 02 86 4e 68   \0Z..+.....0..Nh
02d0 - 74 74 70 73 3a 2f 2f 76-61 75 6c 74 2e 64 61 74   ttps://vault.***
02e0 - 61 66 61 62 72 69 63 2e-69 6e 66 72 61 2e 61 77   *******.*****.aw
02f0 - 73 2e 6c 61 64 62 72 6f-6b 65 73 63 6f 72 61 6c   s.**************
0300 - 2e 63 6f 6d 3a 38 32 30-30 2f 76 31 2f 6b 61 66   .com:8200/v1/kaf
0310 - 6b 61 5f 70 6b 69 5f 64-65 76 2f 63 61 30 68 06   ka_pki_dev/ca0h.
0320 - 03 55 1d 11 04 61 30 5f-82 17 62 72 31 2e 62 72   .U...a0_..br1.br
0330 - 6f 6b 65 72 2e 64 66 2e-6b 61 66 6b 61 2d 64 65   oker.**.kafka-de
0340 - 76 82 14 2a 2e 6c 61 64-62 72 6f 6b 65 73 63 6f   v..*.***********
0350 - 72 61 6c 2e 63 6f 6d 82-2e 63 6f 6d 2e 66 65 65   ***.com..com.fee
0360 - 64 2e 64 61 74 61 66 61-62 72 69 63 2e 64 65 76   d.**********.dev
0370 - 2e 61 77 73 2e 6c 61 64-62 72 6f 6b 65 73 63 6f   .aws.***********
0380 - 72 61 6c 2e 63 6f 6d 30-60 06 03 55 1d 1f 04 59   ***.com0`..U...Y
0390 - 30 57 30 55 a0 53 a0 51-86 4f 68 74 74 70 73 3a   0W0U.S.Q.Ohttps:
03a0 - 2f 2f 76 61 75 6c 74 2e-64 61 74 61 66 61 62 72   //vault.********
03b0 - 69 63 2e 69 6e 66 72 61-2e 61 77 73 2e 6c 61 64   **.*****.aws.***
03c0 - 62 72 6f 6b 65 73 63 6f-72 61 6c 2e 63 6f 6d 3a   ***********.com:
03d0 - 38 32 30 30 2f 76 31 2f-6b 61 66 6b 61 5f 70 6b   8200/v1/kafka_pk
03e0 - 69 5f 64 65 76 2f 63 72-6c 30 0d 06 09 2a 86 48   i_dev/crl0...*.H
03f0 - 86 f7 0d 01 01 0b 05 00-03 82 01 01 00 12 ae 54   ...............T
0400 - 44 4a a0 c0 ea dd 3e 04-03 12 d9 16 37 06 11 06   DJ....>.....7...
0410 - 24 2f 4d b6 ae 57 11 87-e3 95 a8 35 55 98 b9 1d   $/M..W.....5U...
0420 - b8 b1 b6 ed 72 e4 1b 94-86 d1 70 cd 99 48 c5 39   ....r.....p..H.9
0430 - ad 2d 2d a0 4f 75 4c 23-e8 4f ef cc 8c 99 01 0d   .--.OuL#.O......
0440 - 7e d4 f8 98 67 6b 7c 0b-38 34 8b c7 93 28 f1 4b   ~...gk|.84...(.K
0450 - ef 04 3c 47 e4 93 20 cc-7c 41 d3 f8 e2 1b 53 8f   ..<G.. .|A....S.
0460 - 67 68 5c 7b 21 5f ce 8d-81 a1 88 2b 76 17 0e 96   gh\{!_.....+v...
0470 - bd 5a 48 e5 64 bc ed f4-3a eb d2 32 7a 15 19 78   .ZH.d...:..2z..x
0480 - 83 80 8b 58 63 71 e1 c0-5c 6b d5 23 d8 ae b3 a6   ...Xcq..\k.#....
0490 - 6a 75 b3 fb 75 c9 10 21-9c 30 7f 26 3e bd 07 e2   ju..u..!.0.&>...
04a0 - 9d 31 b2 21 6b 63 12 50-dd 8e 65 4a 22 a7 b5 5e   .1.!kc.P..eJ"..^
04b0 - b1 43 6a 28 07 62 a2 00-65 da 0a 40 34 46 27 8d   .Cj(.b..e..@4F'.
04c0 - 30 c6 eb f9 9c ad 64 45-8a a2 98 0c 0f f1 65 20   0.....dE......e
04d0 - c8 b1 e2 e8 ca 8c 42 02-8b 1b 3b 3c ee 49 d7 77   ......B...;<.I.w
04e0 - 3c 5b 8c e6 ba 3c 6f 31-74 56 f0 7b 18 b5 0f 62   <[...<o1tV.{...b
04f0 - 22 17 9a 71 17 15 10 39-91 80 5c aa 79 0c 00 01   "..q...9..\.y...
0500 - 47 03 00 17 41 04 9f 5a-6e 51 10 e8 ed 2b 58 d4   G...A..ZnQ...+X.
0510 - 23 dd ae ca 47 88 48 8a-4b fa ba 75 32 cc 36 a2   #...G.H.K..u2.6.
0520 - dc a6 d6 c5 f9 27 42 aa-ea 48 46 0e 1f 74 40 78   .....'B..HF..t@x
0530 - 16 df c8 48 66 21 34 17-49 81 2f c6 51 4a 07 66   ...Hf!4.I./.QJ.f
0540 - 6e ad cd 6c 89 f5 01 00-2d ac e1 aa 6d 2a 37 2b   n..l....-...m*7+
0550 - 08 4d 1f b6 f1 75 28 21-dc 4f 27 42 51 ad c1 bc   .M...u(!.O'BQ...
0560 - 1b 63 48 e0 3d 4f 8b 35-14 a0 9a f5 bb f6 c1 02   .cH.=O.5........
0570 - 97 0a d7 fe 92 38 fb 6b-6c 9e 4a 12 e3 d5 5d 58   .....8.kl.J...]X
0580 - 5c 4c 8a d0 da e5 7e 52-85 70 84 be 0b c3 0b 46   \L....~R.p.....F
0590 - 87 83 ab bd 71 72 2c 1f-43 d6 8d 3b 3f 5c 82 ac   ....qr,.C..;?\..
05a0 - 41 bc 43 de f9 a0 12 35-2b bd 71 3d cc bb ec d3   A.C....5+.q=....
05b0 - 90 7c f4 a2 94 3d e5 e7-14 e4 cc 12 75 ed 22 a2   .|...=......u.".
05c0 - a0 02 f2 f7 0b 17 e0 28-b8 10 da 6b ef 95 39 81   .......(...k..9.
05d0 - 10 aa e4 77 ed f1 62 f8-06 b5 f6 cc 0b 92 f7 d0   ...w..b.........
05e0 - 24 6f 0e 43 a6 01 3e 7e-2a 79 22 39 39 d0 6b 92   $o.C..>~*y"99.k.
05f0 - cf ae 57 d0 e7 7e 7f ac-15 2b 6b 3d 86 6e 48 c3   ..W..~...+k=.nH.
0600 - b0 ac 7a a0 0c 4c ad 8c-7f aa 8e a5 4e bc 17 52   ..z..L......N..R
0610 - 5a a1 47 4a 63 4c 5f 5d-f8 31 fc 32 13 e9 7e 1b   Z.GJcL_].1.2..~.
0620 - 77 62 de 81 f9 be ef 95-e4 fb c7 46 ab d5 d3 12   wb.........F....
0630 - 07 b6 8f a5 45 ec 06 fc-88 8a 48 93 6f b1 00 d8   ....E.....H.o...
0640 - d1 82 37 c1 3e 02 38 51-0d 00 00 2b 03 01 02 40   ..7.>.8Q...+...@
0650 - 00 25 00 23 30 21 31 1f-30 1d 06 03 55 04 03 13   .%.#0!1.0...U...
0660 - 16 6c 63 67 2d 64 61 74-61 66 61 62 72 69 63 2d   .***-**********-
0670 - 64 65 76 2d 70 6b 69 0e-00 00 00                  dev-pki....
depth=0 O = *******, OU = broker, CN = br1.broker.**.kafka-dev
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = *******, OU = broker, CN = br1.broker.**.kafka-dev
verify error:num=21:unable to verify the first certificate
verify return:1
write to 0x55e3505d4630 [0x55e3505e6bb0] (146 bytes => 146 (0x92))
0000 - 16 03 01 00 07 0b 00 00-03 00 00 00 16 03 01 00   ................
0010 - 46 10 00 00 42 41 04 ba-1d e4 33 ec b3 b1 e8 74   F...BA....3....t
0020 - b3 a9 81 08 1a be 80 f3-8c 95 1e 6c 92 c0 da 42   ...........l...B
0030 - 67 0b 58 8b 61 51 40 24-66 7d e3 49 0f 06 30 99   g.X.aQ@$f}.I..0.
0040 - c5 11 3d 0e dd fc b0 5c-1b 25 fe 07 c3 f2 5f 80   ..=....\.%...._.
0050 - f3 6d 74 1a 3f b2 43 14-03 01 00 01 01 16 03 01   .mt.?.C.........
0060 - 00 30 54 10 a7 a9 32 82-7c e1 d7 31 ab ec e9 10   .0T...2.|..1....
0070 - 48 40 1f 5a 07 40 bd 4e-c6 b8 d2 f0 1e 47 a3 32   H@.Z.@.N.....G.2
0080 - 15 33 af f7 62 51 bb 38-f2 31 d9 a1 f5 59 b0 d2   .3..bQ.8.1...Y..
0090 - 9b 32                                             .2
read from 0x55e3505d4630 [0x55e3505dd8a3] (5 bytes => 5 (0x5))
0000 - 16 03 01 06 7b                                    ....{
read from 0x55e3505d4630 [0x55e3505dd8a8] (1659 bytes => 1403 (0x57B))
0000 - 02 00 00 51 03 01 61 63-cb 9e d8 d6 d9 a2 45 e4   ...Q..ac......E.
0010 - ee 5a 6e 13 9b 8e cd 97-45 be bc fd cc 52 a8 35   .Zn.....E....R.5
0020 - 1f f2 14 8f fd b5 20 61-63 cb 9e 17 5f 7b 16 e4   ...... ac..._{..
0030 - b7 71 ef 5a ad 94 c0 3f-8a a0 7f b9 e6 86 b0 c6   .q.Z...?........
0040 - 39 41 27 c9 c3 e8 ac c0-14 00 00 09 ff 01 00 01   9A'.............
0050 - 00 00 17 00 00 0b 00 04-a4 00 04 a1 00 04 9e 30   ...............0
0060 - 82 04 9a 30 82 03 82 a0-03 02 01 02 02 14 3c f7   ...0..........<.
0070 - da 9a 03 16 0d a7 67 08-fe 67 97 24 3b 09 82 01   ......g..g.$;...
0080 - aa aa 30 0d 06 09 2a 86-48 86 f7 0d 01 01 0b 05   ..0...*.H.......
0090 - 00 30 21 31 1f 30 1d 06-03 55 04 03 13 16 6c 63   .0!1.0...U....lc
00a0 - 67 2d 64 61 74 61 66 61-62 72 69 63 2d 64 65 76   g-**********-dev
00b0 - 2d 70 6b 69 30 1e 17 0d-32 30 30 36 33 30 31 30   -pki0...20063010
00c0 - 32 38 31 35 5a 17 0d 32-33 30 36 32 37 30 32 32   2815Z..230627022
00d0 - 38 34 34 5a 30 48 31 13-30 11 06 03 55 04 0a 13   844Z0H1.0...U...
00e0 - 0a 64 61 74 61 66 61 62-72 69 63 31 0f 30 0d 06   .**********1.0..
00f0 - 03 55 04 0b 13 06 62 72-6f 6b 65 72 31 20 30 1e   .U....broker1 0.
0100 - 06 03 55 04 03 13 17 62-72 31 2e 62 72 6f 6b 65   ..U....br1.broke
0110 - 72 2e 64 66 2e 6b 61 66-6b 61 2d 64 65 76 30 82   r.**.kafka-dev0.
0120 - 01 22 30 0d 06 09 2a 86-48 86 f7 0d 01 01 01 05   ."0...*.H.......
0130 - 00 03 82 01 0f 00 30 82-01 0a 02 82 01 01 00 ab   ......0.........
0140 - 92 1f 67 a1 8a c8 96 6d-0e b9 2e 49 f7 12 63 8b   ..g....m...I..c.
0150 - 3c 8b 28 df ea 54 2f a2-16 e1 3b 58 3e 76 59 1a   <.(..T/...;X>vY.
0160 - 6b 62 5a 66 43 2b 6f fe-af c0 6d 0f ec 5d 6d 40   kbZfC+o...m..]m@
0170 - 6b b4 9b 51 a3 97 52 8b-e7 33 05 6e 87 93 13 8c   k..Q..R..3.n....
0180 - 49 55 bd c6 05 38 91 5a-0a 28 28 2e 86 03 96 a8   IU...8.Z.((.....
0190 - 8f 4a 2f 83 df ad e9 ce-24 83 ad df 64 29 f6 91   .J/.....$...d)..
01a0 - 21 ac 7a d6 a3 d7 64 a1-7e 6b 00 90 04 69 65 f2   !.z...d.~k...ie.
01b0 - c2 78 e4 4f 87 16 d5 bb-7b bf 3d 66 46 a9 4a 13   .x.O....{.=fF.J.
01c0 - de 2e 36 e0 45 be 33 27-b7 67 5e ea 8d 4b 0b 75   ..6.E.3'.g^..K.u
01d0 - 90 d4 3b 62 5e 7a 50 36-07 30 d5 c8 d3 33 c1 06   ..;b^zP6.0...3..
01e0 - 38 c4 f0 47 ad fc 67 3f-8b 8d 67 d9 40 48 d7 d7   8..G..g?..g.@H..
01f0 - 0a 25 22 48 14 00 1f a3-9b 72 00 cb 1c 76 08 74   .%"H.....r...v.t
0200 - c4 93 95 ff a5 58 1a 38-32 1c e6 79 58 6e 09 8f   .....X.82..yXn..
0210 - f5 ba 3f 84 31 cc df 74-b3 24 65 cf 28 46 9d 6d   ..?.1..t.$e.(F.m
0220 - 48 5f 1e 99 8b 2a b7 9b-99 31 7a 61 b5 4e 79 9f   H_...*...1za.Ny.
0230 - 93 6f 6d e1 67 03 e7 3a-45 b9 5e 24 5b f2 2f 02   .om.g..:E.^$[./.
0240 - 03 01 00 01 a3 82 01 a1-30 82 01 9d 30 0e 06 03   ........0...0...
0250 - 55 1d 0f 01 01 ff 04 04-03 02 03 a8 30 13 06 03   U...........0...
0260 - 55 1d 25 04 0c 30 0a 06-08 2b 06 01 05 05 07 03   U.%..0...+......
0270 - 01 30 1d 06 03 55 1d 0e-04 16 04 14 00 20 e6 11   .0...U....... ..
0280 - 31 89 96 4e 3b c2 d6 64-1f 57 79 6a 20 48 d8 43   1..N;..d.Wyj H.C
0290 - 30 1f 06 03 55 1d 23 04-18 30 16 80 14 03 4f 53   0...U.#..0....OS
02a0 - f4 fc 42 9e 46 8d 67 4e-f8 d7 13 b9 b6 1a 6c 2d   ..B.F.gN......l-
02b0 - 59 30 6a 06 08 2b 06 01-05 05 07 01 01 04 5e 30   Y0j..+........^0
02c0 - 5c 30 5a 06 08 2b 06 01-05 05 07 30 02 86 4e 68   \0Z..+.....0..Nh
02d0 - 74 74 70 73 3a 2f 2f 76-61 75 6c 74 2e 64 61 74   ttps://vault.***
02e0 - 61 66 61 62 72 69 63 2e-69 6e 66 72 61 2e 61 77   *******.infra.aw
02f0 - 73 2e 6c 61 64 62 72 6f-6b 65 73 63 6f 72 61 6c   s.**************
0300 - 2e 63 6f 6d 3a 38 32 30-30 2f 76 31 2f 6b 61 66   .com:8200/v1/kaf
0310 - 6b 61 5f 70 6b 69 5f 64-65 76 2f 63 61 30 68 06   ka_pki_dev/ca0h.
0320 - 03 55 1d 11 04 61 30 5f-82 17 62 72 31 2e 62 72   .U...a0_..br1.br
0330 - 6f 6b 65 72 2e 64 66 2e-6b 61 66 6b 61 2d 64 65   oker.df.kafka-de
0340 - 76 82 14 2a 2e 6c 61 64-62 72 6f 6b 65 73 63 6f   v..*.***********
0350 - 72 61 6c 2e 63 6f 6d 82-2e 63 6f 6d 2e 66 65 65   ***.com..com.fee
0360 - 64 2e 64 61 74 61 66 61-62 72 69 63 2e 64 65 76   d.**********.dev
0370 - 2e 61 77 73 2e 6c 61 64-62 72 6f 6b 65 73 63 6f   .aws.***********
0380 - 72 61 6c 2e 63 6f 6d 30-60 06 03 55 1d 1f 04 59   ral.com0`..U...Y
0390 - 30 57 30 55 a0 53 a0 51-86 4f 68 74 74 70 73 3a   0W0U.S.Q.Ohttps:
03a0 - 2f 2f 76 61 75 6c 74 2e-64 61 74 61 66 61 62 72   //vault.********
03b0 - 69 63 2e 69 6e 66 72 61-2e 61 77 73 2e 6c 61 64   **.infra.aws.***
03c0 - 62 72 6f 6b 65 73 63 6f-72 61 6c 2e 63 6f 6d 3a   ***********.com:
03d0 - 38 32 30 30 2f 76 31 2f-6b 61 66 6b 61 5f 70 6b   8200/v1/kafka_pk
03e0 - 69 5f 64 65 76 2f 63 72-6c 30 0d 06 09 2a 86 48   i_dev/crl0...*.H
03f0 - 86 f7 0d 01 01 0b 05 00-03 82 01 01 00 12 ae 54   ...............T
0400 - 44 4a a0 c0 ea dd 3e 04-03 12 d9 16 37 06 11 06   DJ....>.....7...
0410 - 24 2f 4d b6 ae 57 11 87-e3 95 a8 35 55 98 b9 1d   $/M..W.....5U...
0420 - b8 b1 b6 ed 72 e4 1b 94-86 d1 70 cd 99 48 c5 39   ....r.....p..H.9
0430 - ad 2d 2d a0 4f 75 4c 23-e8 4f ef cc 8c 99 01 0d   .--.OuL#.O......
0440 - 7e d4 f8 98 67 6b 7c 0b-38 34 8b c7 93 28 f1 4b   ~...gk|.84...(.K
0450 - ef 04 3c 47 e4 93 20 cc-7c 41 d3 f8 e2 1b 53 8f   ..<G.. .|A....S.
0460 - 67 68 5c 7b 21 5f ce 8d-81 a1 88 2b 76 17 0e 96   gh\{!_.....+v...
0470 - bd 5a 48 e5 64 bc ed f4-3a eb d2 32 7a 15 19 78   .ZH.d...:..2z..x
0480 - 83 80 8b 58 63 71 e1 c0-5c 6b d5 23 d8 ae b3 a6   ...Xcq..\k.#....
0490 - 6a 75 b3 fb 75 c9 10 21-9c 30 7f 26 3e bd 07 e2   ju..u..!.0.&>...
04a0 - 9d 31 b2 21 6b 63 12 50-dd 8e 65 4a 22 a7 b5 5e   .1.!kc.P..eJ"..^
04b0 - b1 43 6a 28 07 62 a2 00-65 da 0a 40 34 46 27 8d   .Cj(.b..e..@4F'.
04c0 - 30 c6 eb f9 9c ad 64 45-8a a2 98 0c 0f f1 65 20   0.....dE......e
04d0 - c8 b1 e2 e8 ca 8c 42 02-8b 1b 3b 3c ee 49 d7 77   ......B...;<.I.w
04e0 - 3c 5b 8c e6 ba 3c 6f 31-74 56 f0 7b 18 b5 0f 62   <[...<o1tV.{...b
04f0 - 22 17 9a 71 17 15 10 39-91 80 5c aa 79 0c 00 01   "..q...9..\.y...
0500 - 47 03 00 17 41 04 9f 5a-6e 51 10 e8 ed 2b 58 d4   G...A..ZnQ...+X.
0510 - 23 dd ae ca 47 88 48 8a-4b fa ba 75 32 cc 36 a2   #...G.H.K..u2.6.
0520 - dc a6 d6 c5 f9 27 42 aa-ea 48 46 0e 1f 74 40 78   .....'B..HF..t@x
0530 - 16 df c8 48 66 21 34 17-49 81 2f c6 51 4a 07 66   ...Hf!4.I./.QJ.f
0540 - 6e ad cd 6c 89 f5 01 00-2d ac e1 aa 6d 2a 37 2b   n..l....-...m*7+
0550 - 08 4d 1f b6 f1 75 28 21-dc 4f 27 42 51 ad c1 bc   .M...u(!.O'BQ...
0560 - 1b 63 48 e0 3d 4f 8b 35-14 a0 9a f5 bb f6 c1 02   .cH.=O.5........
0570 - 97 0a d7 fe 92 38 fb 6b-6c 9e 4a                  .....8.kl.J
read from 0x55e3505d4630 [0x55e3505dde23] (256 bytes => 256 (0x100))
0000 - 12 e3 d5 5d 58 5c 4c 8a-d0 da e5 7e 52 85 70 84   ...]X\L....~R.p.
0010 - be 0b c3 0b 46 87 83 ab-bd 71 72 2c 1f 43 d6 8d   ....F....qr,.C..
0020 - 3b 3f 5c 82 ac 41 bc 43-de f9 a0 12 35 2b bd 71   ;?\..A.C....5+.q
0030 - 3d cc bb ec d3 90 7c f4-a2 94 3d e5 e7 14 e4 cc   =.....|...=.....
0040 - 12 75 ed 22 a2 a0 02 f2-f7 0b 17 e0 28 b8 10 da   .u."........(...
0050 - 6b ef 95 39 81 10 aa e4-77 ed f1 62 f8 06 b5 f6   k..9....w..b....
0060 - cc 0b 92 f7 d0 24 6f 0e-43 a6 01 3e 7e 2a 79 22   .....$o.C..>~*y"
0070 - 39 39 d0 6b 92 cf ae 57-d0 e7 7e 7f ac 15 2b 6b   99.k...W..~...+k
0080 - 3d 86 6e 48 c3 b0 ac 7a-a0 0c 4c ad 8c 7f aa 8e   =.nH...z..L.....
0090 - a5 4e bc 17 52 5a a1 47-4a 63 4c 5f 5d f8 31 fc   .N..RZ.GJcL_].1.
00a0 - 32 13 e9 7e 1b 77 62 de-81 f9 be ef 95 e4 fb c7   2..~.wb.........
00b0 - 46 ab d5 d3 12 07 b6 8f-a5 45 ec 06 fc 88 8a 48   F........E.....H
00c0 - 93 6f b4 00 d8 d1 82 37-c1 3e 02 38 51 0d 00 00   .o.....7.>.8Q...
00d0 - 2b 03 01 02 40 00 25 00-23 30 21 31 1f 30 1d 06   +...@.%.#0!1.0..
00e0 - 03 55 04 03 13 16 6c 63-67 2d 64 61 74 61 66 61   .U....***-******
00f0 - 62 72 69 63 2d 64 65 76-2d 70 6b 69 0e 00 00 00   ****-dev-pki....
write to 0x55e3505d4630 [0x55e3505e6bb0] (37 bytes => 37 (0x25))
0000 - 15 03 01 00 20 f3 2b 68-64 bd fe 8f c4 e9 a2 5b   .... .+hd......[
0010 - 72 01 d0 cb 99 17 0a 0c-e8 fc 26 32 0b b9 39 47   r.........&2..9G
0020 - 01 c9 58 85 fb                                    ..X..
140258794983872:error:141A10F4:SSL routines:ossl_statem_client_read_transition:unexpected message:../ssl/statem/statem_clnt.c:395:
---
Certificate chain
 0 s:O = *********, OU = broker, CN = br1.broker.**.kafka-dev
   i:CN = ***-*********-dev-pki
---
Server certificate
**
**
subject=O = *********, OU = broker, CN = br1.broker.**.kafka-dev

issuer=CN = ***-*********-dev-pki

---
Acceptable client certificate CA names
CN = ***-*********-dev-pki
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Peer signing digest: MD5-SHA1
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3328 bytes and written 332 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.0, Cipher is ECDHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : ECDHE-RSA-AES256-SHA
    Session-ID: 6163CB9E175F7B16E4B771EF5AAD94C03F8AA07FB9E686B0C6394127C9C3E8AC
    Session-ID-ctx:
    Master-Key: 73420B65E6FC92E62A16D5324A4A2E1C3E87F59E9C6FF6A02E2AD0D735020F68B4117A711A90C1A42D9E40CC4743A7D2
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1633930142
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: yes
---
read from 0x55e3505d4630 [0x55e3505cbd80] (8192 bytes => 4016 (0xFB0))

1060 - 00 02 02 2a                                       ...*
read from 0x55e3505d4630 [0x55e3505cbd80] (8192 bytes => 0 (0x0))
4

mmh seems there is a validation issue:

5

Hi @mmuehlbeyer ,

But these are our custom certificates…

6

I see
Cert chain is correctly set/configured?

7

Hi @mmuehlbeyer ,

Yes , The existing certificates are correctly configured in the broker and same certs are using by many clients.

Only when we are trying to add New ca in the existing keystore& trust store we are facing issues .

The client certificate generated with new CA are unable to make connection to the kafka app. even when the keystore & trustore is updated with the new CA ( both old & new CA are existing in keys tore & trust store)

server:~$ openssl s_client -debug -connect app-kafka-test:9092 -tls1
CONNECTED(00000003)
write to 0x55e9592e4120 [0x55e9592f45f0] (7 bytes => 7 (0x7))
0000 - 15 03 01 00 02 02 50                              ......P
140599009273152:error:141E70BF:SSL routines:tls_construct_client_hello:no protocols available:../ssl/statem/statem_clnt.c:1112:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 7 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
read from 0x55e9592e4120 [0x55e9592d8f60] (8192 bytes => 0 (0x0))