Kafka Consumer showing numbers in Scientific Notation format

Clients
1

Hi,

I am producing below set of values through rest proxy but when its consumed through Rest Proxy Consumer it shows numbers in scientific notation format.
Producer Setting:

'Accept: application/vnd.kafka.v2+json' 
'Content-Type: application/vnd.kafka.json.v2+json'

Message Produced:

"value": {
                "Amount8": 15660000.00,
                "Amount9": 12345678.000,
                "Amount10": 17226000.00,
                "Amount11": 12345600.00,
                "Amount12": 1230000.00
            }

Message Consumed Through Rest Proxy Consumer:

"value": {
            "Amount8": 1.566E7,
            "Amount9": 1.2345678E7,
            "Amount10": 1.7226E7,
            "Amount11": 1.23456E7,
            "Amount12": 1230000.0
        }

Message Viewed in Control Center:

"value" : {
  "Amount8": 15660000,
  "Amount9": 12345678,
  "Amount10": 17226000,
  "Amount11": 12345600,
  "Amount12": 1230000
}

Why these conversion is happening? I don’t want numbers to be converted into scientific notation. How to stop it ?

2

I think that this is happening due to how JSON gets handled in REST Proxy, and it isn’t configurable AFAIK. I’m wondering if you’d hit the same issue with a different embedded format, e.g., try using Schema Registry and Content-Type: application/vnd.kafka.avro.v2+json. Other than that, I know that string conversion probably isn’t the most appetizing workaround but you can resort to that if all else fails. Stepping back, how and where are you consuming these messages? If you have client code, it should be able to just handle the values. The question I’d ask is: is the desire to avoid scientific notation strictly for human readability, or is it harming downstream client code?

3

Hi @dtroiano ,

thanks for checking my query.
I am consuming values through Snaplogic Integration Platform Consumer Snaps (which internally builds Kafka Consumer client). In this consumer snap I can configure one of the standard deserializer like Json, String, Avro, etc along with other basic consumer configs. I have less control on consumer client configs here.

For me, problem is downstream service not accepting scientific notation values and failing on processing. Hence we need to ensure values are not sent in scientific notation format.

4

Got it. Potential workaround list in the order that I would go:

  1. Do you need floating point numbers? If you can stick to integral numbers I believe you would avoid this. Have to ask since your example numbers all end in .00
  2. Give Avro a shot
  3. Strings :see_no_evil:

You might also want to raise this as a GitHub issue here. Best case I am missing something and there is a better way that someone will suggest. At the very least it gets the request logged in the right spot.

On a related note, this looks to be an issue elsewhere in the ecosystem, in Kafka Connect. See KAFKA-7900. (This isn’t causing the REST Proxy issue though.)

1 Like