Piping to jq echo'ing escape '\' before every '"'

Tools Command Line tools
1

Hi all…

using jsonschema as the serializer.

docker exec -t kafkacat \
  kcat \
  -b broker:29092 \
  -t json_salescompleted1 \
  -C

output as:

{"SALEDATETIME":"2024-06-24T12:43:26.268+02:00","SALETIMESTAMP":"1719225806268","TERMINALPOINT":"14","NETT":1446.91,"VAT":202.57,"TOTAL":1649.48,"STORE":{"ID":"324213416","NAME":"Warmer"},"CLERK":{"ID":"10018","NAME":"Max"},"BASKETITEMS":[{"ID":"000000030","NAME":"Kellogg's Corn Flakes 750g","BRAND":"Kellogg's","CATEGORY":"Food Cupboard","PRICE":28.99,"QUANTITY":5.0},{"ID":"000000041","NAME":"Castle Lite NRB 24 x 330ml","BRAND":"Castle","CATEGORY":"Beverage","PRICE":269.99,"QUANTITY":4.0},{"ID":"000000009","NAME":"Nescafe Salted Caramel 180g 10 Sachets","BRAND":"Nescafe","CATEGORY":"Food Cupboard","PRICE":74.0,"QUANTITY":3.0}],"FINTRANSACTIONID":"0d0bfdc4-d867-4607-b698-c8b9f0bed6d8","PAYDATETIME":"2024-06-24T12:45:56.268+02:00","PAYTIMESTAMP":"1719225956268","PAID":1649.48}

now tried

docker exec -t kafkacat \
  kcat \
  -b broker:29092 \
  -t json_salescompleted1 \
  -C -J |jq

output as (this made it all colourised and all, great, but why the escape chars)

{
  "topic": "json_salescompleted1",
  "partition": 0,
  "offset": 17898,
  "tstype": "create",
  "ts": 1719225806272,
  "broker": 1,
  "headers": [
    "myPaymentTopicnameHeader",
    "header values are binary"
  ],
  "key": "b4405e45-10ff-44f1-b328-feae152aecd3",
  "payload": "\u0000\u0000\u0000\u0000\u0015{\"SALEDATETIME\":\"2024-06-24T12:43:26.270+02:00\",\"SALETIMESTAMP\":\"1719225806270\",\"TERMINALPOINT\":\"7\",\"NETT\":518.95,\"VAT\":72.65,\"TOTAL\":591.6,\"STORE\":{\"ID\":\"324213416\",\"NAME\":\"Warmer\"},\"CLERK\":{\"ID\":\"10013\",\"NAME\":\"Mohammed\"},\"BASKETITEMS\":[{\"ID\":\"000000021\",\"NAME\":\"Stork Country 40% Fat Spread 1kg\",\"BRAND\":\"Stork\",\"CATEGORY\":\"Food Cupboard\",\"PRICE\":46.99,\"QUANTITY\":2.0},{\"ID\":\"000000006\",\"NAME\":\"Jacobs Kronung Decaf Instant 100g\",\"BRAND\":\"Jabobs\",\"CATEGORY\":\"Food Cupboard\",\"PRICE\":124.0,\"QUANTITY\":3.0},{\"ID\":\"000000020\",\"NAME\":\"Koo Baked Beans in Tomato Sauce 410g\",\"BRAND\":\"Koo\",\"CATEGORY\":\"Food Cupboard\",\"PRICE\":16.99,\"QUANTITY\":1.0},{\"ID\":\"000000044\",\"NAME\":\"Dettol Soap Active 175g\",\"BRAND\":\"Dettol\",\"CATEGORY\":\"Personal Health Care\",\"PRICE\":17.99,\"QUANTITY\":2.0}],\"FINTRANSACTIONID\":\"6d9f98bd-fc74-484a-ba63-c175ac0d3100\",\"PAYDATETIME\":\"2024-06-24T12:48:42.270+02:00\",\"PAYTIMESTAMP\":\"1719226122270\",\"PAID\":591.6}"
}
2

How about if you pipe to jq --raw-output?

3

nope, executed:

docker exec -t kafkacat \
  kcat \
  -b broker:29092 \
  -t json_salescompleted1 \
  -C -J | jq --raw-output

same results, see attached. Not as if I’m doing anything crazy… well at least I don’t think I am…
Also surprised by the xxxx much slower performance of json (because of the serializer) vs Pb.

G

Screenshot 2024-06-24 at 16.08.49
Screenshot 2024-06-24 at 16.08.492722×596 116 KB

4

Backing up: what output (that is valid JSON) are you aiming to see?

I think that the outer JSON is problematic. E.g., if you instead drilled into the payload field by piping to jq --raw-output '.payload' then you would avoid the escape characters.

5

I expected jq to also json format the payload… and not show the escape char before every "

if i change it to:

docker exec -t kafkacat \
  kcat \
  -b broker:29092 \
  -t json_salescompleted1 \
  -C -J | jq --raw-output '.payload'

this is the output., maybe I’m expecting to much :wink:

Screenshot 2024-06-24 at 18.47.41
Screenshot 2024-06-24 at 18.47.412724×992 502 KB

6

This wouldn’t give valid JSON output so I am doubting that jq on its own would do it.

7

… ok, but why ?
lost for the why it’s doing this, not as if i’m doing something funny.

G

8

I’m not sure I’m following the question. Why wouldn’t jq remove escape chars for you? Just my theory: since jq is a JSON processor, I am doubting that it would give you a way to generate invalid JSON.

For example, this is valid JSON:

{
    "some_string": "\"hi\""
}

And this is invalid:

{
    "some_string": ""hi""
}

BTW you can use a tool like https://jsonlint.com/ to test if JSON is valid. If you can use other text munging tools like awk / sed then you can probably arrive at what you expect by just removing backslashes, but then it might be subject to corner cases if there are some backslashes that you want to keep and others that you want to remove.

9

ok, guess our understand of valid json differs… for one valid json for me would be printed via jq in it’s structure, with sum structures shown, key:value’s on a line.

if you look at my output above the entire payload is printed as one long line, with escape chars everywhere, which confuses jq assume.

I do note the double quotes you are escaping.

G