Skip to content

Working with JSON on CLI🔗

Format, parse, transform and extract JSON data (e.g: from an API response using curl) with jq:

jq is like sed for JSON data - you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text.

— ./jq official website

Installation🔗

OS X🔗

1
brew install jq

Debian🔗

1
apt-get install jq

Manala🔗

1
2
3
4
# ansible/group_vars/app.yml
app_patterns:
  apt_packages:
    - jq

1
2
3
4
5
6
# .manala.yaml
manala:
  system:
    apt:
      packages:
        - jq

Usage🔗

Simply use jq by piping a JSON to it:

1
curl https://jsonplaceholder.typicode.com/todos/1 | jq

to get parsed and syntax highlighted JSON.

Of course you can go further and start working with the jq expression language to transform & extract the data you need.

Usage samples🔗

Given:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
{
  "data": [
      {
        "status": "processed",
        "updateId": 1,
        "type": {
          "name": "DocumentsAddition",
          "number": 2000
        },
        "duration": 2.601152779,
        "enqueuedAt": "2020-08-11T10:47:10.848598818Z",
        "processedAt": "2020-08-11T10:47:13.566757416Z"
      },
      {
        "status": "processed",
        "updateId": 2,
        "type": {
          "name": "DocumentsAddition",
          "number": 2000
        },
        "duration": 2.981042727,
        "enqueuedAt": "2020-08-11T10:47:18.147178337Z",
        "processedAt": "2020-08-11T10:47:21.239179936Z"
      },
      {
        "status": "queued",
        "updateId": 3,
        "type": {
          "name": "DocumentsAddition",
          "number": 2000
        },
        "enqueuedAt": "2020-08-11T10:47:26.656031600Z"
      }
  ]
}

Get an object property value🔗

1
jq '.data'

where data is a property of the root JSON object

▶️ Test it

Extract a specific collection item🔗

1
jq '.data[1]'

where data is a property of the root JSON object (.), holding a collection of items (array)

▶️ Test it

Use [-1] to extract the last item.
Use [1, 2] to extract only some items.
Use [.data[1, 2]] to wrap back the items in a collection.
Use .data[1:3] to extract a range.

Unwrap a collection🔗

1
jq '.data[]'

▶️ Test it

Flatten an object🔗

1
jq '.data[1]|flatten'

casting it to an array, removing its keys

▶️ Test it

Use .data[1][] to unwrap all properties values directly (not as an array).

Filter a collection🔗

1
jq '.data[] | select( .status | contains("processed") )'

get only the processed items

▶️ Test it

Use .data[] | select( .status | contains("processed") | not ) to negate the condition and only get non processed items.

Transform items🔗

1
jq '.data | map({ "duration": .duration, "id": .updateId })'

extract duration and id into a normalized object for each item in the collection

▶️ Test it

Use .data | map([ .duration, .updateId ]) to transform to an array instead.

Using variables🔗

Store object properties in $keys var, data length in $count and display them into a JSON object:

1
jq -r '. | keys as $keys | .data | length as $count | { "keys": $keys, "data_count": $count }'

▶️ Test it

To CSV🔗

The @csv formatter allows to convert arrays to csv strings:

1
jq -r '@csv'

▶️ Test it

Note

Use the -r option to output as raw strings instead of JSON strings.

Considering this sample:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
[
    {
        "id": 1,
        "name": "Amanda Poulpe"
    },
    {
        "id": 2,
        "name": "Joseph Petit-suisse"
    },
    {
        "id": 3,
        "name": "Gilbert Sumo"
    }
]

We first need to convert each object in the collection to an array, then convert each to csv:

1
jq -r '.[] | flatten | @csv'

▶️ Test it

A more advanced sample, auto-detecting keys as CSV header:

1
jq -r '. as $rows | ($rows[0] | keys_unsorted) as $keys | $keys, ($rows[] | flatten) | @csv'

Warning

We assume the properties of each rows are the same and ordered the same way. Otherwise, use map to normalize the rows.

▶️ Test it

Complex sample:

1
jq -r '[.data[] | select( .status | contains("processed") )] | map({ "id": .updateId, "duration": .duration }) as $rows | ( $rows[0] | keys_unsorted ) as $keys | $keys, ($rows[] | flatten) | @csv'

gets a CSV of id and duration of every processed items in a collection

▶️ Test it

References🔗

Last update: December 20, 2024