基本介紹

經常用python或java處理json資料，其實在Shell命令列，我們可以用開源JQ工具來處理Json資料。會更加簡潔和高效。

CentOS 安裝JQ

wget -O jq https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64
chmod +x ./jq
cp jq /usr/bin

JQ常用舉例

1、原始JSON資料

cat jsonData.json
{"productId":"2723","click":60,"view":300,"deal"
 
:2,"day":"20130919"}
{"productId":"2728","click":130,"view":800,"deal":10,"day":"20130919"}
{"productId":"3609","click":50,"view":400,"deal":3,"day":"20130919"}
{"productId":"3783","click":375,"view":1200,"deal":50,"day":"20130919"}
{"productId":"3522","click":87,"view":573,"deal":15,"day":"20130919"}

2、格式化輸出

jq -r . jsonData.json
{
  "productId"
 
: "2723",
  "click": 60,
  "view": 300,
  "deal": 2,
  "day": "20130919"
}
{
  "productId": "2728",
  "click": 130,
  "view": 800,
  "deal": 10,
  "day": "20130919"
}
...

2、提取json指定欄位轉csv

這裡我們需要得到productId_day,click,view,deal這種格式的csv檔案。

jq -r '[.productId+"_"+.day,(.click|tostring),(.view|tostring),(.deal|tostring)]|join("
 
,")' jsonData.json
結果：
2723_20130919,60,300,2
2728_20130919,130,800,10
3609_20130919,50,400,3
3783_20130919,375,1200,50
3522_20130919,87,573,15
注意：
  .productId:取productId欄位的值
　.click|tostring:取click欄位的值並轉化為string型別
　[]:陣列
　join(","):陣列中的元素通過","連線。

3、複雜json資料查詢

complexJson='{"uid":"2378","nameInfo":[{"firstName":"nova","lastName":"jos"}]}'
echo ${complexJson} |jq
{
  "uid": "2378",
  "nameInfo": [
    {
      "firstName": "nova",
      "lastName": "jos"
    }
  ]
}
echo ${complexJson} |jq -r '.nameInfo[].lastName'
結果:jos
注意:.nameInfo[].lastName是獲取.nameInfo[] json array中每個json object的lastName欄位。

4、提取指定欄位形成新的json

jq -c '{productId:.productId,click:.click,dayYmd:(.day|tonumber)}' jsonData.json >>newjson.json
結果:
cat newjson.json
{"productId":"2723","click":60,"dayYmd":20130919}
{"productId":"2728","click":130,"dayYmd":20130919}
{"productId":"3609","click":50,"dayYmd":20130919}
{"productId":"3783","click":375,"dayYmd":20130919}
{"productId":"3522","click":87,"dayYmd":20130919}
注意:
 -c/--compact-output:jq 預設是pretty-prints JSON 輸出，加上這個引數，一行一個json物件。

5、csv檔案轉json檔案

cat csvData.csv
1478,john,38
1529,lucy,25
1673,iris,22
jq -R -c 'split(",")|{"uid":.[0],"name":.[1],"age":.[2]|tonumber}' csvData.csv > csv2json.json
cat csv2json.json
{"uid":"1478","name":"john","age":38}
{"uid":"1529","name":"lucy","age":25}
{"uid":"1673","name":"iris","age":22}

主要參考文件