jq (JSON Query utility for Bash CLI scripts)

Define JMESPATH syntax in –query; Extract city and other data from IP address lookup website ipinfo.io, Docker Hub, and other sites

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

The jq utility is like the sed Linux utility, but for JSON. They both reformat and transform text.

The “j” in the jq program name means it queries a JSON-formatted file to produce filtered output.

On a Mac, command brew install jq reveals that the utility has a dependency of the oniguruma regular expression handler and is from Stephen Dolan’s stedolan.github.io/jq

Install jp

On Mac:

brew tap jmespath/jmespath
 brew install jp

Output from an ARM-based mac laptop:

==> Fetching jp
==> Downloading https://ghcr.io/v2/homebrew/core/jp/manifests/1.1.12
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/jp/blobs/sha256:37b85d8d9876ffae1cdf4a3897ca558f2586a826a229d9b85d5799b33e338a89
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:37b85d8d9876ffae1cdf4a3897ca558f2586a826a229d9b85d5799b33e338a89?se=2022-12-24T15%3A45%3A00Z&sig=i%2FoDBRCeCst8TrPsaStWPNkXdv%2FEC00feBgLM5AaDT4%3D&sp=r&spr=https&sr=b&sv=201
######################################################################## 100.0%
==> Pouring jp--1.1.12.arm64_monterey.bottle.tar.gz
🍺  /opt/homebrew/Cellar/jp/1.1.12: 3 files, 3MB

On Ubuntu:
- apt-get install jq
On Windows Git Bash, download jq-win64.exe from github.com/stedolan/jq/releases into a bin folder in the PATH, and specify the program in commands:
- echo '{"foo": 0}' | ./jq-win64.exe .
```
{
"foo": 0
}
```

How to jq

https://stedolan.github.io/jq/manual presents a large (and potentially confusing) list of options.

Here’s an abridged version with examples explained step-by-step.

The ipinfo.io website returns the IP address and its metadata: location Lattitude and Longitude, postal (ZIP code), Unix time zone, carrier (org), State of the Union (region), etc.

curl ipinfo.io

yields:

{
  "ip": "184.153.97.231",
  "hostname": "cpe-184-153-97-231.nyc.res.rr.com",
  "city": "Woodside",
  "region": "New York",
  "country": "US",
  "loc": "40.7454,-73.9054",
  "org": "AS12271 Charter Communications Inc",
  "postal": "11377",
  "timezone": "America/New_York",
  "readme": "https://ipinfo.io/missingauth"
}

NOTE: All JSON responses begin and end with curly braces between { and }.

A dot represents the entire input, but formatted with indents:

curl ipinfo.io | jq ‘.’
.[] returns each element of the array, one at a time,

curl ipinfo.io | jq ‘.[]’
If you only want one city name (with no JSON braces or quote marks):

curl ipinfo.io | jq -r .city

NOTE: jq receives a file piped in, as specified by the pipe character .

To return the response in a variable within a Bash script:

CITY=$( curl -s ipinfo.io | jq -r .city )

-s is so you don’t see the progress, such as:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                              Dload  Upload   Total   Spent    Left  Speed
100   265  100   265    0     0   1007      0 --:--:-- --:--:-- --:--:--  1023

On my computer, I can type ipcity to invoke the above because I have the command in my aliases.zsh file defining keyboard shortcuts:
```
alias ipcity="curl -s ipinfo.io | jq -r .city"
```

More at:

https://thoughtbot.com/blog/jq-is-sed-for-json
https://shapeshed.com/jq-json/

### Docker Hub API

Docker Hub provides an API to access public Docker images.

curl ‘https://registry.hub.docker.com/v2/repositories/library/debian/tags/’

The response starts with:

{"count": 511, "next": "https://registry.hub.docker.com/v2/repositories/library/debian/tags/?page=2", "previous": null, "results": [{"name": "unstable-slim", "full_size": 27750048, "images": [{"size": 31296808, "architecture": "ppc64le", "variant": null, "features": null, "os": "linux", "os_version": null, "os_features": null}, {"size": 26435103, "architecture": "s390x", "variant": null, "features": null, "os": "linux", "os_version": null, "os_features": null}, {"size": 25698949, "architecture": "arm", "variant": "v5", "features": null ....

In JSON, a left bracket ([) begin a list structure:

“results”: [{“name”: “unstable-slim”,:

Note that “name” is an attribute, so would require double-quotes. ???

To return a list of just values beginning with “unstable-slim”:


curl -s 'https://registry.hub.docker.com/v2/repositories/library/debian/tags/'|jq '."results"[]["name"]'

The output at time of this writing:

"rc-buggy-20221219"
"rc-buggy"
"experimental-20221219"
"experimental"
"bookworm-20221219-slim"
"bookworm"
"11.6-slim"
"11.6"
"11-slim"
"11"

More commands

To list the first element from the root (counting from zero):

jq ‘.[0]’

GitHub API

https://stedolan.github.io/jq/tutorial/

JMESPATH

### JMESPATH extension in VSCode

https://www.youtube.com/watch?v=InExHoXJ9Xc

https://docs.aws.amazon.com/cli/v1/userguide/cli-usage-filter.html

JMESPATH –query

jq, grep, awk, sed, etc. filter output from text output type obtained from the server.

However, adding the optional –query switch as part of aws AND az commands does filtering BEFORE sending it to your workstation.

The command that lists everything (922 at time of this writing):

aws ec2 describe-instances
Know that the full range of specifications for –query is defined by the industry standard JMESPATH query format website at:

https://jmespath.org/

https://jmespath.org/specification.html

https://jmespath.org/tutorial.html

https://jmespath.org/examples.html

https://github.com/jmespath/jmespath.py

https://opensourceconnections.com/blog/2015/07/27/advanced-aws-cli-jmespath-query/

https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-output.html#controlling-output-filter

https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-filter.html

jasoneditoronline.org
A gentle yet deep tutorial is at AzureCitadel.com
The command sure to return a lot is described by this blog

AWS EC2 AMI images
- https://www.doaws.pl/blog/2021-12-05-how-to-master-aws-cli-in-15-minutes/how-to-master-aws-cli-in-15-minutes
- this command
- https://blog.ashiny.cloud/2017/09/25/useful-jmespath-in-awscli/
- https://winder.ai/how-to-list-all-amis-for-each-region-in-aws/ don’t work
REMEMBER: There is a different ID for each region for the same AWS EC2 image name. And you want the lastest among multiple versions of the same image, with different dates.
```
#!/bin/bash
if [ -z "$1" ] ; then
 echo "Please pass the name of the AMI"
 exit 1
fi
IMAGE_FILTER="${1}"
declare -a AWS_REGIONS=$( aws ec2 describe-regions --output text --query "Regions[].[RegionName]" | tr "\\n" " " )
 # | tr -d "\\""))
for r in "${AWS_REGIONS[@]}" ; do
 ami=$(aws ec2 describe-images --query 'Images[*].[ImageId]' --filters "Name=name,Values=${IMAGE_FILTER}" --region ${r} --output json | jq '.[0][0]')
 printf "\\"${r}\\" = ${ami}\\n"
done
```

From https://blog.scottlowe.org/2019/03/01/advanced-ami-filtering-with-jmespath/

aws ec2 describe-images --owners 099720109477 \
--filters Name=name,Values="*ubuntu-bionic-18.04*" \
Name=virtualization-type,Values=hvm \
Name=root-device-type,Values=ebs \
Name=architecture,Values=x86_64 \
--query 'sort_by(Images,&CreationDate)[?Name!=`null`]|[?contains(Name,`ubuntu-eks`)==`false`]|[?contains(Name,`minimal`)==`false`]|[-1].ImageId'
"ami-0135afc6d226a70a4"

How many images will be returned if no filtering? 81249

aws ec2 describe-images –owner amazon –output text wc -l

View JSON format to identify the names used to specify filtering:

aws ec2 describe-images –owner amazon –output json

  "Images": [
     {
         "Architecture": "x86_64",
         "CreationDate": "2022-08-03T10:55:58.000Z",
         "ImageId": "ami-0292c068b453b70cc",
         "ImageLocation": "amazon/Deep Learning AMI GPU TensorFlow 2.6.5 (Ubuntu 18.04) 20220802",
         "ImageType": "machine",
         "Public": true,
         "OwnerId": "898082745236",
         "PlatformDetails": "Linux/UNIX",
         "UsageOperation": "RunInstances",
         "State": "available",
         "BlockDeviceMappings": [
             {
                 "DeviceName": "/dev/sda1",
                 "Ebs": {
                     "DeleteOnTermination": true,
                     "Iops": 3000,
                     "SnapshotId": "snap-06eee5b870416b8ad",
                     "VolumeSize": 35,
                     "VolumeType": "gp3",
:...skipping...

Count the number of AMI images: q

aws ec2 describe-images –owner amazon –output json –query “Images”

To avoid the “skipping”, get just the first Image’s json structure:

aws ec2 describe-images –owner amazon –output json –query “Images[0]”

{
 "Architecture": "x86_64",
 "CreationDate": "2022-07-27T10:44:11.000Z",
 "ImageId": "ami-0d0d43dd4e2a847da",
 "ImageLocation": "amazon/Deep Learning AMI GPU TensorFlow 2.6.5 (Amazon Linux 2) 20220726",
 "ImageType": "machine",
 "Public": true,
 "OwnerId": "898082745236",
 "PlatformDetails": "Linux/UNIX",
 "UsageOperation": "RunInstances",
 "State": "available",
 "BlockDeviceMappings": [
     {
         "DeviceName": "/dev/xvda",
         "Ebs": {
             "DeleteOnTermination": true,
             "Iops": 3000,
             "SnapshotId": "snap-0ab22834e0079e294",
             "VolumeSize": 35,
             "VolumeType": "gp3",
             "Throughput": 125,
             "Encrypted": false
         }
     }
 ],
 "Description": "Built with AWS optimized TensorFlow, NVIDIA CUDA, cuDNN, NCCL, GPU Driver, Docker, NVIDIA-Docker and EFA support. For a fully managed experience, check: https://aws.amazon.com/sagemaker",
 "EnaSupport": true,
 "Hypervisor": "xen",
 "ImageOwnerAlias": "amazon",
 "Name": "Deep Learning AMI GPU TensorFlow 2.6.5 (Amazon Linux 2) 20220726",
 "RootDeviceName": "/dev/xvda",
 "RootDeviceType": "ebs",
 "SriovNetSupport": "simple",
 "VirtualizationType": "hvm",
 "DeprecationTime": "2024-07-27T10:44:11.000Z"
}

Architecture”: “x86_64”,
“PlatformDetails”: “Linux/UNIX”,
State: Only ones available or not?

Name contains Description contains text “Tensorflow”
For different states:

aws ec2 describe-images –owner amazon –output json –query “Images[0].State”
Sort by date: https://itecnote.com/tecnote/amazon-web-services-sort-by-date-with-jmespath/

aws lambda list-functions –query “reverse(sort_by(Functions, &LastModified))[:2]”
Get just has lots of AMI images, so this command returns a lot of lines:

aws ec2 describe-images –owner amazon –output text wc -l
Return just the first json

aws ec2 describe-images –owner amazon –output json
VIDEO: “Regular Expressions for Performance Engineers #10 - JMESPath”
https://blog.scottlowe.org/2019/03/01/advanced-ami-filtering-with-jmespath/

Videos:

https://www.youtube.com/watch?v=gdDl5DywQGI

AWS ec2 describ-volumes

https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-filter.html

PROTIP: Specify first root item using [slices]

aws ec2 describe-volumes –query ‘Volumes[0]’

{
 "Attachments": [],
 "AvailabilityZone": "us-west-2c",
 "CreateTime": "2022-12-19T22:00:45.560000+00:00",
 "Encrypted": false,
 "Size": 32,
 "SnapshotId": "",
 "State": "available",
 "VolumeId": "vol-0be808a8215fc5357",
 "Iops": 100,
 "Tags": [
     {
         "Key": "Name",
         "Value": "eks-cluster-with-new-vpc-dynamic-pvc-fdef34a5-b867-4421-9787-b7b14ab810d6"
     },
     {
         "Key": "ebs.csi.aws.com/cluster",
         "Value": "true"
     },
     {
         "Key": "KubernetesCluster",
         "Value": "eks-cluster-with-new-vpc"
     },
     {
         "Key": "CSIVolumeName",
         "Value": "pvc-fdef34a5-b867-4421-9787-b7b14ab810d6"
     },
     {
         "Key": "kubernetes.io/created-for/pvc/namespace",
         "Value": "kubecost"
     },
     {
         "Key": "kubernetes.io/cluster/eks-cluster-with-new-vpc",
         "Value": "owned"
     },
     {
         "Key": "kubernetes.io/created-for/pv/name",
         "Value": "pvc-fdef34a5-b867-4421-9787-b7b14ab810d6"
     },
     {
         "Key": "kubernetes.io/created-for/pvc/name",
         "Value": "kubecost-cost-analyzer"
     }
 ],
 "VolumeType": "gp2",
 "MultiAttachEnabled": false
}

The VolumeId of attached (flatted with []), with no labels:

aws ec2 describe-volumes –query ‘Volumes[3].[VolumeId, State, AvailabilityZone, Size, VolumeType, Attachments[].[InstanceId, State][]][]’
```
[
 "vol-0be808a8215fc5357",
 "available",
 "us-west-2c",
 32,
 "gp2"
]
```
With labels Multi-select:

aws ec2 describe-volumes
–query ‘Volumes[0].{VolumeId: VolumeId, VolumeType: VolumeType, InstanceId: Attachments[0].InstanceId, State: Attachments[0].State}’
```
{
 "VolumeId": "vol-0be808a8215fc5357",
 "VolumeType": "gp2",
 "InstanceId": null,
 "State": null
}
```
Sorted by Date:

aws ec2 describe-volumes
–output table
–query ‘sort_by(Volumes[], &CreateTime)[].{CreateTime: CreateTime, VolumeId: VolumeId, VolumeType: VolumeType}’

–query ‘sort_by(Volumes[], &VolumeId)[].{VolumeId: VolumeId, CreateTime: CreateTime, VolumeType: VolumeType, InstanceId: Attachments[0].InstanceId, State: Attachments[0].State}’

–filters “Name=availability-zone,Values=us-west-2a” “Name=status,Values=available” \

–query ‘sort_by(Volumes[], &VolumeId)[].{VolumeId: VolumeId, VolumeType: VolumeType, InstanceId: Attachments[].InstanceId, State: Attachments[].State}’

“Name=status,Values=available”

aws ec2 describe-volumes
–filters “Name=availability-zone,Values=us-west-2a” “Name=status,Values=attached”
–query ‘Volumes[?Size > 0].{Id:VolumeId,Size:Size,Type:VolumeType}’

Filter:

aws ec2 describe-volumes
–filters “Name=availability-zone,Values=us-west-1a”

Volumes[].CreateTime

--filters "Name=status,Values=null"

aws ec2 describe-volumes
–filters “Name=availability-zone,Values=us-west-2a” “Name=status,Values=attached”
–query ‘Volumes[?Size > 50].{Id:VolumeId,Size:Size,Type:VolumeType}’

https://random.ac/cess/2017/10/31/interactive-aws-cli-query-filtering/

https://how.wtf/aws-cli-query-examples.html

jpterm

https://random.ac/cess/2017/10/31/interactive-aws-cli-query-filtering/

https://www.geeksforgeeks.org/how-to-install-jmespath-term-on-macos/

sudo pip install jmespath-terminal

JMESPATH in Azure

https://www.codemag.com/Article/2001021/Azure-CLI

az vm list

az vm start -g rg-vm -n wm-Ubuntu18

JMESPATH in Python

VIDEO: JMESPATH in PYTHON - JSON query language

sudo pip install jmespath-terminal

jq (JSON Query utility for Bash CLI scripts)

jq (JSON Query utility for Bash CLI scripts)

Install jp

How to jq

More commands

GitHub API

JMESPATH

JMESPATH –query

AWS EC2 AMI images

AWS ec2 describ-volumes

–filters “Name=availability-zone,Values=us-west-2a” “Name=status,Values=available” \

“Name=status,Values=available”

jpterm

JMESPATH in Azure

JMESPATH in Python

More on API Microservices

Others must know: please click to share:

Recommend

Docker (Glossary, Certification helps, Ecosystem)

China approves first foreign video games since crackdown

Google AdWords scam epidemic shows social engineering is evolving

Amazon begins drone deliveries in California and Texas

Report: Data silos cause employees to lose 12 hours a week chasing data

Hedgeup Presale Expected to Gain 600% By January 2023 Ahead of Dash 2 Trade (D2T...

Cosmos and Rocketize Token Are Racing Toward the Top of the Crypto Charts.

4 quick ways to manage money efficiently

五菱版“坦克100”来了！神似吉姆尼，造型设计硬朗续航超300公里...

华为：将进一步加大操作系统投入开源共建基础软件

About Joyk