Tired of filtering semi-structured data from command output? What if I tell you can easily filter from structured or unstructured document data. Yes, you heard it right, we are talking about JSON [JavaScript Object Notation]. JSON is a text-based data format programming language that is used to serialize and transmitting structured or unstructured data or we can say semi-structured data over a network connection.
But Why Json ?
We generally think JSON means data provided general output from any API. But, if we talk about any technology, JSON is a very common programming language or we can say format, which is used as the output format of any resource. While using DevOps tools like Docker, Ansible, or any other tool, we generally get the output in JSON format when we use any output command like Docker inspect or Ansible facts gather.
Giant cloud technology like AWS, provides every AWS CLI output in JSON format and why is it so? As per AWS official documentation, JSON is the default output format of the AWS CLI. Most programming languages can easily decode JSON strings using built-in functions or with publicly available libraries.
Jokes apart, JSON can be easily decoded/filter/parse by most programming languages in an easy manner and it is human readable. Most of the IDE also provides in-build plugins or extra plugins to filter or parse JSON data. Like, visual studio, it has an in-build function that can easily track and display the current position in the top bar of the IDE.
Hidden man – JQ
As per official documentation, JQ is like sed for JSON data – you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text. Let’s explain this is in simple terms, the JQ command-line tool used as a filtering command is used to filter, modify, or even fetch multiple data at once. This is a popular tool that is used to parse or filter widely popular text-based data format programming languages.
Ok ! Ok ! We’ve talked much about JSON and this hidden boy. Let’s now why this tool is so popular in terms of other tools.
Why jq ?
When we talk about output format and at the same time parsing using the command line, JQ is a very popular JSON parsing or filtering tool.
There are some JQ command-line tool alternatives that you can also prefer and get your hands to:
- fx
- Emuto
- Jello
Let’s talk about why JQ is the most popular tool in terms of multiple things.
Multi-OS Support
One of the important aspects of using such kind of tool is because it has an interoperability feature that gives options to set up the command-line tools in any kind of OS flavour. Whether, it is Unix/Linux based or MacOS based, or even windows. You can easily download and start using the JQ tool. To know more about installation, visit the official documentation.
User-friendly
And for me, the most important thing is, if we are creating or discussing open-source, we generally prefer how user-friendly and interactive their documentation is. If you visit the official documentation, you can even check simulate and get an idea of how this actually works.
Multi-Option Output
One more important feature of JQ is interactive output, as we can set output in multiple formats like we want to output in single lines or multiple lines. It also supports multiple colours coding for proper and interactive visualization.
Flexible Options
Apart from other options, the tool or utility can stand its ground only if they provide one and the most important thing which is flexibility. This means, tools should provide a variety of options or features using which we can use utility options to set up our needs.
Useful commands
There are multiple options and features of JQ commands which can be used to design and configure scripts. JQ is not only limited to its options but there are multiple ways through which can use the JQ command as per our needs. Let’s discuss a few of the options provided by JQ.
Multi-line JSON output
Let’s say if you have a single line JSON code and you want to format in a proper manner and readable format, you can use the JQ command to format single-line JSON code. Use the below options to parse JSON in multi-line readable code.
Input:
root@opstree:~# cat test.json
[{"name":"unihal","age":28,"skills":["k8s","aws"]},{"name":"kumar","age":23,"skills":["sls","aws"]},{"name":"rawat","age":21,"skills":["aws","bash"]}]
The above code block contains an example of a file that contains single-line JSON content.
Command:
$ cat test.json | jq '.'
While getting JSON output using the cat command, we can use jq command after using pipe or we can directly use jq '.' test.json to print in multiple lines.
Output:
[
{
"name": "unihal",
"age": 28,
"skills": [
"k8s",
"aws"
]
},
{
"name": "kumar",
"age": 23,
"skills": [
"sls",
"aws"
]
},
{
"name": "rawat",
"age": 21,
"skills": [
"aws",
"bash"
]
}
]
By running the above command, you will get stdout which you can redirect or filter again using other commands like JQ.
Below is the illustration of the above commands that we used to get output in multi readable lines.
Single-line JSON output
If you want to convert multi-line JSON code into a single line of code, you can use the below -c option with the JQ command to gather your output.
Input:
root@opstree:~# cat test.json
[
{
"name": "unihal",
"age": 28,
"skills": [
"k8s",
"aws"
]
},
{
"name": "kumar",
"age": 23,
"skills": [
"sls",
"aws"
]
},
{
"name": "rawat",
"age": 21,
"skills": [
"aws",
"bash"
]
}
]
The above file contains multi-line readable JSON output which can be used to visualize the filter for further use-case.
Command:
$ cat test.json | jq -c '.'
Output:
[{"name":"unihal","age":28,"skills":["k8s","aws"]},{"name":"kumar","age":23,"skills":["sls","aws"]},{"name":"rawat","age":21,"skills":["aws","bash"]}]
Check the below illustration to verify how it will print values in single JSON line output.
Multiple values from JSON
The above command gives a full output which also includes the key-value which we don’t need. If there is a requirement of multiple key values at once but not all at once. JQ provides multi-line keys parse from JSON.
Input:
root@opstree:~# cat test.json
{
"name": "kumar",
"age": 23,
"skills": [
"sls",
"aws"
]
}
The above JSON output contains few entries like the name which is a string type, age is numeric value & skills is an array which contains multiple values.
Command:
cat test.json | jq -r '.name, .age'
In the above command, we filter name & age from JSON file, we can filter multiple accordingly.
Output:
kumar
23
Below is the syntax command which displays that we can print multiple keys values at once which are separated by the new line.
cat test.json | jq -r '.KEY1, .KEY2, .KEYn'
There can be multiple use-cases where we want a few of the value parts from JSON. Below is the diagram representation of multi-value parse from JSON input.
Matching value pattern
If we want JSON output or output, only if there is a matching pattern, means output print only if the key-value matches the value that we provided. Below is the method of displaying an example of this use case.
The below command signifies the method of implementing matching value & key.
- KEY = key part which needs to be check
- VALUE = it checks if matches the values or not
Syntax:
cat test.json | jq ". | select (.KEY| contains("VALUE"))"
Input:
root@opstree:~# cat test.json
{
"name": "kumar",
"age": 23,
"skills": [
"sls",
"aws"
]
}
Command:
cat test.json | jq ". | select (.age | contains(23))"
Output:
{
"name": "kumar",
"age": 23,
"skills": [
"sls",
"aws"
]
}
Key checker
What if we don’t want to modify the values with that of the keys but we want to display only if the key is present. There can be use-cases where we first want to check whether this key is present or not and on the basis of that we want to implement. So, JQ also provided this feature where you can check whether a key part of JSON output is present or not
Input:
root@opstree:~# cat test.json
{
"name": "kumar",
"age": 23,
"skills": [
"sls",
"aws"
]
}
Command:
cat test.json | jq -r ".[] | select(.name)"
As you can see the above command, we check whether the name key is present in the current JSON output. We can check by using an unknown key whether we are getting output or not.
Output:
{
"name": "kumar",
"age": 23,
"skills": [
"sls",
"aws"
]
}
Modify matching key values
JQ command also provided the option to change the value of the key from the current value to a different one we can provide. It’s a good option when we want to modify all the entries without using the external UNIX-LINUX command.
Input :
root@opstree:~# cat test.json
{
"name": "kumar",
"age": 23,
"skills": [
"sls",
"aws"
]
}
Command:
cat test.json | jq ".name=\"rawat\""
The above command provides a representation of modifying a specific key of JSON input. For now, we are modifying the name key to different values.
Output:
{
"name": "rawat",
"age": 23,
"skills": [
"sls",
"aws"
]
}
Conclusion
We covered JSON fundamentals & their use cases in real life. We discussed why JSON is a very popular option for multiple cases. There are lots of JSON parsing tools but why JQ is very popular in terms of command-line utility for JSON parsing. We’ve only covered few of the options that are provided by the JQ command-line utility and this is not even close. There are lots of options and features provided by JQ which we didn’t cover in this blog.
Let us know in the comment section if you know many options related to JQ and also let us know if you want us to cover other alternatives to JQ in our future blog series.
Blog Pundit: Naveen Verma and Sanjeev Pandey
Opstree is an End to End DevOps solution provider
Connect Us
DEVOPS DONE RIGHT 