Generating a JSON map containing shell variables named in a list

jq select examples
jq 'map
bash read json into variable
jq iterate over array
bash json variable
jq tutorial
bash curl json response to variable
jq foreach

My shell-fu is at a below-beginner level. I have a file that contains some lines that happen to be the names of environment variables.

e.g.

ENV_VAR_A
ENV_VAR_B
...

What I want to do is use this file to generate a JSON string containing the names and current values of the named variables using jq like this:

jq -n --arg arg1 "$ENV_VAR_A" --arg arg2 "$ENV_VAR_B" '{ENV_VAR_A:$arg1,ENV_VAR_B:$arg2}'

# if ENV_VAR_A=one and ENV_VAR_B=two then the preceding command would output 
# {"ENV_VAR_A":"one","ENV_VAR_B":"two"}

I'm trying to create the jq command through a shell script and I have no idea what I'm doing :(

Short and sweet (if you have jq 1.5 or higher):

 jq -Rn '[inputs | {(.): env[.]}] | add' tmp.txt

JSON array to bash variables using jq, There's no need for a map : . At the end, we just assemble a valid shell assignment string with ordinary + concatenation, including quotes around the value. For example, if your JSON data is in a file called file.json : could make your declare -- version safe by comparing $key against a list of allowed variable names. To take advantage of JSON using PowerShell, we must be acquainted with two very important cmdlets: ConvertTo-JSON and ConvertFrom-JSON. There are several ways to manipulate JSON files in PowerShell. A good start is performing a simple tour between JSON and PowerShell object realms performing the tasks that we are describing below.

What you want here is an indirect reference. Those can be done with ${!varname}. As a trivial example limited to exactly two lines:

# read arg1_varname and arg2_varname from the first two lines of file.txt
{ read -r arg1_varname; read -r arg2_varname; } <file.txt

# pass the variable named by the contents of arg1_varname as $arg1 in jq
# and the variable named by the contents of arg2_varname as $arg2 in jq
jq -n --arg arg1_name "$arg1_varname" --arg arg1_value "${!arg1_varname}" \
      --arg arg2_name "$arg2_varname" --arg arg2_value "${!arg2_varname}" \
  '{($arg1_name):$arg1_value, ($arg2_name):$arg2_value}'

To support an arbitrary number of key/value pairs, consider instead something like:

# Transform into NUL-separate key=value pairs (same format as /proc/*/environ)
while IFS= read -r name; do                             # for each variable named in file.txt
  printf '%s=%s\0' "$name" "${!name}"                   # print its name and value, and a NUL
done \
  <file.txt \
  | jq -Rs 'split("\u0000")                             # split on those NULs
            | [.[] | select(.)                          # ignore any empty strings
               | capture("^(?<name>[^=]+)=(?<val>.*)$") # break into k/v pairs
               | {(.name): .val}]                       # make each a JSON map
            | add                                       # combine those maps
  '

Bash jq command – Linux Hint, After running the following command, each object of JsonData variable will be printed. Create a JSON file named Students.json with the following content to test the Without deleting the key from JSON data, you can use map function with jq then the index will count from the left side of the list and starting from zero. Assumes the JSON contains an array of objects and each object may contain a "name" property of type string. Counts objects and "name" property values that end with "University". Assumes the file is encoded as UTF-16 and transcodes it into UTF-8. A file encoded as UTF-8 can be read directly into a ReadOnlySpan<byte>, by using the following code:

jq can look up the values from the environment itself.

$ export A=1
$ export B=2
$ cat tmp.txt
A
B
$ jq -Rn '[inputs] | map({key: ., value: $ENV[.]}) | from_entries' tmp.txt
{
  "A": "1",
  "B": "2"
}

A few notes on how this works:

  1. -R reads raw text, rather than trying to parse the input as JSON
  2. -n prevents jq from reading input itself.
  3. inputs reads all the input explicitly, allowing an array of names to be built.
  4. map creates an array of objects with key and value as the keys; . is the current array input (a variable name), and $ENV[.] is the value of the environment variable whose name is the current array input.
  5. from_entries finally coalesces all those {"key": ..., "value": ...} objects into a single object.

Guide to Linux jq Command for JSON Processing, This means that working with JSON via the command line can be It's also possible to download the binary directly or build it from the source. wrap the property name in quotes when accessing it from the jq command: We typically use arrays to represent a list of items. jq 'map(has("name"))' fruits.json� If you downloaded the object using curl, here is my approach to rename some variables to a friendly name as extract data from data arrays. using eval and filters will solve the problem with one line and will produce variables with the desired name

Try something along the following script in bash:

# array of arguments to pass to jq
jqarg=()
# the script to pass to jq
jqscript=""
# just a number for the arg$num for indexing
# suggestion: just index using variable names...
num=1

# for each variable name from the input
while IFS= read -r varname; do

   # just an assertion - check if the variable is not empty
   # the syntax ${!var} is indirect reference
   # you could do more here, ex. see if such variable exists
   # or if $varname is a valid variable name
   if [[ -z "${!varname}" ]]; then
        echo "ERROR: variable $varname has empty value!" >&2
        exit 50
   fi

   # add the arguments to jqarg array
   jqarg+=(--arg "arg$num" "${!varname}")
   # update jqscript
   # if jqscript is not empty, add a comma on the end
   if [[ -n "$jqscript" ]]; then
      jqscript+=","
   fi
   # add the ENV_VAR_A:$arg<number>
   jqscript+="$varname:\$arg$num"
   # update number - one up!
   num=$((num + 1))

# the syntax of while read loop is that input file is on the end
done < input_file_with_variable_names.txt

# finally execute jq
# note the `{` and `}` in `{$jqscript}` are concious
jq -n "${jqarg[@]}" "{$jqscript}"

Just something that hopefully will give you a easier start with your journey in bash.

I guess I would do something unreadable with xargs like:

< input_file_with_variable_names.txt xargs -d$'\n' -n1 bash -c '
   printf %s\\0%s\\0%s\\0 --arg "$1" "${!1}"
' -- |
xargs -0 sh -c 'jq -n "$@" "$0"' "{$(
     sed 's/\(.*\)/\1: $\1 /' input_file_with_variable_names.txt | 
     paste -sd,
)}"

JSON on the command line with jq, A series of how to examples on using jq, a command-line JSON processor. Suppose that we have a file names.json containing the following json. jq can map values and perform an operation on each one. jq project page � jq manual � jq is sed for JSON � bash that JSON (jq) � Parsing JSON with jq. JSON data is used pretty frequently on the web if you’re hitting APIs. This not only includes external data (twitter, weather, marvel database), but often includes internal data to your company. It’s nice to be able to leverage data from anywhere, and it can be frustrating for people to try to parse JSON data.

Parsing JSON with jq, Most of the popular API and data services use the JSON data format, so we'll You can use it to slice and filter and map and transform structured data with you to edit your ~/.bashrc 's' PATH and create the ~/bin_compciv directory, the first part of the name object, simply list the attributes in sequence: Let us take a real life example on the implementation of the JSON in python. A good source for practice purpose is JSON_placeholder, it provides great API requests package which we will be using in our example. To get started, follow these simple steps. Open Python IDE or CLI and create a new script file, name it sample.py.

JSON Configuration Syntax, Please see the Packer Issue Tracker for a list of supported features. when generating portions of a configuration programmatically, since existing JSON Packer expects native syntax for files named with a .pkr.hcl suffix, and JSON syntax It is not necessary to know all of the details of HCL syntax or its JSON mapping in� Stanza to pull the specified environment variables from a ConfigMap. 2: Name of the ConfigMap to pull specific environment variables from. 3: Environment variable to pull from the ConfigMap. 4: Makes the environment variable optional. As optional, the pod will be started even if the specified ConfigMap and keys do not exist. 5

jq Manual (development version), Colors can be configured with the JQ_COLORS environment variable (see below ). If this option is used then no builtin search list is used. This option reads all the JSON texts in the named file and binds an array of the parsed For any filter x , map(x) will run that filter for each element of the input array, and return the� ${thisFile} – the full path of the CMakeSettings.json file ${name} – the name of the configuration ${generator} – the name of the CMake generator used in this configuration; All references to macros and environment variables in CMakeSettings.json are expanded before being passed to the cmake.exe command line. Ninja command line arguments

Comments
  • @CharlesDuffy which further demonstrates the difficulty I'm having with this....I know it doesn't have to be exported to the environment, but if it is an environment variable this syntax works (not sure if should be doing something different). I've already been able to run the command to do exactly what I wanted, but now I need to do it dynamically (based on a whitelist)
  • @CharlesDuffy yes, but I also want the keys of the generated JSON string to be from the file as well.
  • Awesome! How could I go about using arg1_varname to replace the ENV_VAR_A in the JSON string generate by jq? And for final massive props is there anyway I could do this entire command simply from the list itself WITHOUT knowing the number of arguments (just reading the lines from the file)?
  • I have no words, amazing.
  • ...BUT, I have a further requirement, I am trying to run this in a docker container, FROM nginx:stable , I have installed jq and am running CMD echo $( jq -Rn '[inputs] | map({key: ., value: $ENV[.]}) | from_entries' ./config/env ); but I am getting jq: error: ENV/0 is not defined at <top-level>, line 1: [inputs] | map({key: ., value: $ENV[.]}) | from_entries ....any ideas?
  • You need the newest (1.6, at the time of this writing) version of jq for $ENV.
  • If you have jq 1.5, you can use env[.] in place of $ENV[.].
  • Thanks for the heads up, I just took a look at my Docker build and it's installing 1.5 with apt-get install jq so env[.] will work!
  • Heh. I'm working on something that's maybe a little more elegant, but this is actually how I've solved this problem in practice myself in the past. :)
  • You and @CharlesDuffy are both amazingly helpful, thanks so much to both of you.