Accessing Associative Arrays in GNU Parallel

must use subscript when assigning associative array
bash associative array
example bash associative array
bash merge associative arrays
bash for each associative array
associative array in unix
bash convert array to associative array
shell associative array loop

Assume the following in Bash:

declare -A ar='([one]="1" [two]="2" )'
declare -a ari='([0]="one" [1]="two")'
for i in ${!ari[@]}; do 
  echo $i ${ari[i]} ${ar[${ari[i]}]}
done
0 one 1
1 two 2

Can the same be done with GNU Parallel, making sure to use the index of the associative array, not the sequence? Does the fact that arrays can't be exported make this difficult, if not impossible?

A lot has happened in 4 years. GNU Parallel 20190222 comes with env_parallel. This is a shell function that makes it possible to export the most of the environment to the commands run by GNU Parallel.

It is supported in ash, bash, csh, dash, fish, ksh, mksh, pdksh, sh, tcsh, and zsh. The support varies from shell to shell (see details on https://www.gnu.org/software/parallel/env_parallel.html). For bash you would do:

# Load the env_parallel function
. `which env_parallel.bash`
# Ignore variables currently defined
env_parallel --session
[... define your arrays, functions, aliases, and variables here ...]
env_parallel my_command ::: values
# The environment is also exported to remote systems if they use the same shell
(echo value1; echo value2) | env_parallel -Sserver1,server2 my_command
# Optional cleanup
env_parallel --end-session

So in your case something like this:

env_parallel --session
declare -A ar='([one]="1" [two]="2" )'
declare -a ari='([0]="one" [1]="two")'
foo() {
  for i in ${!ari[@]}; do 
    echo $i ${ari[i]} ${ar[${ari[i]}]}
  done;
}
env_parallel foo ::: dummy
env_parallel --end-session

As you might expect env_parallel is a bit slower than pure parallel.

Accessing Associative Arrays in GNU Parallel, Accessing Associative Arrays in GNU Parallel. Question. Assume the following in Bash: declare -A ar='([one]="1" [two]="2" )' declare -a ari='([0]="one" [1]="two")'� Arrays; Accessing Array Elements; Array Assignments; Array from string; Array insert function; Array Iteration; Array Length; Array Modification; Associative Arrays; Destroy, Delete, or Unset an Array; List of initialized indexes; Looping through an array; Reading an entire file into an array; Associative arrays; Avoiding date using printf

Yes, it makes it trickier. But not impossible.

You can't export an array directly. However, you can turn an array into a description of that same array using declare -p, and you can store that description in an exportable variable. In fact, you can store that description in a function and export the function, although it's a bit of a hack, and you have to deal with the fact that executing a declare command inside a function makes the declared variables local, so you need to introduce a -g flag into the generated declare functions.

UPDATE: Since shellshock, the above hack doesn't work. A small variation on the theme does work. So if your bash has been updated, please skip down to the subtitle "ShellShock Version".

So, here's one possible way of generating the exportable function:

make_importer () {
  local func=$1; shift; 
  export $func='() {
    '"$(for arr in $@; do
          declare -p $arr|sed '1s/declare -./&g/'
        done)"'
  }'
}

Now we can create our arrays and build an exported importer for them:

$ declare -A ar='([one]="1" [two]="2" )'
$ declare -a ari='([0]="one" [1]="two")'
$ make_importer ar_importer ar ari

And see what we've built

$ echo "$ar_importer"
() {
    declare -Ag ar='([one]="1" [two]="2" )'
declare -ag ari='([0]="one" [1]="two")'
  }

OK, the formatting is a bit ugly, but this isn't about whitespace. Here's the hack, though. All we've got there is an ordinary (albeit exported) variable, but when it gets imported into a subshell, a bit of magic happens [Note 1]:

$ bash -c 'echo "$ar_importer"'

$ bash -c 'type ar_importer'
ar_importer is a function
ar_importer () 
{ 
    declare -Ag ar='([one]="1" [two]="2" )';
    declare -ag ari='([0]="one" [1]="two")'
}

And it looks prettier, too. Now we can run it in the command we give to parallel:

$ printf %s\\n ${!ari[@]} |
    parallel \
      'ar_importer; echo "{}" "${ari[{}]}" "${ar[${ari[{}]}]}"'
0 one 1
1 two 2

Or, for execution on a remote machine:

$ printf %s\\n ${!ari[@]} |
    parallel -S localhost --env ar_importer \
      'ar_importer; echo "{}" "${ari[{}]}" "${ar[${ari[{}]}]}"'
0 one 1
1 two 2

ShellShock version.

Unfortunately the flurry of fixes to shellshock make it a little harder to accomplish the same task. In particular, it is now necessary to export a function named foo as the environment variable named BASH_FUNC_foo%%, which is an invalid name (because of the percent signs). But we can still define the function (using eval) and export it, as follows:

make_importer () {
  local func=$1; shift; 
  # An alternative to eval is:
  #    . /dev/stdin <<< ...
  # but that is neither less nor more dangerous than eval.
  eval "$func"'() {
    '"$(for arr in $@; do
          declare -p $arr|sed '1s/declare -./&g/'
        done)"'
  }'
  export -f "$func"
}

As above, we can then build the arrays and make an exporter:

$ declare -A ar='([one]="1" [two]="2" )'
$ declare -a ari='([0]="one" [1]="two")'
$ make_importer ar_importer ar ari

But now, the function actually exists in our environment as a function:

$ type ar_importer
ar_importer is a function
ar_importer () 
{ 
    declare -Ag ar='([one]="1" [two]="2" )';
    declare -ag ari='([0]="one" [1]="two")'
}

Since it has been exported, we can run it in the command we give to parallel:

$ printf %s\\n ${!ari[@]} |
    parallel \
      'ar_importer; echo "{}" "${ari[{}]}" "${ar[${ari[{}]}]}"'
0 one 1
1 two 2

Unfortunately, it no longer works on a remote machine (at least with the version of parallel I have available) because parallel doesn't know how to export functions. If this gets fixed, the following should work:

$ printf %s\\n ${!ari[@]} |
    parallel -S localhost --env ar_importer \
      'ar_importer; echo "{}" "${ari[{}]}" "${ar[${ari[{}]}]}"'

However, there is one important caveat: you cannot export a function from a bash with the shellshock patch to a bash without the patch, or vice versa. So even if parallel gets fixed, the remote machine(s) must be running the same bash version as the local machine. (Or at least, either both or neither must have the shellshock patches.)


Note 1: The magic is that the way bash marks an exported variable as a function is that the exported value starts exactly with () {. So if you export a variable which starts with those characters and is a syntactically correct function, then bash subshells will treat it as a function. (Don't expect non-bash subshells to understand, though.)

Bash, aa=([hello]=world [ab]=cd ["key with space"]="hello world"). Access an associative array element echo ${aa[hello]} # Out: world. Listing associative array keys gnu parallel is magic and everyone should use it. Spark likes uncompressed data and does not like combining partitions. Spark is a lot of overhead for simple jobs. Associative arrays in AWK are super powerful. You can access stdin and stdout from inside an R script and thus use it in a pipeline. S3 can handle a lot of files due to smart path

GNU Parallel is a perl program. If the perl program cannot access the variables, then I do not see a way that the variables can be passed on by the perl program.

So if you want to parallelize the loop I see two options:

declare -A ar='([one]="1" [two]="2" )'
declare -a ari='([0]="one" [1]="two")'
for i in ${!ari[@]}; do 
  sem -j+0 echo $i ${ari[i]} ${ar[${ari[i]}]}
done

The sem solution will not guard against mixed output.

declare -A ar='([one]="1" [two]="2" )'
declare -a ari='([0]="one" [1]="two")'
for i in ${!ari[@]}; do 
  echo echo $i ${ari[i]} ${ar[${ari[i]}]}
done | parallel

Exporting the full environment to GNU Parallel - bash - iOS, But reading Accessing Associative Arrays in GNU Parallel I am starting to think: Does it really have to be this way? Is is possible to make a bash function, that� C __gnu_parallel::__accumulate_binop_reduct< _BinOp > General reduction, using a binary operator C std::__add_pointer_helper< _Tp, bool > Add_pointer C std::__add

Arrays and associative arrays, a free sample. Start a free trial to access the full title and Packt library. Associative arrays have been introduced to Bash from Version 4.0. When the indices� Parallel reduction algorithm typically refers to an algorithm which combines an array of elements, producing a single result. Typical problems that fall into this category are: summing up all elements in an array; finding a maximum in an array; In general, the parallel reduction can be applied for any binary associative operator, i.e. (A*B)*C = A*(B*C). With such operator *, the parallel reduction algorithm repetedely groups the array arguments in pairs.

Arrays (Bash Reference Manual), Bash provides one-dimensional indexed and associative array variables. Any variable may be used as an indexed array; the declare builtin will explicitly declare� GNU parallel runs the same (or different) commands with different arguments in parallel possibly using remote computers to help computing. If more than one computer is listed in -S GNU parallel may only use one of these (e.g. if there are 8 jobs to be run and one computer has 8 cores). GNU parallel can be used as a poor-man's version of ClusterSSH:

Bash Reference Manual, Next: GNU Parallel, Previous: Compound Commands, Up: Shell Substring expansion applied to an associative array produces undefined results If command is supplied, it replaces the shell without creating a new process� How to supercharge your bash workflows with GNU parallel. Sorting associative arrays in PHP with array_multisort() Get unlimited access to the best stories on Medium — and support

Comments
  • Maybe I didn't understand what you were looking for, or maybe you hit the accept button too fast. Let me know :)
  • Rici - both answers help me parallelize the processing of the arrays, so both answers are acceptable. I didn't realize clicking on the check mark would limit the number of answers. Thanks for the illuminating answer!
  • Wait, what? I want parallel to do the loop, not have parallel (or env_parallel) just run a function that does the loop. This works too: env_parallel 'echo {} ${ari[{}]} ${ar[${ari[{}]}]}' ::: ${!ari[@]}
  • I have never seen the make_importer func before. Can I use it in GNU Parallel's documentation?
  • One question - the -exp argument to parallel - is that in long form --eof=xp? Doesn't seem like it - is it supposed to be --env?
  • Do try running parallel with '-vv -S localhost --env ar_importer'. How many \ do you see :-)
  • @OleTange: Yep, -exp was a typo. Shows my inexperience with parallel. Fixed, thanks. You're certainly free to reproduce the hack. You can even give me credit; I think I've committed enough hacks in my life that one more won't embarrass me too much. I've never used it with parallel, but I sometimes need to export arrays into subshells; I don't think I've ever published it before. Obviously you only want to run ar_importer once in a target shell, which requires a deeper understanding of parallel options than I possess, but I'm sure you'll have no trouble optimizing it.
  • A further developed version of this hack is today an integral part of env_parallel (a wrapper for GNU Parallel that exports the full environment).