Duplicate output looping through multiple values in while loop bash

Duplicate output looping through multiple values in while loop bash

bash for loop array index
bash loop through array of strings
bash for loop range
bash while loop
bash loop through string
bash while read multiple variables
bash array
bash nested for loop

I'm trying to loop through the contents of 3 files using a while loop in bash. They contain a list of names, aws accounts and aws account numbers.

But the loop isn't correct and it keeps repeating the first name from the list and the first aws environment from the list.

This is the output I see:

AWS user name: aadatiya does not exist in AWS account: company-lab
AWS user name: aadatiya does not exist in AWS account: company-lab
AWS user name: aadatiya does not exist in AWS account: company-lab
AWS user name: aadatiya does not exist in AWS account: company-lab
AWS user name: aadatiya does not exist in AWS account: company-lab
AWS user name: aadatiya does not exist in AWS account: company-lab
AWS user name: aadatiya does not exist in AWS account: company-lab
AWS user name: aadatiya does not exist in AWS account: company-lab

These are the files I'm trying to read from:

aws_users_all="source_files/aws_users_all.txt"

Sample output:

aadatiya
abigailcharles 
tdunphy
broberts

Next file:

aws_env_list="source_files/aws_environments_all.txt"

Sample output:

company-lab
company-stage
company-nonprod
company-prod

Last file: aws_account_numbers="source_files/aws_account_numbers.txt" Sample output:

  123456789191
  987654321211
  456721231213
  123213512321

And this is the code with the incorrect loop:

ofile=source_files/aws_access_keys/company-aws-access-keys-all-accounts.csv
while IFS= read -r aws_user_name
  do 
    while IFS= read -r aws_key
    do
      while IFS= read -r aws_account_num
      do
        user_lives_here=$(aws iam get-user --user-name "$aws_user_name" --profile="$aws_key" 2> /dev/null  | jq -r '.User.UserName')
        if [[ -z "$user_lives_here" ]]; then
          printf "AWS user name: %s does not exist in AWS account: %s\\n(%s)" "$aws_user_name" "$aws_key" "$aws_account_num"
        else
          echo "$aws_user_name,$user_access_key1,$key1_date_created,$key1_last_used,$key1AgeDays,$aws_key,$aws_account_num" >> $ofile
      fi
      done < "$aws_account_numbers"
    done < "$aws_env_list"
  done < "$aws_users_all"

If I take out one level (the account numbers level) the script behaves as expected and produces this output:

AWS user name: aadatiya does not exist in AWS account: company-lab
AWS user name: aadatiya does not exist in AWS account: company-bill
AWS user name: aadatiya does not exist in AWS account: company-stage
AWS user name: aadatiya does not exist in AWS account: company-dlab
AWS user name: aadatiya does not exist in AWS account: company-nonprod
AWS user name: aadatiya does not exist in AWS account: company-prod
AWS user name: aadatiya does not exist in AWS account: company-govcloud-admin-nonprod
AWS user name: abigailcharles does not exist in AWS account: company-lab
AWS user name: abigailcharles does not exist in AWS account: company-bill
AWS user name: abigailcharles does not exist in AWS account: company-stage
AWS user name: abigailcharles does not exist in AWS account: company-dlab
AWS user name: abigailcharles does not exist in AWS account: company-nonprod
AWS user name: abigailcharles does not exist in AWS account: company-prod
AWS user name: abigailcharles does not exist in AWS account: company-govcloud-admin-nonprod

I just commented out this level and it works:

  #while IFS= read -r aws_account_num
  #do
  #done

How can I do this correctly so that I loop through each name, aws account and aws account number so that each entry shows once?


You want one loop that calls read three times, using three different file descriptors.

while IFS= read -r aws_user_name
      IFS= read -r aws_key <&3
      IFS= read -r aws_account_num <&4; do
    ...
done < "$aws_users_all" 3< "$aws_env_list" 4< "$aws_account_numbers"

Duplicate output looping through multiple values in while loop bash, I'm trying to loop through the contents of 3 files using a while loop in bash. They contain a list of names, aws accounts and aws account  Looping through the array based on its length. The idea here is we find the length of the array and loop that many times through it starting from zero and adding one to the index each time. So it will look something like this: #looping for ((i=0; i<${#array3[*]}; i++)); do echo ${array3[i]} done It gives us:


There is a way using paste command:
while read aws_user_name aws_key aws_account_num ;do
    printf "User: %-16s Key: %-22s AccNum: %s\n" \
        $aws_user_name $aws_key $aws_account_num
    # ...
done < <(
    paste "$aws_users_all" "$aws_env_list" "$aws_account_numbers"
)

Sample:

User: aadatiya         Key: company-lab            AccNum: 123456789191
User: abigailcharles   Key: company-stage          AccNum: 987654321211
User: tdunphy          Key: company-nonprod        AccNum: 456721231213
User: broberts         Key: company-prod           AccNum: 123213512321

Nota: this will work simply while each file do contain 1 word by line.

... And if you plan to use space in your source files, you could use another delimiter:

while IFS=@ read -r aws_user_name aws_key aws_account_num ;do
    printf "User: %-16s Key: %-22s AccNum: %s\n" \
        "$aws_user_name" "$aws_key" "$aws_account_num"
    # ....
done < <(
    paste -d@ "$aws_users_all" "$aws_env_list" "$aws_account_numbers"
)
Or playing with IO under pure bash

Near @chepner solution:

exec 3<$aws_env_list
exec 4<$aws_account_numbers
while read aws_user_name ;do
    read -u 3 aws_key
    read -u 4 aws_account_num
    printf "User: %-16s Key: %-22s AccNum: %s\n" \
        "$aws_user_name" "$aws_key" "$aws_account_num"
    # ...
done <$aws_users_all
exec 3<&-
exec 4<&-

Nota: last two lines will close FD 3 and 4.

how can we use multiple variables in single for loop in shell script , bash allows for loops with more than one variable, but only in C like syntax: for ((i​=0 But using a shell loop to process text is often the wrong way to go. Here split the output of awk or other command while [ "$#" -gt 0 ]; do echo "$1:$2" shift 2 done Just duplicate each of the FOUR lines which handle the fields {A, B, }. I'm trying to loop through the contents of 3 files using a while loop in bash. They contain a list of names, aws accounts and aws account numbers. But the loop isn't correct and it keeps repeating the first name from the list and the first aws environment from the list. This is the output I see:


One way to do it is to load all of the input data into arrays before processing it:

IFS= mapfile -t aws_users <"$aws_users_all"
IFS= mapfile -t aws_envs <"$aws_env_list"
IFS= mapfile -t aws_acct_nums <"$aws_account_numbers"

for aws_user_name in "${aws_users[@]}" ; do
    for idx in ${!aws_envs[*]} ; do
        aws_key=${aws_envs[idx]}
        aws_account_num=${aws_acct_nums[idx]}

        # Code using $aws_user_name, $aws_key, and $aws_account_num
        ...
    done
done

Note that mapfile was introduced with Bash 4.0 so this won't work if you are using an older version of Bash. See Read lines from a file into a Bash array for alternatives that work with older versions (but note that the accepted answer is very bad).

Loading the files into memory may seem like a bad idea, but it is unlikely to be a problem in practice. Modern machines have a lot of memory, and Bash is very slow at almost everything. If there is too much data to load into memory then there is probably way too much data to process line-by-line with Bash code.

Bash For Loop Array: Iterate Through Array Values, Arrays are indexed using integers and are zero-based. To declare an array in bash. Declare and an array called array and assign three values:  UNIX shell script for and while loops are key weapons in every shell programmer's arsenal. Although there are two additional looping constructs, select and until, I have almost always been able to efficiently accomplish my UNIX shell scripting objectives with either a for loop or a while loop.


For and Read-While Loops in Bash, This is fundamentally different than the line-by-line command-and-response A read-while loop is a variation of the above, but is safer for reading lines from a file​: When the loop executes, the loop variable, x , takes the value of each of the into the while loop, which prints to stdout using echo , which is redirected to the  UNIX: Loop Through Files In A Directory; Bash read file names from a text file and take action; Bash Shell Check Whether a Directory is Empty or Not; KSH For Loop Examples; HowTo: Skip Bash For Loop; Linux bash for loop; Bash Script: Read One Character At A Time; Shell Script While Loop Examples; Use BASH nullglob To Verify *.c Files Exists or


Advanced Bash-Scripting Guide, During each pass through the loop, arg takes on the value of each Output then piped to "grep", which tests for desired string. do echo $word Example 11-17. while loop with multiple conditions Hint: The function duplicates 'isdigit ()', #+ but adds a test for a mandatory decimal point. The second type of command to loop through stuff is while command. In while command, the body will get executed as long the condition specified in the while command is true. So, the idea here is that while the condition is true, it will keep looping forever. So, it is your responsibility to make sure the condition becomes false at some point in time.


The Complete Guide to Loops in JavaScript, To iterate simply means to repeat an action multiple times. for, for…of, for…in, while, Array. are similar to loops / iterators: Array.values(), Array.keys(), Array.​map(), For loops require 3 statements separated by two semicolons, which can The console output will be the same as the previous example. Using Shell Script To Loop Program Through Multiple Text Files Hello, So I have approximately 300 files of raw data (.txt) files that I am using to perform statistical analysis. I have been able to construct a Fortran program that is able to perform my statistical analysis on a file by file basis.