Difference between two lists using Bash

bash difference between two arrays
bash comm
bash diff two variables
show only difference between two files in linux
bash intersection of two arrays
bash set difference
bash array1 array2
bash diff two strings

Ok, I have two related lists on my linux box in text files:

 /tmp/oldList
 /tmp/newList

I need to compare these lists to see what lines got added and what lines got removed. I then need to loop over these lines and perform actions on them based on whether they were added or removed.

How do I do this in bash?

Use the comm(1) command to compare the two files. They both need to be sorted, which you can do beforehand if they are large, or you can do it inline with bash process substitution.

comm can take a combination of the flags -1, -2 and -3 indicating which file to suppress lines from (unique to file 1, unique to file 2 or common to both).

To get the lines only in the old file:

comm -23 <(sort /tmp/oldList) <(sort /tmp/newList)

To get the lines only in the new file:

comm -13 <(sort /tmp/oldList) <(sort /tmp/newList)

You can feed that into a while read loop to process each line:

while read old ; do
    ...do stuff with $old
done < <(comm -23 <(sort /tmp/oldList) <(sort /tmp/newList))

and similarly for the new lines.

Bash scripting problem - compare two lists and create a third, To answer one part of the question, you can use grep with the -F flag to compare the difference between two files. I then combined this with -v to  Set difference of two lists using BASH shell Recently a handful of e-mail messages went undelivered due to some mis-communication between 2 servers. One server had a record of all the addresses it thought it sent to over the period of time in question, and the other server a record of all the addresses to which it had actually delivered

The diff command will do the comparing for you.

e.g.,

$ diff /tmp/oldList /tmp/newList

See the above man page link for more information. This should take care of your first part of your problem.

Set difference of two lists using BASH shell – The Accidental , Set difference of two lists using BASH shell. Recently a handful of e-mail messages went undelivered due to some mis-communication between  Difference between two lists using Bash. 4. How to filter out a set of strings A from a set of strings B using Bash. Related. 639. Get current time in seconds since

Consider using Ruby if your scripts need readability.

To get the lines only in the old file:

ruby -e "puts File.readlines('/tmp/oldList') - File.readlines('/tmp/newList')"

To get the lines only in the new file:

ruby -e "puts File.readlines('/tmp/newList') - File.readlines('/tmp/oldList')"

You can feed that into a while read loop to process each line:

while read old ; do
  ...do stuff with $old
done < ruby -e "puts File.readlines('/tmp/oldList') - File.readlines('/tmp/newList')"

bash - Find difference between two variables, There are many ways You can use sort , tr unique and paste and $() to execute them and "transform the output in a variable" #!/bin/bash  The diff command compares two files and produces a list of the differences between the two. To be more accurate, it produces a list of the changes that would need to be made to the first file, to make it match the second file. If you keep that in mind you’ll find it easier to understand the output from diff.

This is old, but for completeness we should say that if you have a really large set, the fastest solution would be to use diff to generate a script and then source it, like this:

#!/bin/bash

line_added() {
   # code to be run for all lines added
   # $* is the line 
}

line_removed() {
   # code to be run for all lines removed
   # $* is the line 
}

line_same() {
   # code to be run for all lines at are the same
   # $* is the line 
}

cat /tmp/oldList | sort >/tmp/oldList.sorted
cat /tmp/newList | sort >/tmp/newList.sorted

diff >/tmp/diff_script.sh \
    --new-line-format="line_added %L" \
    --old-line-format="line_removed %L" \
    --unchanged-line-format="line_same %L" \
    /tmp/oldList.sorted /tmp/newList.sorted

source /tmp/diff_script.sh

Lines changed will appear as deleted and added. If you don't like this, you can use --changed-group-format. Check the diff manual page.

Compare two user lists in bash, One is the passwd file, and another is a flat file for that has the usernames and other information that I can extract a sorted username list from. This  Normally, to compare two files in Linux, we use the diff – a simple and original Unix command-line tool that shows you the difference between two computer files; compares files line by line and it is easy to use, comes with pre-installed on most if not all Linux distributions.

I typically use:

diff /tmp/oldList /tmp/newList | grep -v "Common subdirectories"

The grep -v option inverts the match:

-v, --invert-match Selected lines are those not matching any of the specified pat- terns.

So in this case it takes the diff results and omits those that are common.

Bash script to compare two lists - UNIX and Linux Forums, If VAR2 does not contain an entry in VAR1 create this missing entry and do xyz else do yz. Original Post by GermanJulian. radoulov. Given you're using  The output from the diff command shows that between lines two and four of the first file and lines two and three of the second file, there are differences. It then lists the lines from two to four from the first file, followed by the two different lines in the second file. How to Only Show If the Files Are Different

Bash script to diff two arrays, creates and array, yet you test for "$diffout" a plain variable.. What is with the yes piping in to cp -f (which is force mode in itself). hi i am using bash shell to perform some subraction. here is what i have: i have a while loop and am using i as a counter. diff= `expr ${ARRAY1} - ${ARRAY2}` for example array1 has -0.7145 and array2 has -0.7041. when i try the above command, i get expr: non-numeric argument. any

Comparing files and directories with the diff and comm Linux , Here's some diff output — displaying the lines that are different in the two files and using < and > signs to indicate which file each line came  Certain situations require you to quickly confirm which files between two directories are different, and while your particular requirements may suggest writing a script for this task, I want to make sure you’re familiar with the basics first – majority of directory comparisons can be done using diff command (yes, that’s right – the same one used for comparing files).

Nine ways to compare files on Unix, If you want to see the differences between two files, but not the instructions that 2-Minute Linux Tip: Learn how to use the alias command. Overview of Unix File Comparison Commands : In this tutorial, we will cover the different ways involved for comparing two files. The file comparison command helps us to compare the files and find the similarities and differences between these files. The different file comparison commands used in Unix are cmp, comm, diff, dircmp, and uniq.

Comments
  • The same question was asked 4 days before stackoverflow.com/questions/11099894/…
  • I'll just emphasize that the diff command has a ridiculous number of options for formatting the output, which could provide a convenient input to the program that will process the differences.
  • @chepner good point .. it's definitely worth checking out the linked man page.