how to identify people's relationship based on name, address and then assign a same ID through linux comman or Pyspark

prosopagnosia
google images
how can i find a person with just a picture
face recognition app
identify person in photo app
people will finish updating when photos is in the background
face recognition theory
recognizing faces involves what cells

I have one csv file.

D,FNAME,MNAME,LNAME,GENDER,DOB,snapshot,Address
2,66M,J,Rock,F,1995,201211.0,J
3,David,HM,Lee,M,1991,201211.0,J
6,66M,,Rock,F,1990,201211.0,J
0,David,H M,Lee,M,1990,201211.0,B
3,Marc,H,Robert,M,2000,201211.0,C
6,Marc,M,Robert,M,1988,201211.0,C
6,Marc,MS,Robert,M,2000,201211.0,D

I want to assign persons with same last name living in the same address a same ID or index. It's better that ID is made up of only numbers. If persons have different last name in the same place, then ID should be different. Such ID should be unique. Namely, people who are different in either address or last name, ID must be different. My expected output is

D,FNAME,MNAME,LNAME,GENDER,DOB,snapshot,Address,ID
2,66M,J,Rock,F,1995,201211.0,J,11
3,David,HM,Lee,M,1991,201211.0,J,12
6,66M,,Rock,F,1990,201211.0,J,11
0,David,H M,Lee,M,1990,201211.0,B,13
3,Marc,H,Robert,M,2000,201211.0,C,14
6,Marc,M,Robert,M,1988,201211.0,C,14
6,Marc,MS,Robert,M,2000,201211.0,D,15

My datafile size is around 30 GB. I am thinking of using groupBy function in spark based on the key consisting of LNAME and address to group those observations together. Then assign it a ID by key. But I don't know how to do this. After that, maybe I can use flatMap to split the line and return those observations with a ID. But I am not sure about it. In addition, can I also make it in Linux environment? Thank you.


Find and identify photos of people using Photos on Mac, We think we recognise people by their face, but looking at their body can be just as important, new research shows. A people search culls many databases, to find both current and historical information. By cross-referencing and comparing information from many different sources over many years, complex search algorithms sift through and verify data, narrowing the thousands of bits of information down so that they relate to the one person being looked for.


Since you have 30GB of input data, you probably don't want something that'll attempt to hold it all in in-memory data structures. Let's use disk space instead.

Here's one approach that loads all your data into a sqlite database, and generates an id for each unique last name and address pair, and then joins everything back up together:

#!/bin/sh

csv="$1"
# Use an on-disk database instead of in-memory because source data is 30gb.
# This will take a while to run.
db=$(mktemp -p .)

sqlite3 -batch -csv -header "${db}" <<EOF
.import "${csv}" people
CREATE TABLE ids(id INTEGER PRIMARY KEY, lname, address, UNIQUE(lname, address));
INSERT OR IGNORE INTO ids(lname, address) SELECT lname, address FROM people;
SELECT p.*, i.id AS ID
FROM people AS p
JOIN ids AS i ON (p.lname, p.address) = (i.lname, i.address)
ORDER BY p.rowid;
EOF

rm -f "${db}"

Example:

$./makeids.sh data.csv
D,FNAME,MNAME,LNAME,GENDER,DOB,snapshot,Address,ID
2,66M,J,Rock,F,1995,201211.0,J,1
3,David,HM,Lee,M,1991,201211.0,J,2
6,66M,"",Rock,F,1990,201211.0,J,1
0,David,"H M",Lee,M,1990,201211.0,B,3
3,Marc,H,Robert,M,2000,201211.0,C,4
6,Marc,M,Robert,M,1988,201211.0,C,4
6,Marc,MS,Robert,M,2000,201211.0,D,5

It's better that ID is made up of only numbers.

If that restriction can be relaxed, you can do it in a single pass by using a cryptographic hash of the last name and address as the ID:

$ perl -MDigest::SHA=sha1_hex -F, -lane '
   BEGIN { $" = $, = "," } 
   if ($. == 1) { print @F, "ID" }
   else { print @F, sha1_hex("@F[3,7]") }' data.csv
D,FNAME,MNAME,LNAME,GENDER,DOB,snapshot,Address,ID
2,66M,J,Rock,F,1995,201211.0,J,5c99211a841bd2b4c9cdcf72d7e95e46b2ae08b5
3,David,HM,Lee,M,1991,201211.0,J,c263f9d1feb4dc789de17a8aab8f2808aea2876a
6,66M,,Rock,F,1990,201211.0,J,5c99211a841bd2b4c9cdcf72d7e95e46b2ae08b5
0,David,H M,Lee,M,1990,201211.0,B,e86e81ab2715a8202e41b92ad979ca3a67743421
3,Marc,H,Robert,M,2000,201211.0,C,363ed8175fdf441ed59ac19cea3c37b6ce9df152
6,Marc,M,Robert,M,1988,201211.0,C,363ed8175fdf441ed59ac19cea3c37b6ce9df152
6,Marc,MS,Robert,M,2000,201211.0,D,cf5135dc402efe16cd170191b03b690d58ea5189

Or if the number of unique lname, address pairs is small enough that they can reasonably be stored in a hash table on your system:

#!/usr/bin/gawk -f
BEGIN {
    FS = OFS = ","
}
NR == 1 {
    print $0, "ID"
    next
}
! ($4, $8) in ids {
    ids[$4, $8] = ++counter
}
{
    print $0, ids[$4, $8]
}

We identify people by their body, when face is no help, The Real Deal: How to accurately identify people online. September 25, 2019. Accuracy is a core tenet of identity resolution. In this context, it's all about  If the photograph is a family portrait or group shot, try to identify other people in the photo. Look for other photos from the same family line which include recognizable details — the same house, car, furniture, or jewelry. Talk to your family members to see if they recognize any of the faces or features of the photograph.


$ sort -t, -k8,8 -k4,4 <<EOD | awk -F, '  $8","$4 != last { ++id; last = $8","$4 }
                                          { NR!=1 && $9=id; print }' id=9 OFS=,
D,FNAME,MNAME,LNAME,GENDER,DOB,snapshot,Address
2,66M,J,Rock,F,1995,201211.0,J
3,David,HM,Lee,M,1991,201211.0,J
6,66M,,Rock,F,1990,201211.0,J
0,David,H M,Lee,M,1990,201211.0,B
3,Marc,H,Robert,M,2000,201211.0,C
6,Marc,M,Robert,M,1988,201211.0,C
6,Marc,MS,Robert,M,2000,201211.0,D
> EOD
D,FNAME,MNAME,LNAME,GENDER,DOB,snapshot,Address
0,David,H M,Lee,M,1990,201211.0,B,11
3,Marc,H,Robert,M,2000,201211.0,C,12
6,Marc,M,Robert,M,1988,201211.0,C,12
6,Marc,MS,Robert,M,2000,201211.0,D,13
3,David,HM,Lee,M,1991,201211.0,J,14
2,66M,J,Rock,F,1995,201211.0,J,15
6,66M,,Rock,F,1990,201211.0,J,15
$

The Real Deal: How to accurately identify people online, Because many people use the same profile photo on various social You can also use SocialMapper, which can identify or recognize any  Google’s image search tool has been there for a while and it has been extensively used to search the internet using images as search queries. The tool is pretty much accurate at identifying similar images which is why it can also server your purpose for identifying a person from his photograph.


How to Find Out Someone's Name From a Picture, New research reveals that when facial features are difficult to make out, we readily use information about someone's body to identify them — even  The technique of gel electrophoresis separates DNA by size, thus allowing people to be identified based on analyzing the lengths of their DNA. We discuss how gel electrophoresis works, and lab footage is shown of this technique being performed in real time.


Identifying people by their bodies when faces are no help , I don't see any way to determine which of the two girls is identified as Betty. I thought that if I hovered the cursor over a face, it would identify it, but  Toxic people, unfortunately, do not walk around the office with little plastic ID tags on their wrists or ankles, like a rare bird species being tracked by biologists in the wild. I wish they did, though. Toxic people can be hard to identify until it’s too late: until you’ve already bonded, or formed a friendship, or taken up a project


Identify the people in a photo, We are social animals of course and we cannot live without talking to people. So, Here is a trick I followed, when I had to identify good people from bad in my  The image recognition features allow you to: Search by People, Places, or Things using the search box option. Refine your search by Date, People, Places, or Things using the filter options. Note: Image recognition in Amazon Photos is enabled by default, unless you are a resident of Illinois. For residents of Illinois, image recognition is