How to avoid duplicate file upload but keep the uploader unaware of it?

file is a duplicate of a video you have already uploaded
dropbox duplicate files
dropbox upload file with same name
duplicate file finder
duplicate file remover for pc
how to find duplicate video on youtube
dropbox skip duplicates
duplicate file download

First of all, I apologize if the question is not clear, I'm explaining it below.

For every file uploaded, I'm renaming the file and recording the hash values (using sha1_files function, please suggest if there are some better or faster hashing techniques for the file in php) in a separate DB table and checking the hash of every new file to avoid duplicate files.

In this manner, the one uploading a duplicate file will get an error msg and the file won't be uploaded.

My question is, is there any techniques or algorithm by which I can prevent duplicate file upload but the duplicate file uploader will be unaware of it and will find the file in his/her account with a different name than the one already present. However, users won't be able to upload banned files by any means.

Yes, you should use xxhash which is much faster than sha1.

According to their benchmarks:

The benchmark uses SMHasher speed test, compiled with Visual 2010 on a Windows Seven 32-bits box. The reference system uses a Core 2 Duo @3GHz

SHA1-32 is 0.28 GB/s fast, and xxHash is 5.4 GB/s.

The PHP library is only getting a string as input, so you should use the binary library, and have something like this in your PHP:

list($hash) = explode(" ", shell_exec("/path/to/xxHash/xxhsum " . escapeshellarg($filePath)));
echo $hash;

Installing xxhash:

$ wget https://codeload.github.com/Cyan4973/xxHash/tar.gz/v0.6.3 -O xx.tar.gz
$ tar xvzf xx.tar.gz
$ cd xxHash-0.6.3; make

Marriages, Families, and Relationships: Making Choices in a , Threatening to leave, commit suicide, or report them to welfare. Preventing them from getting or keeping a job. However, a victim who is not yet a citizen may be unaware of this legislation and fear that seeking help could result in deportation. and www.endingviolence.org/files/uploads/​ImmigrantWomenPCwheel.pdf  My question is, is there any techniques or algorithm by which I can prevent duplicate file upload but the duplicate file uploader will be unaware of it and will find the file in his/her account with a different name than the one already present. However, users won't be able to upload banned files by any means.

Just add some extra logic in your code possibly using an extra table or extra fields in the existing table (it is up to you, there is more than one way to do it) that saves the file to an alternate location should you discover it is a duplicate rather than sending an error. Not sure, though, if what you are doing is a good idea from the UI design point of view, as you are doing something different with the user input in a way that the user will notice without telling the user why.

Why Does Dropbox Duplicate Files and How to Stop That from , Learn why Dropbox duplicates files and how to prevent it from happening. as to what ends up in your Dropbox and avoid uploading junk like photos you to upload, download and sync content (and even maintain a version history), but it is  Good day everyone, I want to prevent uploading a file if the name and file extension match one of the saved records. My code below works but how can I avoid duplicate entries? "My sample array" Ar

Use an example like this to generate your sha1 hash client side before upload.

Save all your uploaded files with their hash as the filename, or have a database table which contains the hash and your local filename for each file, also save file size and content type.

Before upload submit hash from client side to your server and check for hash in database. If its not present then commence file upload. If present then fake the upload client side or whatever you want to do so the user thinks they have uploaded their file.

Create a column in your users table for files uploaded. Store a serialised associative array in this column with hash => users_file_name as key=>value pairs. Unserialize and display to each user to maintain their own file names then use readfile to serve them the file with the correct name, selecting it server side using the hash

As for your URL question. Create a page for the downloads but include the user in the url as well, so mysite.com/image.php?user=NewBee&image=filename.jpg

Query the database for files uploaded by NewBee and unserialize the array. Then:

$upload = $_GET['image'];
foreach($array as $hash => $filename){
        if($filename == $upload)
              $file = $hash;
    }

Seach database for the path to your copy of that file, then using readfile you can output the same file with whatever namme you want.

header("Content-Description: File Transfer");
header("Content-type: {$contenttype}");
header("Content-Disposition: attachment; filename=\"{$filename}\"");
header("Content-Length: " . filesize($file));
header('Pragma: public');
header("Expires: 0");
readfile($file);

Fix duplicate uploads - YouTube Help, File is a duplicate of a video you have already uploaded” on the Videos page. Changing the filename alone won't prevent the duplicate upload error message. Fix duplicate uploads. If you upload the same video twice, the second upload will be a duplicate. The duplicate will have a status of “Processing abandoned. File is a duplicate of a video you have

You could create an extra table which links files uploaded (so entries in your table with file hashes) with useraccounts. This table can contain an individual file name for every file belonging to a specific user (so the same file can have a different name per user). With current technologies you could also think about creating the file hash in the browser via javascript and then upload the file only if there isn't already a file with that hash in your database if it is you can instead just link this user to the file.

Addition because of comment: If you want the same file to be accessible through multiple urls you can use something like apache's mod_ rewrite. I'm no expert with that but you can look here for a first idea. You could update the .htaccess dynamically with your upload script.

Copying files off camera drive to Camera uploads D, Our users can answer all of your questions on files and folders. Copying files off camera drive to Camera uploads DOES NOT Prompt of DUPLICATES just files up once I plug in the USB I wanted to stop doing this and trust camera uploads. I agree but its drop box coping it up again and not knowing its already there. You could bind to FilesAdded event (Plupload Events), which gets uploader instance and reference to files array as arguments.Then cycle through the files array with your custom conditional and remove extra items with: uploader.removeFile(files[index_of_the_file_to_remove]);

Photos Synced From Finder Will Be Removed, You'll see a window asking you to remove photos. After carefully testing and reviewing nearly twenty duplicate file finder and and you want to keep happy from taking up a lot of memory and not aware of to computer is it re-backing up/​uploading all photos I have stored on my phone, even if they're already in Dropbox? Ideally, Fine Uploader would prevent duplicate submission of the same file. How to go about that is another problem: Client-side hashing is possible, but produces much more overhead. FU could ensure that no files with the same filename and total size are currently in progress or resumable before allowing an upload to continue.

How to Stop Photos Copying Images & Creating Duplicate Files in , How to Stop Importing (Copying) Pictures to the Photos Library in OS X Importing photos into photos app, but without duplicating the image files should not change the setting at all, keep the default Importing choice enabled. I keep trying to delete photos and it turns around and uploads them again. So I was uploading a video and after uploading I notice it was taking a very long time to process. I gave it a very long time and it did not complete processing so I decided to upload the video again. I uploaded the file again, so I notice I had two of the same files so I deleted the older video that was not processing.

FAQs, I recommend that you leave your GPS on "Smart Recording", but if you have already For TCX and GPX uploads, we can handle about two10Mb files. To prevent the accumulation of Distance, export as TCX, select "Calculate If you get this message, or a "duplicate activity" message from Strava, please  How i can handle duplicate files while uploading, for example, if i want to remove an item from an array, then, i can use item.remove(), like wise, if want to check for duplicate files in queue and if want to handle duplicates.

Comments
  • Sorry to ping you once again. I downloaded the zip meant for php, unzipped and saved it in a folder under site's root (using wamp). Then I tried with your code after pointing a $filePath from the saved list of files but I'm getting a blank $hash. Where am I making mistakes? I'm new to these so may sound silly.
  • @NewBee Please first try to get it work via command line, then implement on PHP. I have to go to sleep, so I will answer tomorrow if you have any more questions
  • Can you suggest me a tutorial on how to install these things (github.com/stuartherbert/php-xxhash#how-to-install). I have never worked with php cmd line and installations. Pinging you in every step is not a good idea.
  • Thanks a lot. All of it worked but after $ cd xxHash-0.6.3; make I'm getting the following error
  • -bash: cd: xxHash-0.6.3: No such file or directory cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef xxhash.c xxhsum.c -o xxhsum.exe process_begin: CreateProcess(C:\cygwin64\bin\cc, cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef xxhash.c xxhsum.c -o xxhsum.exe, ...) failed. make (e=5): Access is denied. make: *** [xxhsum] Error 5
  • thanks for responding but I'm asking for that extra logic. Unable to think of anything. There must be some logic to implement this as I'm quite sure all file/image hosts follow this. They don't save the same file twice or reject any duplicate file even if it is from the same user.
  • I'll work on it and update if this solved my problem. :)
  • Yes, I thought of this but I'm finding a problem here. Say the link is mysite.com/image.php?image=filename.jpg for a fresh file then the duplicate link should be mysite.com/image.php?image=filename1.jpg but filename1.jpg doesn't exist on my server. In case I keep it as mysite.com/image.php?image=hash.jpg then the file name becomes same for both users. How can I overcome this?
  • @NewBee I updated my answer. I hope that guides you to a solution.
  • Update my answer with URL solution