Boto3 not uploading zip file to S3 python

upload file to s3 python boto3
boto upload file to s3 directory
boto3 upload file to s3 folder
s3 zip multiple files python
boto3 upload large file to s3
s3 upload file boto3 acl
python zip file
upload csv to s3 python

I'm trying to upload a .zip file to S3 using boto3 for python but the .zip file in my directory is not uploaded correctly. The code downloads all emails of a given user, zips them in the same directory and uploads them to an S3 bucket. The problem is that the file that gets uploaded is not the one I intend to upload. Instead a file with 18kb only appears.

Here's the code:

import sys
import imaplib
import getpass
import email
import shutil
import boto3
import os

username = input("Enter user's first name: ")
surname = input("Enter user's surname: ")
email_address = username + "." + surname + "@gmail.com"
password = getpass.getpass()
directory = username + surname + '/'
def download_emails(server):
    result, data = server.uid('search', None, "ALL")    #search all email and return their uids
    if result == 'OK':
        for num in data[0].split():
            result, data = server.uid('fetch', num, '(RFC822)')    #RFC is a standard for the format of ARPA Internet text messages
            if result == 'OK':
                email_message = email.message_from_bytes(data[0][1])    #raw email text including headers
                file_name = email_message['Subject']       #use dates and file names(can be changed)
                if not os.path.exists(directory):
                    os.makedirs(directory)      #create a dir for user's emails
                try:
                    email_file = open(directory + file_name+'.eml', 'wb')   #open a file for each email and insert the data.
                    email_file.write(data[0][1])
                    email_file.close()
                except:
                    pass

#function to zip all the emails
def archive(zipname, directory):
    return shutil.make_archive(zipname, 'zip', root_dir=directory, base_dir=None)

#function to upload zipped emails to AWS bucket
def upload_to_s3(file_name):
    s3 = boto3.resource('s3',
                aws_access_key_id=accessKey,
                aws_secret_access_key=secretKey,
                aws_session_token=secretToken,
                )

    s3.Bucket('user-backups').put_object(Key=username.title() + " " +
                                surname.title() + "/" + file_name, Body=file_name)
    print("Uploaded")


def main():
    server = imaplib.IMAP4_SSL("imap.gmail.com", 993)   #connect to gmail's imap server
    server.login(email_address, password)   #enter creds
    result, data = server.select('"[Gmail]/All Mail"')  #get all emails(inbox, outbox etc)
    if result == 'OK':
        print("Downloading")
        download_emails(server)
        server.close()
    else:
        print("ERROR: Unable to open mailbox ", result)
    server.logout()
    archive(username + surname, directory)
    upload_to_s3(username + surname + ".zip")
    #os.remove(email_address + ".zip")
    #shutil.rmtree(email_address)
    print("Done")
if __name__ == "__main__":
    main()

The put_object function accepts Body which is either bytes object or a file object. You have currently just passed the plain filename (a string).

From documentation:

Body (bytes or seekable file-like object) -- Object data.

So the fix should be to pass the file object. Consult this to know how to do that.

amazon s3 - Boto3 not uploading zip file to S3 python, I'm trying to upload a .zip file to S3 using boto3 for python but the .zip file in my directory is not uploaded correctly. The code downloads all  Upload Zip Files to AWS S3 using Boto3 Python library September 13, 2018 1 minute read Menu. Boto3; Solution; Example Code; References; Support Jun; Learn how to upload a zip file to AWS S3 using Boto3 Python library. Boto3. According to boto3 document, these are the methods that are available for uploading.

Just use s3.client.upload_file.

upload_file(Filename, Bucket, Key, ExtraArgs=None, Callback=None, Config=None)

def upload_to_s3(file_name):
  s3 = boto3.client('s3')
  Key = username.title() + " " + surname.title() + "/" + file_name
  try: 
     s3.meta.client.upload_file('/path/to/file', 'user-backups', Key)
  except Exception as e:
     print(e)

Upload Zip Files to AWS S3 using Boto3 Python library, Click here to view 'How to upload a zip file to AWS S3'. Thank you for visiting. aws Python. Get link; Facebook; Twitter; Pinterest; Email; Other  Tutorial on how to upload and download files from Amazon S3 using the Python Boto3 module. Learn what IAM policies are necessary to retrieve objects from S3 buckets. See an example Terraform resource that creates an object in Amazon S3 during provisioning to simplify new environment deployments.

You can check out this article for more information.

There are a number of ways to upload. Check out this boto3 document where I have the methods listed below:

The managed upload methods are exposed in both the client and resource interfaces of boto3:

S3.Client method to upload a file by name: S3.Client.upload_file()
S3.Client method to upload a readable file-like object: S3.Client.upload_fileobj()
S3.Bucket method to upload a file by name: S3.Bucket.upload_file()
S3.Bucket method to upload a readable file-like object: S3.Bucket.upload_fileobj()
S3.Object method to upload a file by name: S3.Object.upload_file()
S3.Object method to upload a readable file-like object: S3.Object.upload_fileobj()

I made it work using s3.client.upload_file.

upload_file(Filename, Bucket, Key, ExtraArgs=None, Callback=None, Config=None) . Upload a file to an S3 object.

import boto3
s3Resource = boto3.resource('s3')

try: 
    s3Resource.meta.client.upload_file('/path/to/file', 'bucketName', 'keyName')
except Exception as err:
    print(err)

How to upload a zip file to #aws #s3 using #boto3? #python, The code below shows, in Python using boto, how to upload a file to S3. except​: # Not all file objects implement fileno(), # so we fall back on this file.seek(0, os. Uploading Files¶ The AWS SDK for Python provides a pair of methods to upload a file to an S3 bucket. The upload_file method accepts a file name, a bucket name, and an object name. The method handles large files by splitting them into smaller chunks and uploading each chunk in parallel.

Example: Upload a File to AWS S3 with Boto, Bucket names cannot have '/' in them. When you use a url like this "​anikets3bucket/folder1" the tool simply treats the parts behind / as "directory". In this tutorial, I will be showing how to upload files to Amazon S3 using Amazon’s SDK — Boto3. Get Your Access Key and Access Secret Once you have an account with Amazon Web Services, you

How to upload a file in S3 bucket using boto3 in python, Here we'll be dealing with files but you can read more on s3 here. Boto is the Amazon Web Services (AWS) SDK for Python, which allows Python developers to  Upload and Download files from AWS S3 with Python 3. July 28, 2015 Nguyen Sy Thanh Son. 3. boto3 python s3 s3. 3.

AWS S3 bucket file upload with python and Boto3, by the AWS SDK for Python to download files are similar to those provided to upload files. The file object must be opened in binary mode, not text mode. s3  You can use method of creating object instance to upload the file from your local machine to AWS S3 bucket in Python using boto3 library. Here is the code I used for doing this: import boto3 s3 = boto3.resource('s3') s3.create_bucket(Bucket= 'anikets3bucket') s3.Object('anikets3bucket','abcd.txt').upload_file(Filename='C:\\Users\\Aniket

Comments
  • The thing is it is uploaded as a .zip file. I get a zip file in my directory that's 23MB in size. When I upload it it's like a totally different file. And I'm passing the filename + extension in upload_to_s3.
  • @davidb Try this and share your findings
  • @davidb can u share your code on how u use zipfile to upload? It seems that we have to know the file names within the zipped file to do it?