Downloading a directory tree with ftplib

python ftp download all files in a directory
python ftp download latest file
ftplib pasv
ftplib proxy
ftp rename python
python ftp upload
python ftp server
ftp library

This will not download the contents of sub-directories; how can I do so?

import ftplib
import configparser
import os

directories = []

def add_directory(line):
 if line.startswith('d'):
  bits = line.split()
  dirname = bits[8]
  directories.append(dirname)

def makeDir(archiveTo):
 for dir in directories:
  newDir = os.path.join(archiveTo, dir)
  if os.path.isdir(newDir) == True:
   print("Directory \"" + dir + "\" already exists!")
  else:
   os.mkdir(newDir)

def getFiles(archiveTo, ftp):
 files = ftp.nlst()
 for filename in files:
  try:
   directories.index(filename)
  except:
   ftp.retrbinary('RETR %s' % filename, open(os.path.join(archiveTo, filename), 'wb').write)

def runBackups():

 #Load INI
 filename = 'connections.ini'
 config = configparser.SafeConfigParser()
 config.read(filename)

 connections = config.sections()
 i = 0

 while i < len(connections):
  #Load Settings
  uri = config.get(connections[i], "uri")
  username = config.get(connections[i], "username")
  password = config.get(connections[i], "password")
  backupPath = config.get(connections[i], "backuppath")
  archiveTo = config.get(connections[i], "archiveto")

  #Start Back-ups
  ftp = ftplib.FTP(uri)
  ftp.login(username, password)
  ftp.cwd(backupPath)

  #Map Directory Tree
  ftp.retrlines('LIST', add_directory)

  #Make Directories Locally
  makeDir(archiveTo)

  #Gather Files
  getFiles(archiveTo, ftp)

  #End connection and increase counter.
  ftp.quit()
  i += 1

 print()
 print("Back-ups complete.")
 print()

this should do the trick :)

import sys
import ftplib
import os
from ftplib import FTP
ftp=FTP("ftp address")
ftp.login("user","password")

def downloadFiles(path,destination):
#path & destination are str of the form "/dir/folder/something/"
#path should be the abs path to the root FOLDER of the file tree to download
    try:
        ftp.cwd(path)
        #clone path to destination
        os.chdir(destination)
        os.mkdir(destination[0:len(destination)-1]+path)
        print destination[0:len(destination)-1]+path+" built"
    except OSError:
        #folder already exists at destination
        pass
    except ftplib.error_perm:
        #invalid entry (ensure input form: "/dir/folder/something/")
        print "error: could not change to "+path
        sys.exit("ending session")

    #list children:
    filelist=ftp.nlst()

    for file in filelist:
        try:
            #this will check if file is folder:
            ftp.cwd(path+file+"/")
            #if so, explore it:
            downloadFiles(path+file+"/",destination)
        except ftplib.error_perm:
            #not a folder with accessible content
            #download & return
            os.chdir(destination[0:len(destination)-1]+path)
            #possibly need a permission exception catch:
            ftp.retrbinary("RETR "+file, open(os.path.join(destination,file),"wb").write)
            print file + " downloaded"
    return

source="/ftproot/folder_i_want/"
dest="/systemroot/where_i_want_it/"
downloadFiles(source,dest)

recursive ftp directory downloader with python · GitHub, Downloads an entire directory tree from an ftp server to the local destination. :​param ftp_handle: an authenticated ftplib.FTP instance. :param path: the folder on  Downloads an entire directory tree from an ftp server to the local destination :param ftp_handle: an authenticated ftplib.FTP instance :param path: the folder on the ftp server to download

This is a very old question, but I had a similar need that i wanted to satisfy in a very general manner. I ended up writing my own solution that works very well for me. I've placed it on Gist here https://gist.github.com/Jwely/ad8eb800bacef9e34dd775f9b3aad987

and pasted it below in case i ever take the gist offline.

Example usage:

import ftplib
ftp = ftplib.FTP(mysite, username, password)
download_ftp_tree(ftp, remote_dir, local_dir)

The code above will look for a directory called "remote_dir" on the ftp host, and then duplicate the directory and its entire contents into the "local_dir". It invokes the script below.

import ftplib
import os

def _is_ftp_dir(ftp_handle, name, guess_by_extension=True):
    """ simply determines if an item listed on the ftp server is a valid directory or not """

    # if the name has a "." in the fourth to last position, its probably a file extension
    # this is MUCH faster than trying to set every file to a working directory, and will work 99% of time.
    if guess_by_extension is True:
        if name[-4] == '.':
            return False

    original_cwd = ftp_handle.pwd()     # remember the current working directory
    try:
        ftp_handle.cwd(name)            # try to set directory to new name
        ftp_handle.cwd(original_cwd)    # set it back to what it was
        return True
    except:
        return False


def _make_parent_dir(fpath):
    """ ensures the parent directory of a filepath exists """
    dirname = os.path.dirname(fpath)
    while not os.path.exists(dirname):
        try:
            os.mkdir(dirname)
            print("created {0}".format(dirname))
        except:
            _make_parent_dir(dirname)


def _download_ftp_file(ftp_handle, name, dest, overwrite):
    """ downloads a single file from an ftp server """
    _make_parent_dir(dest)
    if not os.path.exists(dest) or overwrite is True:
        with open(dest, 'wb') as f:
            ftp_handle.retrbinary("RETR {0}".format(name), f.write)
        print("downloaded: {0}".format(dest))
    else:
        print("already exists: {0}".format(dest))


def _mirror_ftp_dir(ftp_handle, name, overwrite, guess_by_extension):
    """ replicates a directory on an ftp server recursively """
    for item in ftp_handle.nlst(name):
        if _is_ftp_dir(ftp_handle, item):
            _mirror_ftp_dir(ftp_handle, item, overwrite, guess_by_extension)
        else:
            _download_ftp_file(ftp_handle, item, item, overwrite)


def download_ftp_tree(ftp_handle, path, destination, overwrite=False, guess_by_extension=True):
    """
    Downloads an entire directory tree from an ftp server to the local destination

    :param ftp_handle: an authenticated ftplib.FTP instance
    :param path: the folder on the ftp server to download
    :param destination: the local directory to store the copied folder
    :param overwrite: set to True to force re-download of all files, even if they appear to exist already
    :param guess_by_extension: It takes a while to explicitly check if every item is a directory or a file.
        if this flag is set to True, it will assume any file ending with a three character extension ".???" is
        a file and not a directory. Set to False if some folders may have a "." in their names -4th position.
    """
    os.chdir(destination)
    _mirror_ftp_dir(ftp_handle, path, overwrite, guess_by_extension)

Downloading Every File on an FTP Server, dir to get a list of the contents in an input directory. As mentioned above, this doesn't directly output anything to a list or some other data structure  path - path to the new directory being copied from the server. directory - the destination directory where are files will be copied to. So this line creates the directory into which the files in path will be copied to – borisp Oct 10 '15 at 5:51. Thank you for the answer.

this is an alternative. you can try using ftputil package. You can then use it to walk the remote directories and get your files

ftptool · PyPI, Higher-level interface to ftplib. A os.walk interface is implemented for walking the directory tree: You could upload and download from str using these two: Downloads an entire directory tree from an ftp server to the local destination :param ftp_handle: an authenticated ftplib.FTP instance :param path: the folder on the ftp server to download

Using ftputil, a fast solution could be:

def download(folder):
    for item in ftp.walk(folder):
        print("Creating dir " + item[0])
        os.mkdir(item[0])
        for subdir in item[1]:
            print("Subdirs " +  subdir)
        for file in item[2]:
            print(r"Copying File {0} \ {1}".format(item[0], file))
            ftp.download(ftp.path.join(item[0],file), os.path.join(item[0],file))

Download files and folder from FTP using Python, You can use this Python script to download / clone entire FTP directory and sub directories from remote FTP Host. Recursively fetch files from an FTP server directory. Here, it's downloading all the zip files found in or beneath the parent directory. - recurse_ftp.py

Using ftp.mlsd() instead of ftp.nlst():

import sys
import ftplib
import os
from ftplib import FTP


def fetchFiles(ftp, path, destination, overwrite=True):
    '''Fetch a whole folder from ftp. \n
    Parameters
    ----------
    ftp         : ftplib.FTP object
    path        : string ('/dir/folder/')
    destination : string ('D:/dir/folder/') folder where the files will be saved
    overwrite   : bool - Overwrite file if already exists.
    '''
    try:
        ftp.cwd(path)
        os.mkdir(destination[:-1] + path)
        print('New folder made: ' + destination[:-1] + path)
    except OSError:
        # folder already exists at the destination
        pass
    except ftplib.error_perm:
        # invalid entry (ensure input form: "/dir/folder/")
        print("error: could not change to " + path)
        sys.exit("ending session")

    # list children:
    filelist = [i for i in ftp.mlsd()]
    print('Current folder: ' + filelist.pop(0)[0])

    for file in filelist:
        if file[1]['type'] == 'file':
            fullpath = os.path.join(destination[:-1] + path, file[0])
            if (not overwrite and os.path.isfile(fullpath)):
                continue
            else:
                with open(fullpath, 'wb') as f:
                    ftp.retrbinary('RETR ' + file[0], f.write)
                print(file[0] + '  downloaded')
        elif file[1]['type'] == 'dir':
            fetchFiles(ftp, path + file[0] + '/', destination, overwrite)
        else:
            print('Unknown type: ' + file[1]['type'])


if __name__ == "__main__":

    ftp = FTP('ftp address')
    ftp.login('user', 'password')
    source = r'/Folder/'
    dest = r'D:/Data/'
    fetchFiles(ftp, source, dest, overwrite=True)
    ftp.quit()

How to download a folder containing multiple files on an ftp server to , FTP protocol doesn't support directory download. Most FTP clients (AFAIK, not including standard command line tools "ftp") allow to download directories by  Python 101: Downloading a File with ftplib. There are lots of different ways to download a file from the internet using Python. One popular way is to connect to an FTP server and download your files that way. So that is what we will be looking at in this article.

How to quickly download or upload directory tree over FTP, The fastest way to download or upload directory tree over FTP using just plain console is to execute lftp command. directory has been the remote source directory in most of your program, but the os.walk() function cannot walk a remote directory. You need to iterate over the returned files yourself, using a callback supplied to the retrlines function.

Using wget to recursively download whole FTP directories, Java code example to download a whole directory from a FTP server The important point in this method is to re-create directory structure of  Easily find large folders or subfolders in the Windows Explorer-like tree view. The hierarchical treemap chart in 2D shows you which file types are found in which folders. It will also visualize the size of each folder for easy disk space management.

Java FTP Download a complete folder from server, Download the Directory Structure. The FTP/SFTP Folder Compare requires that you download the files (to your local system) from the server. You may download​  Hi, This script works like a charm but I would like to download the whole root, not just a specified directory. The following setting is not working: source = "/" Can somebody please tell me how to set the right source variable?

Comments
  • also, you may not want to exit in the first ftplib.error_perm except in case your User account has permission issues. pass could work.
  • Awesome. Works like a charm. Should be a library for this!
  • Is it really that complicated to the point where I should be considering packages? I'm thinking it'll just take 10 lines of really complicated code.