The file download tool to download multiple files from the remote servers to local machine.

• Python version: 3.7

activating dev environment: source ./dev/bin/activate

deactivating: deactivate

Installing dependencies: pip install -r requirements.txt

Linting code: pylint

create ftp password and put into folder ftp/passwd to run from localhost

• Supported protocols: HTTP(S) and FTP. There's a downloader.py file that can help to add new protocols easily.
• To avoid memory issue when the file is big, I use the streaming mechanism.
• To support download multiple files, I use the thread pool executor, so the requests can run in parallel in multiple cores machine.
• To avoid partial data, any file that is raised an exception while downloading will be removed.

• Handling unique file name better. I only grab the file name by the last part of url. There are some ways to improve:

  • grouping files to folders by the domains from the urls.
  • splitting the url and concatenate to the file name.

• Allowing a user to input the destination folder. I designed the code to have an option to set the destination folder, if I can spend more time, I will handle the destination folder from user input. Currently, a downloaded file is saved to folder files inside the source code.

• Guessing file type if the url does not have an extension of the file. There is a library named python-magic that can help to detect the file type by content of the file, but the supported file types are limited.

• Adding integration tests.