The file download tool to download multiple files from the remote servers to local machine.
- Python version: 3.7
activating dev environment:
source ./dev/bin/activate
deactivating:
deactivate
Installing dependencies:
pip install -r requirements.txt
Linting code:
pylint
create ftp password and put into folder ftp/passwd to run from localhost
- Supported protocols: HTTP(S) and FTP. There's a downloader.py file that can help to add new protocols easily.
- To avoid memory issue when the file is big, I use the streaming mechanism.
- To support download multiple files, I use the thread pool executor, so the requests can run in parallel in multiple cores machine.
- To avoid partial data, any file that is raised an exception while downloading will be removed.
-
Handling unique file name better. I only grab the file name by the last part of url. There are some ways to improve:
- grouping files to folders by the domains from the urls.
- splitting the url and concatenate to the file name.
-
Allowing a user to input the destination folder. I designed the code to have an option to set the destination folder, if I can spend more time, I will handle the destination folder from user input. Currently, a downloaded file is saved to folder files inside the source code.
-
Guessing file type if the url does not have an extension of the file. There is a library named python-magic that can help to detect the file type by content of the file, but the supported file types are limited.
-
Adding integration tests.