Skip to content

Parallel Data Fetching from Web and Synchronize & Aggregate the Fetched Information by Order

Notifications You must be signed in to change notification settings

cankocagil/Parallel-File-Downloader-CLI-App

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Parallel File Downloader CLI App

CLI application downloads an index file to obtain a list of text file URLs and download these files in parallel by multi-threading.

  • If the requested file is found in the server, the response is a 200 OK message. When this is the case, our program establishes <connection_count> parallel connections with the server including the file, downloads non-overlapping parts of the file through these connections, constructs and saves the file under the directory in which our program runs

  • Tested on Unix/Linux and Mac machines.

Test URLs:

Run with just Index file:

python ParallelFileDownloader.py www.cs.bilkent.edu.tr/~cs421/fall21/project1/index1.txt
python ParallelFileDownloader.py www.cs.bilkent.edu.tr/~cs421/fall21/project1/index2.txt

Run with Index file and Number of Parellel Connections:

python ParallelFileDownloader.py www.cs.bilkent.edu.tr/~cs421/fall21/project1/index1.txt 3
python ParallelFileDownloader.py www.cs.bilkent.edu.tr/~cs421/fall21/project1/index2.txt 5

About

Parallel Data Fetching from Web and Synchronize & Aggregate the Fetched Information by Order

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published