Giter Club home page Giter Club logo

multipart-download's Introduction

multipart-download Build Status

NPM

Speed up download of a single file with multiple HTTP GET connections running in parallel

Class: MultipartDownload

MultipartDownload is an EventEmitter.

start(url[, options])

  • url <string> Url of file to be downloaded
  • options <StartOptions> Download options (Optional)
    • numOfConnections <number> Number of HTTP GET connections to use for performing the download (Optional)
    • writeToBuffer <boolean> Store downloaded data to buffer (Optional)
    • saveDirectory <string> Directory to save the downloaded file (Optional)
    • fileName <string> Set name of the downloaded file (Optional)
    • headers <Object> Set custom HTTP headers (Optional)

Starts the download operation from the url.

Multiple HTTP GET connections will only be used if the target server supports partial requests. If the target server does not support partial requests, only a single HTTP GET connection will be used regardless of what the numOfConnections is set to.

If the numOfConnections parameter is not provided, a single connection will be used.

If the writeToBuffer parameter is set to true, the downloaded file will be written into a buffer.

If the saveDirectory parameter is provided, the downloaded file will be saved to the saveDirectory.

If the fileName parameter is provided, the downloaded file will be renamed to fileName. If the fileName parameter is not provided, the downloaded file will maintain its original file name.

If the headers parameter is provided, the headers will be included in the HTTP request.

Event: 'error'

  • err <Error> Emitted error

Event: 'data'

  • data <string> | <Buffer> Chunk of data received
  • offset <number> Offset for the chunk of data received

The file being downloaded can be manually constructed and manipulated using the data and offset received.

Event: 'end'

  • output <string> Downloaded file buffer or downloaded file saved path

output is the buffer of the downloaded file if the writeToBuffer parameter is set to true.

output is the location of the saved file if the saveDirectory parameter is provided.

output will be null if writeToBuffer is not set to true or saveDirectory parameter is not provided.

start(url[, numOfConnections, saveDirectory]) โ— DEPRECATED and REMOVED in v1.0.0

Example

Download without writing to buffer or saving to file

const MultipartDownload = require('multipart-download');

new MultipartDownload()
  .start('https://homepages.cae.wisc.edu/~ece533/images/cat.png', {
    numOfConnections: 5
  })
  .on('error', (err) => {
    // handle error here
  })
  .on('data', (data, offset) => {
    // manipulate data here
  })
  .on('end', () => {

  });

Download and write to buffer

const MultipartDownload = require('multipart-download');

new MultipartDownload()
  .start('https://homepages.cae.wisc.edu/~ece533/images/cat.png', {
    numOfConnections: 5,
    writeToBuffer: true
  })
  .on('error', (err) => {
    // handle error here
  })
  .on('data', (data, offset) => {
    // manipulate data here
  })
  .on('end', (buffer) => {
    console.log(`Downloaded file buffer: ${buffer}`);
  });

Download and save to file

const os = require('os');

const MultipartDownload = require('multipart-download');

new MultipartDownload()
  .start('https://homepages.cae.wisc.edu/~ece533/images/cat.png', {
    numOfConnections: 5,
    saveDirectory: os.tmpdir(),
    fileName: 'kitty.png'
  })
  .on('error', (err) => {
    // handle error here
  })
  .on('data', (data, offset) => {
    // manipulate data here
  })
  .on('end', (filePath) => {
    console.log(`Downloaded file path: ${filePath}`);
  });

multipart-download's People

Contributors

dependabot[bot] avatar zulhilmizainuddin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

multipart-download's Issues

Emit error events instead of throwing exceptions

'MultipartDownload' class is inherited from and extends the 'EventEmitter' class, so it should emit 'error' events in case of app fail.
Try/catch statement in JS can't intercept async errors and throwing exceptions is pretty useless.
For example error in multipart-download.ts#L68 will never be handled, as network request was processed asynchronously, This one can be easily reproduced and tested if your dev server doesn't return 'accept-ranges' header.
I think it's good idea to swap all throw statements with this.emit('error', errMsg)...

Please use cb form of writeStream.end()

Hello! I found your lib is very useful, but faced some little bugs.
First use callback form of calling writeStream.end() in ../src/models/multipart-download.d.ts (.js), it's reducing a lot errors caused by async behaviour when using this lib heavily.
So, instead emitting 'end' event right after, you should call this in end method callback:

new partial_download_1.default()
...
.on('end', () => {
  if (options.saveDirectory) {
    writeStream.end(() => {
      // do the job here
      if (++endCounter === options.numOfConnections) {
        this.emit('end', filePath);
      }
    });
  }
 });

I know, i should PR this, but i'm very lazy to mess with typescript, lol.

Lockfile: multipart-download do not releases file after download

I'm using "multipart-download" in a complex scenario, where downloaded file should be streamed as http response and then removed (offtopic: it's would be great if this lib could expose option to write into Buffer, not only to filesystem), but there is problem - downloaded file can't be released and available for erasing only when node process shuts down.

Following sample of code shows that file can't be removed once it's have been completely downloaded, at least on Windows 10 x64 2015 LTSB.

const MultipartDownload = require('multipart-download')
const fs = require('graceful-fs')

const removeFile = filePath => {
  console.log('remove file called')
  fs.unlink(filePath, err => {
    if (err)
      return console.error(err)
  })
}

new MultipartDownload()
  .start('... any direct file url goes there ...', {
    numOfConnections: 5,
    saveDirectory: process.cwd(),
    fileName: 'file.bin'
  })
  .on('end', (filePath) => {
    console.log(`Downloaded file path: ${filePath}`)
    setTimeout(() => { /* just to keep node running for 1 min */ }, 60000)
    // Locked! File actually is only marked for removal, and will be really erased once node process stops
    setTimeout(() => removeFile(filePath), 100)
  })

Note that there is no difference between 'fs' and 'graceful-fs' modules (even if i replace all fs-require sections inside 'multipart-download' lib), problem still exists.
I'm solved this partially using 'cluster' module and running 'multipart-download' as a separate process.
Probably related to same problems in different modules: Multi-threaded-downloader #187 issue

Add cancel support

It makes sense that if the user begins downloading a large file, he'll be able to cancel the download at any time.

Buffer is always null

Code example

const MultipartDownload = require('multipart-download');

const multipartDownload = (url) => {
  return new Promise((resolve, reject) => {
    new MultipartDownload()
      .start(url, {
        numOfConnections: 4,
        writeToBuffer: true,
      })
      .on('error', (err) => {
        console.log(err);
        reject(err);
      })
      .on('end', (buffer) => {
        console.log(buffer);
        resolve(buffer);
      });
  });
};

module.exports = multipartDownload;

Add get total file size

It would be really great if you add a way to get total amount of data in order to calculate percent downloaded or download speed.
Thanks.

`headers` option not work

According to src/models/operation-factory.ts, options.headers will not be passed to request in any case.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.