Giter Club home page Giter Club logo

torrent-stream's Introduction

torrent-stream

Travis Build branch Dependency Status devDependency Status

The streaming torrent engine that peerflix uses

npm install torrent-stream

How can I help?

  1. Open issues on things that are broken
  2. Fix open issues by sending PRs
  3. Add documentation

Usage

torrent-stream is a node module that allows you to access files inside a torrent as node streams.

var torrentStream = require('torrent-stream');

var engine = torrentStream('magnet:my-magnet-link');

engine.on('ready', function() {
	engine.files.forEach(function(file) {
		console.log('filename:', file.name);
		var stream = file.createReadStream();
		// stream is readable stream to containing the file content
	});
});

You can pass start and end options to stream to slice the file

// get a stream containing bytes 10-100 inclusive.
var stream = file.createReadStream({
	start: 10,
	end: 100
});

Per default no files are downloaded unless you create a stream to them. If you want to fetch a file without creating a stream you should use the file.select and file.deselect methods.

When you start torrent-stream it will connect to the torrent dht and fetch pieces according to the streams you create.

Full API

engine = torrentStream(magnet_link_or_buffer, opts)

Create a new engine instance. Options can contain the following

{
	connections: 100,     // Max amount of peers to be connected to.
	uploads: 10,          // Number of upload slots.
	tmp: '/tmp',          // Root folder for the files storage.
	                      // Defaults to '/tmp' or temp folder specific to your OS.
	                      // Each torrent will be placed into a separate folder under /tmp/torrent-stream/{infoHash}
	path: '/tmp/my-file', // Where to save the files. Overrides `tmp`.
	verify: true,         // Verify previously stored data before starting
	                      // Defaults to true
	dht: true,            // Whether or not to use DHT to initialize the swarm.
	                      // Defaults to true
	tracker: true,        // Whether or not to use trackers from torrent file or magnet link
	                      // Defaults to true
	trackers: [
	    'udp://tracker.openbittorrent.com:80',
	    'udp://tracker.ccc.de:80'
	],
	                      // Allows to declare additional custom trackers to use
	                      // Defaults to empty
	storage: myStorage()  // Use a custom storage backend rather than the default disk-backed one
}

engine.on('ready', fn)

Emitted when the engine is ready to be used. The files array will be empty until this event is emitted

engine.on('download', [piece-index])

Emitted every time a piece has been downloaded and verified.

engine.on('upload', [piece-index, offset, length])

Emitted every time a piece is uploaded.

engine.on('torrent', fn)

Emitted when the metadata has been fetched.

engine.on('idle', fn)

Emitted when all selected files have been completely downloaded.

engine.files[...]

An array of all files in the torrent. See the file section for more info on what methods the file has

engine.destroy(cb)

Destroy the engine. Destroys all connections to peers

engine.connect('127.0.0.0:6881')

Connect to a peer manually

engine.disconnect('127.0.0.1:6881')

Disconnect from a peer manually

engine.block('127.0.0.1:6881')

Disconnect from a peer and add it to the blocklist, preventing any other connection to it

engine.remove([keep-pieces], cb)

Completely remove all saved data for this torrent. Optionally, only remove cache and temporary data but keep downloaded pieces

engine.listen([port], cb)

Listen for incoming peers on the specified port. Port defaults to 6881

engine.swarm

The attached peer-wire-swarm instance

engine.swarm.downloaded

Shows the total bytes downloaded. With this you can know how much you downloaded and how many bytes you still have to download to reach the end of the file.

file = engine.files[...]

A file in the torrent. They contains the following data

{
	name: 'my-filename.txt',
	path: 'my-folder/my-filename.txt',
	length: 424242
}

file.select()

Selects the file to be downloaded, but at a lower priority than streams. Useful if you know you need the file at a later stage.

file.deselect()

Deselects the file which means it won't be downloaded unless someone creates a stream to it

stream = file.createReadStream(opts)

Create a readable stream to the file. Pieces needed by the stream will be prioritized highly. Options can contain the following

{
	start: startByte,
	end: endByte
}

Both start and end are inclusive

License

MIT

torrent-stream's People

Contributors

adammw avatar amilajack avatar arestov avatar aron123 avatar asapach avatar bsuh avatar dcerisano avatar feross avatar giacomocerquone avatar ivshti avatar jaruba avatar jasnell avatar linusu avatar mafintosh avatar novalagung avatar patmooney avatar ralphtheninja avatar roeycohen avatar thermatk avatar transitive-bullshit avatar watdafox avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

torrent-stream's Issues

question

hi

how do you allocate the file size in tempdir ?

Cache bitfield to disk

Currently every restart will result in all stored data being reverified.
We should cache the bitfield to disk once in while so this is not needed.

Seed/upload code path is inefficient

I've been doing some benchmarks on the following config (might've mentioned a couple of times already):

  • Single-core 2 GHz, 2 GB of memory
  • Ubuntu 14.04 Server x86_64 VM
  • Node v0.10.25
  • Deployed in the cloud, so has virtually unlimited up/down bandwidth
  • Using torrent-stream 0.8.0 (earlier versions work similarly), with mostly default config (connections: 50).

Here's the observed behavior:

  • The torrent size is around 4 GB.
  • When peers start requesting pieces, the node process memory grows until it consumes all available physical memory and spills into swap, until it either runs out of virtual memory, or stops to a crawl.
  • It seems like the memory is being allocated and free-ed, so GC is working.
  • It doesn't look like a memory leak, but I wouldn't reject the possibility.
  • The upload speed remains pretty good (several Mbytes/s, seems like mostly limited by disk i/o), but the system becomes unresponsive overall and slowly gets crushed by peer requests.
  • Tweaking connections limit helps a bit, but not for long.

I've reduced the problematic piece of code to these lines:

torrent-stream/index.js

Lines 422 to 429 in 7cf7f46

wire.on('request', function(index, offset, length, cb) {
if (pieces[index]) return;
engine.store.read(index, function(err, buffer) {
if (err) return cb(err);
engine.emit('upload', index, offset, length);
cb(null, buffer.slice(offset, offset+length));
});
});

When commenting them out, basically switching to leech mode, everything goes back to normal: memory remains stable and download speed is fine.

My theory is that the engine accepts more requests than it can handle and drowns under their weight. Probably introducing a throttle control and upload limits would help the issue.

Any ideas what to try next while I'm investigating?

property length of undefined

hi

yestarday module was working today i always have

"TypeError: Cannot read property 'length' of undefined"

files array seems empty...

thanks

dowloading do not start

hi

many torrents do not start downloading, same torrent in my usual client works (deluge or transmission)

i m connected thru router but with dmz enabled, no vpn or things like that...
an idear ?

thanks

Lazy torrent files structure building

Build it only for files with recieved minimum 100 kb of data.

It is possible situation where we need only 1 file from each of 10 torrent with 1 000 files (music discography torrents). So to get 10 files we will build structure with 10 000 files

Possible memory leaks?

I noticed this behavior but don't know whether this is an issue or not:

After starting createReadStreaming file from torrent, for example some .mp4 and monitoring memory, I saw that memory usage grows to 300Mb on the average - when I'm programmatically call .destroy() and .remove() on the engine instance then memory stops growing but it doesn't decrease back to low level which it was before streaming.

Update to bittorrent-dht 1.0.0

I just updated bittorrent-dht to 1.0.0. It now follows the spec, contains a proper routing table, answers find_node requests from other nodes, and more efficiently crawls the DHT.

Previously, to find peers we would do a very aggressive crawl where we stuck every new node into a big queue and queried them all, not paying any attention to how close they were to the target info hash. So we had lots of pointless traffic to far away nodes that weren't helping us at all. Now, we only recurse on the K closest nodes, so we find the correct nodes with a lot less UDP traffic.

There may be some perf regressions or problems, so it would be great if you could update peerflix and let me know if you find any bugs.

Request random pieces if all selections have been met

In order to be better swarm citizens we should probably think about changing piece strategy in all critical pieces have been downloaded.

A simple way of doing this would be do fetch random pieces if the selection array is empty.

Peer Exchange support

Support for Peer Exchange protocol.

According to Wikipedia, there seem to be three different implementations available.

  • Vuze – introduced in Azureus
  • BitComet – proprietary – introduced in BitComet
  • MainLine – introduced in µTorrent

I think we should start by supporting the MainLine implementation, but I have trouble finding some specs document.

Choosing mp4 file

Hi,

First of all torrent-stream is awesome !!
I'm building a web app using torrent-stream but i'm having some troubles when i try to stream a torrent with multiple files (.srt, .nfo, .mp4 in a directory).
I'm looking at file.select but i can't find a solution ...

Thanks

More modularization!

As illustrated by #25 it would probably be a good idea to futher modulalize index.js into smaller pieces (in seperate npm modules if possible)

Update parse-torrent to 1.4

I noticed torrent-stream is using an old version of parse-torrent that still has the length 0 bug fixed in this commit webtorrent/parse-torrent@3f7395b

I tested upgrading to the latest version and it doesn't seem to have any regressions. Would be nice to get this fixed!

Assuring read stream has ended

I'm utilizing the read stream to pipe data down to a lame mp3 decoder then to the speakers. Im using the read streams destroy() function when done listening to the file, is this releasing the saved data and disconnecting from peers?

Detect download finished

I was wondering if there is any way of detecting when the torrent has been completely downloaded (no more pieces are expected).

That is considering the case we don't have the length specified in the torrent file (eg. because we are using a magnet link)

Consider adding a "free rider" and "findpeers" options

Free rider: disable storing pieces (only mem[index] is used in memory) and disable sending 'have' to other peers, other peers can still see you of course but you minimize your participation to a torrent, the rational is for a project like Peersm where other peers are relaying data for you and then should not keep what they are relaying and neither advertise themselves. Small changes that I can provide if there is an interest.

Findpeer: there is a TODO comment in the code, I don't know exactly why the 10000 value was chosen but it causes in my case an average delay of 2mn before things start, maybe it could become an option (in the meantime routing tables are added???)

Virtual pieces

Peerflix/torrent-stream's biggest flaw in terms of performance has always been related to torrents with large pieces. The new algorithm is terrific at downloading pieces in order and focusing on the critical pieces, but this still doesn't solve the problem that you have to go through 16 MB on torrents where the piece is 8MB (8MB for the end chunk of data and 8MB for the start), when you just have to download 1-2MB. This is a huge regression.

My proposal is the following: introduce an experimental working mode called "virtual pieces" where: 1) piece verification is disabled and 2) the torrent object's pieces and pieceLength fields are replaced with different values so that the entire torrent-stream works as if the torrent's pieces are, let's say 512KB.

Since in both cases the piece size is a multiple of the blocks size (16KB), there will be no problem simply operating under the assumption that the torrent's pieces are 512KB and 1) mapping the values on every wire.request and 2) mapping the values on .peerPieces, .have(), .bitfield() etc.

This is an elegant solution since the algorithm can always operate at a fixed ideal granularity. Verification can still be implemented - if the pieces are 8MB, you'll need 16 virtual pieces downloaded before you can verify the "real" piece.

Only uploading and almost not downloading

Hey,
Right now I have this code:

engine.on('ready', function() {
    engine.files.forEach(function(file) {
        console.log('filename:', file.name);
        var stream = file.createReadStream();
    });

    var server = http.createServer(function(req, res) {
        var file = engine.files[2];
        res.setHeader('Content-Length', file.length); // this is important for vlc
        file.createReadStream().pipe(res);
    });
    server.listen('2020');
});
engine.on('download', function(index) {
    console.log("download - " + index);
});
engine.on('upload', function(index, offset, length) {
    console.log("upload - " + index);
});
engine.on('error',function(err,info){
    console.log(err,info);
});

When I start the app and open the url http://127.0.0.1:2020 from VLC it just shows the first 40/50 seconds of the video. After that the sound keeps repeating for a minute or so and then VLC closes the stream.
Also in my console log it always says upload - indexnumber and just 1 or 2 times download - indexnumber.

Can somebody help me with solving this problem?

stream events improperly emmited?

'use strict';

var torrent = require('torrent-stream'),
    fs = require('fs'),
    path = require('path'),
    files,
    engine;

engine = torrent('magnet:', { tmp: path.join(__dirname, 'hello') });


engine.on('ready', function() {
  files = engine.files.length;
  engine.files.forEach(function(file, index) {
    var input = file.createReadStream();
    var filename = file.path.split(path.sep);
    var output;

    console.log('Downloading:', path.basename(file.path));
    if (filename.length > 1) {
      try {
        fs.readdirSync(filename[0]);
      } catch (err) {
        fs.mkdirSync(filename[0]);
      }
    }
    output = fs.createWriteStream(path.join(__dirname, file.path));
    input.pipe(output);
    (function(i) {
      output.on('finish', function() {
        if (++i === files) {
          console.log('done!');
          // engine.destroy();
        }
      });
    })(index);
  });
});

done! gets immediately displayed when all the files are available even if they're not fully downloaded yet. I tried listening to input.on('end') as well but it acts the same. Is there any other event I can listen to?

Algorithm ideas

This is more of a discussion, not an issue.

BTW, great job on the algorithm in torrent-stream. It's a great evolution of the peerflix algorithm, it's much lighter on the CPU and much more "sequential" - playback starts at only about 50-70% of the downloaded amount required to start playback on the original peerflix.

Something I've noticed (and the reason for the less CPU usage) is the absence of the calcOffset mechanism. Despite this the algorithm is way better. I was wondering - since a peer now reserves a sequential set of blocks, can the number of the reserved blocks depend on the peer's speed, so that download goes smoother and we rely less on hotswaps?

The actual limit of maximum requests can be implemented in peer-wire-swarm (and set to 5) via a mechanism similar to async.queue, while MAX_REQUESTS in torrent-stream/index will depend on the peer's individual speed - e.g. anywhere from 5 to 10, so that peers would finish up their consecutive set of reserved blocks for rather consistent amount of time.

Stop streaming when post variable is set/a link is visited

I want to stop streaming when the user clicks on a button which will set the post variable url as close.
Right now I have this, but it says engine is not defined. Which is true. But I am having a hard time solving this.

var url = req.body.torrent;
var indexOfMovie;
if(url=="close"){
    res.sendfile('index.html');
    engine.remove();
    console.log("page closed");

}else{
    var engine = torrentStream(url,{
        path: 'tmp'
    });

    engine.on('ready', function() {
        engine.remove();            
        var server = http.createServer(function(req, res) {
                server.on("listening", function () { server.close(); });
                if(url!="close"){
                    var file = engine.files[indexOfMovie];
                    res.setHeader('Content-Length', file.length);
                    file.createReadStream().pipe(res);
                }
        });
        server.listen('2020');

    }); 
}

This should probably be done by stopping the server and removing the engine. I tried that with server.close() if server is listening but that isn't doing much.

Performances enhancement

I did not investigate for now but prefer to write it in order to keep it in memory and in the TODO list.

We have integrated with Peersm a modified version of torrent-stream ("free rider" slight modifications, I will fork a repo if there is an interest), the demo links for streaming are the first two of http://www.peersm.com/?links-en, you can try them, it's working well but a bit slow, see below why.

Two of our servers (which have a good bandwidth) are seeding the video file (magnet:?xt=urn:btih:49C338AA58DEE04262DEC7AE42D6FD48962C16A5) and some of our adsl connections (using rtorrent for the servers, not mastering it but apparently it does not know how to send metadata info when you use magnet links, that's why adsl connections are there too).

Unlike usual bittorrent clients, torrent-stream has no problem to find the seeders, but does not realize it should better get pieces from our servers rather than keeping trying from the adsl connections.

And unlike torrent-stream, once (if) usual bittorrent clients succeed to know from whom they can download, they choose the servers.

The algorithm is OK, torrent-stream does detect that the seeders are slow and therefore download pieces quasi sequencially, but it does not get this would be much faster retrieveing all pieces from the servers.

[Error: Torrent is missing required field: info]

I'm trying to access files by http http://torcache.net/torrent/F2A20457467776FBB5895039BA0DD0CFC2DD2D99.torrent

but everytime I'm having such an error - [Error: Torrent is missing required field: info]

What can be the problem?

Here's code extract:

var engine = torrent('http://torcache.net/torrent/F2A20457467776FBB5895039BA0DD0CFC2DD2D99.torrent', {
  connections: 100,
  path: '/tmp/myfile'
});

File download progress

Is there a way to find a file's download progress? I've posted a question that could be related--I know that if I know all the blocks associated with a file that I could listen for the piece-indexes emitted by engine.on('download'), but I was wondering if torrent-stream has a built-in method for finding a file's download progress.

Blocks associated with file

In a multi-file torrent (read: not typical movie torrents as used with peerflix, popcorn time, etc., i.e. music albums,) how would I find the blocks (piece-indexes) associated with each file?

Specifically: engine.on('download') emits a piece-index; how do I know where that lies within a file? That is, is there a way to tell that a file consists of blocks 52-79?

Thanks for reading. I'll document as I figure this out.

Hash doesn't match on big torrents

I've been playing around with this and I came across where once I downloaded a .torrent file for a big download it will not start downloading. I checked to see what was making this problem and found out the hashes do not seem to match.

So far I test low as 3.87GiB (4157627274 Bytes)

file allocation

hi

sorry to ask here but how do you allocate file space on disk ?

i search a way to pre allocate a file then add data to it while downloading then serve it thru htttp server
, the same process you have if i understand... but i m a little bit lost in your code :p

thanks

Inconsistent peer handling

Peers that are discovered via dht go through the blocklist, but those that come from trackers don't. Also, the former generate peer event and the latter don't.

Build errro in bignum

When trying to build this I get an error in bignum as the module does not appear to be included.

..\bignum.cc(10): fatal error C1083: Cannot open include file: 'openssl/bn.h': No such file or directory [C:\U
ard\Documents\GitHub\torrent-stream\node_modules\bittorrent-tracker\node_modules\bignum\build\bignum.vcxproj]

Can't destroy - has no method 'destroy'

I can't destroy an engine.

TypeError: Object #<Object> has no method 'destroy'
  at EventEmitter.engine.destroy (/vagrant/popcrn/node_modules/torrent-stream/index.js:559:34)

DHT does not work (properly)

Hello,

I haven't really dug into it but I have tested it several times and am 100% sure - when I start a torrent, it barely connects to 1-2 peers and those connections die quickly. I tried this with two ISP's. Since I remember an older version working perfectly, I went into the commit log and discovered the "using bittorrent-dht instead of vendored one" commit.

So I did

git checkout a4aed6fdb1be5e23639a99dd06e36b24747f28ad

There is a dramatic difference - in this case, everything works as expected.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.