Comments (20)
FMA test data were seeded for one month (1 channel, 156 torrents, 23 GB total).
The music data are still available inside Tribler.
from tribler.
Some of the .torrent
crawled can be found in my dropbox. Currently it has 3658 .torrents.
I'm crawling mininova now, but I got blocked so maybe it will take some time to get more torrents on this site. AFAIK mininova now hosts legal torrents only.
from tribler.
"Performance analysis of a Tor-like onion routing implementation", Quinten Stokkink, Harmjan Treep, http://arxiv.org/abs/1507.00245
@qstokkink student Wouter is aiming to repeat your work and do a CPU performance analysis automagically after each pull request.
from tribler.
@Pathemeous Found interesting code from Quinten: devel...qstokkink:devel
from tribler.
If you want to re-use our code you should extract the start_profiling and stop_profiling functions from our code and call them respectively before and after whatever you want to profile (and make sure it's in the filter). The tunnel_piecharts.R script can then parse profiling files in this format. We integrated this into the Gumby pipeline with a script added to the .conf file.
from tribler.
Thanks @qstokkink, I will have a look.
from tribler.
We do already have memory profiling and manhole (telnet into the process to inspect it in real time) It would be cool to have the profiling integrated into gumby's instrumentation.py so all the experiments can use it straight away.
from tribler.
A simple 1 week starting experiment without Tribler, Tor-stack, and no Gumby.
Goal is to first detect if there are Libtorrent bottlenecks. Experimental setup consists of just Libttorrent on Ubuntu. Libtorrent with seeding of 50GB..250GByte and testing the local download performance. With 1000 swarms or so, it is expected that Libtorrent grinds to a halt in normal settings.
Outcome: Spend 2 weeks to create graph to show performance development, as you're seeding more GBytes and swarms.
Experiment:
- seed 10 GByte, 50 swarms
- download for 1 minute
- write down downloaded speed MBytes progress
- Repeat: double amount of GBytes & swarms
- create scalability graph: seeding size versus download speed
- tweak until above works :-)
- document after 2 weeks effort
- Move to Gumby+Tribler_tor stack..
from tribler.
https://github.com/Pathemeous/tribler
from tribler.
Progress: first script to seed and control Libtorrent from Python.
Step closer to a PullRequest tester in Jenkins with 1 TByte seeding test.
Ardhi has 3000+ Linux .iso torrents, seems sufficient for a future test.
Measure in an easy to build setting the cost of 1 TByte seeding (or MaxHardDiskCapacity). Start Libtorrent for 1 hour with various seeding size settings and measure the total consumed bandwidth. Goal is to identify the overhead. For instance, set the seeding upload bandwidth to just 10 KByte or so. This need to be subtracted to obtain the DHT, PEX, and othe control overhead protocol traffic.
Docs: The limits of the number of downloading and seeding torrents are controlled via active_downloads, active_seeds and active_limit in session_settings change the default of 5 active seed to 10000 :-)
from tribler.
- Graph plot of startup time (time since launch until first seeding occurs)
- Graph (bar) plot of total amount of bandwidth used while seeding for 1 hour.
Interesting result is the following: - Graph plot of the effective amount of bandwith used (total minus overhead of torrent management)
from tribler.
@ardhipoetra great, thanks for sharing!
You might want to rate-limit your requests and maybe use HTTP proxies for the crawling process?
from tribler.
In the end, I used rate-limit my request and it works.
As @devos50 requested, here is the link to the collection, zipped. You can put that in the bbq.
As for the crawler, I made the repository on https://github.com/ardhipoetra/legal-torrent-crawler
from tribler.
Currently, Tribler will create an introduction point for every torrent it is seeding. This could potentially overload the exit nodes, especially now that exit nodes are also running the PexCommunity
. To minimize this problem it was thinking about somehow limiting the number of introduction points that we create. We could do this from the TriblerTunnelCommunity
or maybe using libtorrent's auto-management feature (which allows us to limit the number of active seeds). Using auto-management seems to make the most sense. @qstokkink @devos50 What do you think?
from tribler.
Sure. Why not?
from tribler.
@drew2a Just a small reminder. Please setup a seedbox for a Tribler channel with lots of Creative Commons music. (e.g. different then superapp; overlap). Simple static dump. For demo purposes only. Show that we can seed lots of stuff: https://github.com/mdeff/fma
EDIT: then please use that to setup a demo channel with markdown and real content #3615
from tribler.
For further development. An idea.
As I see, there are two types of Tribler's users:
- Channel's creators (who wants to seed content by creating a channel)
- Normal users (who wants to download)
So, what if we developed a tool that makes it easier to create and seed a channel?
Like:
$./create_and_seed.sh <folder>
Where is a folder with the following structure:
my channel
├ sub_directory
| ├ file1
| ├ file2
| └ README.md
├ sub_directory2
| ├ file3
| └ file4
└ README.md
The behavior:
- Create a channel, using
my channel
as a channel name - Create a markdown preview for a folder in case of the presence of
*.md
-file in this folder. - Start to seed a content
@ichorid what do you think?
from tribler.
@ichorid what do you think?
What are the fileX
things? Torrents? Or actual files that should become individual torrents?
from tribler.
What are the fileX things? Torrents? Or actual files that should become individual torrents?
Actual files (discussed offline).
from tribler.
I did an experiment:
- Have generated
1GB
of data divided into1024
torrents (generate_test_data.py) - Seed them (seeder.py)
- Have downloaded
3
different torrents (picked randomly) from another PC. All downloads have been completed within the range of [5..30] seconds.
No trackers were used.
Libtorrent version: 1.2.10
from tribler.
Related Issues (20)
- UI Layout for the Add Torrent button is corrupted HOT 2
- Reports to Sentry are silently ignored on some Tribler installations
- ProactorEventLoop bug reproduced: UDP server stops accepting datagrams from any clients after a single client disconnects HOT 1
- `SearchResultsModel` performs queries without text
- [7.13.3] PermissionError: [Errno 13] Permission denied: <filepath>
- Health notification is not being sent to the GUI
- [7.13.3] ValueError: bit must be 0 or 1, got 4
- `Stale.yml` Node.js 16 actions are deprecated
- `Gatekeeper.yml` gatekeeper Label check failed: required 1 of 'PR: safe to check', but found 0. HOT 1
- is_frozen() returns the wrong result when the Tribler binary is built with cx_freeze instead of PyInstaller
- [7.14 release branch] PyInstaller refuses to build Tribler on Mac
- `guitest` is failing HOT 1
- After upgrade to 7.14.0 http_port keeps changing and api does not work HOT 27
- GUI tests are order dependent
- No matching distribution found for libtorrent==1.2.19 for macOS HOT 1
- [7.14.0] ValueError: invalid literal for int() with base 10 (in DownloadManager.set_proxy_settings) HOT 1
- [7.14] TypeError: 'NoneType' object is not subscriptable
- Streaming in Ubuntu
- Failed to load /org/gtk/libgtk/icons/16x16/status/image-missing.png: Unrecognized image file format HOT 1
- [main] KeyError: 'ratio' (in `DownloadWidgetItem.__lt__`)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tribler.