With the introduction of _iter_category and _iter_categories the candidates are no longer kept active though in and outgoing walks in other communities.

We decided to drop this cross community activity feature entirely. The following cleanup can be performed:

• Candidates become community specific
• remove Candidate is_any_active, associate, is_associated, disassociate, in_community, all_inactive methods
• rewrite Candidate is_all_obsolete into is_obsolete
• remove community parameters
• remove global Dispersy candidate list (the code still uses this every now and then, must be rewritten to use community candidates only)

This issue is related to #38. Though if possible we want to fix #38 before doing a code cleanup.

Goal: make code easier to understand for new developers and reduce pylint errors/warnings

Currently there are some automatic warning (not all need fixing):
http://jenkins.tribler.org/jenkins/job/Test_dispersy_devel/616/violations/

First pylint warning is moving functionality away from the main dispersy.py in order to reduce it, from the current 4631 SLOC:
http://jenkins.tribler.org/jenkins/job/Test_dispersy_devel/616/violations/file/tribler/Tribler/dispersy/dispersy.py/

Feedback is needed: what to change

We should get a usb/serial controlled power strip to be able to automate power cycling all the NAT boxes before or after a NAT experiment run.

Something similar to this would be good enough:
http://www.amazon.com/PwrUSB-Basic-controlled-Automation-automatically-remotely/dp/B008AZW9P6

We should test the performance difference between between calling the logger with and without if __debug___.

If the performance difference is negligible we won't need to use if __debug__, and we can also remove the exception we make for this from the pylint config file (#8).

I was planning to add a peercache to Dispersy. Looking at the code, the bootstraptribler.txt file seemed like a logical choice to start with.

I'm planning to add another column, which determines if we should or should not sync with a peer. For our normal bootstrappeers, it would turn sync off and for all cached peers sync would be turned on.

By modifying get_bootstrap_candidates() in bootstrap.py, I could make it return either a BootstrapCandidate or a normal Candidates depending on the sync on/off.

Most of the other code would still work, as the 1% probability of selecting a bootstrap/peercache peer should be fine, and the create_introduction_request turns off sync depending on the instance-type of the candidate, i.e. whenever it gets a BootstrapCandidate.

However, I'm a bit concerned about the

assert all(not sock_address in self._candidates for sock_address in self._dispersy._bootstrap_candidates.iterkeys()), "none of the bootstrap candidates may be in self._candidates"

lines i'm seeing in the code. These will definitely cause problems.
I get why these were added, however is it possible to remove them from the get_candidate iterators etc? I guess we should only check this assert whenever we add a new candidate to the _candidates right?

Currently Tribler is using the attach_profiler decorate from Dispersy.
I have no problems with this decorator, it is actually quite handy. However, could we think of a better solution than directly importing the decorator from Dispersy?

Maybe a shared library containing a couple of these decorators etc?

A basic tutorial showing how to configure, instantiate, and perform basic operations with dispersy would be very helpful.

We are experiencing some problems with the Dispersy.stop() method in regard to the timeout property. If the callback is not able to stop during this time it will return before running all expired tasks. Which by itself is I guess expected.

However, if one of those not yet run expired tasks is a dispersythread_data_came_in task, things will break. The Dispersy.stop() method now has cleaned/removed all singletons, and hence we get stracktraces like:

E         dispersy/callback:679  _loop                     | 
Traceback (most recent call last):
  File "/home/jenkins/workspace/Test_tribler_full-ui-run_61x/tribler/Tribler/dispersy/callback.py", line 677, in _loop
    result = call[0](*call[1], **call[2])
  File "/home/jenkins/workspace/Test_tribler_full-ui-run_61x/tribler/Tribler/dispersy/endpoint.py", line 136, in dispersythread_data_came_in
    timestamp)
  File "/home/jenkins/workspace/Test_tribler_full-ui-run_61x/tribler/Tribler/dispersy/dispersy.py", line 1786, in on_incoming_packets
    self._on_batch_cache(meta, batch)
  File "/home/jenkins/workspace/Test_tribler_full-ui-run_61x/tribler/Tribler/dispersy/dispersy.py", line 1858, in _on_batch_cache
    messages = list(self._convert_batch_into_messages(batch))
  File "/home/jenkins/workspace/Test_tribler_full-ui-run_61x/tribler/Tribler/dispersy/dispersy.py", line 2061, in _convert_batch_into_messages
    yield conversion.decode_message(candidate, packet)
  File "/home/jenkins/workspace/Test_tribler_full-ui-run_61x/tribler/Tribler/dispersy/conversion.py", line 1408, in decode_message
    return self._decode_message(candidate, data, verify, False)
  File "/home/jenkins/workspace/Test_tribler_full-ui-run_61x/tribler/Tribler/dispersy/conversion.py", line 1349, in _decode_message
    decode_functions.authentication(placeholder)
  File "/home/jenkins/workspace/Test_tribler_full-ui-run_61x/tribler/Tribler/dispersy/conversion.py", line 1170, in _decode_member_authentication
    members = [member for member in self._community.dispersy.get_members_from_id(member_id) if member.has_identity(self._community)]
  File "/home/jenkins/workspace/Test_tribler_full-ui-run_61x/tribler/Tribler/dispersy/dispersy.py", line 705, in get_members_from_id
    if public_key]
  File "/home/jenkins/workspace/Test_tribler_full-ui-run_61x/tribler/Tribler/dispersy/member.py", line 370, in __init__
    super(Member, self).__init__(public_key, private_key)
  File "/home/jenkins/workspace/Test_tribler_full-ui-run_61x/tribler/Tribler/dispersy/member.py", line 154, in __init__
    assert DispersyDatabase.has_instance(), "DispersyDatabase has not yet been created"
AssertionError: DispersyDatabase has not yet been created
E         dispersy/callback:679  _loop                     

Can we implement a fix for this? Maybe not process new messages while in the "stopping" state?

We need a README with A general description of the code structure/design, links to protocol documentation, etc.

Line endpoint.py:212 calls convert_packet_to_meta_message() which expects a packet without the optional Swift tunnel prefix. If this gets called with a tunnel-prefixed message it will result in a get_community() call with the wrong CID. The tunnel prefix is not removed before the call.

convert_packet_to_meta_message() dumps 2 bytes (Dispersy version field) from the beginning of the packet, to get the CID, then calls get_community(cid) (which might have side-effects). Assuming CID is deadbeaf:

• without tunnel prefix: 0001deadbeaf => deadbeaf (correct)
• with tunnel prefix: ffffffff0001deadbeaf => ffff0001dead

This might also apply for: endpoint.py:377, endpoint.py:396, and endpoint.py:142.

For example, this affects TrackerDispersy, which tracks communities when its get_community() is called.

I want to start encrypting packets which are send to candidates. However, I want to change the encryption-key depending on which candidate.

Therefore, I was thinking of implementing a new endpoint which accepts any other endpoint as a parameter. Next, when sending packets using the send method, this new endpoint would use a callback to figure out if packets send to this candidate should be encrypted or not.
However, as it is now, the endpoint does not know for which community these packets are send. Therefore, it is a bit difficult to encrypt packets for only one community.

Any ideas how I can improve on this idea?

During the DAS4 runs I often get this error:
Looking at the code, this can only be caused by calling the _on_timeout method twice or if the _on_timeout method is still called after the unregistering it.

"callback.py", line 613, in _loop
    result = call[0](*call[1], **call[2])
"requestcache.py", line 103, in _on_timeout
    cache = self._identifiers[identifier]
KeyError: 33437

Before we started using nosetests, all unittests were run with both __debug__ enabled and disabled. Unfortunately nosetests doesn't (easily?) allow us to run in -O optimized mode. However, I would like to see this feature again to avoid problems like the one fixed in #39.

When Dispersy was living on SVN, it used keyword expansion (see revision.py) to feed Dispersy (and logging information) with the branch and revision number of the running code.

Git provides similar features through 'smudge' and 'clean' filters (see http://git-scm.com/book/ch7-2.html#Keyword-Expansion)

We need to remove revision.py (since it obviously can not working on git). And possibly replace it with the filters that Git provides.

First discussed here #106 (comment).

We should improve the pop behavior, it doesn't make sense.

• We could remove the cleanup_delay entirely, although this would mean that we no longer have the ability to ensure that an identifier/number isn't used for some time. It is currently used for the introduction request, adding 4.5 seconds before the identifier can be used again. Without the cleanup_delay the randomness in the numbers will have to prevent clashes.
• We could add the identifier to a different dictionary or set, removing it from RequestCache_identifiers. This would result in has, get, and pop no longer being able to find the identifier. We would need a new way to see if an identifier is unclaimed though, for instance by introducing is_used or something similar that checks _identifiers and the new dict/set.
• I'm sure there are other options.

Anyway, removing the cleanup_delay entirely is the simplest change, and would result in a simplification of the RequestCache. Are there cases where the delay is essential?

Dispersy assesses its LAN address once after startup, while the WAN address keeps being continuously assessed. This allows the WAN address to keep working in roaming scenario's while the LAN address breaks.

A quick and simple fix would be to reassess (by calling _guess_lan_address) the LAN address whenever the WAN IP address (so we will ignore the port number) changes.

http://jenkins.tribler.org/jenkins/job/Test_BootstrapServers/

This test only shows console output that may not be trivial to read. A simple image with a red/green success-rate per tracker would improve readability.

A few months ago we discovered that a bug in handling incoming sequence numbers had caused peers to incorrectly accept messages. This in turn caused peers to return incorrect messages when asked for messages with a specific sequence number (range). This resulted in most peers sending endless stream of missing-sequence and incorrect responses.

This is what happened:

1. messages are received and gathered in batches
2. the whole batch is decoded (packet string -> Message instance)
3. global time and sequence numbers are checked for the whole batch
4. community check is performed for the whole batch (i.e. check_undo)
5. the whole batch is stored in the database
6. the whole batch is handled by the community (i.e. on_undo)

The problem was that some messages were dropped or delayed in step 4. This created gaps in the sequence numbers, but this had already been checked and passed in the previous step 3.

A few month ago I solved this by simply adding a step 4.5, which did the same as step 3. Removing or delaying the messages that had become invalid by the gaps. However this is currently only done on the undo_{own,other} messages. Hence we need a proper fix for this problem.

Tribler LaunchManyCore currently defines the autoload for the HardKilledCommunity.
Shouldn't this be moved to the dispersy constructor? Or is it Tribler specific?

An infinite loop bug was introduced in: 11c85c9#community.py

Caused by:

if (result == exclude_candidate) or (exclude_tunnel and result.tunnel):
    continue

I am extending test_candidates.py to reproduce before fixing.

I got a community which uses the RequestCache to store requests. When receiving a message, I check if the RequestCache has any requests with this identifier stored, if not i accept it.

However, if the RequestCache does not have this type of Cache stored but a different one. The has-method will return false, but the set-method will not allow me to store the new Cache.

Can we improve upon this? Why is the request-cache global?

As reported by user VVS on our forum

Traceback (most recent call last):
  File "/home/vvs/tribler-6.2.0-rc1/Tribler/dispersy/dispersy.py", line 2703, in store_update_forward
    messages[0].handle_callback(messages)
  File "/home/vvs/tribler-6.2.0-rc1/Tribler/dispersy/dispersy.py", line 2492, in on_introduction_response
    self.wan_address_vote(payload.destination_address, candidate)
  File "/home/vvs/tribler-6.2.0-rc1/Tribler/dispersy/dispersy.py", line 1066, in wan_address_vote
    for candidate in [candidate for candidate in community.candidates.itervalues() if candidate.wan_address == self._wan_address]:
UnboundLocalError: local variable 'community' referenced before assignment

Dispersy is light-weight and uses only a two-way handshake. We have no design yet which preserves our ligh-weight character and prevents IPv4 spoofing.
Dispersy uses a requestCache to store outstanding requests and as a mechanism to issue timeouts on pending requests. The request cache has a 16-bit random identifier, unique only per community and message type (intro-req, missing-proof, etc)

When a peer is send a request, no check is done if the IPv4 address of the response is matching.

By sending a flood of 65k to a single peer it is possible to obtain a false match with an outstanding RequestCache entry.
Code pointer: https://github.com/tribler/dispersy/blob/master/dispersy.py#L2402

Complication: the private-search community now uses the request cache as a "transaction number", with multiple peers involved.

See screenshot, after running for 10 minutes it seems a lot of packets are dropped.
Could be related to issue #38 or #70. But never seen this popup before.

A peer sending a request with conversion N should ideally get a response with conversion N.

We can only do this for messages that using CandidateDestination, the candidate instance should hold a list of valid conversions where the highest can be used when encoding the message.

Currently, the acceptable_global_time property is checked in then check_full_sync method.
However, in communities which have sync turned off (searchcommunity), torrent messages which are created by on demand by other peers are dropped as the current global_time is higher than the default range.

Whenever an exception occurs in a method which is scheduled using the call method, I get a traceback consisting of the method "calling" the call method.

Could we expose the original traceback?
An example can be found in the Tribler/Main/Utility/GuiDBHandler.py, there I include the traceback into the exception reporting and am printing it to console before raising the exception.

A better solution seems to be to provide the raise statement with the traceback:
http://docs.python.org/2/reference/simple_stmts.html#grammar-token-raise_stmt as suggested here:
http://stackoverflow.com/questions/1603940/how-can-i-modify-a-python-traceback-object-when-raising-an-exception/13898994#13898994

Outcome: a tutorial with one or all three ORM packages used
Goal: use higher-level relational algebra, instead of hand-coded SQL
Motivation: less code, easy to expand and re-use

Options Sqlalchemy, Storm, Django:

• http://www.sqlalchemy.org/
• https://storm.canonical.com/
• https://www.djangoproject.com/

One of many discussions on this: http://stackoverflow.com/questions/53428/what-are-some-good-python-orm-solutions

FYI: originally created by Johan in the Tribler project Tribler/tribler#130

Boudewijn and I (I'm sure Niels too :)) think that we should move to the standard python logging system.

Pros:

• Dprint is a custom made logging framework that only a few people understand.
• We will be able to integrate better with the rest of the python ecosystem (nose comes to mind).
• It will be easier for newcomers to develop and debug dispersy.
• We will trim a bit dispersy's codebase.

The estimate is around a day of work.

Its not a fun task to do but I volunteer to do it.

Johan, do you approve? :)

Problem: sending fake identities to victim will pollute internal data structures
Solution: as a first line of defense we use rate control

Each overlay has a list of neighbors and Dispersy is not different.
The incoming and re-visit list of peers is easy to pollute from any Internet location to any victim. Simply create new fake identities and send them at full speed to a single UDP port. The victim will quickly have only fake nodes in it's caches and will need to do a fall-back to a central bootstrap server, after many failed connection attempts.
Or worse not detect that all neighbors come from a single IPv4 address or /24 block.
Dispersy connects and syncs with them, without knowing it is no longer connected to any honest peer.

Proof-of-concept implementation of an attack: http://jenkins.tribler.org/job/Experiment_AllChannel+Channelcommunity_attacker/18/
This attack does not create fake identities, only pollute candidate list with 1 identity on several ports.

Other tickets have priority, but in 2014 we can hopefully address this known vulnerability.

Dispersy should apply rate control at three levels: /32. /24 and /16 blocks of addresses. We need to ensure that a single block or several blocks dominate the neighbor list. Exact details: t.b.d.

I have a question regarding the MetaObject object. All payload objects have to extend this and implement an Implementation class.

But why? What's the benefit of this approach vs a normal constructor?

So the tests/experiments finish early and we get less cruft to go trough to find the problems.

We need to decide on the Dispersy release policy.

We are starting to get projects using it, should we do a Dispersy release every time we do a Tribler release?

#83 (comment).

It would be better to use CandidateDestination instead or MemberDestination. A simple 'get_candidate_from_member' method should be sufficient, and reduces code complexity by removing the entire MemberDestination policy.

Searching for MemberDestination shows that it is only used for the 'dispersy-signature-request' message.

As Boudewijn told me, we can do without them and will make dispersy's start()/stop() methods ( #6 ) easier to implement (less things to clean up manually)

We should remember the order that communities are defined using Dispersy.define_auto_load() and use its reverse order when unloading communities when ``Dispersy.stop()` is called.

This will prevent bugs from occurring when communities depend on each other, i.e. in a future database cleanup we want AllChannelCommunity to create a database that all ChannelCommunities will need.

I think we should add an exception if someone is trying to send a packet larger than 65k.
Currently, this will +/- crash the endpoint as it will never be sent.

Unfortunately we do not know the Tribler / Dispersy versions of peers we are connected to. A simple way to solve this is to add a message that requests this information.

It is possible to add some debug numbers to such a message, such as number of dropped / delayed packets, etc.

Similar to using --strict when calling tool/main.py

Should it be fixed, disabled, removed?

Dispersy currently does not unload all communities, as call cannot handle generators if not called from dispersythread.
This also causes Dispersy to not close databases etc.

Currently logger.py is calling config. This has to be moved to the main.py and unit-test base class, as this is currently disabling loggers when Dispersy is used as a library, e.g. in Tribler and Gumby.

(This issue is duplicate of #38. This was closed after it fixed a bug causing the trackers to no longer respond.)

We would expect more candidates sooner. Especially the AllChannelCommunity takes much longer to obtain a good number of candidates than we would expect.

Strangely enough the walk success rate is relatively high (around 90%). This contradicts the lack of available candidates.

This behavior must be either solved or explained. Please investigate.

Currently the id_ parameter can be specified as either unicode or string. This results in warnings, as seen below.

/Tribler/dispersy/callback.py:399: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if tup[1] == id_:

Solutions:

1. Allow only strings
2. Allow only unicode strings
3. Allow both, but always convert them to either string or unicode string
4. Others?

I prefer to using strings slightly over using unicode. I am not a fan of the third option, as this results in unnecessary 'under the hood' conversions.

Since we enabled the Dispersy debug output in our nose runs, we have large amounts of output. Perhaps too much.

The logger provided DEBUG, INFO, WARNING, ERROR, and CRITICAL levels. Currently most of Dispersy's output is DEBUG while a few select are INFO (i.e. start, stop, IP change, candidate statistics).

There are three options:

1. Keep it as it is (debug output will remain very large)
2. Move all debug to info (no distinction can be made between what is currently info)
3. Put the SQL statements behind a separate if flag.
4. Diable the SQL statement output entirely.

Communities attempt to clean their candidates at a regular interval. However, because is_obsolete is global a candidate which is active in another community will not be cleaned.

This makes sense, as we don't want to garbage collect this specific candidate instance. However, the community should remove it from its local candidate list and unvote its wan address vote.

Can we change this behaviour and pass is_obsolete the community as an argument.

Currently three naming conventions are used within community.py:

• Name without 'dispersy' prefix
  For example: get_member, acceptable_global_time, ...
• Name with 'dispersy' prefix
  For example: dispery_enable_candidate_walker, dispersy_store, ...
• Name with 'create_dispersy' prefix
  For example: create_dispersy_authorize, create_dispersy_signature_request, ...

Time permitting we can rename (some of) these methods/properties to use the same convention. At the very least we should choose one of the above conventions and ensure that any new methods/properties use the same convention.

We would expect more candidates sooner. Especially the AllChannelCommunity takes much longer to obtain a good number of candidates than we would expect.

Strangely enough the walk success rate is relatively high (around 80%). This contradicts the lack of available candidates.

This behaviour must be either solved or explained. Please investigate.

Obviously it should return either True or False depending on success or failure.

• Endpoint.close should return either True or False,
• Database.close should return either True or False,
• Callback stop already returns True or False.

Also, Tribler's LaunchManyCore should output a warning when Dispersy is unable to shutdown.

Wrote a unit test
http://jenkins.tribler.org/jenkins/job/Test_dispersy_trunk/78/testReport/junit/dispersy.tests.test_candidates/TestCandidates/test_tracker_yield_introduce_candidates/

The problem can be observed in: