Moving libp2p dependencies to peerDependency would allow to specify minimal required libp2p version on which gossipsub will work.

Originally posted by @vasco-santos in #83

The go-libp2p-gossipsub implementation has a throttled state that a message can go to if too many messages get validated at once. What would this look like in our implementation?

Trying to update libp2p and running latest v0.6.2 version of this package I get the following typing issues:

node_modules/libp2p-gossipsub/src/index.d.ts:10:28 - error TS7016: Could not find a declaration file for module 'time-cache'. '/Users/adam/Projects/iovlabs/rif-communications-pubsub/node_modules/time-cache/src/index.js' implicitly has an 'any' type.
  Try `npm install @types/time-cache` if it exists or add a new declaration (.d.ts) file containing `declare module 'time-cache';`

10 import TimeCache = require('time-cache');
                              ~~~~~~~~~~~~

node_modules/libp2p-gossipsub/src/index.d.ts:12:25 - error TS7016: Could not find a declaration file for module 'libp2p-interfaces/src/pubsub'. '/Users/adam/Projects/iovlabs/rif-communications-pubsub/node_modules/libp2p-interfaces/src/pubsub/index.js' implicitly has an 'any' type.
  Try `npm install @types/libp2p-interfaces` if it exists or add a new declaration (.d.ts) file containing `declare module 'libp2p-interfaces/src/pubsub';`

12 import Pubsub = require('libp2p-interfaces/src/pubsub');
                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

There are more complex options in gossipsub like scoreParams, scoreThresholds and directPeers. We should have examples for their usage, otherwise people will need to look into the code.

Let's consider a simple network with 3 peers {A, B, C}, with 2 connections established A<->B and B<->C

In this scenario, if A subscribes to a given topic and C publishes messages to that topic, A should be able to receive the messages.

It seems to me that https://github.com/ChainSafe/js-libp2p-gossipsub/blob/v0.8.0/ts/index.ts#L1034 this should not break here, otherwise we will not grab fanout peers. However, this does not seem to fix the issue, as I tried to do a small hacking in the code but messages were not sent to the peer.

Other than that, should we make canRelayMessage options to be true by default? I would expect the default behaviour to relay messages, but users might turn that off if they like.

Test scenario to test: libp2p/js-libp2p-interfaces#83 (use this branch as a dep)

@wemeetagain any ideas on what is wrong here?

cc @Gozala

Some fixes have gone into master that we need in order to release a new version of js-IPFS, could a new version of this module be published please?

see https://travis-ci.com/github/ChainSafe/js-libp2p-gossipsub/jobs/343776073

Occasionally CI tests fail on this test:

  gossip

    1) should send gossip to non-mesh peers in topic

  0 passing (2s)
  1 failing

  1) gossip

       should send gossip to non-mesh peers in topic:

      AssertionError: expected 1 to be above 1
      + expected - actual

      at Context.<anonymous> (test/gossip.js:49:39)

Restarting the tests usually "fixes" the problem.

When I startup libp2p-gossipsub as part of libp2p I get unhandled promise rejection error as follow:

(node:77341) UnhandledPromiseRejectionWarning: Error: Heartbeat timer is already running
    at Heartbeat.start (/Users/adam/Projects/iovlabs/rds-ipfs/node_modules/libp2p-gossipsub/src/heartbeat.js:41:27)
    at Gossipsub.<anonymous> (/Users/adam/Projects/iovlabs/rds-ipfs/node_modules/libp2p-gossipsub/src/index.js:340:28)
    at Generator.next (<anonymous>)
    at fulfilled (/Users/adam/Projects/iovlabs/rds-ipfs/node_modules/libp2p-gossipsub/src/index.js:24:58)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

Do you have some ideas why is it happening? Can it affect something?

Currently, it seems that everything works as expected even with this error, but it raises concern...

An initial implementation of the message cache data structure used in the gossip sub protocol has been implemented. Due to the ambiguity of the associated types, tests need to be written to make sure that this implementation is somewhat equivalent to the Go implementation. The tests need to follow the tests written in Go.

API docs need to be generated and added to the README, as discussed on ChainSafe/gossipsub-js#15.

In index.js, upon receiving a protobuf encoded RPC message, we process each kind of control message in handleIHave, handleIWant, handleGraft and handlePrune. We need to test these methods.

• Test for handleIHave

• Test for handleIWant

• Test for handleGraft

• Test for handlePrune

An important aspect of the gossipsub implementation is to be able to accommodate nodes running floodsub. We should have a test suite that proves that, as discussed on ChainSafe/gossipsub-js#15

Add manually-created or auto-generated typescript definitions.

gossipsub v1.1 features direct peers and peer exchange require gossipsub to initiate dialing new peers. We're also going to want access to the PeerStore to add the contents of PeerRecords.

Should gossipsub be passed a reference to the global Libp2p object so it can dialProtocol? Is there a better / different way to approach this?

If we have a reference to the global object, we might pick the connectionManager, registrar, peerStore, etc. from one place, rather than passing each submodule in explicitly.

@vasco-santos

Add to the already existing heartbeat tests, tests containing more than one node in the network and with publish and subscribe actions around, in order to confirm the content of the messages.

In the context of finding an issue with the heartbeat on ChainSafe/gossipsub-js#21

Currently, this test sends 10k messages, which overloads local / CI infrastructure.
It needs to be tweaked to be able to run without performance issues.

All released versions of libp2p-gossisub use logical OR in the guarding condition of the function _processRpcControlMessage:

_processRpcControlMessage (peer: Peer, controlMsg: ControlMessage): void {
    if (!controlMsg) {
      return
    }

    const iwant = this._handleIHave(peer, controlMsg.ihave)
    const ihave = this._handleIWant(peer, controlMsg.iwant)
    const prune = this._handleGraft(peer, controlMsg.graft)
    this._handlePrune(peer, controlMsg.prune)

    if (!iwant || !ihave || !prune) {
      return
    }

    const outRpc = createGossipRpc(ihave, { iwant: [iwant], prune })
    this._sendRpc(peer, outRpc)
}

I assume logical AND should be used instead, as an outbound control message may include any non-empty combination of IWANT, IHAVE, and PRUNE:

_processRpcControlMessage (peer: Peer, controlMsg: ControlMessage): void {
    if (!controlMsg) {
      return
    }

    const iwant = this._handleIHave(peer, controlMsg.ihave)
    const ihave = this._handleIWant(peer, controlMsg.iwant)
    const prune = this._handleGraft(peer, controlMsg.graft)
    this._handlePrune(peer, controlMsg.prune)

    if (!iwant && !ihave && !prune) {
      return
    }

    const outRpc = createGossipRpc(ihave, { iwant: iwant && [iwant], prune })
    this._sendRpc(peer, outRpc)
}

Note that composing the iwant array in outRpc message requires an additional check now.

The go-libp2p-gossipsub reference implementation of the corresponding function also uses this logic:

func (gs *GossipSubRouter) HandleRPC(rpc *RPC) {
    ctl := rpc.GetControl()
    if ctl == nil {
        return
    }

    iwant := gs.handleIHave(rpc.from, ctl)
    ihave := gs.handleIWant(rpc.from, ctl)
    prune := gs.handleGraft(rpc.from, ctl)
    gs.handlePrune(rpc.from, ctl)

    if len(iwant) == 0 && len(ihave) == 0 && len(prune) == 0 {
        return
    }

    out := rpcWithControl(ihave, nil, iwant, nil, prune)
    gs.sendRPC(rpc.from, out)
}

I came across this issue in a use case where IHAVE messages received by a subscribing peer were not answered by IWANT messages which, in consequence, led to lost publication messages.

Thanks a lot for fixing this issue in a patch release!

Hubertus

BTW, the pull request for [WIP] Gossipsubv1.1 is also affected.

Gossipsubv1.1 has the following spam protection:

In flight IWANT requests, sent as a response to an IHAVE advertisement, are probabilistically tracked. For each IHAVE advertisement which elicits an IWANT request, the router tracks a random message ID within the advertised set. If the message is not received (from any peer) within a period of time, then a behavioural penalty is applied to the advertising peer through P₇. This measure helps protect against spam IHAVE floods by quickly flagging and graylisting peers who advertise bogus message IDs and/or do not follow up to the IWANT requests.

In go-libp2p-pubsub, they can attribute the invalid message to a specific reason. This lets them properly score the peer/message based on the reason: https://github.com/libp2p/go-libp2p-pubsub/blob/master/score.go#L563

Unfortunately, based on the current design of js-libp2p-pubsub, we are not able to determine why a message is invalid. Only that it is (it returns a boolean, valid/invalid). Earlier in the gossipsubv1.1 effort, we extended our extension of validate to include topic validators and a function to process topic validator errors: here, but this is insufficient to track other causes of a message being invalid, eg: invalid or missing message signature.

A fix for this very likely requires modifying js-libp2p-pubsub in some way.

One possibility is to have validate throw an Error using js-errcode.
We could then extend validate to handle errors of different codes, and make our scoring code quite similar to the above go-libp2p-pubsub example.

Here is another relevant section of go-libp2p-pubsub: https://github.com/libp2p/go-libp2p-pubsub/blob/8945f91465bc3ba418688684a2c29c384cd79647/pubsub.go#L924

Open to other ideas of how to proceed.

See:
#87 (comment)
libp2p/js-libp2p#675
#83 (comment)

Once we can more easily rely on emitted events to trigger peer IP refreshes, use that. (Instead of currently running a synchronous peer-wide IP update, applied periodically)

As always, testing needs to be done in order to make sure this works as intended.
Here are some cases that need to be tested:

• Make sure that the gossipsub protocol is properly mounted on top of both nodes, A and

• Check that peer A can dial into peer B (vice-versa)

• Have peer B subscribe to a topic in peer A

• Have peer A publish to topic and make sure that peer B receives it

• Have peer B unsubscribe from topic that peer A publishes to

• Unmount gossipsub protocol from peers.

Hi hi,

I noticed that Go cannot publish messages that JS receives when using elliptic keys. That's because Go inlines the public key into the peer ID. We already worked on this issue with just general connections, but it appears it's broken in PubSub additionally.

This PR: libp2p/js-peer-id#101 will allow the issue to be fixed and enable finishing up this TODO: https://github.com/libp2p/js-libp2p-pubsub/blob/master/src/message/sign.js#L76

We recently moved the seenCache to the gossipsub layer, this is a sanity check to see if this is good.
In reading thru the go-libp2p-pubsub code, I see that their seenCache is actually at the pubsub layer!
Their processing goes in the following order (here):

1. filter blacklisted peers
2. filter unsigned messages in strictSigning mode
3. filter messages sent by others claiming to be from self
4. filter seenCache messages
5. push message onto topic validation queue, filter if queue full
6. add message to seen cache
7. process success case

We don't need to mimic the go-libp2p-pubsub implementation exactly but we should probably respect the behavior as closely as possible.

After the other tests are completed, we need to make sure that this JS implementation of gossipsub is interoperable with the Go reference implementation.

• Using a libp2p daemon, test whether any peer running either JS or Go implementations of gossipsub can receive and handle each others messages. Based these tests on the ipfs pubsub interop tests.

• Get started on these interop tests provided by libp2p

Need to test the gossipsub protocol for multiple nodes (>2).

• Mount gossipsub protocol on 3 nodes, A, B and C

• Establish connections between all peers

• Subscribe to different topics on different nodes

• Publish to different topics on different nodes

Before deploying gossipsub into the wild in the libp2p network, it would be great to have a suite of stress tests

Currently, we have a @ts-ignore hiding the fact that time-cache has no types.
This causes type errors for downstream users that don't have the time-cache types accessible.

Browser tests for go-gossipsub-copied tests are disabled because they create many independent peers, and it is slow and leads to timeouts in a browser environment.

Figure out how to appropriately run these tests.

Currently we can either disable gossipIncoming option or gossipUnverified messages.
Proposed solution would be to add hooks (optional param) when subscribing to topics, where we can pass function that decodes message and verifies it. If that hook throws, we ignore and don't gossip that message.

Proposed alternative

gossip.subscribe("topic", (message) => transformedMessage)

gossip.on("topic", (transformedMessage) => {}) 

Good work!!

Is this implementation of the updated spec? v1.1?

I am not aware of the differences between the two spec versions, but wanted to ask here.

Thanks

After asking about tagging peers on twitter, I was redirected to 2 issues related to the connection manager functionality in go-libp2p (1 and 2). These issues brought to my attention the js equivalent of the go connection manager, js-libp2p-connection-manager. This module manages connections to peers. However, it is lacking some of the functionality that the Go connection manager has.

Currently, I have copied over the peer class from the js-libp2p-floodsub as a temporary measure. However, I would have to manually add a lot of the functionality and I don't think it is worth the extra effort. My current thoughts are either to use js-libp2p-connection-manager as a super class and add the needed functionality (tagging,etc) and later on make a PR to the js-libp2p-connection-manager. Or make a PR to the js-libp2p-connection-manager to include some of this functionality before using it in our codebase.

Currently, only the default gossipsub multicodec ('/meshsub/1.0.0') is used to initialize the gossipsub protocol. We would like to initialize the gossipsub protocol with an application-supplied multicodec instead.

One known use-case is for applications that specify/require non-standard gossipsub multicodecs.

Debugging for an unrelated problem (ChainSafe/lodestar#3110) I noticed the cache keeps items for too little.

@wemeetagain noticed it keeps items for 30 seconds.

• Consider dropping time cache completely for a simpler solution
• Consult with other teams a proper timeouts or follow the spec https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/p2p-interface.md#the-gossip-domain-gossipsub

See below the message block 2046119 (new) FdqanLvU71cfZa2GBXOd7TOPJ8c, it's (new) and repeated

                  >>> Gossipssub - block 2046119 (new)           16.68s late FdqanLvU71cfZa2GBXOd7TOPJ8c Lighthouse/v1.5.0-90d5ab1/aarch64-linux 16Uiu2HAmBi4CiFxB9zQbEx4mkbGiE9bwSaaZ9nGZX8RhRGPXL2fh
Eph 63941/8 4.700 [CHAIN]            info: Received block: 2046119
Eph 63941/8 6.014 []                 info: Synced - block: 2046120   … (empty) - head: 2046119 0xe042…490b - finalized: 2046048 0x7916…f846 (63939) - peers: 14
                  >>> Gossipssub - block 2046114 (already known) 80.33s late AhdyuZn7FP8Ml6pgd2o6IZPzPSs Lighthouse/v1.5.0-90d5ab1+/x86_64-linux 16Uiu2HAmCWxz9iXTrstyF1YDbvmpN6ngG2R3cTw6bTAXMLZEN2b8
                  >>> Gossipssub - block 2046114 (already known) 80.34s late AhdyuZn7FP8Ml6pgd2o6IZPzPSs Lighthouse/v1.5.0-90d5ab1/x86_64-linux 16Uiu2HAmJj6y2ces4gjhRzPFXBEqRdB6NoKWSh3B1Mgevr41c4WA
                  >>> Gossipssub - block 2046114 (already known) 80.34s late AhdyuZn7FP8Ml6pgd2o6IZPzPSs rust-libp2p/0.31.0 16Uiu2HAm3gcqtY6W5717jFQXSfcHkMzGYAag8XuYgcqgDCebeFbz
                  >>> Requesting peer beacon_block IWANT 16Uiu2HAm2KKqQj4PdHEZoRegyRzTegTqDBuzHcPwp4foesYtzkvE [ 'L9aYtlGZxVxHBuNjtVR/3gHfth4' ]
                  >>> Gossipssub - block 2046115 (new)           72.12s late AH+MpdqfVsFB/nogYJhkIEdrWZM Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
Eph 63941/9 0.364 [SYNC]             info: INFURA HEAD 2046121, delay 0.3640000820159912s
                  >>> Requesting peer beacon_block IWANT 16Uiu2HAm2KKqQj4PdHEZoRegyRzTegTqDBuzHcPwp4foesYtzkvE [ 'DYodRUy6nfRhpLYdNLTTYqYqFCs' ]
                  >>> Gossipssub - block 2046121 (new)            0.88s late M1lw4SOUyZFV8rGS79oDyupTZCA Prysm/v1.4.1/a9ee3ee06afdffa2a9b0e759d4142642e7488416 16Uiu2HAm2KKqQj4PdHEZoRegyRzTegTqDBuzHcPwp4foesYtzkvE
Eph 63941/9 0.899 [CHAIN]            info: Received block: 2046121
Eph 63941/9 0.901 [CHAIN]           error: Block error slot=2046121, errCode=BLOCK_ERROR_PARENT_UNKNOWN
Eph 63941/9 1.265 [ETH1]            error: Error updating eth1 chain cache message=The user aborted a request.
AbortError: The user aborted a request.
    at abort (/home/lion/Code/eth2.0/lodestar/node_modules/node-fetch/lib/index.js:1418:16)
    at AbortSignal.abortAndFinalize (/home/lion/Code/eth2.0/lodestar/node_modules/node-fetch/lib/index.js:1433:4)
    at AbortSignal.dispatchEvent (/home/lion/Code/eth2.0/lodestar/node_modules/event-target-shim/src/event-target.mjs:337:35)
    at abortSignal (/home/lion/Code/eth2.0/lodestar/node_modules/@chainsafe/abort-controller/src/abort-signal.ts:68:12)
    at AbortController.abort (/home/lion/Code/eth2.0/lodestar/node_modules/@chainsafe/abort-controller/src/abort-controller.ts:26:9)
    at Timeout._a (/home/lion/Code/eth2.0/lodestar/packages/lodestar/src/eth1/provider/jsonRpcHttpClient.ts:107:18)
    at listOnTimeout (internal/timers.js:557:17)
    at processTimers (internal/timers.js:500:7)
Eph 63941/9 1.934 [CHAIN]            info: Received block: 2046120
                  >>> Gossipssub - block 2046115 (already known) 76.10s late AH+MpdqfVsFB/nogYJhkIEdrWZM Prysm/v1.4.4/29d48dfe7eb38959752048aec700f31c3abf7484 16Uiu2HAmH3MSFR4ijLotNJ6qML15uVRqGxkr5S19UUJrp3vmaTSQ
Eph 63941/9 6.000 []                 info: Synced - block: 2046121 0xa78e…0393 - head: 2046121 0xa78e…0393 - finalized: 2046048 0x7916…f846 (63939) - peers: 13
                  >>> Gossipssub - block 2046115 (already known) 82.79s late AH+MpdqfVsFB/nogYJhkIEdrWZM Lighthouse/v1.5.0-90d5ab1/x86_64-linux 16Uiu2HAmJj6y2ces4gjhRzPFXBEqRdB6NoKWSh3B1Mgevr41c4WA
                  >>> Gossipssub - block 2046115 (already known) 82.79s late AH+MpdqfVsFB/nogYJhkIEdrWZM rust-libp2p/0.31.0 16Uiu2HAm3gcqtY6W5717jFQXSfcHkMzGYAag8XuYgcqgDCebeFbz
Eph 63941/10 0.424 [SYNC]             info: INFURA HEAD 2046122, delay 0.4240000247955322s
                  >>> Gossipssub - block 2046116 (new)           74.16s late OLcXFZm0n6fyIJoIAFo538tsDMg Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
Eph 63941/10 6.001 []                 info: Synced - block: 2046122   … (empty) - head: 2046121 0xa78e…0393 - finalized: 2046048 0x7916…f846 (63939) - peers: 14
                  >>> Gossipssub - block 2046116 (already known) 78.10s late OLcXFZm0n6fyIJoIAFo538tsDMg Prysm/v1.4.4/29d48dfe7eb38959752048aec700f31c3abf7484 16Uiu2HAmH3MSFR4ijLotNJ6qML15uVRqGxkr5S19UUJrp3vmaTSQ
                  >>> Gossipssub - block 2046120 (new)           31.11s late 1zuLxwl+Rng+EuPfSDwtZySj0D4 Lighthouse/v1.5.0-90d5ab1/aarch64-linux 16Uiu2HAmBi4CiFxB9zQbEx4mkbGiE9bwSaaZ9nGZX8RhRGPXL2fh
Eph 63941/11 1.066 [SYNC]             info: INFURA HEAD 2046123, delay 1.065999984741211s
                  >>> Gossipssub - block 2046117 (new)           75.91s late NfWh1gqsh3z2rFOZBXpom78S1iQ Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046116 (already known) 88.62s late OLcXFZm0n6fyIJoIAFo538tsDMg Lighthouse/v1.5.0-90d5ab1/x86_64-linux 16Uiu2HAmJj6y2ces4gjhRzPFXBEqRdB6NoKWSh3B1Mgevr41c4WA
                  >>> Gossipssub - block 2046116 (already known) 88.75s late OLcXFZm0n6fyIJoIAFo538tsDMg rust-libp2p/0.31.0 16Uiu2HAm3gcqtY6W5717jFQXSfcHkMzGYAag8XuYgcqgDCebeFbz
Eph 63941/11 6.001 []                 info: Synced - block: 2046123   … (empty) - head: 2046121 0xa78e…0393 - finalized: 2046048 0x7916…f846 (63939) - peers: 14
                  >>> Gossipssub - block 2046118 (new)           70.53s late mNPQQTZJ5WUwTRI/pl6ZE8c+8vA Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
Eph 63941/12 0.704 [SYNC]             info: INFURA HEAD 2046124, delay 0.7039999961853027s
                  >>> Gossipssub - block 2046119 (new)           65.38s late FdqanLvU71cfZa2GBXOd7TOPJ8c Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA

I brought up TypeScript support in the Libp2p Roadmap document. Raul supports having TypeScript declarations for js-libp2p. I created a repo for this purpose and also found this other repo as well. The end goal would be to merge it into js-libp2p, just like we want to do with this library.

In either case, these will need to be integrated into this repo.

Any thoughts?

cc @GregTheGreek, @ansermino

Problem

in ts/index.ts, libp2p is required as a direct dependency. but it is declared as a devDep.

Solution

move libp2p to dep or peerDep

Gossipsub1.1 spec was extended to include: Validation queue protection with Random Early Drop

References:

• libp2p/specs#292
• libp2p/go-libp2p-pubsub#384

0.9.1 bumped the libp2p-interfaces dep from 0.10.x to 0.11.x. The current libp2p release depends on 0.10.x so the types are incompatible.

Errors manifest as:

src/runtime/libp2p-nodejs.js(40,7): error TS2419: Types of construct signatures are incompatible.
  Type 'new (libp2p: Libp2p, options?: Partial<GossipInputOptions> | undefined) => Gossipsub' is not assignable to type 'new (...args: any[]) => PubsubBaseProtocol'.
    Construct signature return types 'Gossipsub' and 'PubsubBaseProtocol' are incompatible.
      The types of 'peers' are incompatible between these types.
        Type 'Map<string, import("/Users/alex/Documents/Workspaces/ipfs/js-ipfs/node_modules/libp2p-gossipsub/node_modules/libp2p-interfaces/dist/src/pubsub/peer-streams")>' is not assignable to type 'Map<string, import("/Users/alex/Documents/Workspaces/ipfs/js-ipfs/node_modules/libp2p-interfaces/dist/src/pubsub/peer-streams")>'.
          Type 'import("/Users/alex/Documents/Workspaces/ipfs/js-ipfs/node_modules/libp2p-gossipsub/node_modules/libp2p-interfaces/dist/src/pubsub/peer-streams")' is not assignable to type 'import("/Users/alex/Documents/Workspaces/ipfs/js-ipfs/node_modules/libp2p-interfaces/dist/src/pubsub/peer-streams")'.
            Types have separate declarations of a private property '_rawOutboundStream'.
src/runtime/libp2p-browser.js(35,7): error TS2322: Type 'typeof Gossipsub' is not assignable to type 'new (...args: any[]) => PubsubBaseProtocol'.

cc @vasco-santos

I don't have access to the chainsafe travis account so I am creating this issue. The libp2p team suggests that we use aegir.

cc'ing @GregTheGreek and @priom.

I posted an issue in the libp2p github to tell them that we will be adding some extra functionality to some of the JS libp2p libraries in order to make them work like the same Go libp2p libraries.

These are the libraries that we will need to work on (may be extended):

• js-libp2p-connection-manager with go-libp2p-connmgr and corresponding interface

  • Peers in the JS implementation aren't tagged

  • JS implementation doesn't have a grace period

  • JS implementation has a different peer weighting mechanism than the Go implementation. Also the ranges of the weights differ.

  • Go implementation has a trimming heuristic for removing extra peer connections. JS implementation evicts connections on demand. There's an issue discussing these differences.

• js-libp2p-peer-info. There isn't an associated Go module but there is the js-libp2p module uses this in addition to the peer store for managing information about peers. The Go equivalent is go-libp2p-host.

• [ x ] The creation of a separate router module. The concept of a router doesn't really get introduced until gossipsub. In floodsub, we can get away without using a router. There is a go-libp2p-pubsub-router module that manages this for the Go implementations of pubsub protocols.

• Add support for determining the protocol a peer supports to js-peer-info. This issue addresses this problem. We might have to add this functionality to the swarm (inconsistently, called switch in JS) module since it handles dialling into peers.

• Add interoperability tests with go-libp2p-pubsub. The interoperability repository is still in development.

When the message cache implementation is stable and fully tests (e.g when #3 is complete), it makes sense to refactor it into its own npm module. Several of these data structures that were created for libp2p such as the time cache have their own modules and are widely used for different projects.

This repo is named gossipsub-js rather than the more standard js-libp2p-gossipsub.

It would be nice to conform to the naming of all of the other libp2p repos.
Perhaps we should id all known links that would need to be updated if we updated the repo name here.

With the new spec of the gossipsub protocol already finished and implemented in go-libp2p, other implementations are starting to update to v1.1

While it is backwards compatible with v1.0, there are important updates regarding security, as well as bootstrapping that makes it important to update sooner rather than later.

@wemeetagain what are your expectations for this update?

Gossipsub v1.1 checklist

• explicit peer agreements - #99
• peer exchange (PX) on prune - #100
• graft/prune backoff #95
• flood publishing - #89
• adaptive gossip dissemination - #94
• outbound mesh quotas - #91
• peer scoring - #83 #87
  • score thresholds - #83
  • heartbeat maintenance #91
  • opportunistic grafting - #96
  • score function - #83
  • topic parameter calculation and decay - #83
  • extended validators - #93
• spam protection measures
  • GRAFT messages for unknown topics are ignored - #91
  • IWANT message responses are limited in the number of retransmissions to a certain peer - #104
  • IHAVE messages are capped to a certain number of IHAVE messages and aggregate number of message IDs advertised per heartbeat - #104
  • In flight IWANT requests, sent as a response to an IHAVE advertisement, are probabilistically tracked. #107
  • Invalid message spam, either directly transmitted or as a response to an IHAVE message is penalized by the score function #113

Polishing

• refactor libp2p-pubsub libp2p/js-libp2p-pubsub#61 libp2p/js-libp2p-pubsub#62 libp2p/js-libp2p-pubsub#63 libp2p/js-libp2p-pubsub#64 libp2p/js-libp2p-pubsub#65 libp2p/js-libp2p-pubsub#66
• refactor gossipsub to use libp2p-pubsub v0.6.x #110
• refactor internal maps to use string peer ids where appropriate #111
• allow node configurability of constants #112
• copy go-libp2p-pubsub gossipsub tests
• final close reading of go-libp2p-pubsub, ensure nothing was missed #102 #108
• migrate ts/pubsub.js to libp2p-pubsub repo

Currently, all received messages in a subscribed topic are automatically gossiped.
We would like to add an option to not automatically gossip, rather, we would rely on manually re-publishing a message if gossiping is desired.

One known use-case is for validating incoming messages before gossiping.

This probably looks like adding a boolean option, passed in the constructor, similar to how #40 was exposed.

Some nodes are implemented in golang using (go-libp2p-pubsub), but they never arrive using this js library. Is there any known issue ?

Since the eventual goal of this library is to be used within the Libp2p js project, this library should be licensed under the MIT License. All Libp2p repositories are licensed under this license.

Also, since there is a lot of intersection between the gossipsub protocols and floodsub protocols, it is going to be easier to use open-sourced components in order to build out this library.

There seems to be issues when I set the policy to StrictNoSign.

I was able to reproduced in the 2-nodes.spec.js, Publish 10 msg to a topic test fails then policy is set: D4nte#1

When setting the policy on the js-libp2p example: https://github.com/libp2p/js-libp2p/blob/master/examples/pubsub/1.js Only the first message gets through.

Am I using the API correctly?

Since in Gossipsub we need to handle peers that are either using Floodsub or Gossipsub, we need to find a way to find which protocol a peer is running.

Here are several ways to go about doing this:

• Use js-peer-info and use peerInfo.protocols to determine which protocols are peer supports. The main issue with this approach is that peerInfo.protocols is not really used anywhere and has little to no tests for it in js-peer-info. As per this issue, it could be removed and thus can be seen as not stable to use.
• The other way is suggested in this issue. Run ls on multistream (see this) to determine the list of available protocols.

More work will need to be done to determine the best approach.

Control message piggybacking is a way to reduce the amount of messages that get passed around in gossipsub. It is an optional feature but I think should be implemented in the future to give developers the option to use it in their applications.

Unfortunately, didn't test out releasing w/ Aegir in #77
There are a few issues with the current setup when trying to release with Aegir

1. .eslint.js getting picked up by aegir for its linting step. Can be mitigated by either running with --lint=false or renaming .eslint.js to eg: .eslint.ts.js
2. tsc never gets run during the release cycle, so aegir fails on the test and other steps. Can be mitigated by either checking in built js in src/ or adding a tsc step in aegir

In order to make sure our implementation is the best it can be, we will need to benchmark each step of the gossipsub protocol.

• Subscribing to peers

• Publishing to peers

In GossipSub, we need to store the latest publish message and periodically run the heartbeat operation. The first can be easily done with default JS functionality but I don't think this is the case for the second. An initial Google search brings up using Promises to get similar functionality to Go's time.sleep but it's still not the same since it only pauses the function in which the promise is called.

We need to find a way to do this deterministically and make sure that we don't get unnecessary hangs in our code.