libp2p / go-libp2p Goto Github PK

View Code? Open in Web Editor NEW

5.7K 178.0 1.0K 51.42 MB

libp2p implementation in Go

License: MIT License

Go 99.64% Shell 0.35% Standard ML 0.01%

libp2p networking-stack p2p

go-libp2p's Introduction

The Go implementation of the libp2p Networking Stack.

Background
Roadmap
Usage
- Examples
- Dashboards
Contribute
- Supported Go Versions
Notable Users

Background

libp2p is a networking stack and library modularized out of The IPFS Project, and bundled separately for other tools to use.

libp2p is the product of a long, and arduous quest of understanding -- a deep dive into the internet's network stack, and plentiful peer-to-peer protocols from the past. Building large-scale peer-to-peer systems has been complex and difficult in the last 15 years, and libp2p is a way to fix that. It is a "network stack" -- a protocol suite -- that cleanly separates concerns, and enables sophisticated applications to only use the protocols they absolutely need, without giving up interoperability and upgradeability. libp2p grew out of IPFS, but it is built so that lots of people can use it, for lots of different projects.

To learn more, check out the following resources:

Roadmap

Our roadmap for go-libp2p can be found here: https://github.com/libp2p/go-libp2p/blob/master/ROADMAP.md This document represents current projects the go-libp2p team is focused on and provides an estimation of completion targets. It is a complementary roadmap to the overarching libp2p project roadmap: https://github.com/libp2p/specs/blob/master/ROADMAP.md

Usage

This repository (go-libp2p) serves as the entrypoint to the universe of packages that compose the Go implementation of the libp2p stack.

You can start using go-libp2p in your Go application simply by adding imports from our repos, e.g.:

import "github.com/libp2p/go-libp2p"

Examples

Examples can be found in the examples folder.

Dashboards

We provide prebuilt Grafana dashboards so that applications can better monitor libp2p in production. You can find the dashboard JSON files here.

We also have live Public Dashboards that you can check out to see real time monitoring in action.

Contribute

go-libp2p is MIT-licensed open source software. We welcome contributions big and small! Take a look at the community contributing notes. Please make sure to check the issues. Search the closed ones before reporting things, and help us with the open ones.

Guidelines:

read the libp2p spec
ask questions or talk about things in our discussion forums, or open an issue for bug reports, or #libp2p-implementers on Filecoin slack.
ensure you are able to contribute (no legal issues please -- we use the DCO)
get in touch with @libp2p/go-libp2p-maintainers about how best to contribute
have fun!

There's a few things you can do right now to help out:

Go through the modules below and check out existing issues. This would be especially useful for modules in active development. Some knowledge of IPFS/libp2p may be required, as well as the infrastructure behind it - for instance, you may need to read up on p2p and more complex operations like muxing to be able to help technically.
Perform code reviews.
Add tests. There can never be enough tests.

Supported Go Versions

We test against and support the two most recent major releases of Go. This is informed by Go's own security policy.

Notable Users

Some notable users of go-libp2p are:

Kubo - The original Go implementation of IPFS
Lotus - An implementation of the Filecoin protocol
Drand - A distributed random beacon daemon
Prysm - An Ethereum Beacon Chain consensus client built by Prysmatic Labs
Berty - An open, secure, offline-first, peer-to-peer and zero trust messaging app.
Wasp - A node that runs IOTA Smart Contracts built by the IOTA Foundation
Mina - A lightweight, constant-sized blockchain that runs zero-knowledge smart contracts
Polygon Edge - A modular, extensible framework for building Ethereum compatible networks
Celestia Node - The Go implementation of Celestia's data availability nodes
Status go - Status bindings for go-ethereum, built by Status.im
Flow - A blockchain built to support games, apps, and digital assets built by Dapper Labs
Swarm Bee - A client for connecting to the Swarm network
Elrond Go - The Go implementation of the the Elrond network protocol
Sonr - A platform to integrate DID Documents, WebAuthn, and IPFS and manage digital identity and assets.
EdgeVPN - A decentralized, immutable, portable VPN and reverse proxy over p2p.
Kairos - A Kubernetes-focused, Cloud Native Linux meta-distribution.
Oasis Core - The consensus and runtime layers of the Oasis protocol.

Please open a pull request if you want your project (min. 250 GitHub stars) to be added here.

go-libp2p's People

Contributors

Stargazers

Watchers

Forkers

wemeetagain dardevelin tilgovi daviddias harlantwood dignifiedquire mnp stebalien davidar izogain raulpercy alexsteele curtiszimmerman neuroradiology creantan realimage ldnvnbl peersafe david415 zjiekai ludovic-guegan-aldebaran geoah johndpope hackergrrl richardlitt peckjerry justindrake bobquest33 ikvm tptee izouxv zippy marten-seemann peterjerry nevercgoodbye jonnycrunch ericxtang naszam developer-dashboard dondyabla benderpan hunterchen enzoh eric-greencomb wigy-opensource-developer api-all gurry devopsmi farck verysimplyms terasum bigdata207 minxinping0105 metacurrency w1r2p1 skyformat99 pplam crystalsilver snail007 penguinxr2 jammyjaccy mashatan bdjibril avive sysbot bozzcq kotevode froyobin mitacha cocooncrash suzaku dmbreaker stanxii misrab rargulati romatotti alholm chrisdostert preetsinojiya eternityforest wanghhao charlieyqin cloud-robotics ahnan4arch upperwal deckbsd iotserv mark-rushakoff ringtail red7hj ctrox pythonai blockchainb monkcage qidu nisdas halokid versus forvos liyongming1982

go-libp2p's Issues

libp2p reuse

libp2p strikes me as an excellent package to have extracted out of ipfs for greater re-use in other projects. (which I believe was the plan!)

However, many other projects want the flexibility of being able to "go get" and of using standard go vendoring support for their dependencies. Do you think of gx here (and therefore a dependency on ipfs also) would put people off from re-using libp2p?

It's certainly made me think before using. I fully understand why you have done it, and I really like gx actually, but for a "core" library like this, I think better adoption would be achieved by working hard to minimise dependencies.

TestRateLimiting random failure

https://travis-ci.org/ipfs/go-libp2p/jobs/113809892

Unsure about Connecting behaviour

I'm making this issue as a note to go investigate and test what happens in the following situation:

the node is not connected to peer P at the time we start this
ipfs routine A calls host.Connect(C1, P)
ipfs routine B calls host.Connect(C2, P)
ipfs routine A gets cancelled, so C1 gets cancelled.

does routine B error out? or does it continue trying to connect to the peer?

Ping command creates a lot of goroutines

To replicate: while true; do; ipfs ping ID -n 1; sleep 0.5; done

Results in creation of following goroutine on receiving side:

gx/ipfs/QmZHf9MrEVGXNppqymCAGnvjYeHfVvg8kJkVtDQ8PhLRMT/yamux.(*Stream).Read(0xc82a8b3ad0, 0xc822fdefe0, 0x20, 0x20, 0x0, 0x0, 0x0)
    /home/kubuxu/go/src/gx/ipfs/QmZHf9MrEVGXNppqymCAGnvjYeHfVvg8kJkVtDQ8PhLRMT/yamux/stream.go:125 +0x3f1
gx/ipfs/QmZFdZY2Pjib1dT5632ZEgtDcaB36H3MGR6vP2diiZMyKS/go-smux-yamux.(*stream).Read(0xc82a8b3ad0, 0xc822fdefe0, 0x20, 0x20, 0xc800000000, 0x0, 0x0)
    /home/kubuxu/go/src/gx/ipfs/QmZFdZY2Pjib1dT5632ZEgtDcaB36H3MGR6vP2diiZMyKS/go-smux-yamux/yamux.go:20 +0x4b
gx/ipfs/QmduCCgTaLnxwwf9RFQy2PMUytrKcEH9msohtVxSBZUdgu/go-peerstream.(*Stream).Read(0xc83e9bce00, 0xc822fdefe0, 0x20, 0x20, 0xc83d4b5f80, 0x0, 0x0)
    /home/kubuxu/go/src/gx/ipfs/QmduCCgTaLnxwwf9RFQy2PMUytrKcEH9msohtVxSBZUdgu/go-peerstream/stream.go:73 +0x6c
gx/ipfs/QmYT5cN3i8sg7HtwjiWJKybcfs6FvHL5zUoEJMdSuqR6GY/go-libp2p/p2p/net/swarm.(*Stream).Read(0xc83e9bce00, 0xc822fdefe0, 0x20, 0x20, 0x7f7aabfc51d8, 0x0, 0x0)
    /home/kubuxu/go/src/gx/ipfs/QmYT5cN3i8sg7HtwjiWJKybcfs6FvHL5zUoEJMdSuqR6GY/go-libp2p/p2p/net/swarm/swarm_stream.go:30 +0x4d
gx/ipfs/QmYT5cN3i8sg7HtwjiWJKybcfs6FvHL5zUoEJMdSuqR6GY/go-libp2p/p2p/metrics/stream.(*meteredStream).Read(0xc841cf6f40, 0xc822fdefe0, 0x20, 0x20, 0x22, 0x0, 0x0)
    /home/kubuxu/go/src/gx/ipfs/QmYT5cN3i8sg7HtwjiWJKybcfs6FvHL5zUoEJMdSuqR6GY/go-libp2p/p2p/metrics/stream/metered.go:37 +0x75
io.ReadAtLeast(0x7f7aac00eee8, 0xc841cf6f40, 0xc822fdefe0, 0x20, 0x20, 0x20, 0x0, 0x0, 0x0)
    /usr/lib/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7f7aac00eee8, 0xc841cf6f40, 0xc822fdefe0, 0x20, 0x20, 0x0, 0x0, 0x0)
    /usr/lib/go/src/io/io.go:315 +0x62
gx/ipfs/QmYT5cN3i8sg7HtwjiWJKybcfs6FvHL5zUoEJMdSuqR6GY/go-libp2p/p2p/protocol/ping.(*PingService).PingHandler(0xc820011090, 0x7f7aac00ee40, 0xc841cf6f40)
    /home/kubuxu/go/src/gx/ipfs/QmYT5cN3i8sg7HtwjiWJKybcfs6FvHL5zUoEJMdSuqR6GY/go-libp2p/p2p/protocol/ping/ping.go:37 +0xc1
gx/ipfs/QmYT5cN3i8sg7HtwjiWJKybcfs6FvHL5zUoEJMdSuqR6GY/go-libp2p/p2p/protocol/ping.(*PingService).PingHandler-fm(0x7f7aac00ee40, 0xc841cf6f40)
    /home/kubuxu/go/src/gx/ipfs/QmYT5cN3i8sg7HtwjiWJKybcfs6FvHL5zUoEJMdSuqR6GY/go-libp2p/p2p/protocol/ping/ping.go:29 +0x34
gx/ipfs/QmYT5cN3i8sg7HtwjiWJKybcfs6FvHL5zUoEJMdSuqR6GY/go-libp2p/p2p/host/basic.(*BasicHost).SetStreamHandler.func1(0x7f7aac00ee80, 0xc841cf6f40, 0x0, 0x0)
    /home/kubuxu/go/src/gx/ipfs/QmYT5cN3i8sg7HtwjiWJKybcfs6FvHL5zUoEJMdSuqR6GY/go-libp2p/p2p/host/basic/basic_host.go:150 +0x83
created by gx/ipfs/QmYT5cN3i8sg7HtwjiWJKybcfs6FvHL5zUoEJMdSuqR6GY/go-libp2p/p2p/host/basic.(*BasicHost).newStreamHandler
    /home/kubuxu/go/src/gx/ipfs/QmYT5cN3i8sg7HtwjiWJKybcfs6FvHL5zUoEJMdSuqR6GY/go-libp2p/p2p/host/basic/basic_host.go:116 +0x490

They seem to die out after some time.

Here is goroutine frequency list during pining for prolong amount of time: https://ipfs.io/ipfs/QmWVcqMQKazS7UXr4HS3sPqJYvGJAJfqQSLpeFtVrJjuYP

Connection Closing

we need to decide how to choose when to disconnect from a given peer. Right now, we never disconnect after making a connection unless that connection gets dropped.

improve code coverage

our current code coverage is... not pleasant:

ok      github.com/ipfs/go-libp2p/p2p/crypto    0.014s  coverage: 43.0% of statements
?       github.com/ipfs/go-libp2p/p2p/crypto/pb [no test files]
ok      github.com/ipfs/go-libp2p/p2p/crypto/secio  0.004s  coverage: 0.0% of statements
?       github.com/ipfs/go-libp2p/p2p/crypto/secio/pb   [no test files]
?       github.com/ipfs/go-libp2p/p2p/discovery [no test files]
?       github.com/ipfs/go-libp2p/p2p/host  [no test files]
ok      github.com/ipfs/go-libp2p/p2p/host/basic    0.019s  coverage: 36.3% of statements
?       github.com/ipfs/go-libp2p/p2p/host/routed   [no test files]
?       github.com/ipfs/go-libp2p/p2p/metrics   [no test files]
?       github.com/ipfs/go-libp2p/p2p/metrics/conn  [no test files]
ok      github.com/ipfs/go-libp2p/p2p/metrics/stream    0.004s  coverage: 87.5% of statements
?       github.com/ipfs/go-libp2p/p2p/nat   [no test files]
?       github.com/ipfs/go-libp2p/p2p/net   [no test files]
ok      github.com/ipfs/go-libp2p/p2p/net/conn  0.676s  coverage: 46.7% of statements
?       github.com/ipfs/go-libp2p/p2p/net/filter    [no test files]
ok      github.com/ipfs/go-libp2p/p2p/net/mock  2.474s  coverage: 77.3% of statements
ok      github.com/ipfs/go-libp2p/p2p/net/swarm 6.616s  coverage: 75.5% of statements
ok      github.com/ipfs/go-libp2p/p2p/net/swarm/addr    0.003s  coverage: 83.7% of statements
?       github.com/ipfs/go-libp2p/p2p/net/transport [no test files]
ok      github.com/ipfs/go-libp2p/p2p/peer  0.018s  coverage: 31.0% of statements
ok      github.com/ipfs/go-libp2p/p2p/peer/addr 0.002s  coverage: 100.0% of statements
ok      github.com/ipfs/go-libp2p/p2p/peer/queue    0.541s  coverage: 87.5% of statements
?       github.com/ipfs/go-libp2p/p2p/protocol  [no test files]
ok      github.com/ipfs/go-libp2p/p2p/protocol/identify 0.590s  coverage: 75.6% of statements
?       github.com/ipfs/go-libp2p/p2p/protocol/identify/pb  [no test files]
ok      github.com/ipfs/go-libp2p/p2p/protocol/ping 0.018s  coverage: 74.4% of statements
ok      github.com/ipfs/go-libp2p/p2p/protocol/relay    0.212s  coverage: 79.6% of statements
ok      github.com/ipfs/go-libp2p/p2p/test/backpressure 3.683s  coverage: 0.0% of statements
ok      github.com/ipfs/go-libp2p/p2p/test/reconnects   0.742s  coverage: 0.0% of statements
?       github.com/ipfs/go-libp2p/p2p/test/util [no test files]

Lets work on getting that looking better!

bench test fails

» GO15VENDOREXPERIMENT=1 go test ./p2p/crypto/bench_test.go     ◉ ◼◼◼◼◼◼◼◼◼◼
# command-line-arguments
p2p/crypto/bench_test.go:20: undefined: GenerateKeyPair
p2p/crypto/bench_test.go:20: undefined: RSA
p2p/crypto/bench_test.go:35: undefined: GenerateKeyPair
p2p/crypto/bench_test.go:35: undefined: RSA
FAIL    command-line-arguments [build failed]

p2p/net/swarm: Fails to build due to dependency vendoring dependencies

github.com/ipfs/go-libp2p/p2p/net/swarm $ go build
# github.com/ipfs/go-libp2p/p2p/net/swarm
./swarm.go:112: cannot use PSTransport (type streammux.Transport) as type streammux.Transport in argument to peerstream.NewSwarm:
    streammux.Transport does not implement streammux.Transport (wrong type for NewConn method)
        have NewConn(net.Conn, bool) (streammux.Conn, error)
        want NewConn(net.Conn, bool) (streammux.Conn, error)

Looks like swarm.go imports github.com/jbenet/go-peerstream which has its own vendored version of streammux.Conn which of course makes it a distinct type from the unvendored streammux.Conn we're expecting.

The solution would be for @jbenet to unvendor things from go-peerstreap (like go-libp2p did recently, yay) or to avoid dependencies which have vendoring policies inconsistent with this project.

abstraction for transport level streams

Some transports we will add in the future (example: QUIC) will support opening new streams at the transport level instead of requiring we use a stream multiplexer. This won't work with our current level of abstractions, we will need some thought around how to make this work.

mocknet flaky under stress

I beleive there is a bug in the mocknet hosts somewhere. When putting it under stress i get strange failures (messages appear to go places they arent wanted). Switching to using testnet tcp backed hosts fixes the problems.

Implement (finish) circuit switching relay

circuit switching is a very basic form of relaying that we already have some code in place for in p2p/protocol/relay. The code isnt currently functional and needs some attention.

Implementing circuit switching means that we can provide nearly 100% connectivity to users. Anyone who can connect to a public relay node should be able to communicate with any other user in the network (who can also connect to a public relay).

Tasks:

finish and test code in p2p/protocol/relay
determine how this code will be used
- Host addon
- if X dials fail, try using a relay?
- Try using a relay and dialing normally at the same time, drop relay if normal connection succeeds?
add config option to set your node as a relay
config option for rate limiting relayed connections. (bandwidth of each, and total count)
set all gateway nodes as open relays

Punching NAT behind PPPoE does't work

In my case NAT is behind PPPoE tunnel, and go-ipfs is unable to punch through it. I don't know if it is possible at all but leaving it here as it is more and more common setup.

Handling connection helpers

libp2p will soon have multiple modules that it can use to 'help' in connecting to other peers. For example, relaying a connection to a peer, or sending a message out to ask that peer to connect to you.

We need to decide when to resort to those methods. Do we try to do one of those immediately? after ten seconds of normal dialing? Try relay after normal dialing fails entirely? We also need to figure out where these mechanisms should be plugged in to.

more context wiring

There are many different long running calls that we don't pass contexts into, for example, the IdentifyConn call in p2p/protocol/identify. We should make an effort to have contexts wired throughout the entire codebase

You could open a issue :P

https://github.com/ipfs/go-libp2p/blob/master/TODO

Can we add vendor to .gitignore?

Hit >80% code coverage on all libp2p packages

Similar to ipfs/kubo#3053 we want to have really good test coverage on libp2p. Its the core on which ipfs and other future applications rest, and if libp2p doesn't work, the apps on top of it don't stand much of a chance. Here is a list of all the libp2p package (in this repo and outside of this repo) that we need to have coverage for:

Allow peerstore to write to a go-datastore

I am working with @whyrusleeping to allow peerstore data to be stored to the hard disk when it takes up a certain amount of space as described in go-ipfs issue #2484

I'm opening this issue to discuss adding support for the go-datastore interface.

Relay should report dial failures

No buildable Go source files when running "go get"

$ go version
go version go1.6.2 linux/arm
$ echo $GOPATH
/home/pi/go
$ go get -d github.com/ipfs/go-libp2p
package github.com/ipfs/go-libp2p: no buildable Go source files in /home/pi/go/src/github.com/ipfs/go-libp2p

NAT Traversal testing

We really need to start actually testing NAT traversal in a variety of different scenarios. I have investigated a few different ways of doing this virtually, and theyre all 'really hard'. My current solution for the near term is to build a physical network with a way to automate the different NAT traversal scenarios of each router in the network for testing.

Whatever method is chosen, we can use this issue to track the implementation.

libp2p spec link in CONTRIBUTE is a broken link.

Nov 18 sync up notes

These are the notes of the meeting had on Wed Nov 18, might be useful for more people:

go-lib2p has two main entry points, p2p/net/swarm and p2p/host
host is what is known in libp2p spec as libp2p.Node. Calling it Node is makes more sense than host, because ultimately, libp2p abstracts the host to enable true addressing per process
routed host is the libp2p.figuredItOut() idea. The figuredItOut idea is when libp2p finds a way to contact a peer through its means (transports available, routing mechanisms, or even in the future have some machine learning to know which are the best ways to contact a peer at a given time)
example on how to use go-libp2p right now https://gist.github.com/whyrusleeping/4fe7dd6caa090881c8c6
go-libp2p uses gx, meaning you must install gx and gx-go
- go get -u github.com/whyrusleeping/gx
- go get -u github.com/whyrusleeping/gx-go

decrease dial concurrency when LOW_MEM flag set

In ipfs we have an IPFS_LOW_MEM env var that we can use to tell ipfs to try and use less memory for its normal operations. I think that we should respect that down into libp2p and decrease the dial limits in p2p/net/swarm/swarm_dial (roughly).

mocknet newStream notification issue

When opening a new stream using the mocknet (p2p/net/mock) the connection handler doesnt get called until a write actually occurs. I beleive this is a bug.

404-error for "The IPFS Network Spec, which grew into libp2p"

Weblink to "The IPFS Network Spec, which grew into libp2p" yields 404-error.

Set of examples

It would be really helpful, if can get an folder with the following examples:

Start two libp2p Nodes (create and make them listen)
Set up a handler for a protocol (through multistream)
Dial from one Node to the other, when both Nodes have TCP and uTP set up + SPDY (Yamux can be there too)
Dial from one Node to the other having set up secio too

go-ipfs cleanup

Either:

components of go-ipfs are to be yanked out of it (e.g. gateway, ipfswatch, seccat)
go-ipfs as a collection of such tools + UX stuff, it is the libp2p parts that are to be yanked out

Which one is preferable?
Ref: ipfs/kubo#2006

Moving DHT into libp2p from ipfs

The DHT package in IPFS seems to be pretty self sufficient so I was wondering if there is any plan to move it in go-libp2p.

Disclaimer: I'm pretty new to go-ipfs/go-libp2p and I'm still trying to wrap my head around gx which is my main issue with DHT being in IPFS. But in any case, DHT would be an amazing addition for libp2p. :D

ps. libp2p is the most amazing piece of spec and implementation for anything p2p I have had the pleasure of gazing upon and I really want to thank you for it.

secio errors don't get surfaced to dialer

In debugging secio with @diasdavid I noticed that the errors we receive in secio don't get propogated nicely back up to the dialer nicely. Not sure yet why this is

swap out metrics lib

we should switch out my forked go-metrics library with codahales metrics lib (that we use in go-ipfs)

Build failure (gx dependencies)

Since commit 7b48e5b I am unable to build this package. More specifically ./p2p/net/swarm/addr. The first problem is:

# github.com/ipfs/go-libp2p/p2p/net/swarm/addr
p2p/net/swarm/addr/addr.go:70: cannot use split[0] (type "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr".Multiaddr) as type "github.com/jbenet/go-multiaddr".Multiaddr in argument to manet.IsIP6LinkLocal:
    "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr".Multiaddr does not implement "github.com/jbenet/go-multiaddr".Multiaddr (wrong type for Decapsulate method)
        have Decapsulate("gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr".Multiaddr) "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr".Multiaddr
        want Decapsulate("github.com/jbenet/go-multiaddr".Multiaddr) "github.com/jbenet/go-multiaddr".Multiaddr

When I look at the code, the failure is normal. The package gx/ipfs/QmQB7mNP3QE7b4zP2MQmsyJDqG5hzYE2CL8k1VyLWky2Ed/go-multiaddr-net uses github.com/jbenet/go-multiaddr while libp2p uses the incompatible package gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr.

Both are very similar but the Multiaddr type from the gx package is different from the Multiaddr type from the github package.

We should probably update gx/ipfs/QmQB7mNP3QE7b4zP2MQmsyJDqG5hzYE2CL8k1VyLWky2Ed/go-multiaddr-net to use gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr instead of github.com/jbenet/go-multiaddr

Also, this might not be the right place for this discussion, but I am worrying about the go package names that encode the download location. Both github and gx. Go has a feature to enable any domain (not just GitHub) to be part of an import path (see Remote import paths). Perhaps we should use the ipfs.io domain for our packages and have gx integrate into that scheme. The package name would be the same, regardless of where it comes from, IPFS or GitHub.

Examples broken

$ go build ./examples/hosts/
# github.com/libp2p/go-peerstream
../../../workspace/gopath/src/github.com/libp2p/go-peerstream/listener.go:112: cannot use conn (type transport.Conn) as type conn.Conn in argument to func literal:
	transport.Conn does not implement conn.Conn (missing ID method)

abuse upnp to do local peer discovery

In networks that don't have mdns, but do allow upnp (does this ever happen?) we can abuse upnp to find other peers on the local network. If we add our peer ID to the upnp entry, other nodes can do a upnp listing and find us that way.

Race in dial

WARNING: DATA RACE
Write at 0x00c4202a8270 by goroutine 29:
  gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm.(*DialSync).DialLock.func1()
      /home/kubuxu/go/src/gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm/dial_sync.go:8
4 +0x16d

Previous read at 0x00c4202a8270 by goroutine 9:
  gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm.(*activeDial).wait()
      /home/kubuxu/go/src/gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm/dial_sync.go:4
1 +0xa5
  gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm.(*DialSync).DialLock()
      /home/kubuxu/go/src/gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm/dial_sync.go:9
1 +0x133
  gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm.(*Swarm).gatedDialAttempt()
      /home/kubuxu/go/src/gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm/swarm_dial.go:
191 +0x404
  gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm.(*Swarm).Dial()
      /home/kubuxu/go/src/gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm/swarm_dial.go:
161 +0x141
  gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm.(*Network).DialPeer()
      /home/kubuxu/go/src/gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm/swarm_net.go:3
7 +0x229
  gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/host/basic.(*BasicHost).dialPeer()
      /home/kubuxu/go/src/gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/host/basic/basic_host.go
:282 +0x26f
  gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/host/basic.(*BasicHost).Connect()
      /home/kubuxu/go/src/gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/host/basic/basic_host.go
:275 +0x13c
  github.com/libp2p/go-libp2p-kad-dht.connectNoSync()
      /home/kubuxu/go/src/github.com/libp2p/go-libp2p-kad-dht/dht_test.go:97 +0x223
  github.com/libp2p/go-libp2p-kad-dht.connect()
      /home/kubuxu/go/src/github.com/libp2p/go-libp2p-kad-dht/dht_test.go:103 +0x64
  github.com/libp2p/go-libp2p-kad-dht.TestValueGetSet()
      /home/kubuxu/go/src/github.com/libp2p/go-libp2p-kad-dht/dht_test.go:166 +0x414
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:610 +0xc9

Goroutine 29 (running) created at:
  gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm.(*DialSync).DialLock()
      /home/kubuxu/go/src/gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm/dial_sync.go:85 +0x410
  gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm.(*Swarm).gatedDialAttempt()
      /home/kubuxu/go/src/gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm/swarm_dial.go:191 +0x404
  gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm.(*Swarm).Dial()
      /home/kubuxu/go/src/gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm/swarm_dial.go:161 +0x141
  gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm.(*Network).DialPeer()
      /home/kubuxu/go/src/gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/net/swarm/swarm_net.go:37 +0x229
  gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/host/basic.(*BasicHost).dialPeer()
      /home/kubuxu/go/src/gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/host/basic/basic_host.go:282 +0x26f
  gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/host/basic.(*BasicHost).Connect()
      /home/kubuxu/go/src/gx/ipfs/QmYD8kZ8i5LVz5BBv6TdrWYRE7YZRs1dZXC3zmBHvQTwPC/go-libp2p/p2p/host/basic/basic_host.go:275 +0x13c
  github.com/libp2p/go-libp2p-kad-dht.connectNoSync()
      /home/kubuxu/go/src/github.com/libp2p/go-libp2p-kad-dht/dht_test.go:97 +0x223
  github.com/libp2p/go-libp2p-kad-dht.connect()
      /home/kubuxu/go/src/github.com/libp2p/go-libp2p-kad-dht/dht_test.go:103 +0x64
  github.com/libp2p/go-libp2p-kad-dht.TestValueGetSet()
      /home/kubuxu/go/src/github.com/libp2p/go-libp2p-kad-dht/dht_test.go:166 +0x414
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:610 +0xc9

Update README URLs based on HTTP redirects

This is an issue to track the changes which were initially started in #17. We move around repositories a lot, and links don't get updated. I am opening this issue to remind myself to look into using dkhamsing/frankenstein to make this work; currently blocked by https://github.com/dkhamsing/frankenstein/issues/17.

TestConnHandler is flaky

Every so often this one fails: https://travis-ci.org/libp2p/go-libp2p/jobs/156598013

External gateway detection seems off

Several users have reported (and I have noticed occasionally) that the addresses we advertise for other peers to connect to us on arent quite right. For example, peers might see an address with our correct external address, but without the right external port. We should investigate and test this.

make it build

doesn't build yet. working on it.

HSUISHFDIUHDFIUFHDIFHFIOHFIOSJFIOIFDJSOIDFJSIODO golang packages

Intermittent "context deadline exceeded" when no deadline set

We are seeing intermittent dial attempt failed: context deadline exceeded with no deadline set. This is with local/nearby nodes and no suspicious network weather.

The error happens here: https://github.com/mediachain/concat/blob/3a07b2cb8d7104b570fd5bcccb5c8f48f429cc38/mcnode/proto.go#L359

The context is a plain Context.

@vyzo can provide more details -- happens every few restarts. Any clue where to look?

Example projects that use libp2p

I come from the OpenBazaar project and I am new to go-libp2p and go-ipfs. Are the examples that use libp2p in a production environment? Where's the best place to learn libp2p by example?

InterfaceListenAddresses called way too often

InterfaceListenAddresses gets called quite a lot, and its a method that makes syscalls to resolve 0.0.0.0 addresses into real IP addresses. These should probably be cached in some way to aleviate the stress on the kernel we cause with so many syscalls here

host tests timed out

https://travis-ci.org/libp2p/go-libp2p/jobs/163584504

tcp.go:196: d.rd.DialContext undefined (type reuseport.Dialer has no field or method DialContext

Sort of self explanatory.

missed deps

github.com/satori/go.uuid
github.com/whyrusleeping/go-logging
github.com/jbenet/go-datastore
p2p/host/basic/basic_host_test.go:12:2: cannot find package "command/vendor/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" in any of:
/usr/local/go/src/command/vendor/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context (from $GOROOT)
p2p/peer/peer_test.go:14:2: cannot find package "command-line-/vendor/QmNsoHoCVhgXcv1Yg45jtkMgimxorTAN36fV9AQMFXHHAQ/go-base58" in any of:
/usr/local/go/src/command-line-/vendor/QmNsoHoCVhgXcv1Yg45jtkMgimxorTAN36fV9AQMFXHHAQ/go-base58 (from $GOROOT)
p2p/peer/addr_manager_test.go:7:2: cannot find package "command-line-/vendor/QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" in any of:
/usr/local/go/src/command-line-/vendor/QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr (from $GOROOT)
p2p/metrics/stream/metered_test.go:8:2: cannot find package "com/vendor/QmciEePSP8wpGYZp8fsPFi49Ya7xQMUFwFj2z5fDpfZnhC/randbo" in any of:
/usr/local/go/src/com/vendor/QmciEePSP8wpGYZp8fsPFi49Ya7xQMUFwFj2z5fDpfZnhC/randbo (from $GOROOT)
p2p/net/mock/mock_test.go:17:2: cannot find package "command-l/vendor/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" in any of:
/usr/local/go/src/command-l/vendor/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context (from $GOROOT)

@whyrusleeping

logger subsystems are off

one of them is github.com/ipfs/go-libp2p/p2p/net/swarm/addr There might be others that are also equally weird

basichost can remember invalid protocol preference

If i connect node A to node B, and they speak bitswap 1.1.0, node B will remember that node A supports that protocol. if node A is then downgraded to only support bitswap 1.0.0, node B will still attempt to only communicate over bitswap 1.1.0, resulting in a failure to communicate.

update NAT library

We're currently using an old version of github.com/fd/go-nat, There have since been improvements on both it and the underlying library it uses (jackpal/go-nat-pmp). Lets update!

Build Failure: Running gx install hangs fetching ipfs objects

I am trying to build go-libp2p in a fresh workspace.
After running make, the obvious next step in building seems to be gx install.
Alas, it hangs while fetching dependencies from ipfs; running with --verbose hangs here:

  - fetching QmZ4Qi3GaRbjcx28Sme5eMH7RQjGkt8wHxt2a65oLaeFEV via ipfs api

Dialer.Dial takes remote peerID but does not confirm that it is peer that it connected to

It potentially exposes libp2p users to MitM attack if the check isn't done manually after the Dial. We do it in go-libp2p-swarm code but if someone uses Dial directly it might be a problem.

It currently isn't done as secio handshake is done lazily on first usage.