ipfs / js-ipfs Goto Github PK
View Code? Open in Web Editor NEWIPFS implementation in JavaScript
Home Page: https://js.ipfs.tech
License: Other
IPFS implementation in JavaScript
Home Page: https://js.ipfs.tech
License: Other
https://github.com/ipfs/js-ipfs#per-component-view
Network layer - See hhttps://github.com/diasdavid/js-libp2p
This issue focus on detailing the improvements we are committing to do for js-ipfs milestone 1
.
Currently, @nginnever is leading the charge on data-importing + files API. It would be excellent to have a couple of extra hands with the list of tasks below.
https://github.com/ipfs/js-ipfs-data-importing
The commands
command simply prints out all the available commands (and their subcommands). It is part of tooling.
Read only mounting works now:
in one term:
mkdir -p foo
echo "bar" > foo/bar
echo "baz" > foo/baz
# foo/baz: added block /XLM1ZETht3wv8vUPXMkx3JZGP5T9txAz782
# foo/bar: added block /XLMCA8WXBNRBwFhzRnHgHFLwGmQzkAQELH7
# foo: added tree /XLNUfxWCtidC125VaSgtM6yM1VRs7PcGvzq
ipfs add -r foo
ipfs mount /ipfs
in another:
# finding files by hash
cat /ipfs/XLMCA8WXBNRBwFhzRnHgHFLwGmQzkAQELH7
# bar
cat /ipfs/XLM1ZETht3wv8vUPXMkx3JZGP5T9txAz782
# baz
# listing directories
ls /ipfs
# ls: ipfs: Permission denied
ls -l /ipfs/XLNUfxWCtidC125VaSgtM6yM1VRs7PcGvzq
# -r--r--r-- 0 root wheel 4 Dec 31 1969 bar
# -r--r--r-- 0 root wheel 4 Dec 31 1969 baz
# path resolution
cat /ipfs/XLNUfxWCtidC125VaSgtM6yM1VRs7PcGvzq/bar
# bar
cat /ipfs/XLNUfxWCtidC125VaSgtM6yM1VRs7PcGvzq/baz
# baz
# symlinks work :)
ln -s /ipfs/XLNUfxWCtidC125VaSgtM6yM1VRs7PcGvzq foo
ls foo
# baz
# bar
cat foo/bar
# bar
wow, thanks unix :) โค๏ธ
ipfs mount <ipfs-path> <os-path>
On a mac, el capitan. IPFS installed fine. All command line tests run fine.
Video streaming works fine
Only ada.jpg loads on the photo demo (I may not have tried all)
Trying to load http://localhost:5001/webui. It stalls on
http://localhost:5001/ipfs/QmRyWyKWmphamkMRnJVjUTzSFSAAZowYP4rnbgnfMXC9Mr/#/?_k=***
It fails on three browsers : Firefox, Safari and Chrome. Chrome reports "waiting for localhost".
I have this error:
12:23:08.514 ERROR core/serve: Path Resolve error: context deadline exceeded gateway_handler.go:472
Regards
Laurent
https://github.com/diasdavid/node-spdy-stream-muxer
https://github.com/diasdavid/abstract-stream-muxer
https://github.com/indutny/spdy-transport
TEST
https://github.com/ipfs/js-ipfs/blob/master/tests/test-http-api/test-version.js#L38-L47
LOG
Debug: internal, implementation, error
AssertionError: expected [Error: Client request error: Parse Error] to not exist
at /home/travis/build/ipfs/js-ipfs/tests/test-http-api/test-version.js:40:27
at /home/travis/build/ipfs/js-ipfs/node_modules/ipfs-api/src/request-api.js:23:14
at finish (/home/travis/build/ipfs/js-ipfs/node_modules/wreck/lib/index.js:146:20)
at wrapped (/home/travis/build/ipfs/js-ipfs/node_modules/hoek/lib/index.js:867:20)
at ClientRequest.onError (/home/travis/build/ipfs/js-ipfs/node_modules/wreck/lib/index.js:155:16)
at ClientRequest.g (events.js:260:16)
at emitOne (events.js:77:13)
at ClientRequest.emit (events.js:169:7)
at Socket.socketOnData (_http_client.js:307:9)
at emitOne (events.js:77:13)
at Socket.emit (events.js:169:7)
at readableAddChunk (_stream_readable.js:146:16)
at Socket.Readable.push (_stream_readable.js:110:10)
at TCP.onread (net.js:523:20)
https://github.com/diasdavid/webcrypto
WebCryptoAPI polyfil to work in Node.js in the Browser (so that you don't have to be concerned about moving crypto code between a browser and server side app)
Context: nodejs/node#2833
https://github.com/jbenet/node-multihashing
https://github.com/jbenet/node-multihash
Use all the functions in multihash.
Wait, why, how is this different from Node crypto?
This module just makes working with multihashes a bit nicer. node-multihash is only for encoding/decoding multihashes, and does not depend on other libs. This module will depend on various implementations for each hash. For now, it just uses crypto, but will use sha3 and blake2, etc.
The next major subsystem to do is the DHT.
The rough plan in order of importance:
Note that this module should work on top of a msgproto packet stream. msgproto handles all standard encapsulation (network, integrity, signing, etc).
IPFS Repo is the storage driver for IPFS.
I've started working on this on #45, but as the HTTP API is in flux/being written (between a RPC vs REStful discussion), I'm going to defer this until that part is spec.
Whole spec ipfs/specs#65
HTTP API spec - https://github.com/ipfs/api
Add to cli: ipfs pin [-r] <ipfs-path>
Modules that are part of the Record System are about to change due to recent finding ipfs/specs#24 (comment)
Once we decided on a good solution for this issue, the development will continue and I will update the issue
It is time to give js-multiaddr a spin and upgrade it to support uTP, WebSockets and IPFS.
multiaddr spec - https://github.com/jbenet/multiaddr
js impl - https://github.com/jbenet/js-multiaddr
go impl - https://github.com/jbenet/go-multiaddr
Note how uTP, WebSockets and IPFS are not yet recognized by multiaddr https://github.com/jbenet/js-multiaddr/blob/master/src/protocols.js#L29-L37. We need to add support for:
After this, we need to add support to js-mafmt to be able to validate uTP, WebSockets and IPFS too
While we are at it, we should also support WebSockets on the go-multiaddr, //cc @whyrusleeping
I understand you want to modularize the project, but breaking up into a huge tree of git submodules is a little insane. Node already uses a module system by default, so we can put the same module structure you designed into directories. Each one as its own repo just makes the project a pain to maintain and to contribute to. If we move the modules into directories we will barely need to change the code, since the modules can still be loaded via require
.
There are projects with similar scope/size, and they all work great with the standard Node way of organizing modules. (E.g. Dat, which I know you've contributed to).
Overall, the current structure adds painful management overhead, will likely scare off contributors, and doesn't provide anything over the built-in Node module system. Should I go ahead and refactor?
@diasdavid Bitswap doesn't have a repository associated with it. (Neither does Peer routing, but I'm not sure if that is top-level and if it needs one yet or not).
Has work been started on it yet? @mafintosh is keen to help now, if so, or to just start cracking.
(repo, config, fs, etc)
One of the core pieces to get jsipfs working, is having a IPLD importer (file -> chunker -> merkleDAG), so that we can create MerkleDAG structures that can be used to be moved around bitswap.
The goals of the importers are described here - https://github.com/ipfs/specs/blob/master/overviews/implement-ipfs.md#ipld-data-importing
Probably it is possible to use some of the work developed by @mafintosh on his hyperdrive module, which although focuses on rabin chunking, should be easy to detach and use another chunking algorithm.
This takes me to my next point. It would be great to have a set of primitives to digest, chunk and parse files, so that different chunkers can be replaced/added without any trouble. Similar to the efforts we've been doing for libp2p interfaces.
ipfs-repo
is the module that implements and offers the functionality offer in repo spec
git repo: https://github.com/ipfs/js-ipfs-repo
It should work natively using the fs or in the browser using local storage
js-ipfs-repo should be able to grab an ipfs repo generated by go-ipfs and manipulate it without breaking interopability.
https://github.com/diasdavid/node-lib2p-swarm
This module is currently under a major refactoring, keep posted for updates.
The project currently uses Make to automate installation. This hurts portability, since Windows developers would need Cygwin or something similar. Grunt is a much better alternative since Node developers are used to it and it is fully cross-platform (since it's Node-based).
Resolving #11 would eliminate the need for a build system since the project could be installed via NPM.
This issue focus on detailing the improvements we are committing to do for js-ipfs milestone 1
.
js-ipfs-api
is the IPFS HTTP-API client library used by js-ipfs cli and a ton of other applications that want to interact with an IPFS node through HTTP (mostly go-ipfs nodes). Currently, for js-ipfs
we have to use a branch of js-ipfs-api
since IPFS 0.3.11 and 0.4.0 versions different in how the http-api endpoints work. We want to support 0.4.0 http api spec, which is standardised on https://github.com/ipfs/http-api-spec.
You can find js-ipfs-api
at https://github.com/ipfs/js-ipfs-api
feat/files-api
has more docs than masterWhat are the things that occur during a ipfs init
? (From key generation, if those base files are c&p or fetched from the network, etc). It would be good to make this part of IPFS spec itself.
The files
API offers an interface to work with unixfs, a Merklelized Data Structure that represents a typical FileSystem used in a Unix OS. with files, dirs and symlinks. This API is what enables IPFS to add and retrieve files without having to ask the users to convert their data into a graph data structure.
The swarm API enables IPFS users to learn about their connected peers and connect to other peers if desired
jsipfs node
is the interface that enables us to start and stop a daemon and also get some information about it
We ought to gracefully use port zero (like go-ipfs).
1) http api "before all" hook:
Uncaught AssertionError: expected [Error: listen EADDRINUSE 127.0.0.1:9090] to not exist
at tests/test-http-api/index.js:18:27
at src/http-api/index.js:55:18
at node_modules/hapi/lib/server.js:269:20
at done (node_modules/hapi/node_modules/items/lib/index.js:23:21)
at Server.onError (node_modules/hapi/lib/connection.js:163:16)
at emitErrorNT (net.js:1253:8)
@RichardLitt, thank you so much for helping me review stuff, as I explain in IRC, there isn't exactly PR to review, since I'm pretty much the only one maintaining the code and I just keep iterating over master till we hit a alpha version.
There is a simple 'roadmap' in this repo's Readme, however, if it is the first time you are dialing in to libp2p, it might not be clear what is going on. For that, I've a WIP of the libp2p spec, it does explain a lot, unfortunately not all, yet. You can find it here - ipfs/specs#19
In terms of reviewing code, the best place to start would be by the Swarm components, which handles dialing, protocol muxing, stream muxing and Identify. It is what every other component uses so that they can open streams between peers.
Once js-ipfs
milestone 1 is reached, we will start seeing more applications that will use IPFS and perhaps even depend fully on it in order to work. It is very important that we standardise as most as possible the core-API, so that we save our users from countless upgrading hurdles.
This will also help us identify how IPFS should be used directly. While we are at it, we must update js-ipfs-api
to expose the same interface, so that apps can exchange between a full node and one accessible through an http-api without a problem.
As discussed on the Oct 26 Sprint meeting, we will be renaming the prefix of any of our github repos that contain any JavaScript module, from 'node-' to 'js-'.
This decision came from hearing the feedback from the IPFS community, where there was always some confusion with the words 'node' and 'ipfs' and also a the lack of clarity if a module is supposed to be Node.js only. 'node' is a very overloaded word in the IPFS context.
This doesn't mean however that we will be leaving the npm ecosystem, npm will still be the tool of choice when it comes to publishing and installing this JavaScript modules.
We will be rotating the repo names starting of today, this should not change the behaviour when installing one of these modules or break any CI tools, since we always published to npm without the prefix.
List of repos suffering this change (let us know if we miss any):
Interact with raw blocks in the datastore. It uses IPFS-repo to store and get blocks
https://github.com/diasdavid/node-libp2p-mdns-discovery
Node.js libp2p mDNS discovery implementation. mDNS is used to discover IPFS peers in a local area network.
The pin command let's you control what hashes (blocks) are not garbage collected, meaning that they are not removed unless you specifically tell the IPFS node to do so.
https://github.com/diasdavid/node-ipfs-railing
Simple module to parse a list of bootstrap peers list, attempt to connect to them and return which ones are reachable and valid, so that we can populate our DHT initially. This process is known on the P2P literature as Railing
https://github.com/diasdavid/node-ipfs-random-walk
Node.js IPFS implementation of the discovery service through DHT random walking
In order to refresh the peers in our k-buckets for the Kademlia DHT, we perform 'random-walks' on the DHT, finding leaf peers, so that we can consider them for our k-buckets.
As with ipfs/kubo#9, the app fails to build on Windows due to the dependency on fuse (fuse4js).
https://github.com/diasdavid/node-multistream
Follows https://github.com/jbenet/multistream/tree/master/multistream spec
This issue focus on detailing the improvements we are committing to do for js-ipfs milestone 1.
ipfs-blocks
holds the Block data structure and the block-service
.
https://github.com/ipfs/js-ipfs-blocks
I'm not sure what this repo is doing. Can I use any of the stuff here? Can I install and help out? If so, how?
The top description is also confusing: ipfs implementation in node https://github.com/jbenet/ipfs
. Does that mean that https://github.com/jbenet/ipfs is a more up-to-date version of this? Or is that just a link to what IPFS is?
I want to get up on speed with js-ipfs and ran into a few small issues.
package.json
reads "main": "src/index.js"
I changed it locally to "main": "src/ipfs-core/index.js"
to be able to follow the README usage:
var IPFS = require('ipfs')
var node = new IPFS()
otherwise yields error: ipfs module not found
missing dependency
Should this be added to package.json?
'fs-blob-store'
jsipfs daemon
Initializing daemon... error listen EAFNOSUPPORT :::4001
if run a second time
Initializing daemon... This repo is currently being used by another daemon
Then I rm -r ~/.ipfs
and ipfs init
using go-ipfs, but can't get jsipf daemon working because of the EAFNOSUPPORT error
I want to run js-ipfs daemon in the browser, but can't even get it to work in node. Is it usable to a degree even if just for basic operations like fetching objects? I tried browserify but get all sort of dependency issues like:
Error: Cannot find module 'buffer!./pb/crypto.proto'
That I'm not sure how to resolve.
Thanks for any pointers on how to get js-ipfs daemon up and running, ideally in the browser!
https://github.com/diasdavid/node-ipfs-kad-router
Module to perform lookup for peers in a Kademlia DHT. For a given 'key' it must return a pq(priority queue) of peers that are the closest to that key - https://github.com/diasdavid/node-ipfs-kad-router/blob/master/src/index.js#L79
The go is to go through these modules, understand what stays, what leaves and what is missing compared to the go-ipfs. (probably will draw some ASCII art after I get this into my head, sometimes helps to think and communicate better)
The object API enables IPFS users to create graph structures with IPFS were links are MerkleLinks (multihashes)
It would be pretty sweet to grab this func and also the logic to 'inject' a go-ipfs repo (found in index.js and browser.js) and make a utils file that can be used across tests that can 'create and teardown a go-ipfs generated repo' and 'create and teardown a fresh baked repo.
This issue focus on detailing the improvements we are committing to do for js-ipfs milestone 1
.
unixfs
is one of the Data Structures that gets mounted on top of the MerkleDAG, it is what enables ipfs to handle files, like adding and getting them to IPFS. Currently this module is pretty close to finished, only requiring a second (or third) pair of eyes to go through the code, increase the code coverage and make sure everything feels right.
https://github.com/ipfs/js-ipfs-unixfs
feat/files-api
has more docs than masterThis issue focus on detailing the improvements we are committing to do for js-ipfs milestone 1
.
ipfs-repo
is one of the core pieces of js-ipfs
. It enables js-ipfs to use different storage backends without having to introduce any complex logic for the other IPFS subsystems. Although this module is quite stable at the time, its performance has room for improvement and documentation needs some love.
https://github.com/ipfs/js-ipfs-repo
Aye mates!
This is your captain speaking, the IPFS Node.js ship as been to set sail and good winds are blowing! Thank you for all the messages and enthusiasm shared so far, we truly appreciate your support and since some of you are eager to get their hands on code and help with the development, I'll keep this issue updated every week with what has been going on and where you can help the most. To make sure you don't miss any updates, click the 'watch' button on the repo.
Plan:
Where you can help during this week:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.