Hey everyone, just a heads up that there is a repo called https://github.com/ipfs/archives which is a place to coordinate efforts to archive datasets out there. The more help we get on this, the better it will go, so if you're an archivist and interested, watch the repo and help us out! ❤️ 📖 📚

Would it be useful to add a possible level of indirection and apply chunking to large link lists?

The real world directory object for all the RFCs is 421 kB in size for instance:

% ipfs object stat /ipns/QmbuG3dYjX5KjfAMaFQEPrRmTRkJupNUGRn1DXCgKK5ogD/archives/RFCs
NumLinks: 7715
BlockSize: 421110

One could imagine large directories with frequent small updates benefitting from being chunked themselves.

I would like to see the command API served over raw TCP or unix sockets, essentially whatever the Go net package supports. Requests would be encoded in JSON, written to the stream, and the same JSON encoded responses as used in the HTTP api would be read off the stream.

My reason for this is that I want to write fairly low level IPFS integration stuff and so far driving an HTTP client has become much more of a hassle than simply opening a socket.

I don't think this would be too much work to implement, and I volunteer to do it if I can get some guidence on where to start amending things in go-ipfs.

re: ipfs/team-mgmt#34

I am interested in starting some more discussion aimed at developing with Python in the IPFS ecosystem. This would include a hangouts vid-chat and/or irc discussion as part of our monday sprints. Some possible topics off of the top of my head:

• IPFS Extensions for popular web frameworks e.g. django, flask, etc.
• Eventually; a partial or full implementation of the IPFS protocol in Python (or C? -- that's a scary thought)
• ???

I'm particularly interested in the case of web apps that use IPFS as a backend and can scale simply by adding more nodes/processes. In general I think this would require a pub/sub or messaging framework for coordination (see discussions: ipfs/notes#64, ipfs/notes#40, ipfs/apps#10), but I think for now there should exist a nice library for small webapps that would allow one to use IPFS as a simple backend database.

Some existing projects:

• Git on IPFS
• WebRecorder.io on IPFS
• We've also started an api client for python which is mostly stable but could still use some more work.

I am excited to hear if anyone would be interested in this, and if anyone has any more suggestions/ideas put 'em here.

Iancus-MacBook-Pro:~ aurel$ go get -u github.com/jbenet/go-ipfs/cmd/ipfs
Iancus-MacBook-Pro:~ aurel$ ipfs version
-bash: ipfs: command not found

Many things apparent in the ipfs effort were pioneered a long time ago by the AFS (now OpenAFS) project, including storage location transparency; a global, automatic namespace mounted on the clients; and cross-realm authentication and permission management. At a minimum, their papers should be studied and cited.

In fact, it might also make sense to reuse much of their work -- ipfs could be implemented as an alternative AFS file server and so reuse all the effort of the AFS file system client kernel and FUSE modules. (A shim VLDB server may be required for full interoperability, but this is trivial.) RxRPC is a little long in the tooth at this point but may still provide a good foundation for both client/server and server/server interactions.

I would be happy to answer questions about the above, too; feel free to contact me as well as post here. :)

Referencing the Whole Earth Catalogue
This is often considered to be like the internet before the internet, and was an inspiration for many internet pioneers. It also sounds very wholesome and nice. More friendly than the stark "InterPlanetary", which implies span, but not connectedness.

I think this would be a nice association for a p2p cryptosystem!

Hey, Quick note to all IPFS watchers here.

I signed up The IPFS Project to support https://savecrypto.org -- an EFF-led petition asking President Obama to stand up against backdoor keys for encryption. This is one of many actions necessary to ensure encryption is not horribly compromised in the US.

Since this issue is central to much of what we're doing here, I figured it would be good to also alert everyone watching. Please sign and/or have your orgs endorse. This sort of action is heard. Remember the amazing SOPA/PIPA victory.

IPFS in good company:

Hi,

I like this & would like to run a node in a headless Virtualbox instance that runs on my Win10 desktop. Kindly advise as to which distro would be ideal for installation & I'll spin one up & (with your help) get this party started.

Personally, I'd like to host website media using it- for some of my sites.

Please advise.

Thanks,

-N

I'm actually not sure where to ask this, so I'm asking here. I went to the ipfs website and downloaded the Mac OSX binary, and moved it to /usr/local/bin/.

$ ipfs did not register as a valid command. What am I missing?

These conventional languages have a plethora of libraries available, and have a large programmer base. As go-ipfs reaches alpha, we can start with alternative implementations in other languages (similar to py-ipfs). Java/CPP are battle tested and can catch up quickly too.

This repository was moved from https://github.com/jbenet/ipfs to https://github.com/ipfs/ipfs -- it is the main repository for the project, and will be more of a "map" going forward.

doesn't look very nice.
Can we get one with just the logo and a transparent background?

Thank you for this project first of all.
I am trying to get my head around a couple of things. I have already read the paper (draft3) and watched some of the videos.

My first question is: how are IPFS objects stored? and where?
I have noticed this directory in my machine .ipfs/datastore and it seem to contain logfiles, a manifest and a database file. However, none of these files is readable (I see only bytes) and I cannot find any tool to open them. I am assuming that IPFS objects are stored in the database file in some format. Otherwise, how else are they stored? This is a bit confusing because in the paper it looks like there are IPFS files containing JSON-like information about the DAG graph. However, I am not clear where these files are and what formats they have.

My second question is related to locations. IPFS uses DHTs, so when a peer receives a requests it looks for that hash (or multihash) and if found, it checks where it is in the machine (say /User/Jonny/Documents/image_of_fish.jpg). If that is how it works internally, where could I find this mapping under the .ipfs/ directory? If that is not the case, how does the system know where the data is in the machine?

Maybe a bit of more clarification on what is stored under .ipfs/ and in what format would really help.

Thank you!
Ivan

We need to list all the repos with a short description of what each one is for. I'm going to start this out as a standalone markdown file and link to it from the readme.

I will use this issue to note thoughts regarding Linked Data in the IPFS context. Note this is just brainstorming.

The power of the semantic web is worth considering. While it hasn't really "taken off," it's TRTTD when it comes to data structuring.

@msporny created the wonderfully simple JSON-LD. Since IPFS is a tree dag structure, the JSON-LD spec (or a simplified version of it) might fit IPFS really, really well. This would give IPFS all the power of the semantic web with very little overhead.

This would mean the addition of a @context link (doesn't have to be that key, or even in the Links structure, could be its own field).

I hesitate to say there MUST always be a context in objects, as I'm sure this would hamper IPFS use. A strong design decision is to give the user entirely free reign on data format.

But perhaps there is some middle ground. At the very least, we should support the optional addition of an @context type of thing. Will continue to think about it.

Actually, @msporny, i'm really curious to hear your thoughts. Check out this project (paper, talk) and lmk your thoughts.

Hi guys,
I translated the English article into the French wikipedia. As I'm no expert in data storage nor P2P technology, I reckon there is mistranslation, and a proofreading should be done for accuracy-sake.

Thanks.

The design of BItSwap is viable for p2p relationships, but the exchange of ledgers is not trustworthy. Bad things can happen when an attacker creates Sybil nodes and sends crafted ledgers in order to erase debt/add credit, or possibly create a DoS attack against a user by creating debt. The consensus of debt/credit exchanged among peers can't be trusted in a Byzantine environment. If it could, viable cryptocurrency would have been around before the invention of the blockchain (a PoW blockchain is still the only Sybil-proof consensus mechanism).

Additionally, tying debt only to a node ID possibly allows nodes to erase debt by simply generating a new ID.

A viable solution for dealing with unknown/untrusted peers is to tie the exchange to cryptocurrency micropayments. This could either be remuneration (a node charges a price for certain data, and peers pay to download), or fidelity bonds (a downloader proves they have destroyed or donated money before data is exchanged). Since it doesn't really matter what blockchain is being used, Bitcoin, Filecoin, or both could be supported.

I am only addressing a replacement for the p2p debt/credit ledger exchange, cryptocurrency payments aren't necessary if data is being exchanged instead. Ledgers will work fine if they were created on the local node (you can trust yourself).

Hi guys, awesome project! I have a question though.

If the nodes in the DAG are stored as immutable content addressable blobs, surely if I write a single byte to a (logical) file somewhere down in the hierarchy, then because its content hash changes, anything which references it (tree node, for example) must also be updated, and it in turn will have a new hash, and so on all the way up to the root key after the mutable namespace?

In other words, every change anywhere in a tree will cause you to have to recursively recompute a new DAG for the ancestors. Isn't this kinda expensive? Think mounting a VM disk image where lots of writes are happening? I think this is why Ivy chose a log-based approach. Or have I missed something?

[EDIT] of course, I'm assuming "commit on file close after writing" semantics

Hello,

because the URL structures used in the IPFS-HTTP gateways, many static websites can become "broken" when using relative paths in their links.

Even if we can manually fix the links in the browser address bar, manually fixing the CSS and JS links is very annoying.

I imagine possible "fixes":

• Making native the IPFS protocol in the most popular browsers.
• Adding code to the gateways to modify on-the-fly the served HTML files (this "feature" should have the option of being enabled and disabled).
• And other variations of the previous ideas.

There is a small mention of IPNS at the bottom of ipfs.io, but how will hash to "nice name" mapping work?

Hi,

is it possible to have a datum replicated? Say I have a file I care very much and I would like to be sure it is replicated at least on two nodes: is it possible? I briefly looked at examples but I did not see this option.
If it is not implemented is it possible to implement it? Does it make any sense in the IPFS specs?

Thanks in advance
Bye
Piero

The project directory in this repo has many sections, maybe some of them (like Applications) should be merged with awesome-ipfs for example by linking awesome-ipfs here instead of having duplicate data

project-directory.md will also have to be updated when that is done.

this happens after graceful shutdown of my node: Error: Post http://127.0.0.1:5001/api/v0/version?enc=json&stream-channels=true: dial tcp 127.0.0.1:5001: connectex: No connection could be made because the target machine actively refused it.

Can you help @jbenet or others?

Why not stick to writing a single implementation (eg. go-ipfs) and just have some interface for interacting with the ipfs daemon with implementations in many languages?

Apologies if this is the wrong place, but is there any plans to implement an ACL on the HTTP gateway to define which data is allowed to be pulled via that gateway? (I.e has to have originated from your own nodes or some-such)

For instance if running a public HTTP>IPFS gateway to allow embedding on a content on a webpage, one may not want that gateway to allow just anything on the IPFS network as otherwise it is unfortunately likely to be abused.

In an Ideal world it wouldn't be an issue but we don't live in an ideal world and thus some nodes have bandwidth restrictions or are in jurisdictions where law enforcement is likely to be less than understanding about what IPFS actually is. I.e all they'll see is content was pulled from a HTTP server under your control.

Hey,

IPFS looks amazing, though its missing one key aspect of a file system, and that is permissions. Lets say that I create a file that I only want to share with my friends (or some list of nodes). The simple example would be a social profile. It would be nice to have the classic unix user/group model implemented in IPFS.

I'm not even sure where to start on something like this without a central authority to enforce the permissions, but it seems like it would be a useful thing to have.

Let's use this to ask questions that will then be added to the faq in README.md (which will eventually be put in http://ipfs.io too)

IPFS Objects have the following format:

// An IPFS Object
message Object {
  optional bytes data = 1; // opaque data
  repeated Link links = 2; // refs to other objects
}

// An IPFS Link
message Link {
  optional bytes hash = 1;
  optional string name = 2;
  optional uint64 size = 3; // cumulative
}

Wondering whether to kill Link.name entirely, and let the Object.data name links if desired.

Is there any metadata being passed around along with IPFS traffic? It would be exceptionally useful to aggregate the total storage used/free amongst all IPFS nodes and chart it somewhere. Average bandwidth as well. Being able to see the network grow by charting this data, and also an indicator of a network-wide attack on IPFS would be features of this data.

Protobufs has a successor you might want to use instead:

http://kentonv.github.io/capnproto/

Salutations. I have a quick question about policy, apologies if this is the wrong place to ask it.

How long does data stay alive on gateway.ipfs.io after it's accessed via gateway.ipfs.io/ipfs/$hash then the original data source goes offline?

Thanks,
Mike

The BitSwap protocol is not yet finished. Chances are it won't exactly as described. Still needs modeling + tweaking. (Just a warning for people reading the paper-- that's only large question remaining.)

Does IPFS provide/plan to provide control over HTTP headers when hitting an IPFS endpoint?

Hello Everyone!

There's many people watching this repo, so figured I'd throw a quick update here.

The go-ipfs implementation is almost at an alpha stage. A few days ago we streamed video across nodes. This means the DHT, BitSwap, and MerkleDAG are all working wonderfully together. And what's more, this is all with ipfs mounted at /ipfs (with FUSE for now).

All of this is thanks to the hard work of awesome contributors like @whyrusleeping @perfmode @mappum @Bren2010 and more. Huge thanks.

There is a lot to do finish implementing, we need to make the subsystems more robust, and we need to polish the entire thing. In particular, we still need to make the UX of installing and using IPFS awesome. If you know Go, and aren't scared of raw systems, come help us out! We're targeting to have a first release later this month.

Feel free to talk here, via email, on twitter, or come hang out at #ipfs on freenode.

Thanks!

Juan

Hello...

I have recently read the 2014 Filecoin and I understand from the IPFS website that Filecoin will follow the V1.0 release of IPFS. I am considering the use of IPFS and possibly Filecoin in a telco communications environment for a modified type of messaging; and for high latency network file stores.

Several questions come to mind...

Questions:

Q1: is there an implementation of the Filecoin system?

Q2: if so, then have there been any performance analysis tests run that you can publish for file size boundary cases?

Q3: I am interested in Proof of Retrievability Challenges... Is there a discussion of the different cases to be considered?

Q4: I have looked for an implementation of the Gateway.ipfs.io executable and I don't believe it is at the IPFS Github repository. Did I miss it?

If there is a more appropriate place to ask these questions please direct me to that location.

With kind regards,
jds

In the README, in the section "Who designed it", the "many people" link is broken
( https://github.com/ipfs/ipfs/blob/master/contributors.mdis )

I'm not sure if you're using git itself or just the idea. If you aren't actually using git, please ignore the next part :)

git uses sha1 for checking integrity. There are known flaws in SHA1. Isn't that an issue for your filesystem? Have you considered using a modified version of git?

http://ipfs.io/docs/getting-started/ says

> curl "http://ipfs.io/ipfs/$hash"

needs to say

> curl "http://gateway.ipfs.io/ipfs/$hash"

Hi everyone,

I've been reading about IPFS and some questions immediately popped up. These are mostly from the user perspective and very naive as in "How would [insert name] run on IPFS?".

Straight to questions:

• Would it be possible to implement something like inotify to get notified when new files are created?
• It's clear that IPFS works best for static content. Do you have any ideas for dealing with the following use-cases: AJAX & friends (partial content update), web-services (computation on servers, software not available), "real-time" multi-user systems (forums, chats, etc...). Mostly things that fit the centralized model quite well, not so easily the distributed model.

Some points regarding the questions:

• Starting from the bottom. Multi-user systems will easily branch into multiple website versions. Would this ever be unified or one would have to jump between different website versions to track the different conversations? How would logins (privacy?) work in this case?
• For web-services, "centralized" servers would still be necessary to perform the computation. Consider the situation where the software running in the "backend" is not available (publicly).
• Still in the web-service case, how would "upload" work at all? Files would be dropped in a folder monitored by the "server" perhaps? Thinking of a website like www.virustotal.com where users upload files that will be scanned by several anti-virus software.
• Still on the anti-virus software example. After processing the user supplied files, how would the user know that the analysis report is available?

Thanks and apologies if some of these questions have been answered elsewhere.

http://ipfs.io/docs/getting-started/

<p>This should bring up a console like this:</p>
<p><img src="" alt=""></p>

This issue will be to announce events to the watchers of this repo. We may have events that are not announced here. for best results, watch https://github.com/ipfs/community (we will also have a calendar soon).

So far, two new recurring events just popped up:

• Portland: IPFS Enthusiasts -- starts 2015-10-14 -- http://calagator.org/events/1250469161 -- discuss here: ipfs/community#49 cc @brixen
• Copenhagen: IPFS Meetup -- starts 2015-11-28 -- http://www.meetup.com/Copenhagen-IPFS-Meetup/events/225899409/?a=socialmedia discuss here: ipfs/community#51 cc @NeoTeo

One offs:

• Palo Alto or San Francisco: an IPFS Meetup on 2015-10-20 -- details TBD. discuss here -- ipfs/community#53

If you announce something here, please keep it to a short entry like those above, making sure to list location, time, and links to events and to an issue in https://github.com/ipfs/community/issues/ for further discussion.

Currently ipns resolves this way: ipns_addr >> ipfs_obj
What about:
ipns_addr >> HEAD -> ipfs_obj, prev_head -> prev_ipfs_obj, prev_head -> ... -> 1st_ipfs_obj, null

Why is this better?
Because ipns owner won't be capable of erasing the truth once stated in the network.
E.g. if we keep sins per each person, even if you hack all ipns owners you won't be able to erase your sins.

@whyrusleeping pointed me to IPRS but I hardly get it.

Problems

For quickly mutating ipns (years of streaming media) you will have a very long chain of commits to keep somewhere.

Update ipfs cat /ipfs/Qmcqtw8FfrVSBaRmbWwHxt3AuySBhJLcvmFYi3Lbc4xnwj/readme to "Use at your own discretion!"

Current text is:
"Warning:
| This is alpha software. use at your own discretion! |
| Much is missing or lacking polish. There are bugs. |
| Not yet secure. Read the security notes for more. |"

We've reached a pretty stable API, and IPFS now runs pretty reliably. People are already using IPFS from other languages, primarily JS through https://www.npmjs.com/package/ipfs-api

There's been talk about organizing an effort to get API bindings for more languages. Maybe we can start with:

• C/C++
• C#
• Erlang
• Go - https://github.com/ipfs/go-ipfs-api
• Haskell
• Java - https://github.com/ipfs/java-ipfs-api
• Javascript - https://github.com/ipfs/node-ipfs-api
• Lua
• PHP
• Python - https://github.com/ipfs/python-ipfs-api
• Ruby
• Rust
• ObjC
• Swift
• Scala - https://github.com/ipfs/scala-ipfs-api

The API is very simple -- it is just a REST-like HTTP + JSON API. Do we have any volunteers to help out with languages listed above (or others)? Please respond here if you can dedicate a good chunk of time. (I'll prepare a guideline doc for implementors in the meantime.) Then we can have a bunch of people working on this at the same time, which will make it go way faster. And then we can release them all in one go!

The first implementation, in Node is at https://github.com/jbenet/node-ipfs

Will be used for many of the Demos in #2.

i'd be into working on one. maybe a python implementation of a smaller submodule. ipfs-datastore?

Means nodes rarely see each other and operate mostly locally. Hovewer sooner or later all data gets distributed to all nodes using the time when they are online simultaneously, including between nodes that are never see each other directly (only though other nodes).

Hi there,

I've been running and IPFS daemon on my server for a few days now, and I noted regular crashes (one or two times a day).

It always starts with "panic: runtime error: slice bounds out of range".

At first I was running the master (0.3.8-dev) i386 version from Gobuilder downloaded last week, and the same behavior continues with the same version but from monday (2015-09-28).

The crash logs are very, very large (1.3Mo and 1.2Mo for today's (2015-09-29) crashes), and github doesn't want me to attach them to the issue, but I could create a gist here: https://gist.github.com/p4bl0-/16fb68dc8262bbe3b2c0 (no other public pastebin I found accepted files this size!).

Sorry to not be able to help further, I don't know how to read Go traces.