My original post to ipfs/kubo#1720:

This might be an extreme case, but consider someone wishing to make Wikipedia available on IPFS:

$ zcat dumps.wikimedia.org/enwiki/20150901/enwiki-20150901-all-titles-in-ns0.gz | wc -l
 11944438

It seems like a bad idea to have a merkledag node with 12M links, but that would be the representation using unixfs Data_Directory. It seems like a similarly bad idea to have a merkledag node past even 1k links, and directories with a thousand files occur more commonly in practice.

Alternate directory representations (Data_PrefixTreeDirectory and/or Data_HashTableDirectory) might be a solution, using intermediate merkledag nodes to ultimately reference all of a large directory's children in a way that permits efficient traversal and selective retrieval. The distinction between directory representations could be transparent to users, with implementations free to choose whichever data structure it deems suitable.

Going back to the example: the list of Wikipedia pages is 67 MB gzipped, before adding hashes. A user shouldn't have to download ~400 MB of protobufs just to find the hash of a single article page.

What's a sensible upper limit for a merkledag node's link count or encoded size in bytes? Is there precedent for reducing merkledag fan out? What other components would need to know about a new unixfs DataType?

@whyrusleeping replied:

We've been thinking about doing fanout on unixfs directories for some time now. But the 'best' way to do it isnt immediately apparent. There are a few different strategies,

the first is to just add all entries past the first N to a specially named link. And go down that way until we have stored all the entries. This is really convenient for listing directories, as you can just keep loading more until you hit the end, providing output the whole way. The downside is that searching for a certain directory becomes.... hard.

The second way is to shard them git style, and have K links at each level, each with log(k) characters in their name, and recurse that downward until each node has a sane number of links. This method works a lot better for lookups on large amounts of links, but kinda sucks for listing directories.

1k links is totally fine, our max block size is 256k (soft limit), and links are 47 bytes, with that we can store around 5500 links in each node and still be under 256k.

My reply:

COW filesystems are also worth exploring.

ZFS implements POSIX directories on top of "ZAP", the ZFS Attribute Processor, which provides a generalized key-value data structure on top of the underlying COW storage layer. ZAP objects, and thus POSIX directories, are encoded into COW objects for storage using two distinct formats.

The smaller format ("microzap") lumps an entire directory together into a single COW object containing a big boring hash table. It's used if the resulting object would fit into a single block (<= 128 KB) and the keys/filenames would all fit into fixed-sized structs (64 bytes each, so filenames must be <= 50 bytes). In practice this means most directories under 2000 files can be stored trivially, and every directory modification requires writing an entire replacement directory object to the storage layer. This seems analogous to Data_Directory, though the microzap structure uses defined sorting to optimize lookups.

The larger format ("fatzap") is a hash table guaranteed to span at least two objects. The top object is a pointer table, which references 2^N leaf objects, each acting as a hash table bucket. Leaf objects contain a header describing their contents ("I store objects whose hash starts with 10111") and the corresponding hash table, which in the case of a POSIX directory includes filenames pointing to file objects. There are a couple other less common bits of indirection too (leaf chaining for pointer table collisions and external pointer tables past 2^13 leaves), but those are details. The key feature is locality: ZFS can find what it's looking for, either for retrieval or modification, without touching too many blocks or even too many bits inside a block.

ZFS directories work well in practice having these two formats. POSIX readdir() does not guarantee that its results have any particular sorting, and hash ordering on-disk has better properties for lookups than lexical ordering, so that's what ZFS uses.

One other design note: ZAP objects/ZFS directories use a salted hash, meaning that multiple ZFS directories containing the same files will be stored and returned in different orders. I don't know the exact rationale behind this ZFS feature, but Go maps were changed from "undefined" to random iteration because an undefined yet stable ordering allowed people to depend on it.

@jbenet replied:

Thanks @willglynn these are great suggestions!

Maybe we should probably move this discussion into the https://github.com/ipfs/specs repo? it may be good to move discussion about general system changes to that repo. I'm sorry, i know it's very far from complete, but maybe with some help we can get it there!

there's one tricky thing to solve around this: how to discern whether a request should access the underlying ipfs objects, or get the "virtual" unioned directory instead. today we achieve this by separating ipfs {add, cat, ls} (unixfs) from ipfs object {get, put, data, links} (merkledag). in the future, i'd like to find some resolution to this in a way that doesn't create N+1 interfaces (where N is the number of "datastructures layered on ipfs" or "applications"), and instead just 2. (( tricky indeed )).

• There are seemingly arbitrary differences in error messages that respond as a string, and those which respond with a JSON object with Message and Code. CF ipfs/kubo#2293
• protobuf doesn't work as an encoding sometimes. The usage seems arbitrary and inconsistent.
• xml isn't specified as a possible encoding, and encoding isn't specified as an option in most of the commands, although it is in some.

Some commands return a hash of a created object as Key, some return it as Hash. This is pretty obnoxious. We should pick one and use it everywhere.

Are peerinfo objects authenticated? I am assuming each peerinfo object contains a map of services to multiaddresss. Does it also contain a signature on the corresponding map from the peer?

thank you, @jbenet :)

we should come up with a good permissions model for the api-- i.e. be able to grant permissions to other peers, or HTTP API consumers, or to specific apps to strict subsets of the API. (say RO, or on subset of commands, or forcing a given option to be set, etc).

I didn't see any specs for multikey so here are my notes what I'd love to see in it and how it could look.

We need a way to represent key types but also how are those keys stored, for example password they might protected. As keys might be getting much bigger (QC and hash based signature crypto) we also need a way to express keys bigger than 256 bytes. There are three options for that:

• use two bytes for size (64KiB = 524Kibit)
• use one byte multiplied by 4 (1KiB = 8Kibit)
• use two bytes multiplied by 4 (256KiB = 2Mibit)

(maybe something even bigger).
Each of them hash its pros and cons.

As format goes I would see it as:

[key protection schema+key type][crypto type][size][protected key]

First byte in 3 lower bits would include information if it is public, private or secret key (3bits = 8 values, rest left for future) and in 5 higher bits it would include information about how the key was protected, for example: no protection, scrypt+AES256, scrypt+salsa20, pure AES and so on.

Next byte would point onto crypto schema of key itself, it would depend on key type.

In case of symmetric key it might be AES128, AES256, salsa20.
In case of private and public for example RSA1024, RSA2048, ed25519, curve25519, ECDSA.

Questions:

• - ~~ in case of public/private keys not all of them are suited for both signing and encryption. Should we have key types for such keys. How about two cryptos being used in pair, example : ed25519+curve25519 (one is suited for signing and one for encrypting), should we have crypto key type or just have two keys in full multikey wrapping? ~~ no longer a problem as for ed25519 <-> curve25519 public key conversion
• - were should be designation of Hash-based message authentication codes placed (probably as symmetric but then we have symmetric keys that some are suited for encryption and some signing)
• - list of protection schemes and cryptos to include in initial spec.

Edit: Point of key protection schema is to allow sending for example password protected private keys. In case that just exchange of public keys took place, no-protection schema should be expected.

16:14 <•jbenet> validity may depend on the node's own position in spacetime.
16:15 <•jbenet> aside from the obvious time notion, it is possible in the future people may do things like geofencing records, like "when in Rome, resolve to A. otherwise, B".
16:15 <•jbenet> this of course is up to the node to decide + opt to follow, but a bunch of other interesting things can fall out of app-defined validity constraints.

I'm trying to figure out what is the protocol to achieve mDNS and DHT interop so that exchange doesn't have to understand the way that the resource/peer is found. Mainly my questions are:

• Do we populate the DHT routing table with mDNS entries found?
  • If yes, what is the criteria to drop/keep these mDNS entries? Or do we overload the protocol to have a virtual infinite bucket for mDNS entries? What if this value is abused?
  • If not, what is the criteria to opt for mDNS? Do XOR metric to all of the peers (mDNS available + DHT) and simply send to whatever is close? Is there any criteria in this process to prefer local network peers over remote ones?

This issues follows on my original proposal #90 (see the full scheme https://github.com/nicola/interplanetary-paths) discussed this weekend with the team in New York and conversations with @Stebalien, @jbenet, @diasdavid, @mildred (ping @dignifiedquire )

READ first: because there is so much text already, I wrote #important whenever it is the important part to read (of mine of course - feel free to use this in your posts)

Background

In this issue, I argue that there are two path schemes that we should offer to traverse IPLD objects:

• a low-level path scheme to traverse data blocks
• higher-level path scheme to traverse data objects.

Different layers of abstraction

We can abstract the different forms of data in different layers. For example, imagine we have file1.jpg in the folder dir.

Layer 4 (application)

The nice path that an application like unixfs should offer to their final user should allow to do the following /$hash/dir/file1.jpg.

Layer 3 (IPLD object path)

Let's assume that the unixfs application decides to structure their data in this way:

/$hash === {
  dir: {
    files: {
       file1.jpg: Link{@link: hash},
       ...
       file10000.jpg: Link{@link: hash}
    }
  }
}

Note: in the case of unixfs, we could aim at merging Layer 3 with Layer 4, but for the sake of the argument, I just made unixfs a bit more complex than it should.

Layer 2 (IPLD block path)

However, since the folder is very big, our chunker (this can be implemented in many ways, let's assume that The Nicola IPLD Chunker works this way) is going to split the IPLD object in multiple IPLD objects, that we are going to call IPLD blocks

/$hash === {
  dir: {
    files: {
      shard1: {
        file1: Link{@link: hash},
        ...,
      },
      shard2: {
        file5000: ...,
        ...,
      }
    }
  }
}

Summary of the 4 layers and their paths

• Layer 4: Application (e.g. unixfs): /$hash/dir/file1.jpg
• Layer 3: IPLD object: /$hash/dir/files/file1.jpg
• Layer 2: IPLD blocks: /$hash/shard3/files/shard1/file1/0/0/0//
• Layer 1: JSON primitives ({key: value})
• Layer 0: Data primitives (strings, numbers, links)

Differences between 2 and 3

In other words the two key layers for IPLD are the 2nd and the 3rd.
The reason why they should have different path schemes is because they are both important to the final application developer (depending whether they are writing higher or lower level application).

The difference between the two is that one traverses the actual IPLD data blocks, while the other one abstracts that. From the previous example:

• Layer 3: /hash/file3 would be a path of the final representation
• Layer 2: /hash/shard1/file1 is what we will have to use instead.

Note on different path schemes

Also, the way the separator will traverse either layer may have different meaning, for example in Layer 3, maybe there is no need to have transparent links, while it can be important for Layer 2.

So for example

{
  file1: Link({@link: hash, permission: 0777})
}
hash === {
  name: "Nicola"
}

• Layer 3:
  • /file/name === Nicola
  • /file/permission === undefined
• Layer 3
  • /file//name === Nicola (note the // meaning that we are traversing a link)
  • /file/permission === 0777

Reworking examples in the current spec

For simplicity call high-ipld high level, and low-ipld the low level (the low level is the current IPLD)

> low-ipld cat QmCCC...CCC/cat.jpg
{
  "data": "\u0008\u0002\u0012��\u0008����\u0000\u0010JFIF\u0000\u0001\u0001\u0001\u0000H\u0000H..."
}

> high-ipld cat QmCCC...CCC/cat.jpg
\u0008\u0002\u0012��\u0008����\u0000\u0010JFIF\u0000\u0001\u0001\u0001\u0000H\u0000H..."

> low-ipld cat --json QmCCC...CCC/doge.jpg
{
  "subfiles": [
    {
      "@link": "QmPHPs1P3JaWi53q5qqiNauPhiTqa3S1mbszcVPHKGNWRh"
    },
    {
      "@link": "QmPCuqUTNb21VDqtp5b8VsNzKEMtUsZCCVsEUBrjhERRSR"
    },
    {
      "@link": "QmS7zrNSHEt5GpcaKrwdbnv1nckBreUxWnLaV4qivjaNr3"
    }
  ]
}

> high-ipld cat QmCCC...CCC/doge.jpg
\u0008\u0002\u0012��\u0008����\u0000\u0010JFIF\u0000\u0001\u0001\u0001\u0000H\u0000H..."

> low-ipld cat --json QmCCC...CCC/blogpost
{
    "shards": [
        {
          "@link": "QmPHPs1P3JaWi53q5qqiNauPhiTqa3S1mbszcVPHKGNWRh"
        },
        {
          "@link": "QmPCuqUTNb21VDqtp5b8VsNzKEMtUsZCCVsEUBrjhERRSR"
        },
        {
          "@link": "QmS7zrNSHEt5GpcaKrwdbnv1nckBreUxWnLaV4qivjaNr3"
        }
      ]
}

> high-ipld cat QmCCC...CCC/blogpost
"This is a very long blogpost..."

Notes on implementation

Two paths options

• Solution 1: Different path scheme
  Maybe it can be done with different prefixes (/ipld1 , /ipld2)
• Solution 2: Same path with different separator
  By using a different separator than /, for example ., then we can mix the two path schemes. Assume that / will traverse the high level representation and . the lower level. I am not sure how this would work

Notes about this conversation

At the beginning my perception of IPLD was that pathing would resolve the high level representation, so that if I have a JSON, I could just be able to traverse it /friends/0/name, however the current IPLD pathing may not allow that.

Also, blocks and objects are in reference to file system concepts, they are open for better naming

#important

In IPRS, the validity scheme of a Record is also a MerkleDAG link to an obj, but what should be that obj structure to accept several validity schemes?

@whyrusleeping @jbenet ??

A week or so ago, @nicola @dignifiedquire @Stebalien and I discussed lots of IPLD things in detail.

Among them, we discussed pathing again. One result here was that we revisited the "two pathing delimiters" and link-properties problem. @Stebalien and I got to something in the end, and many approve (including @dignifiedquire and @diasdavid). @mildred i would be very interested in your thoughts. And I believe this is more in line with your original thinking too, or at least things you and I considered.

In the meeting we recalled many observations from previous conversations, including:

• we want the format to be as absolutely simple as possible, requiring very little conceptual overhead
• pathing with two different delimiters gets complicated for users
• the // shorthand for /@link/ increases conceptual complexity
• breaking apart into layers (Layers 2, 3, 4) helps address graph shaping problems
• the {"@link": "hash"} construction is getting around the problem that JSON does not allow representing a different value type natively (eg a link value). Other formats (CBOR, YML) do not have this problem.

Right now we have:

{
  "mode": 0444,
} // hash = Qmfoo

{
  "foo": {
    "@link": "Qmfoo",
    "mode": 0755,
  }
}

> /foo
{
  "@link": "Qmfoo",
  "mode": 0755,
}

> /foo/mode
0755

> /foo/@link/mode
0444

> /foo//mode
0444

Recalled and new observations where

• The "link properties" problem is introduced by using an object to represent links.
• We use an object to represent links because other JSON formats (like EJSON) use objects like {"$date": "2016-01-01"} to represent other Value Types.
• The "link objects" (such as a directory entry with a mode in a directory) are different from "link values". The current way to do things mixes these two.
• We'll already have to re-path in layer 4 or layer 3 to address things like /foo/bar/baz representing a unixfs directory traversal. (ie. the shape of the data, and without // or /@link/ traversals)
• Users will have to use /@link/ a lot all over the place.
• There is no way to address the Link Value on it's own.

We found a very nice middle ground between #1 in https://github.com/ipfs/ipld-examples/ and the current way to do things.

The proposal is to move to something like:

{
  "mode": 0444,
} // hash = Qmfoo

{
  "foo": {
    "entry": {"&": "Qmfoo"},
    "mode": 0755,
  }
}

> /foo/entry
{"&": "Qmfoo"}

> /foo/mode
0755

> /foo/entry/mode
0444

• Instead of using {"@link": "<hash>"} to represent a Link Value, use {"&": "<hash>"}
• Link Values are not objects, and SHOULD NOT include other properties.
• Users can still create "link objects" to add properties to links
• / is used for all traversal (local properties or link resolution)
• No need for a // shorthand
• Users are free to use link or whatever other propety name they wish. this is not a reserved keyword

To be clear, this is what #1 proposed all along, which has been championed by many of us along the way. It took a while for us to discover all the subtle, intricate problems that emerged otherwise.

Why @ instead of @link:

Why @? We did not want to use @link because we do not want to have things like:

{
  "link": {"@link": "Qmfoo"}
  "mode": 0755
}

Well then, maybe use &?

We can use any symbol for the Link Value. & was proposed (and seems to be winning favor) because:

• @ is just a hold-over from JSON-LD.
• & is commonly used across programming languages to describe "references"
• $ is overused and could create problems.

However & may have problems:

• It has meaning in URL query strings. (not sure this is a problem, i dont think it shows up in paths anywhere, but it might...)

CBOR formatting simplification

@dignifiedquire mentioned that this significantly simplifies the CBOR formatting implementation. I'll let him describe more as I don't recall the comments very well.

How would a third party utilize libp2p? The simple way would be to directly use libp2p but then you wouldn't be using multi-multiplexing and you would introduce redundancy if multiple software was running the full network stack; correct? Would a better alternative be to have a standalone networking demon that all the different programs wishing to the p2p networking would connect to?

Abstract

Introduction

Cross compatibility through several implementations and runtimes is historically an hard goal to achieve. Each framework/language offers different testing suits and implement a different flavour of testing (BDD, TDD, RDD, etc). We need a better way to test compatibility across different implementations.

Instead of the common API tests, we can achieve cross implementation testing by leveraging interfaces offered through the network and defined by the Protocol. We call this Protocol Driven Development.

In order for a code artefact to be PDD compatible

• Expose a connection (duplex stream) interface, may be synchronous (online, interactive) or asynchronous.
• Implement a well defined Protocol spec

Objectives

The objectives for Protocol Driven Development are:

• Well defined process to test Protocol Spec implementations
• Standard definition of implementation requirements to comply with a certain protocol
• Automate cross implementation tests
• Have a general purpose proxy for packet/message capture

Process

In order to achieve compliance, we have to follow four main steps:
1 - Define the desired Protocol Spec that is going to be implemented
2 - Design the compliance tests that prove that a certain implementation conforms with the spec
3 - Once an implementation is done, capture the messages traded on the wire using that implementation, so that the behaviour of both participants can be replicated without the agent
4 - Create the Protocol Compliance Tests (consisting on injecting the packets/messages generated in the last step in the other implementations and comparing outputs)

Protocol Spec

Should define the goals, motivation, messages traded between participants and some use cases. It should not cover language or framework specifics.

Protocol Compliance Tests Spec

Defines what are the necessary “use stories” in which the Protocol implementation must be tested to assure it complies with the Protocol Spec. For e.g:

# Protocol that should always ACK messages of type A and not messages of type B
> A
< ACK
> B
> B
> B

A test would pass if the messages transmitted by an implementation follow the expected format and order and it would fail if format and order are not respected, plus if any extra message is transmitted that is was not defined.

Tests should be deterministic, so that different implementations produce the same results:

┌─────────┐     ┌─────────┐    ┌───────────────┐
│input.txt│──┬─▶│go-impl  │───▶│ output.go.txt │
└─────────┘  │  └─────────┘    └───────────────┘
             │  ┌─────────┐    ┌───────────────┐
             └─▶│node-impl├───▶│output.node.txt│
                └─────────┘    └───────────────┘

So that a diff between two results should yield 0 results

$ diff output.go.txt output.node.txt
$

Interchange Packet/Message Capture

Since most of these protocols define leverage some type of encoded format for messages, we have to replicate the transformations applied to those messages before being sent. The other option is capturing the messages being sent by one of the implementations, which should suffice the majority of the scenarios.

Protocol Compliance Tests Suite

These tests offer the last step to test different implementations independently. By sending the packets/messages and evaluating their responses and comparing across different implementations, we can infer that in fact they are compatible

Example use case - go-multistream and node-multistream tests

(experimenting, will report soon :) )

no file at ipfs/specs/ipns-interplanetary-naming-system.

Goal

Having a well-defined specification for the IPFS Core API will let us build JS, Go, etc. modules that define the core interface, allowing many different implementations. This is very important for IPFS App developers: rather than hard-code for an HTTP daemon or a Go ephemeral node or a localhost daemon, they can simply develop against the Core API and swap out IPFS Nodes as needed. This will be especially useful once js-ipfs runs in the browser and apps will want to swap between HTTP and a local node.

Steps

• meeting with at least @jbenet @diasdavid @whyrusleeping to discuss core spec
• update the core spec with API details (use go-ipfs as a base)
  • ongoing work here: ipfs/interface-ipfs-core
• write a JS ipfs-core-api module that exposes the core interface (with tests)
• update js-ipfs core to adhere to ipfs-core-api
• update js-ipfs-api asking that it adhere to the ipfs-core-api
• write a Go ipfs-core-api package that exposes the core interface (with tests)
  • ipfs/kubo#2876
• update go-ipfs-api to adhere to ipfs-core-api

Steps updated by @lgierth 1-Sep-2016

In the top-level README.md, the link to the IPNS is broken.

This might seem a joke now but as project is called Inter-Planetary I think we should correct for light propagation times. In case of Mars the delay would be ~20mins but we don't know how humanity might evolve so it is better to be prepared for it.

Correcting for space would be quite simple, you need to know your position in space and position in space of record origin, calculate distance between them and divide buy speed of light. This gives additional time the record should be valid.

Light cones, blue area is space-time in which record is known and valid:

I've been thinking off and on ever since I discovered IPFS about how i could leverage its awesomeness. Obviously it's perfect for storing and accessing static content, but the sticking point always seems to be that it's “not dynamic”.

It occurred to me today that, when people make this argument, it's generally not the data that needs to be dynamic, it's how the data links together that allows an app or website to be dynamic. For example, people aren't constantly editing their tweets on Twitter (their tweets aren't dynamic), they create new tweets that generally relate somehow to other tweets. The ability to dynamically associate data seems to be what's missing.

From a database perspective, this is generally accomplished through the use of a join table. Since all data within IPFS is already uniquely identified (Merkle-links), and IPLD does an amazing job of distributing and routing data, it seems that the benefits of a join table could be accomplished through a small (conceptually) addition to IPLD.

I’m not familiar with the actual IPFS/IPLD implementation, so please forgive my ignorance. Conceptually my idea goes something like this…

IPLD should already be implementing a kind of join table, but, instead of linking two objects together, it’s linking an object to an IPFS node that can return that object. Now imagine expanding this capability slightly so it can also manage the linking of one object to another.

I’ll use a simple hypothetical command line example below to explain how this could be used.

> echo “Hi, my name is Bob” | ipfs add -q
QmR5wc8AuuJhLKTZMfH9eV6SYwao68eGuDitSkbAsB3UDu

> echo “Hi Bob, my name is Mary” | ipfs add -q
QmPn73mc9DqmhpeyE4T4TxuWGFXjAaLta49PSiUxcavRNr

> echo “Hi Bob, my name is Sally” | ipfs add -q
QmVgaeqotXMg2hs4tfHgRGm8EqWDF72hqf7aowyBoozYWz

> ipfs associate QmR5wc8AuuJhLKTZMfH9eV6SYwao68eGuDitSkbAsB3UDu QmPn73mc9DqmhpeyE4T4TxuWGFXjAaLta49PSiUxcavRNr

> ipfs associate QmR5wc8AuuJhLKTZMfH9eV6SYwao68eGuDitSkbAsB3UDu QmVgaeqotXMg2hs4tfHgRGm8EqWDF72hqf7aowyBoozYWz

> ipfs get_associates QmR5wc8AuuJhLKTZMfH9eV6SYwao68eGuDitSkbAsB3UDu
QmPn73mc9DqmhpeyE4T4TxuWGFXjAaLta49PSiUxcavRNr
QmVgaeqotXMg2hs4tfHgRGm8EqWDF72hqf7aowyBoozYWz

Linked objects would not automatically be returned when you fetch an object. Instead, it would be up to the client to explicitly query for a list of related objects, and they would then iterate through that list of hashes fetching each “child” normally.

With this new capability you can imagine how trivial it would be to add a comments section to a blog post or really any other scenario that normally relies on a relational database for its “dynamic” content.

Thoughts? Comments? Has this idea already been discussed elsewhere?

I think we're planning on auto-generating this from the Go implementation (at least for now). Discussion in ipfs/kubo#785. I just wanted to put a reference here, because I think it's likely that folks interested in the API spec (but not necessarily interested in how it's generated) will looks for issues in this repository.

WIP specs are stored in a pull-request and stay there for a long time. This is quite hidden. Might be a good idea to link those specs in the README to help people discover them.

Stub for issue

protocol/network#Encryption:

IPFS uses cyphersuites like TLS.

NOTE: we do not use TLS directly, because we do not want the CA system baggage. Most TLS implementations are very big. Since the IPFS model begins with keys, IPFS only needs to apply ciphers. This is a minimal portion of the whole TLS standard.

So you rolled your own transport encryption? I find this surprising, and I'm… skeptical.

This needs to be a new section (or an entirely separate document) rather than an aside.

(Also, I'll add that if all you want is to connect to something, exchange RSA public keys, and get an encrypted transport stream, TLS is entirely capable of providing that without involving CAs.)

The type Record appears several times through the spec with different formats

type Record struct {
  Value     []byte
  Expires   time.Duration
  Signature []byte
}

Record Node {
  Scheme   Link // link to a validity scheme
  Value    Link // link to an object representing the value.
  Version  Data // record version number
  Validity Data // data needed to satisfy the validity scheme
}

type Record struct {
  Scheme    Link // link to the validity scheme
  Signature Link // link to a cryptographic signature over the rest of record
  Value     Data // an opaque value
}

type Record struct {
  Scheme    Link // link to the validity scheme
  Signature  Link // link to a cryptographic signature over the record
  Expires   Data // datetime at which record expires
  Value     Data // an opaque value
}

Shouldn't it be just the last one?

Let's collect the IPLD bigger picture here

cc (@jbenet, @mildred, @dignifiedquire, @Stebalien)

• new org - (#128): move implementations and specs under https://github.com/ipld
• spec refining:
  • cycles/relative links (ipld/specs#1) - If we add relative links in merkle links, then we can have mutual relationships.
  • attributes in / (ipld/specs#2)
  • Find a name space: '/ipld' or '/ipfs' or just hash
  • Use of IPLD pointers as links (ipld/specs#3): right now implementation only support hashes
• query - way to find attributes (or value) that is easy to express (instead of programmatically traversing a tree)
  • selectors (ipfs-inactive/2016-IPFS-Workshop-Lisbon#5 and ipfs/notes#12)
    • design/find a nice language
    • research on relational db in directed graph
    • must be simple, fast, reliable
    • must create virtual graphs
  • full query language (later 2017) - more complex query than simple selectors
    • could be unreliable (unless specified)
    • design a query language
    • should create virtual graphs
• programmable data & transformations (#102)
  • IPC (inter-planetary computation) - decentralized compilable computation
  • Transformation - subclass of IPC (but pure, functional and return a graph/value)
  • IPLD objects can be input to IPC or can define transformations in the data itself
  • talk to the functional programming language research folks
  • could be writing contracts & smart signature (?)
    • infinite computation should be possible
    • possible languages to consider
      • web assembly
      • haskell
      • javascript
      • morte
      • LLVM IR
        • maybe there is a need for multipl (multi-programming or multi-language)
• verifiable computation on IPLD
  • consider simplifying lambda auth as a lib in haskell
  • consider extending IPRS, IPC(IPLD)
• abstract paths (#91)
  • should be able to express abstract paths via transformations
  • could do sharding (consequence of programmable data)
    • is sharing application specific?

I will clean this up in different issues

So, I'm putting this here for lack of a better place to put it...

When discussing IPLD, @jbenet brought up the concept of graph transformation programs. Basically, an IPLD object could point to to a portable/secure (e.g. compiled to webasm) program that could take the IPLD object as an input and could lazily generate some a different object as output. This would allow arbitrarily flexible abstractions (literally, turing complete). This is really useful.

However, @jbenet then noted that this could be used to add support for encrypting objects. That is, a decryption program could be written as a graph transformation that takes a key in addition to the object. After thinking about this, I realized that this isn't actually possible to do securely. In general, it's impossible to securely run untrusted code (even in a sandbox) on private data (cipertext/key). There will always be side channels by which that untrusted code can communicate with the outside world (at the end of the day, the user is a viable side channel).

I'm leaving this here as a not to future persons so nobody tries this. Basically, crypto, even decryption, must always be done by a small, carefully verified library.

We should make sure to note that a nodes keypair is not necessarily the keypair they must use for managing an ipns namespace.

Currently, we have /ipfs and /ipns. Soon, I assume we'll add /ipld (/ipld/HASH/a/b is will not point to the same object as /ipfs/HASH/a/b). So, my question is, what does /ipns/KEY/a/b point to? If we say that IPNS links can map to either an IPLD link xor an IPFS link, there's no ambiguity (replace /ipns/KEY with the value stored in the DHT, either /ipfs/HASH or /ipld/HASH). Unfortunately, this doesn't allow one to easily say "I want to talk about this IPFS file at the IPLD level".

My primary motivation is the API/gateway. That is, I need to be able to tell the gateway how to interpret the path I'm handing it. (Really, I'm trying to come up with a design that moves most of the API's functionality to the gateway).

Possible solutions:

1. /ipns+ipfs/KEY and /ipns+ipld/KEY.
2. /ipns/KEY and /ipns+ipld/KEY (IPFS is assumed).
3. /ipns/KEY/ipfs and /ipns/KEY/ipld (multiaddress?).

P.S. Does this problem even make sense or am I missing something?

I am currently working on py-ipfs-api and it is difficult to see what the library is missing from the Core API spec. It seems that the specification is not up-to-date and does not reflect the most import elements of the API. Could someone please revise the specifications to better reflect what is actually required of a full implementation?

It would be great to synthesize all of the ipfs spec into a set of proto3 and grpc files. not sure if doable, but getting close would be nice.

• Where could I read on implementing something inherently dynamic like a chat in IPFS?
  • Should every message modify an IPNS entry to the latest message in a linked list of blocks?
  • Should there be an authority that owns an IPNS entry to keep a chat working? There could be a set of authorities that should simultaneously accept new entry otherwise, but I couldn't find how IPFS is implemented, so I'm unsure if this is possible currently.

Such example would be nice to have to understand the proper way to develop for IPFS networks and it would serve like a proof that dynamic content is servable over IPFS.

As far as I've been able to discern, this only exists in the minds of @jbenet @diasdavid @whyrusleeping . Let's get this documented!

Now that we have marsheling and unmarsheling handled by IPLD implementations (and since a Record is a MerkleDAG Obj that uses IPLD spec), we don't need this anymore on the Record spec:

func Marshal(r *Record) ([]byte, error) {
  return recordProtobuf.Marshal(r)
}

func Unmarshal(r *Record, d []byte) (error) {
  return recordProtobuf.Unmarshal(r, d)
}

Originally in interplanetary paths I included the possibility of having relative paths.

Here I am proposing to add in the specs (and in the js-ipld implementation), the ability to resolve relative paths.
Having relative paths saves space (../, instead of a full hash), helps avoiding splitting things in multiple pieces (../ instead of creating a new object) and finally, allow circular relationships: merkle graph!

{
  nicola: { '/': hashNicola },
  david: { '/': hashDavid }
}

{
  name: "Nicola",
  friends: [{'/':'../david' }]
}

{
  name: "David",
  friends: [{'/':'../nicola' }]
}

After learning how a private key can be cracked through plain text + signature sampling (see: #24 (comment)), I've been searching to understand what are other implications and ways to overcome them, taking inspiration from PKI (SSL, TLS, X509, etc) and also from TUF, Notary and Docker Content of Trust.

Problem Statement: If an attacker is able to crack down a private key, they will be able to: 1) Impersonate us and therefore abuse our credibility in their favor, 2) Capture private communication. We can avoid number 2 by having a crypto channel that ensures Perfect Forward Secrecy, making the number 1 the issue the most pressing one.

In the IPFS context, if an attacker could 1) impersonate us, they would be able to: a) publish content as if it was us, b) add provider records that point to a given peer, which could cause a DOS on that peer if a lot of peers try to contact it at the same time for Records that it doesn't even possess.

To illustrate the several problems and possible candidate solutions, I'll go through 3 different scenarios.

Single key-pair

This scenario is simple to illustrate. If we use the same key-pair for everything and if it gets compromised, then an attacker gets full privileges and there is no way other than accepting defeat and rebooting the whole node. Using the same key for everything increases the possibility for crypto analysis and side channel attacks

Multiple 'chained' key-pair (hopefully the right way to go)

In this scenario, we would create a chain of keys with different privileges, using the MerkleDAG data structure:

• id-key-pair (aka root-key-pair):
  • also used to generate the peerId
  • if it gets cracked, the peer must be rebooted and with the process, reprovide every block (however this is highly unlikely)
  • used to sign the nounce between peers so that authenticity can be assured when providing a record
  • we must avoid assuming that this authenticity is offered by the transport (for example with TLS), as libp2p will support different transports with different capabilities.
• record-store-key-pair:
  • signed by id-key-pair
  • created to avoid having to use the id-key-pair signing all the records, so that if it gets compromised, we don't have to reboot the peer
  • if it gets cracked, the attacker will be able to create records, but won't be able to provide them because it will be missing the id-key-pair.
  • if can set a larger validity for the record-store-key-pair (like every month) to make it that is very hard for an attacker to have time to crack it. (and peer-Id can be rotated every 6 months, for e.g, making it really hard for brute forcing)
• record-key-pair:
  • signed by record-store-key-pair
  • used to sign a single record (ensuring freshness for every record)
  • just like the record, this key has validity and stop beings valid when the record is not valid anymore, so if it gets cracked, it won't be able to be used. Also has the same benefit of record-store-key-pair, if it gets cracked, an attacker can generate records for a short period of time, but it would be able to store them in the network because it won't be able to prove it is the actual node by missing the id-key-pair

Multiple 'non chained' key-pair

Very similar to the above with the exception that the record-store-key-pair would not be signed by the id-key-pair, making it impossible to associate who created the record to who is providing it, enabling attackers that compromise the record-store-key-pair to add records to the network pointing to a peer that they do not control for DOS attacks.

Other notes:

• File blocks/chunks are not signed, only hashed.

Following up the conversation that we had a couple of weeks ago, I took some time (with the help of @Stebalien) today and wrote down a path scheme that would keep the single /:

https://github.com/nicola/merkle-paths

I will put up a very simple implementation some time soon, I want to hear feedbacks now!
Also, I added some examples, amongst those an attempt on how to do cyclic graphs using relative paths

/cc @jbenet @noffle @diasdavid @mildred

Following discussions on #61 and #62, I noticed that on #61 we decided to encode links using the multiaddress format, and on #62 we defined what we wanted for the textual representation.

My question is: do we want to really use multiaddress to encode IPLD paths or do we want to use a similar encoding scheme but different (because the domain is different).

I came up with the following idea. Let's say we have a path that looks like:

/ipfs/HASH/foo//bar/baz/@link/boo

We could encode it as:

<id for ipfs><hash length><multihash>
  <id for internal link><length>"foo"
  <id for external // link>
  <id for internal link><length>"bar"
  <id for internal link><length>"baz"
  <id for external @link>
  <id for internal link><length>"boo"

This is compatible with the multiaddress scheme, but do we implement it there or do we do it elsewhere. If so, where?

Whoever wants to get more familiar with Python3 I/O (as I am actually heavily doing... 👍 ) can read here
Starting from page 77, it get quite fun!

Also this repo can be quite interesting starting.

And also considering the differences at struct

links in the new libp2p spec parts are broken, see pages like https://github.com/ipfs/specs/tree/master/protocol/network

https://datatracker.ietf.org/doc/draft-benfield-http2-p2p/

Hi all, introducing myself to the state of IPFS through the protocol spec. I've found that there are several dead links in that spec which could use a fix.

A few examples, but not necessarily all:

• networking -> moved to libp2p?
• routing -> moved to libp2p?
• block exchange?

Important note: This is not an entirely new proposal, it has been discussed several times over the course of the last 9 months in different situations with different views and opinions across the IPFS team. This comes to formalize the process so that we can commit and move it forward :)

Several of our users have been mislead to think of IPFS of File System instead of a Graph Database (fair point that is the name of the project after all:D). This mostly comes from the fact that the cat, get and addcommands come as first order commands and are the ones mostly used by demos.

Now, with the addition of the 'Files API', another layer of complexity and indirection was added. The common reaction to it is "Wait, what is Files API, weren't we adding Files all this time?".

With this, we miss the chance to lead our users to understand what a great Graph Database primitives IPFS offers and we also make it really hard for users to understand what Files API is, specially when it has such a generic name.

So, were is the proposal (this is not just me, although I'm the one writing it) that we've discussed through several points in time.

Rename the Files API to mfs, this will enable us to have one non generic keyword to call it that we can use with our users and also something that the community will be able to search for or aspect specially, since it has very technical details.

Move the cat, get and add commands to under the files umbrela.

In practical terms, this means:

# currently, the file manipulation commands are:
ipfs cat
ipfs get
ipfs add
ipfs files mkdir
ipfs files flush
ipfs files read
ipfs files cp
ipfs files ls
ipfs files stat
ipfs files rm
ipfs files write

Next:

# Manipulation of files
ipfs files cat
ipfs files get
ipfs files add

# mfs - Mutable File System
ipfs mfs mkdir
ipfs mfs flush
ipfs mfs read
ipfs mfs cp
ipfs mfs ls
ipfs mfs stat
ipfs mfs rm
ipfs mfs write

In ipfs/kubo#1320, I proposed a hypothetical fanout graph with:

Object    Type            Link
                       Name Hash
========  ===========  ==== ========
<hash-5>  fanout       d*   <hash-6>
<hash-6>  fanout-leaf  d    <hash-7>

and suggested non-recursive resolution of /ipfs/<hash-5>/d should
return /ipfs/<hash-7>.

On Wed, Jun 03, 2015 at 02:32:47PM -0700, Juan Batiz-Benet wrote:

btw, the fanout links thing should work like a union or a unixfs
file-- each node should work as a valid root (i.e. no fanout and
fanout-leaf distinction).

Agreed on valid-root-ness. I haven't looked into our layout
implementation. Fanout-leaf-ness is really just a way to identify
whether a link points to another fanout node, or if it points to an
independent child. If that information isn't contained in the fanout
node (e.g. if it requires you to lookup the linked hash and check it's
type), then non-recursive resolution for /ipfs/<hash-5>/d would look
like:

1. Fetch <hash-5> from the Merkle DAG.
2. Check it's type. It's a fanned-out directory (i.e. not a
   multi-chunk file, or a single-chunk file, or a single-chunk
   directory).
3. Find the fanout strategy used by this object. In this case it
   turns out to be first-name-character fanout.
4. Lookup the fanout child for our name (‘d’). The name starts with a
   ‘d’, so follow the ‘d*’ (or whatever) link to drill down the fanout
   tree to <hash-6>.
5. Repeat (1), but for <hash-6>. Since <hash-5> was a fanout object,
   we recurse here even though --recursive wasn't set.
6. Repeat (2), but for <hash-6>. It's a fanned-out directory (if
   we're not distinguishing leafs from non-leafs).
7. Repeat (3), but for <hash-6>. This time it's “just use the full
   name”.
8. Repeat (4), but for <hash-6>. The full name is ‘d’, so follow the
   ‘d’ link to <hash-7>.
9. Repeat (1), but for <hash-7>.
10. Repeat (2), but for <hash-7>. It's a unixfs-dir, so we've left the
    fanout and should use <hash-7> in the response.

I'd like to be bailing out after (8), before fetching <hash-7>.
That's going to require some way to distinguish “follow this link to
get another fanout object” (what we had in step 4) with “follow this
link to leave the fanout tree (what we had in step 8). I don't really
care if that information is encoded at the object level (e.g. “all
links from this node are to fanout objects”) or in special per-link
metadata (e.g. “this link points to another fanout object”), but I
want it somewhere in the linking object so I don't have to fetch the
linked object to find out. In this case “first-name-character fanout”
isn't compatible with links to non-fanout objects and “just use the
full name” isn't compatible with links to fanout objects, so that's
enough to figure this out without a <hash-7> fetch. Are all fanout
strategies going to be so obvious? Or do we need a separate way
to distinguish links that point to fanout nodes from links that point
to non-fanout nodes?

I'm trying to implement fs-repo for js and I'm not really sure the spec reflects the current go implementation. This is how the blocks are stored right now in my machine (0.3.8-dev).

blocks/
├── 1220025c
│   └── 1220025cfbe6331dd3b9403dbcd9b4810149d2fef328c8f8445454d0e684acf350be.data
├── 122003ba
│   └── 122003baa5a30b05babad30b53ee3f4a6f967f82cc0e4cf3510a145145a926211d4b.data
├── 1220053b
│   └── 1220053b4dd1c32d8400f1588a36c57a567278bb4462d403e53dac5db8ce07d13d11.data
...

While in the spec directory partitioning looks like:

# locations of the blocks
.ipfs/blocks/1114/f6/23/e0ec7f8719fb14a18838d2a3ef4e550b5e53
.ipfs/blocks/1220/fe/38/9b55ea958590769f9046b0f7268bca90a92e4a9f45cbb30930f4bf89269d

Full p2p connectivity requires NAT traversal (including the possibility of relaying messages between peers, e.g. STUN, TURN), so no p2p networking layer can be complete without incorporating a notion of p2p message routing.

In IPFS, p2p routing is solely the responsibility of the "Routing" layer (e.g. a Kademlia DHT).

Distinction

The line between Network and Routing layers is blurred significantly, and thus we must clearly distinguish the two responsibilities of Routing:

1. Peer Routing, for peer-to-peer discovery and connectivity
2. Content Routing, for data storage, discovery, and retrieval

To make this distinction more concrete, this essentially means splitting the Routing interface:

type Routing interface {
  PutValue(Key, Value)
  GetValue(Key) Value

  PutProvider(Key)
  GetProviders(Key) []peer.ID

  FindPeer(peer.ID) peer.Addr
}

into:

type ContentRouting interface {
  PutValue(Key, Value)
  GetValue(Key) Value

  PutProvider(Key)
  GetProviders(Key) []peer.ID
}

type PeerRouting interface {
  FindPeer(peer.ID) peer.Addr
}

And allowing the Network layer to use PeerRouting. This cleanly
separates the layers and avoids confusion.

Important Routing Implementation Details

It should be noted that p2p content routing systems (beyond the context of ipfs) usually combine both Peer Routing and Content Routing. Consider, "Kademlia" which is the combination of "Kademlia Peer Routing" (address ring, buckets, xor distance, etc), and the "Kademlia DHT", which uses "Kademlia Peer Routing". From an implementation standpoint, it is much cleaner to think of the whole system.

Forcing implementors to completely separate these two components is difficult, and perhaps counter-productive. Thus IPFS implementations should be careful to make it possible for Routing layer implementations to solve both problems, and export two components, PeerRouting and ContentRouting. The Network layer can then be a client of the PeerRouting part.

(We probably should do review with PRs?)

To have a secure, simple and user-friendly way of storing and managing keypairs for use by ipfs. As well as the ability to share these keys, encrypt, decrypt, sign and verify data.

I think this is two separate things: key management, and performing cryptographic operations. Instead of all in ipfs key, let's split it into:

ipfs key [ gen | ls | info | rm | rename | send ]
ipfs crypt [ sign | verify | encrypt | decrypt ]

Key management:

ipfs key gen

LGTM

ipfs key send

• note that the key will be encrypted before sending (relevant in case comm channel isn't).

otherwise LGTM.

Crypto ops:

ipfs crypt encrypt/decrypt

• i think default to stdin/stdout.
• maybe have -i, --input for input ?
• cipher? default to chacha, include salsa20
• mode? default to CTR. probably better not to allow ECB
• chose the IV? random? birthday problem. could potentially use the hash, but that's convergent.
• send the IV? i believe it doesnt have to be secure, only unique, so typically sent along.

We need to model this after common systems. in fact, i'd love to be able to make these tools non-ipfs specific, and do cat foo | crypt <args> | ipfs add.

We will also need to make additions to support keys in other commands, these changes are as follows:

• No, no changing other tools! compose them (or even alias them). instead of having ipfs add -r <file> support encryption, we wrap ipfs add to do ipfs encrypt <opts> | ipfs add <other opts>.

Structures

these are only for the ipfs crypt part, not the ipfs key

Signed DAG:

• sig only needs to cover the hash
• include a node with the fill public key so verification does not require finding it elsewhere.
• i.e.:

links:
  - name: @type, hash: <hash to signed dag definition>
  - name: content, hash: <hash>
  - name: key, hash: <pkhash>
data:
  <sig>

(@type is json-LD style)

Encypted DAG:

• not encrypted with public key, should encrypt with symmetric key. well i guess we should allow pki, but maybe using something like: https://gist.github.com/jbenet/12609db43c1d09ab519d (if it's right, just came up with it, so likely horribly wrong).

What is two stage cache?

It is cache which is defined by two times instead of one. First defines how long the entry can be cached, second how long the entry is "safe" meaning how long we can use it without trying to validate if it is actual.

If we try getting cached entry there are few cases that can happen:

• no entry available in cache or it is older than the first time => resolve entry, save it and return it
• entry available and it is younger than the "safe" time => return the entry
• entry available, it is older thank the "safe" time, younger than the first time => return the entry, run resolution in background if it is not already running, after it resolves it is reinserted into cache.

Why is it better?

Normal cache (with one time) has problem, if let's say the cache time is 1 minute we will have cache miss every 1 minute. It is the case currently with IPNS which ruins UX.

How it compares with current caching strategy of IPNS?

Currently entries are cached for a 60 seconds in normal cache. It is an equivalent of two stage cache with both times equal to 60 seconds (60, 60), cache is valid 60 seconds, re-resolve after 60 seconds.

What can it improve?

UX could be directly improved by changing it to (60, 30). This means if user uses entry after 30 seconds from resolution it will start re-resolving in background. This also increases consistency, because resolutions will be run more frequently.

We could also increase consistency without changing UX but increasing load on the network by making it (60, 0). This means that if entry is read for cache it will start re-resolving.

Due to how many possible configurations there are I suggest that we allow IPRS/IPNS entries to set their cache times in records them selves, defaulting to (60, 30) which would better in both UX and consistency than current (60, 60).

cc @jbenet @whyrusleeping @noffle

I think the inclusion of name inside the merkledag link is very limited for a particular use case, and even that is not general enough. I understand that it makes file-system use case easier to do, but even that I am not sure if it is really necessary. It could always be done through a two-level link: one which is just a link and another, which contains then metadata.

It feels to me that this is also something which is duplicated partially on the files layer.

Is this a legacy thing?

Anyway, while reading the paper, the section "path lookup performance" I see that you are even using in flattened tree name as a path. So that you add / between path segments. This has some assumptions about a platform used which IPFS is running on.

Initially I assumed that name is not a path so issues like how to store path separators uniformly and normalize them do not exist. But this seems not to be true. For example, in this case it would be much better if name would be an array of strings, in the most common case just an array with one item. But in such flattened tree view arrays would have multiple items.

But this is just a special case. Maybe name should not be a string, but each link could contain arbitrary metadata? Then flatten tree could store arrays, files layer could store file names. And some other service providing object-oriented file-system identified by tags could use metadata to store sets of tags on relations between objects/files.

Record Validity Systems

This is a discussion on why IPFS supports pluggable validity schemes. (Warning: very rough draft!)

(Note: the application of these records are things like routing systems (dht records) or name systems (dns /ipns records))

Different applications call for different validity schemes, depending on hard constraints. Some examples are discussed below. Since IPFS seeks to support a wide variety of use cases (e.g. fast, in-datacenter replication; slow, trusted blockchain-consensus replication; dns; ...) we must allow users to pick their own record validity schemes. Since these records are authored by a user, and record consumers have some relation to the author, it is reasonable to expect that agreement upon which validity scheme to use is pre-agreed upon.

(It is worth exploring ways to express the validity verification computation generically, so that new validity schemes can be used without requiring a code update to older clients. This is out of scope for this discussion.)

So far, this Validity System record flexibility has been used mainly for IPNS, but it is perhaps the case that all routing records should be equally flexible.

Vector Clocks or Log Time or Blockchain Time

Distributed systems are often organized with discrete logical events, or epoch intervals. These are correlated with physical time, but not dependent on the problem of measuring and agreeing on physical time, as that would require distributed consensus itself. These discrete time steps can be either relative (as in lamport's vector clocks), or pseudo-global (as in blockchain time). It depends largely on the use case constraints (latency / synchrony tradeoffs).

Regardless, these systems share in common that there is an append-only log of events, where each event represents a different "time epoch" and a timestamp can be made to refer to this epoch. Tecords may be given validity in terms of these logical clocks, in terms

[ 1 ]<---[ 2 ]<---[ 3 ]<---[ 4 ]<---[ 5 ]

Examples:

• "valid from T1 to T2"
• "valid before T"
• "valid after T"

These validity constraints can even be expressed in terms of variables, where the variables are decided when the record actually enters into effect. An example of this is: a bitcoin transaction created and signed at time T1, enters blockchain at time T2 and is valid from T2+1000 to T2+1000+10.

(Good old) NTP Timestamps

Fast applications where clock skew is non-criticial, security is not paramount (or where trust is fully delegated), may opt to use NTP Timestamps. For example, most humans operate on NTP Timestamps. This works well in practice.

((There are cases -- e.g. multi-party distributed consensus protocols -- where a "physical timestamp" is not available, not precise enough (skew), not trusted, or not in the application model. Thus systems opt for logical timestamps of other sorts. discussed above))

In practice these systems function like the generalized vector clocks, but the timestamps are derived from some physical measurement, and meant to convey meaning outside of the system itself. (hence the special distinction)

Examples:

• "valid before 2015-01-01 00:00:00.0"
• "valid after 2015-01-01 00:00:00.0"

Time-To-Live (TTL)

A slightly different way to use discrete, incrementing time is to reference the constraint as a time to live. I describe this separately because it is a substantially different use case, and the constraint expressed is necessarily relative. The record enters operation or "becomes valid" under well-defined conditions (e.g. a record is served, media begins to play, a transaction enters into effect).

It is worth noting that TTL validity is expressed relatively, and thus it can be signed without exacty knowledge of the time at which it will enter validity.

Examples:

• "valid from epoch 100 to 100+5" (logical)
• "valid for 10 more network hops"
• "valid for 100 seconds" (NTP)
• "received a DNS UDP packet with a record valid for 100s"
• "a rented movie began playing at time Y and is valid until Y+24hrs"
• "a bitcoin transaction was added to the blockchain at time Y"

Log Head Validity

Another validity scheme for append-only log datastructures is to use the "last" log entry as the most valid. This is divorced from a concept of regular intervals of time, and depends instead on state distribution, meaning that the state of the log needs to be replicated to the program deciding upon the validity of a record. When built as signed merkle-dags (logs using cryptographic hashes to reference previous entries, and digital signatures for authorship), the records may be sent over any (untrusted, asynchronous) channel.

It is worth noting that this scheme suffers from an important attack vector: an attacker may not be able to forge records, but may be able to serve old records, and supress communication of new records. A MITM can then ensure victims perceive an outdated version of reality (invalid as far as the record author is concerned). In git, this is preserving exploits, silencing bugfixes; in bitcoin, this is serving alternate or outdated (as far as work-majority is concerned) blockchain heads.

Examples:

• "the last commit in the branch" (git)
• "the last block in the blockchain" (bitcoin)