ipfs-persistence-consortium's Introduction

IPFS Consortium

A agreement among trusted parties to assist with persistance of IPFS objects.

How To Use

Clone the source

$ git clone https://github.com/pipermerriam/ipfs-persistence-consortium.git

cd into the newly cloned repository

Install Requirements

You can do this with pip. Recommended to do this in a virtual environment

$ pip install -r requirements.txt

Setup a config file

Setup a config.json in the root directory of the project that looks like the following.

{
    "peers": [
        "QmP24Zx4uEh1ZhmPsERb4ZwBDVdXjHiFqRSkHyqeSJQEsW"
    ]
}

The peers key needs to be an array of IPFS ids.

Run the script

$ python ipfs_consortium.py

This will loop through all of the peers and pin their files locally. This script can be setup to run periodically with something like cron.

How it works

For each peer in the peers list, the script uses ipfs name resolve to find the ipfs address that has been published from that peer.
This address is expected to point to a directory of JSON files. Each of these json files should contain a top level array of IPFS hashes. These hashes represent the files that this peer wishes to have mirrored.
Each of these hashes is pinned.

How to ask others to mirror your IPFS content

Create a directory to contain your IPFS manifest JSON files.

$ mkdir ~/ipfs-manifests

In this directory, create any number of *.json files. These files should contain a single top level array of IPFS file hashes.

// manifest1.json
[
    "QmYwxuzozCBSD3FLP5Rg1pAXA4qjDtNrdGjyNzqEtr8TKu",
    "QmTrXyK24CAymWj5eATjxrisc8QaJocnGSY11BFFDEvqtH"
]

Add this file into IPFS

$ ipfs add -r ipfs-manifests
added QmPSM44WZ6rFW7d81pMQYG7BzDJRjhrWh7s69dTUpgmtRY ipfs-manifests/manifest-1.json
added QmbsC66VNjNdBU8simUfmxj2dUdpyWgkJKRiVBxBE9sBXQ ipfs-manifests

Take the hash associated with the ~/ipfs-manifests directory and publish it on IPNS

$ ipfs name publish QmbsC66VNjNdBU8simUfmxj2dUdpyWgkJKRiVBxBE9sBXQ
Published to QmP24Zx4uEh1ZhmPsERb4ZwBDVdXjHiFqRSkHyqeSJQEsW: QmbsC66VNjNdBU8simUfmxj2dUdpyWgkJKRiVBxBE9sBXQ

Give other members your IPFS id and ask them to add it to their peers list. You can get this your IPFS id with:

{
    "ID": "QmfGr14Jmu3fUvN74MGG7C8Av8buDbtLVK7Bs4yiaS4chX",
    "PublicKey": "CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCQxRzWJvimlHD0sHY8+n5X4TYhj2tNLVfiJZLJz1vZRMwSTqgmMyrFEh2o+4B21EtfR6tf0eKLwUerfPR0DUWRjJv4YL1+SidNxESsqIaENPBsaLwhGRFXM3PYeW+UjlTjrEybOf2cCY1h8+9XrlCMLHPROzS+QHbAW4Elz7CfqmbMbhDVXEuIVkDQxvXPVgVQEFwkKYexwfHbeLa2n5WTunsSec6GlEUfbwQOlmb/iMYfu2HfPztmwS1wXk1d4WvsUiB53gqljp7jsTPaE5YD4wyyUlQDeulO1zCdDjq2gQDxjtZ0fUzLfJU3dsdnpPcP1KQzvvSEAxNedjAsRRQTAgMBAAE=",
    "Addresses": [
        "/ip4/127.0.0.1/tcp/4001/ipfs/QmfGr14Jmu3fUvN74MGG7C8Av8buDbtLVK7Bs4yiaS4chX",
        "/ip4/10.0.1.48/tcp/4001/ipfs/QmfGr14Jmu3fUvN74MGG7C8Av8buDbtLVK7Bs4yiaS4chX"
    ],
    "AgentVersion": "go-ipfs/0.4.0-dev",
    "ProtocolVersion": "ipfs/0.1.0"
}

Setting up an EC2 instance running ipfs

EC2_setup.md

ipfs-persistence-consortium's People

Contributors

Stargazers

Watchers

ipfs-persistence-consortium's Issues

Hey Piper!

Hey @pipermerriam :-D! Giveth and Swarm City are looking at building something similar to this (and once we get it working we will likely include others (First on the list is Aragon, then we will start on boarding more groups) @sophiii is leading it with help from @sponnet and @jbaylina

A few brief questions:

Did you end up going forward with this? If not, why not?!
Are their other repos we should look at?
Have others done this?
Any words of wisdom?

Much love,
Griff

Missing file creation in README

In the section How to ask others to mirror, only the directory is created, but not the manifest-1.json file, and most importantly, it does not explain what to put into the file.

Distributed Part-Seeding

I had a similar idea for a service that can help persist IPFS objects which I think can improve on what you've started here in a number of ways.

Make a particular IPFS hash a token that all persistance consortium clients will pin, the content of the hash doesn't have to have any real meaning but then you can use findprovs with that hash to find other persistance consortium clients on the DHT.
Get around the trust factor by only seeding certain blocks of any given set. You don't really need to worry about the legality of the content if you're only storing a random 10% of the blocks, and so instead of having to have trusted peers you can have it be a trustless decentralised network of services.
Make each server have some kind of webservice or RPC so a client program can be made which can query the DHT for servers, and ask them to persist an object, and watch findprovs so it knows when the object is fully uploaded.
Instead of pinning whole files or individual blocks, just use the IPFS cache. This will give each object a limited amount of time to be sent around the network, but it also means you don't have to worry about running out of harddrive space and the service can be run completely hands off.

Recommend Projects

pipermerriam / ipfs-persistence-consortium Goto Github PK