Giter Club home page Giter Club logo

ipfs-store's Introduction

IPFS-Store

IPFS-Store aims to provide an API on top of IPFS with the following features:

  • IPFS proxy
  • Search capabilities
  • Pinning strategy (replication strategy)

IPFS-Store-_New_frame.jpg

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

Prerequisites

  • Java 8
  • Maven
  • Docker (optional)

Build

  1. After checking out the code, navigate to the root directory
$ cd /path/to/ipfs-store/ipfs-store-service/
  1. Compile, test and package the project
$ mvn clean package
  1. Run the project

a. If you have a running instance of IPFS and ElasticSearch

Executable JAR:

$ export IPFS_HOST=localhost
$ export IPFS_PORT=5001
$ export ELASTIC_HOST=localhost
$ export ELASTIC_PORT=9300
$ export ELASTIC_CLUSTERNAME=elasticsearch

$ java -jar target/ipfs-store-exec.jar

Docker:

$ docker build  . -t kauri/ipfs-store:latest

$ export IPFS_HOST=localhost
$ export IPFS_PORT=5001
$ export ELASTIC_HOST=localhost
$ export ELASTIC_PORT=9300
$ export ELASTIC_CLUSTERNAME=elasticsearch

$ docker run -p 8040:8040 kauri/ipfs-store

b. If you prefer build all-in-one with docker-compose

$ docker-compose -f docker-compose.yml build
$ docker-compose -f docker-compose.yml up

WIKI: Getting-started

API Documentation

Overview

Persistence

Represents the writting operations.

Raw

Enable to store any kind of content. The API uses HTTP multipart to sent the data or file over the request.

Operation Description Method URI
store Store raw content into IPFS POST /ipfs-store/raw/store
index Indexraw content POST /ipfs-store/raw/index
store_index Store & Index raw content POST /ipfs-store/raw/store_index
JSON

Enable to store JSON document.

Operation Description Method URI
store Store json content into IPFS POST /ipfs-store/json/store
index Index json content POST /ipfs-store/json/index
store_index Store & Index json content POST /ipfs-store/json/store_index
Query

Represents the read operations.

Operation Description Method URI
fetch Get content GET /ipfs-store/query/fetch/{hash}
search Search content POST /ipfs-store/query/search
search Search content GET /ipfs-store/query/search

Configuration

Represents the configuration operations.

Operation Description Method URI
create_index Create an index in ElasticSearch POST /ipfs-store/config/index/{index}

WIKI: API-Documentation

Details

[Persistent / Raw] Store content

Store a content (any type) in IPFS

  • URL: /ipfs-store/raw/store

  • Method: POST

  • Header: N/A

  • URL Params: N/A

  • Data Params:

    • file: [content]

  • Sample Request:

$ curl -X POST \
    'http://localhost:8040/ipfs-store/raw/store' \
    -F 'file=@/home/gjeanmart/hello.pdf'
  • Success Response:
    • Code: 200
      Content:
{
    "hash": "QmWPCRv8jBfr9sDjKuB5sxpVzXhMycZzwqxifrZZdQ6K9o"
}

[Persistent / Raw] Index content

Index IPFS content into the search engine

  • URL /ipfs-store/raw/index
  • Method: POST
  • Header:
Key Value
content-type application/json

  • URL Params N/A

  • Data Params

    • request:
Name Type Mandatory Default Description
index String yes Index name
id String no Identifier of the document in the index. id null, autogenerated
content_type String no Content type (MIMETYPE)
hash String yes IPFS Hash of the content
index_fields Key/Value[] no Key/value map presenting IPFS content metadata 
{
  "index": "documents", 
  "id": "hello_doc",
  "content_type": "application/pdf",
  "hash": "QmWPCRv8jBfr9sDjKuB5sxpVzXhMycZzwqxifrZZdQ6K9o",
  "index_fields": [
    {
      "name": "title",
      "value": "Hello Doc"
    }, 
    {
      "name": "author",
      "value": "Gregoire Jeanmart"
    }, 
    {
      "name": "votes",
      "value": 10
    }, 
    {
      "name": "date_created",
      "value": 1518700549
    }
  ]
}
  • Sample Request:
curl -X POST \
    'http://localhost:8040/ipfs-store/raw/index' \
    -H 'content-type: application/json' \  
    -d '{"index":"documents","id":"hello_doc","content_type":"application/pdf","hash":"QmWPCRv8jBfr9sDjKuB5sxpVzXhMycZzwqxifrZZdQ6K9o","index_fields":[{"name":"title","value":"Hello Doc"},{"name":"author","value":"Gregoire Jeanmart"},{"name":"votes","value":10},{"name":"date_created","value":1518700549}]}'

  • Success Response:

    • Code: 200
      Content:
{
    "index": "documents",
    "id": "hello_doc",
    "hash": "QmWPCRv8jBfr9sDjKuB5sxpVzXhMycZzwqxifrZZdQ6K9o"
}

[Persistent / Raw] Store & Index content

Store content in IPFS and index it into the search engine

  • URL /ipfs-store/raw/store_index

  • Method: POST

  • Header: N/A

  • URL Params N/A

  • Data Params

    • file: [content]
    • request:
Name Type Mandatory Default Description
index String yes Index name
id String no Identifier of the document in the index. id null, autogenerated
content_type String no Content type (MIMETYPE)
index_fields Key/Value[] no Key/value map presenting IPFS content metadata 
{
  "index": "documents", 
  "id": "hello_doc",
  "content_type": "application/pdf",
  "index_fields": [
    {
      "name": "title",
      "value": "Hello Doc"
    }, 
    {
      "name": "author",
      "value": "Gregoire Jeanmart"
    }, 
    {
      "name": "votes",
      "value": 10
    }, 
    {
      "name": "date_created",
      "value": 1518700549
    }
  ]
}
  • Sample Request:
curl -X POST \
  http://localhost:8040/ipfs-store/raw/store_index \
  -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
  -F file=@/home/gjeanmart/hello.pdf \
  -F 'request={"index":"documents","id":"hello_doc","content_type":"application/pdf","index_fields":[{"name":"title","value":"Hello Doc"},{"name":"author","value":"Gregoire Jeanmart"},{"name":"votes","value":10},{"name":"date_created","value":1518700549}]}'

  • Success Response:

    • Code: 200
      Content:
{
    "index": "documents",
    "id": "hello_doc",
    "hash": "QmWPCRv8jBfr9sDjKuB5sxpVzXhMycZzwqxifrZZdQ6K9o"
}

[Persistent / JSON] Store content

Store a JSON document in IPFS

  • URL: /ipfs-store/json/store
  • Method: POST
  • Header:
Key Value
content-type application/json

  • URL Params: N/A

  • Data Params:

    • payload: {json}
Name Type Mandatory Default Description
payload JSON yes JSON Document to store
  • Sample Request:
$ curl -X POST \
    'http://localhost:8040/ipfs-store/json/store' \
    -H 'Content-Type: application/json' \
    -d '{
      "field1": "val1",
      "field2": 10,
      "field3": {
          "test": true
      }
    }'
  • Success Response:
    • Code: 200
      Content:
{
    "hash": "QmdUNaxwiGT7fzdt6gVpMDFAmjf7dDxMwux16o4s1HyCnD"
}

[Persistent / JSON] Index content

Index IPFS JSON document into the search engine

  • URL /ipfs-store/json/index
  • Method: POST
  • Header:
Key Value
content-type application/json

  • URL Params N/A

  • Data Params

    • request:
Name Type Mandatory Default Description
index String yes Index name
id String no Identifier of the document in the index. id null, autogenerated
content_type String no Content type (MIMETYPE)
hash String yes IPFS Hash of the content
index_fields Key/Value[] no Key/value map presenting IPFS content metadata 
{
  "index": "json_documents", 
  "id": "json_doc",
  "content_type": "application/json",
  "hash": "QmdUNaxwiGT7fzdt6gVpMDFAmjf7dDxMwux16o4s1HyCnD",
  "index_fields": [
    {
      "name": "field1",
      "value": "val1"
    }, 
    {
      "name": "external_field",
      "value": 10
    }, 
    {
      "name": "date_created",
      "value": 1518700549
    }
  ]
}
  • Sample Request:
curl -X POST \
    'http://localhost:8040/ipfs-store/json/index' \
    -H 'content-type: application/json' \  
    -d '{"index":"json_documents","id":"json_doc","content_type":"application/json","hash":"QmdUNaxwiGT7fzdt6gVpMDFAmjf7dDxMwux16o4s1HyCnD","index_fields":[{"name":"field1","value":"val1"},{"name":"external_field","value":10},{"name":"date_created","value":1518700549}]}'

  • Success Response:

    • Code: 200
      Content:
{
    "index": "json_documents",
    "id": "json_doc",
    "hash": "QmdUNaxwiGT7fzdt6gVpMDFAmjf7dDxMwux16o4s1HyCnD"
}

[Persistent / JSON] Store & Index content

Store a JSON document in IPFS and index it into the search engine

  • URL /ipfs-store/json//store_index
  • Method: POST
  • Header:
Key Value
content-type application/json

  • URL Params N/A

  • Data Params

    • payload: {json}
Name Type Mandatory Default Description
payload JSON yes JSON Document to store 
-   `request: `
Name Type Mandatory Default Description
index String yes Index name
id String no Identifier of the document in the index. id null, autogenerated
content_type String no Content type (MIMETYPE)
index_fields Key/Value[] no Key/value map presenting IPFS content metadata 
{
  "payload": {
    "field1": "val1",
    "field2": 10,
    "field3": {
      "test": true
    }
  },
  "index": "json_documents",
  "id": "doc",
  "content_type": "application/json",
  "index_fields": [
      {
        "name": "type",
        "value": "json"
      },
      {
        "name": "title",
        "value": "json_sample"
      }
  ]
}
  • Sample Request:
curl -X POST \
  http://localhost:8040/ipfs-store/jsonstore_index \
  -H 'content-type: application/json' \
  -d '{
      "payload": {
        "field1": "val1",
        "field2": 10,
        "field3": {
          "test": true
        }
      },
      "index": "json_documents",
      "id": "doc",
      "content_type": "application/json",
      "index_fields": [
          {
            "name": "type",
            "value": "json"
          },
          {
            "name": "title",
            "value": "json_sample"
          }
      ]
  }'

  • Success Response:

    • Code: 200
      Content:
{
    "index": "json_documents",
    "id": "doc",
    "hash": "QmWPCRv8jBfr9sDjKuB5sxpVzXhMycZzwqxifrZZdQ6K9o"
}

[Query] Get content

Get content on IPFS by hash

  • URL http://localhost:8040/ipfs-store/query/fetch/{hash}

  • Method: GET

  • Header: N/A

  • URL Params N/A

  • Sample Request:

$ curl \
    'http://localhost:8040/ipfs-store/query/fetch/QmWPCRv8jBfr9sDjKuB5sxpVzXhMycZzwqxifrZZdQ6K9o' \
    -o hello_doc.pdf

  • Success Response:

    • Code: 200
      Content: (file)

[Query] Search contents

Search content accross an index using a dedicated query language

  • URL http://localhost:8040/ipfs-store/query/search
  • Method: GET or POST
  • Header:
Key Value
content-type application/json
  • URL Params
Name Type Mandatory Default Description
index String no Index to search (if null: all indices)
pageNo Int no 0 Page Number
pageSize Int no 20 Page Size / Limit
sort String no Sorting attribute
dir ASC/DESC no ASC Sorting direction
query String no Query URL encoded (for GET call)
  • Data Params

The search operation allows to run a multi-criteria search against an index. The body combines a list of filters :

Name Type Description
name String Index field to perform the search
names String[] Index fields to perform the search
operation See below Operation to run against the index field
value Any Value to compare with
Operation Description
full_text Full text search
equals Equals
not_equals Not equals
contains Contains the word/phrase
in in the following list
gt Greater than
gte Greater than or Equals
lt Less than
lte Less than or Equals 
{
  "query": [
    {
      "name": "title",
      "operation": "contains",
      "value": "Hello"
    },
    {
      "names": ["author", "title"],
      "operation": "full_text",
      "value": "Gregoire"
    },
    {
      "name": "votes",
      "operation": "lt",
      "value": "5"
    }
  ]
}
  • Sample Request:
curl -X POST \
    'http://localhost:8040/ipfs-store/query/search?index=documents' \
    -H 'content-type: application/json' \  
    -d '{"query":[{"name":"title","operation":"contains","value":"Hello"},{"name":"author","operation":"equals","value":"Gregoire Jeanmart"},{"name":"votes","operation":"lt","value":"5"}]}'

curl -X GET \
  'http://localhost:8040/ipfs-store/query/search?index=documents&page=1&size=2&query=%7B%22query%22%3A%5B%7B%22name%22%3A%22votes%22%2C%22operation%22%3A%22lt%22%2C%22value%22%3A%225%22%7D%5D%7D' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \

  • Success Response:

    • Code: 200
      Content:
{
  "content": [
    {
      "index": "documents",
      "id": "hello_doc",
      "hash": "QmWPCRv8jBfr9sDjKuB5sxpVzXhMycZzwqxifrZZdQ6K9o",
      "content_type": "application/pdf",
      "index_fields": [
        {
          "name": "__content_type",
          "value": "application/pdf"
        },
        {
          "name": "__hash",
          "value": "QmWPCRv8jBfr9sDjKuB5sxpVzXhMycZzwqxifrZZdQ6K9o"
        },
        {
          "name": "title",
          "value": "Hello Doc"
        },
        {
          "name": "author",
          "value": "Gregoire Jeanmart"
        },
        {
          "name": "votes",
          "value": 10
        },
        {
          "name": "date_created",
          "value": 1518700549
        }
      ]
    }
  ]
}
],
"sort": null,
"firstPage": false,
"totalElements": 4,
"lastPage": true,
"totalPages": 1,
"numberOfElements": 4,
"size": 20,
"number": 1
}

[Configuration] Create an index

Create an index in ElasticSearch

  • URL: /ipfs-store/config/index/{index}
  • Method: POST
  • Header:
Key Value
content-type application/json

  • URL Params: N/A

  • Data Params: N/A

  • Sample Request:

$ curl -X POST \
    'http://localhost:8040/ipfs-store/config/index/MyIndex' \
    -H 'content-type: application/json'
  • Success Response:
    • Code: 200
      Content: N/A

Advanced Configuration

WIKI: Configuration

Clients

WIKI: Clients

Examples

WIKI: Examples

ipfs-store's People

Contributors

enshore avatar

Watchers

James Cloos avatar iasinDev avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.