Giter Club home page Giter Club logo

hyperdrive-service's Introduction

@hyperspace/hyperdrive

A companion service for Hyperspace that provides FUSE/CLI access to Hyperdrives.

This service creates a "root drive" for you, which is a single, private Hyperdrive that's mounted at ~/Hyperdrive. You can think of this root drive as a replacement for your normal home directory (where you might have Documents, Videos, and Pictures folders, for example).

The CLI gives you commands for interacting with Hyperdrives, both inside and outside of FUSE. Here are some of the included commands:

  • The mount and unmount commands allow you to attach/detach other other Hyperdrives to/from your root drive.
  • The seed and unseed commands let you decide whether you'd like to advertise a drive on the Hyperswarm DHT.
  • The import and export commands are for users who don't want to use FUSE, but still want to move data in and out of Hyperdrives.

For Hyperdrive Daemon Users

This module replaces the functionality of the hyperdrive-daemon. If you've previously used that daemon, running Hyperspace for the first time will migrate your content in ~/.hyperdrive to be compatible with this service.

Also, @hyperspace/hyperdrive will only run in the foreground -- we switched to this model to keep things easy to use and debug. If you'd like to run it persistently (such that this service and Hyperspace auto-start on reboot), you can set this up using your system's process manager (like systemd).

Installation

Before installing the Hyperdrive service, you'll want to first install hyperspace. Once Hyperspace is installed, run:

npm i @hyperspace/hyperdrive -g

After the NPM installation, you should have access to the hyperdrive CLI tool.

Getting Started

In another terminal, spin up Hyperspace by running hyperspace (no other flags needed).

If you're planning on using FUSE, you have to perform a one-time setup step to do the FUSE installation. This will prompt you for sudo access:

$ hyperdrive fuse-setup

Once FUSE has been configured, you're ready to start the Hyperdrive service:

$ hyperdrive start

API

@hyperspace/hyperdrive exposes an API for programmatically interacting with your root drive. To create a client:

const HyperdriveServiceClient = require('@hyperspace/hyperdrive/client')
const client = new HyperdriveServiceClient()

const client = new HyperdriveServiceClient(opts = {})

Create a new client for interacting with the Hyperdrive service.

If you don't provide any options, options will be loaded from a configuration file inside of the ~/.hyperspace directory and a Hyperspace client will be created automatically.

Options include:

{
  mnt: string, // The FUSE mountpoint
  key: Buffer, // Your root drive key.
  client: HyperspaceClient // A hyperspace client.
}

await client.mount(path, opts = {})

Mounts a Hyperdrive inside of your root drive. path must be contained within your root drive's mountpoint (typically ~/Hyperdrive).

Options include all options to Hyperdrive's mount method, such as:

{
  key: Buffer, // The key of the drive you're mounting.
  version: number, // The drive version (if you're creating a static mount)
}

await client.unmount(path)

Unmount the drive mounted at path. path must be contained within your root drive's mountpoint (typically ~/Hyperdrive).

await client.seed(path, opts = {})

Start announcing a mounted drive on the Hyperswarm DHT.

Options include:

{
  remember: boolean, // true if this network configuration should be persisted across restarts.
}

await client.unseed(path, opts = {})

Stop announcing a mounted drive on the Hyperswarm DHT.

Options include:

{
  remember: boolean, // true if this network configuration should be persisted across restarts.
}

await client.info(path)

Returns info about the drive mounted at path.

The info takes the form:

{
  key: Buffer, The drive's key
  discoveryKey: Buffer, // The drive's discovery key.
  writable: boolean, // true if the drive is writable.
  mountPath: string, // The path of the enclosing mountpoint.
  announce: boolean, // true if the drive is currently being announced on the DHT.
  lookup: boolean    // true if the drive is currently being looked up on the DHT.
}

await client.stats(path)

Return drive storage/networking statistics.

The stats take the form:

{
   storage, // Storage info about each mountpoint in the drive.
   network, // Networking info about each mountpoint in the drive.
}

where storage has the same structure as that returned by Hyperdrive.stats and network has the form:

{
  '/': {
    metadata: {
      key,
      discoveryKey,
      length: number, // The Hypercore's length,
      byteLength: number // The Hypercore's byteLength
      peers: [Peer] // An Array of Hypercore Peer objects.
    },
    content: {
      // Same as above
      ...
     }
  },
  '/mountpoint1': {
     // Same as above
     ...
  }
}

const { progress, drive } = client.import(key, dir, opts = {})

Imports a Hyperdrive into Hyperspace.

If you're using FUSE, you probably don't need to explictly import/export, because you can replicate this functionality using the filesystem alone.

progress is an instance of mirror-folder. drive is the Hyperdrive that you're importing into.

Options include:

{
  watch: false // Watch for changes.
}

Note: This imported drive will not appear inside your root drive unless you explicitly mount it with hyperdrive mount (mount path) (imported drive key)

const { progress, drive } = client.export(key, dir, opts = {})

Exports a Hyperdrive into a local directory.

If you're using FUSE, you probably don't need to explictly import/export, because you can replicate this functionality using the filesystem alone.

Options include:

  watch: false // Watch for changes.

CLI Commands

The hyperdrive CLI tool includes a handful of subcommands that wrap the API methods described above. Running hyperdrive help will give more complete usage info:

$ ./bin/run/run help
A Hyperspace service that for managing Hyperdrives over FUSE.

VERSION
  @hyperspace/hyperdrive/1.0.0 linux-x64 node-v12.9.1

USAGE
  $ hyperdrive [COMMAND]

COMMANDS
  autocomplete   display autocomplete installation instructions
  create         Create a new drive mounted at the specified path
  export         Export a Hyperdrive into a directory.
  force-unmount  Forcibly unmount the root filesystem (useful if it was not cleanly unmounted).
  fuse-setup     Perform a one-time configuration step for FUSE.
  help           display help for hyperdrive
  import         Import a directory into a Hyperdrive.
  info           Display information about the drive mounted at the given mountpoint.
  mount          Mount a drive at the specified mountpoint.
  seed           Seed a Hyperdrive on the network.
  start          Start the Hyperdrive service.
  stats          Get the networking stats for the drive mounted at a path.
  unmount        Unmount a drive. The root drive will be unmounted if a mountpoint is not specified.
  unseed         Stop seeding a Hyperdrive on the network.

License

MIT

hyperdrive-service's People

Contributors

andrewosh avatar bendik avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

hyperdrive-service's Issues

Error running updating drive with `hyperdrive import`

Just trying to update a drive with some updated a hyperdrive that was created with the original hyperdrive daemon, and getting the following error:

drive name shortened, as it's a development drive

Importing /Users/Paul/Projects/FederatedWiki/development/client/client into ad8cb16...ca (Ctrl+c to exit)...

Importing | ======================================== | 100% | 25/25 Files(node:9183) UnhandledPromiseRejectionWarning: PathAlreadyExists: Path /plugins already exists.
    at /Users/Paul/.nvm/versions/node/v14.15.1/lib/node_modules/@hyperspace/hyperdrive/node_modules/hyperdrive/index.js:636:45
    at Get._callback (/Users/Paul/.nvm/versions/node/v14.15.1/lib/node_modules/@hyperspace/hyperdrive/node_modules/mountable-hypertrie/index.js:308:16)
    at Get._update (/Users/Paul/.nvm/versions/node/v14.15.1/lib/node_modules/@hyperspace/hyperdrive/node_modules/hypertrie/lib/get.js:75:8)
    at onnode (/Users/Paul/.nvm/versions/node/v14.15.1/lib/node_modules/@hyperspace/hyperdrive/node_modules/hypertrie/lib/get.js:113:10)
    at onnode (/Users/Paul/.nvm/versions/node/v14.15.1/lib/node_modules/@hyperspace/hyperdrive/node_modules/hypertrie/index.js:334:5)
    at processTicksAndRejections (internal/process/task_queues.js:80:21)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:9183) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node proc)
(node:9183) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Importing | ======================================== | 100% | 55/55 Files

The /plugins directory was part of the original import when the drive was created, and the contents have not changes since the last update back in August. And the drive contains a lot more than 55 files!

Running with --trace-warnings gives a little extra information, if that helps.

Importing /Users/Paul/Projects/FederatedWiki/development/client/client into ad8cb16...ca (Ctrl+c to exit)...

Importing | ======================================== | 100% | 21/21 Files(node:9162) UnhandledPromiseRejectionWarning: PathAlreadyExists: Path /plugins already exists.
    at /Users/Paul/.nvm/versions/node/v14.15.1/lib/node_modules/@hyperspace/hyperdrive/node_modules/hyperdrive/index.js:636:45
    at Get._callback (/Users/Paul/.nvm/versions/node/v14.15.1/lib/node_modules/@hyperspace/hyperdrive/node_modules/mountable-hypertrie/index.js:308:16)
    at Get._update (/Users/Paul/.nvm/versions/node/v14.15.1/lib/node_modules/@hyperspace/hyperdrive/node_modules/hypertrie/lib/get.js:75:8)
    at onnode (/Users/Paul/.nvm/versions/node/v14.15.1/lib/node_modules/@hyperspace/hyperdrive/node_modules/hypertrie/lib/get.js:113:10)
    at onnode (/Users/Paul/.nvm/versions/node/v14.15.1/lib/node_modules/@hyperspace/hyperdrive/node_modules/hypertrie/index.js:334:5)
    at processTicksAndRejections (internal/process/task_queues.js:80:21)
    at emitUnhandledRejectionWarning (internal/process/promises.js:168:15)
    at processPromiseRejections (internal/process/promises.js:247:11)
    at processTicksAndRejections (internal/process/task_queues.js:94:32)
(node:9162) PathAlreadyExists: Path /plugins already exists.
    at /Users/Paul/.nvm/versions/node/v14.15.1/lib/node_modules/@hyperspace/hyperdrive/node_modules/hyperdrive/index.js:636:45
    at Get._callback (/Users/Paul/.nvm/versions/node/v14.15.1/lib/node_modules/@hyperspace/hyperdrive/node_modules/mountable-hypertrie/index.js:308:16)
    at Get._update (/Users/Paul/.nvm/versions/node/v14.15.1/lib/node_modules/@hyperspace/hyperdrive/node_modules/hypertrie/lib/get.js:75:8)
    at onnode (/Users/Paul/.nvm/versions/node/v14.15.1/lib/node_modules/@hyperspace/hyperdrive/node_modules/hypertrie/lib/get.js:113:10)
    at onnode (/Users/Paul/.nvm/versions/node/v14.15.1/lib/node_modules/@hyperspace/hyperdrive/node_modules/hypertrie/index.js:334:5)
    at processTicksAndRejections (internal/process/task_queues.js:80:21)
(node:9162) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
    at emitDeprecationWarning (internal/process/promises.js:180:11)
    at processPromiseRejections (internal/process/promises.js:249:13)
    at processTicksAndRejections (internal/process/task_queues.js:94:32)
Importing | ======================================== | 100% | 55/55 Files

import subcommand requires a key

A key is required as an argument without explanation. An explanation would be helpful about how to create one, or is it a bug as the original hyperdrive did not require one?

hyperdrive start fails

osxfuse installed via brew and restart done once security extension was approved

> npm i @hyperspace/hyperdrive -g

> hyperdrive fuse-setup
Configuring FUSE...
Password:
FUSE successfully configured:
  * Your root drive will be mounted at ~/Hyperdrive when the daemon is next started.
  * If your mountpoint ever becomes unresponsive, try running `hyperdrive force-unmount`.

> hyperdrive start
Could not start the Hyperdrive service. Is Hyperspace running?
Error: Error: RPC stream destroyed
    at RPC._destroy (/usr/local/lib/node_modules/@hyperspace/hyperdrive/node_modules/arpeecee/index.js:187:27)
    at WritableState.updateNonPrimary (/usr/local/lib/node_modules/@hyperspace/hyperdrive/node_modules/streamx/index.js:187:16)
    at WritableState.update (/usr/local/lib/node_modules/@hyperspace/hyperdrive/node_modules/streamx/index.js:172:70)
    at updateWriteNT (/usr/local/lib/node_modules/@hyperspace/hyperdrive/node_modules/streamx/index.js:479:6)
    at processTicksAndRejections (node:internal/process/task_queues:79:21)

> hyperspace
fish: Unknown command: hyperspace

Tag management

When mounting a drive, it is possible to give the version like; --version=123
But howto manage the versions ?
I'm correct that the 'tag' command is missing ?
So we can list/create/delete tags on a hyperdrive and thus can mount all tagged versions.

hyperdrive fuse-setup fails on macOS Catalina due to permission error

Running hyperdrive fuse-setup on macOS Catalina fails on the first go. In the terminal window, you see the error message Could not configure the FUSE module: Error 1: … @hyperspace/hyperdrive/bin/commands/fuse-setup.js:66:37….

Separately, a macOS alert tells you that system extensions by Benjamin Fleischer have been blocked.

To reproduce

  1. On macOS Catalina, run hyperdrive fuse-setup

What should happen

FUSE setup should succeed.

What actually happens

FUSE setup fails due to a permission error installing system extensions.

Workaround

Manually go into System Preferences and grant permission for Bejamin Fleischer to install system software on your computer.

Suggestions/discussion

  • Document this issue under known issues.
  • Can this extension be signed so the error does not occur?
  • Looking into OSXFuse, it appears that the open source project is no longer maintained and the license is no longer open source (osxfuse/osxfuse#616)

Initial mount sync is very slow (~20 minutes)

See update


I have hyperspace and @hyperspace/hyperdrive running on two computers (running Pop!_OS 20.04 and macOS). When I mount drives created on the other, neither computer syncs the contents.

To reproduce

On both Computer A and Computer B:

  1. In one terminal instance: hyperspace
  2. In another terminal instance: hyperdrive fuse-setup (only at first install) followed by hyperdrive start

On Computer A:

  1. From the ~/Hyperdrive folder/drive: hyperdrive create computer-a
  2. Use hyperdrive info in the computer-a drive to get its key (<key-a>)
  3. echo 'Hello' > ~/Hyperdrive/computer-a/hello.txt

On Computer B:

  1. From the ~/Hyperdrive folder/drive: hyperdrive mount computer-a <key-a>

(You can also reverse the instructions and create a drive on Computer B and mount it on Computer A, the outcomes are the same.)

What should happen

The hello.txt file should sync to Computer B.

What actually happens

Nothing.

Notes

  • behaviour does not change if I mount the computer-a drive in Computer B using the --seed flag.
  • If there is some way to get you more useful diagnostic information, please let me know.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.