Issue: REXray not taking config.yaml file when it’s placed in /etc/rexray/config.yaml or ~/.rexray/config.yaml
Version: 0.2.0-rc5+23

my aws keys are not in sudo rexray service module types

i tried changing to .yml and still no go.

here is my config.yaml file:

--- 
AwsAccessKey: AblahblahblahQ
AwsSecretKey: 7blahblahblahS
StorageDrivers: 
  - ec2

If updating the dl.bintray.com/emccode/rexray/install file is separate from the standard CI process, then having a single line comment in the top of the file would be nice to track how to do the update at a later point.

TBD

It is important to be able to import the RR package in the future. This may cause friction compared to the single instance daemon mode that has been introduced in 0.2. To allow this the following can be done today. Let's make sure that we either set a permanent abstraction between CLI accessing RR and packages importing RR and provide similar functionality re consuming configuration files.

import rexray "github.com/emccode/rexray/rexray/commands"

rexray.UpdateLogLevel()
rexray.InitDriverManagers()
sdm := rexray.GetSdm()

I am building REX-Ray deployment into Docker Machine. In this case I will be doing it across multiple types of OSs.

This is going at a fast pace, so as of today, what's the best route to install REXray? from Bintray? from github releases? Is it going to be a tar.gz file or will it be the binary?

What is the long term approach going to be?

Would it be best to have a "latest" tag so the latest version is always pulled?

root@ec2-54-219-167-85:/home/ubuntu# curl -sSL https://raw.githubusercontent.com/emccode/rexray/master/install.sh | sh -
sh: 14: [: x86_64: unexpected operator

I can resolve the problem using #!/bin/bash instead of sh. Doing more testing to figure out what's going on..

We just ran into a situation where when starting rexray with sudo rexray start remotely through SSH, docker machine ..blah.. sudo rexray start it would hang indefinitely.

In troubleshooting I tried running something similar locally, sudo rexray start | sudo tee /tmp/temp. The tee is interpreting the stdin/stdout's and waiting for them to close apparently so acting similar to the remote SSH. If you run that command it hangs.

With all the changes, the systemd configuration is broken, and uses the -daemon argument, which no longer exists.

It seems to be fixed with ExecStart=/opt/rexray/rexray start --foreground=true though.

@clintonskitson suggested to start rexray by doing sudo var1=val var2=val rexray service start to populate the configuration. The configuration is populated, but rexray cannot provision a new volume.

to reproduce the issue:

sudo rexray service install
sudo /etc/init.d/rexray start
sudo rexray service stop
sudo AWS_ACCESS_KEY=mykey AWS_SECRET_KEY=mysecretkey rexray service start

ubuntu@fun21:~$ rexray volume create --size=20 --volumename="wefw"
FATA[0000] AWS was not able to validate the provided access credentials (AuthFailure)
ubuntu@fun21:~$ rexray service module types
[
  {
    "id": 1,
    "name": "AdminModule",
    "defaultConfigs": [
      {
        "address": "tcp://:7979"
      }
    ]
  },
  {
    "id": 2,
    "name": "DockerRemoteVolumeDriverModule",
    "defaultConfigs": [
      {
        "address": "unix:///run/docker/plugins/rexray-remote.sock",
        "config": {
          "LogLevel": "info",
          "StorageDrivers": [],
          "VolumeDrivers": [
            "docker"
          ],
          "OsDrivers": [
            "linux"
          ],
          "MinVolSize": 0,
          "RemoteManagement": false,
          "DockerVolumeType": "",
          "DockerIops": 0,
          "DockerSize": 0,
          "DockerAvailabilityZone": "",
          "AwsAccessKey": "mykey",
          "AwsSecretKey": "mysecretkey",
          "AwsRegion": "",
          "RackspaceAuthUrl": "",
          "RackspaceUserId": "",
          "RackspaceUserName": "",
          "RackspacePassword": "",
          "RackspaceTenantId": "",
          "RackspaceTenantName": "",
          "RackspaceDomainId": "",
          "RackspaceDomainName": "",
          "ScaleIoEndpoint": "",
          "ScaleIoInsecure": false,
          "ScaleIoUseCerts": true,
          "ScaleIoUserName": "",
          "ScaleIoPassword": "",
          "ScaleIoSystemId": "",
          "ScaleIoSystemName": "",
          "ScaleIoProtectionDomainId": "",
          "ScaleIoProtectionDomainName": "",
          "ScaleIoStoragePoolId": "",
          "ScaleIoStoragePoolName": "",
          "XtremIoEndpoint": "",
          "XtremIoUserName": "",
          "XtremIoPassword": "",
          "XtremIoInsecure": false,
          "XtremIoDeviceMapper": "",
          "XtremIoMultipath": "",
          "XtremIoRemoteManagement": false
        }
      }
    ]
  },
  {
    "id": 3,
    "name": "DockerVolumeDriverModule",
    "defaultConfigs": [
      {
        "address": "unix:///run/docker/plugins/rexray-local.sock",
        "config": {
          "LogLevel": "info",
          "StorageDrivers": [],
          "VolumeDrivers": [
            "docker"
          ],
          "OsDrivers": [
            "linux"
          ],
          "MinVolSize": 0,
          "RemoteManagement": false,
          "DockerVolumeType": "",
          "DockerIops": 0,
          "DockerSize": 0,
          "DockerAvailabilityZone": "",
          "AwsAccessKey": "mykey",
          "AwsSecretKey": "mysecretkey",
          "AwsRegion": "",
          "RackspaceAuthUrl": "",
          "RackspaceUserId": "",
          "RackspaceUserName": "",
          "RackspacePassword": "",
          "RackspaceTenantId": "",
          "RackspaceTenantName": "",
          "RackspaceDomainId": "",
          "RackspaceDomainName": "",
          "ScaleIoEndpoint": "",
          "ScaleIoInsecure": false,
          "ScaleIoUseCerts": true,
          "ScaleIoUserName": "",
          "ScaleIoPassword": "",
          "ScaleIoSystemId": "",
          "ScaleIoSystemName": "",
          "ScaleIoProtectionDomainId": "",
          "ScaleIoProtectionDomainName": "",
          "ScaleIoStoragePoolId": "",
          "ScaleIoStoragePoolName": "",
          "XtremIoEndpoint": "",
          "XtremIoUserName": "",
          "XtremIoPassword": "",
          "XtremIoInsecure": false,
          "XtremIoDeviceMapper": "",
          "XtremIoMultipath": "",
          "XtremIoRemoteManagement": false
        }
      }
    ]
  }
]

See https://github.com/docker/docker/blob/master/docs/extend/plugin_api.md where the SPEC file has been updated for JSON.

It would be nice to allow for supplying multiple module configurations per file. This would allow for supplying the configuration for deploying combinations of vdm's and storage platforms that can be started using existing methods.

Being able to allow the usage of a storage profile as an option when consuming storage would be useful.

docker create --driver=rexray --opt=profile=fastbytes --size=16

This would leverage profile based consumption logic to decide which of the underlying platforms to consume from.

Currently the failover capability in REX-Ray is passed through to the underlying storage platform. This means that depending on the platform, Volumes can be locked to that platform until there is manual intervention to allow for a Volume to be unmounted and unmapped from the host.

There are exceptions to this however. As an example, the EC2 driver is aware of hosts being powered on or off, and allows Mount commands to progress if the EC2 instance is powered off. This means that pre-emption is built into the platform and enabled natively through the driver. But this is only in the case of an instance being powered off and not among any failures at higher application layers.

In the case of ScaleIO, there is a flag AllowMultipleMappings which can be flipped in the ScaleIO driver which would negate any need for pre-emption and could be advertised as an optional parameter. This of course comes with risk when volumes are managed appropriately that multiple hosts can be exposed to the same non-clustered FS.

Otherwise, proper pre-emption is a necessary part of being safe which involves being able to identify the proper situation for unmapping and mapping a volume.

following along from https://github.com/docker/compose/releases/tag/1.4.0

curl -L https://github.com/docker/compose/releases/download/1.4.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose

we should have rexray be similar.

curl -L https://github.com/emccode/rexray/releases/download/0.2.0-rc1/rexray-`uname -s`-`uname -m`.tar.gz > /tmp/rexray-`uname -s`-`uname -m`.tar.gz

this means we should get rid of the v from the leading address and follow standard uname returns.

When RR starts, it would be nice to add the version information in the log file.

The ScaleIO platform only allows names up to 31 characters through the platform API. This means anonymous volumes that get created based on GUIDs don't work properly, ie -v /path.

Would be great to specify an unmount of "all" volumes.

There is a parameter in a Dockerfile where it specifies a certain amount of external volumes be used. Currently in Docker 1.7+, you can specify extra volumes using the runtime flag of -v, but this does not override volumes that are specified in the Dockerfile. Once you pass the --volume-driver flag, all volumes that are associated are passed to the plugin. In the following case there is one volume that is specified attached to /var/lib/mysql.

docker run -ti --name mysql7 -p 3306:3306 -e MYSQL_PASS="87654321" --volume-driver=rexray -v wpmsql01c:/var/lib/mysql tutum/mysql

If you look at the Dockerfile you will see that the volume statement includes two volumes as VOLUME ["/etc/mysql", "/var/lib/mysql"]. This causes Docker to try and create a new volume for /etc/mysql since this volume based on mount point is not specified at runtime. A new GUID is created and a new volume is created and attach to this mount point.

The error as Error response from daemon: Plugin Error: {"Error":"problem getting response: The given name exceeds the allowed length of 31 characters."} is coming up since the ScaleIO driver currently only supports up to 31 characters in the name of a volume.

FIX: Ensure that you specify the same number of volumes or more than the number specified in the Dockerfile under volumes. The mount point path in the Dockerfile must correlate to the mount point path under the -v flags.

Daemon drivers allow you to specify different REST endpoints for different use cases. Since this does not perform introspection to detect the appropriate driver to be used, you must specify them in REXRAY_DAEMONDRIVERS.

here are my steps to recreate

wget

Download the file using wget, tar, move, +x, install

$ wget https://bintray.com/artifact/download/akutz/generic/rexray-linux_amd64-0.2.0-rc2%2B18.tar.gz
$ tar xzfv rexray-linux_amd64-0.2.0-rc2+18.tar.gz
$ sudo mv rexray /bin/rexray
$ sudo chmod +x /bin/rexray
$ sudo rexray service install
update-rc.d: /etc/init.d/rexray: file does not exist
FATA[0000] installation error exit status 1

curl

Download the file using wget, tar, move, +x, install

$ curl -LOk https://bintray.com/artifact/download/akutz/generic/rexray-linux_amd64-0.2.0-rc2%2B18.tar.gz
$ tar xzfv rexray-linux_amd64-0.2.0-rc2+18.tar.gz
$ sudo mv rexray /bin/rexray
$ sudo chmod +x /bin/rexray
$ sudo rexray service install
ERRO[0000] error reading configuration file(s) from default path(s)  error=open : no such file or directory global=/etc/rexray/config.yaml user=/root/.rexray/config.yaml
ERRO[0000] error reading configuration file(s) from default path(s)  error=open : no such file or directory global=/etc/rexray/config.yaml user=/root/.rexray/config.yaml
ERRO[0000] error reading configuration file(s) from default path(s)  error=open : no such file or directory global=/etc/rexray/config.yaml user=/root/.rexray/config.yaml
panic: open /root/.rexray/usage.template: permission denied

goroutine 1 [running]:
github.com/emccode/rexray/rexray/commands.initUsageTemplates()
    /home/travis/gopath/src/github.com/emccode/rexray/rexray/commands/usage.go:25 +0x38e
github.com/emccode/rexray/rexray/commands.init.1()
    /home/travis/gopath/src/github.com/emccode/rexray/rexray/commands/rexraycli.go:113 +0x40
github.com/emccode/rexray/rexray/commands.init()
    /home/travis/gopath/src/github.com/emccode/rexray/rexray/commands/usage.go:139 +0x4c9
main.init()
    /home/travis/gopath/src/github.com/emccode/rexray/rexray/rexray.go:28 +0x40

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 6 [runnable]:
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:20
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

Because sudo doesn't always play nice with /etc/environment and so many people place configuration information there even when inappropriate or not fully understood, REX-Ray should manually load /etc/environment in order to relieve users of possible issues due to Linux systems being so disparate and supporting so many different configuration options.

Would be nice to add debug statements into the volume driver manager to allow for visibility of incoming requests.

Honestly, I don't even know if this is feasible, but I thought it would be useful if you could mount an S3 bucket directly as a volume. There already exists the s3-fuse file system: https://github.com/s3fs-fuse/s3fs-fuse

When dependent packages get updated, a build can hit an error unless you manually clear out the vendor directory of existing packages. I propose that we defaultly remove the packages, and have a flag that disables this functionality to ensure a build can happen quickly if desired.

It looks like the RPM and WGET installation is failing due to the % that is present in the links.

https://dl.bintray.com/emccode/rexray/stupid/latest/%rexray-latest-x86_64.rpm

[root@mdm2 centos]# wget https://dl.bintray.com/emccode/rexray/stupid/latest/%rexray-latest-x86_64.rpm
--2015-09-20 04:09:39--  https://dl.bintray.com/emccode/rexray/stupid/latest/%25rexray-latest-x86_64.rpm
Resolving dl.bintray.com (dl.bintray.com)... 108.168.194.92, 108.168.194.91
Connecting to dl.bintray.com (dl.bintray.com)|108.168.194.92|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2015-09-20 04:09:39 ERROR 404: Not Found.

[root@mdm2 centos]# rpm -ivvh https://dl.bintray.com/emccode/rexray/stupid/latest/%rexray-latest-x86_64.rpm
Retrieving https://dl.bintray.com/emccode/rexray/stupid/latest/%rexray-latest-x86_64.rpm
curl: (22) The requested URL returned error: 400 Bad Request
error: skipping https://dl.bintray.com/emccode/rexray/stupid/latest/%rexray-latest-x86_64.rpm - transfer failed

Hi @clintonskitson (et. al),

Currently both the install and install.sh scripts are hosted on Bintray. I'm thinking that the latter should actually only be hosted on GitHub with the former pointing to https://raw.githubusercontent.com/emccode/rexray/master/install.sh instead of https://dl.bintray.com/emccode/rexray/install.sh. The reason for this is so that we can more easily update the actual installer -- it will only take a commit.

If you agree, I'd like to make this change ASAP as it will require replacing the static installer, resetting its download count. I want that to happen well before the release of 0.2.0.

In storage/xtremio/storage.go the package is declared as package scaleio.
Shouldn't it be package xtremio ?

Would it be a good idea to not retrieve volumes that do not have names? For instance "".

Other idea is to have a flag that retrieves everything, including volumes with "" no names.

Based on release notes in RC4, it looks like the log should be at /var/log/rexray.log. I have ran it from rexray start and also /etc/init.d/rexray start and no luck. Am I missing something?

It looks like this parameter is being ignored currently.

[root@mdm2 rexray]# mv /etc/rexray/config.yml /etc/rexray/config.yml-old
[root@mdm2 rexray]# rexray --config=/etc/rexray/config.yml-old volume
FATA[0000] Error: Driver Volume discovery failed: AWS was not able to validate the provided access credentials (AuthFailure) 
[root@mdm2 rexray]# mv /etc/rexray/config.yml-old /etc/rexray/config.yml
[root@mdm2 rexray]# rexray --config=/etc/rexray/config.yml-old volume
- name: ""
  volumeid: vol-5e0144a7
  availabilityzone: us-west-1b
  status: available
  volumetype: standard
  iops: 0
  size: "8"
  networkname: ""
  attachments: []
- name: ""
  volumeid: vol-4d0045b4
  availabilityzone: us-west-1b
  status: available
  volumetype: standard
  iops: 0

/var/lib/docker/volumes/testa16

The volume mount point that is used should be outside of /var/lib/docker/volumes and likely associated to the volume manager. Something like /var/lib/rexray/volumes would suffice.

moby/moby#15880 (comment)

when performing a rexray get-volume in AWS, it gets all EBS volumes. That becomes a problem because a container cannot mount a volume if it's in a different AZ than the host.

It would be good if rexray only discovered volumes within the AZ the host is in

Would it make sense to have a VirtualBox storage driver for REX-Ray?

I started working on a VirtualBox volume driver plugin over at appropriate/docker-volumes-virtualbox that uses the vboxwebsrv SOAP service, but I had to put it on the backburner before getting to the point of having the VM format and mount the volume. Part of this was due to the tediousness of the vboxwebsrv SOAP interface and part of it was simply due to lack of time.

Since REX-Ray is already handling the formatting and mounting part as far as I understand it, I was wondering if it would make more sense for me to repurpose my code to be a REX-Ray storage driver and thereby avoid having to do some of that extra work.

Do REX-Ray storage drivers need to be integrated with this project, or is there a way to build them as a separate binary that lives in its own project?

It would be nice to allow the REX-Ray admin to set configurable limitations per module.

• size of volume created
• type of options or limits on options
• limits on create/remove operations
  and more

Would be nice to directly correlate a commit hash to a binary even with actual releases. Having rexray version return version and commit for all releases would help here.

TBD

Since the new foundational enhancements with an admin API and logging, we now need think about hiding configuration parameters in JSON conversion and logging.

Considering the default behavior is to expose TCP admin session, a simple call like http://ip:7979/r/module/instances is going to expose too much information. I would assume that returning this kind of information upon instantiation may be appropriate, but that returning it after that point should not be possible.

[
  {
    "id": 1,
    "typeId": 1,
    "name": "AdminModule",
    "config": {
      "address": "tcp://:7979"
    },
    "description": "The REX-Ray admin module",
    "started": true
  },
  {
    "id": 2,
    "typeId": 2,
    "name": "DockerRemoteVolumeDriverModule",
    "config": {
      "address": "unix:///run/docker/plugins/rexray-remote.sock",
      "config": {
        "LogLevel": "info",
        "StorageDrivers": [
          "ec2"
        ],
        "VolumeDrivers": [
          "docker"
        ],
        "OsDrivers": [
          "linux"
        ],
        "MinVolSize": 0,
        "RemoteManagement": false,
        "DockerVolumeType": "",
        "DockerIops": 0,
        "DockerSize": 0,
        "DockerAvailabilityZone": "",
        "AwsAccessKey": "A",
        "AwsSecretKey": "jSv",

In order to simplify how someone would figure out how to leverage RR, we should include an example file or include it in the README. @kacole2

There are cases where it is very nice to allow bulk commands by way of the CLI. For example, if someone would like to remove all mapped volumes this should be possible via nested command as such.

rexray volume unmount $(rexray volume map -q)

Currently this would be accomplished with a command like ./rexray volume map | grep "volumeid:" | awk '": " { system("./rexray volume unmount --volumeid="$2); }'.

just a suggestion, so take it how you wish.

I would like to see rexray volume get be rexray volume list. or keep the get for specific volumes but have a list or ls shorthand.

Hi there,

I tried to configure Rex-ray(0.1 and 0.2) on a CoreOS(801) with Docker 1.8.1.
And here is what I get. https://gist.github.com/Arkan/55ed62a965db1b9a4ee3
As you can see, the first time, when the volume doesn't exist, it is created, and then mounted correctly.
The second time, the volume exists, and there is an error with Docker(or REX-ray?)
Do you have any idea on what's going on here?

Thanks

TBD

again, just another suggestion.

remove sounds like "i'm going to remove this from my inventory" or "remove this from this machine". Perhaps delete makes it feel like "are you sure you want to do this?"

rexray still functions in v0.2 just as it used to in v0.1x when ENV variables were used for configuration. I can still declare ENV variables and rexray still creates volumes. However, the ENV variables are not populated in the configuration output with rexray service module types.

It would be nice to add the ability to lock RR releases for dependent packages and tie the glide.yaml file to specific commits, preferably tied to other supported releases.

Running the sh install method is erroring out. It looks like because lsb-release exists, it is trying to do an apt package install. I am not very savvy with why the lsb-release would be something tagged to determine a valid apt install.

core@ip-10-0-1-223 ~ $ curl -sSL https://dl.bintray.com/emccode/rexray/install | sh -
sudo: dpkg: command not found

REX-Ray has been installed to /usr/bin/rexray

sh: line 45: /usr/bin/rexray: No such file or directory

elif [ "$ARCH" = "x86_64" -a \
       -e "/etc/debian-release" -o \
       -e "/etc/debian-version" -o \
       -e "/etc/lsb-release" ]; then

core@ip-10-0-1-223 ~ $ cat /etc/lsb-release 
DISTRIB_ID=CoreOS
DISTRIB_RELEASE=723.3.0
DISTRIB_CODENAME="Red Dog"
DISTRIB_DESCRIPTION="CoreOS 723.3.0"

The daemon should not initialize if there are no storage drivers present to be used. This way when running rexray as a service, it will retry starts until a storage driver can be initialized successfuly.

The download_rexraycli link is "https://github.com/emccode/rexray/releases/download/latest/rexray-Linux-x86_64", while it seems that this url doesn't exist.
I changed it manually to "https://github.com/emccode/rexray/releases/download/v0.1.150418/rexray-Linux-x86_64"

Problem after uninstalling and reinstalling RR under Ubuntu.

root@ec2-54-219-167-85:/home/ubuntu# curl -sSL https://dl.bintray.com/emccode/rexray/install | sh -
(Reading database ... 77285 files and directories currently installed.)
Preparing to unpack rexray-latest-x86_64.deb ...
/var/lib/dpkg/info/rexray.prerm: line 2: /usr/bin/rexray: No such file or directory

Removal of files by rm /var/lib/dpkg/info/rex* looks to fix the problem.