Giter Club home page Giter Club logo

node-smartdc's Introduction

node-smartdc is a node.js client library and set of CLI tools for using with the Joyent SmartDataCenter API, for example the Joyent Compute Service.

This repository is part of the Joyent Triton project. See the contribution guidelines and general documentation at the main Triton project page.

(Note: Current releases and the #master branch of this are for SmartDataCenter (SDC) version 7. It is not 100% backward compatible with SDC 6.5. For 100% compatility with SDC 6.5, you must install a "6.5.x" version of this module.)

Installation

To use the CLI tools (a number of sdc-* commands) you may want to install globally:

npm install -g smartdc

The CLI commands typical work with JSON content. We suggest you also install the json tool for working with JSON on the command line. The examples below use json heavily.

npm install -g json

CLI Setup and Authentication

There are CLI commands corresponding to almost every action available in the SmartDataCenter API; see the Joyent CloudAPI documentation for complete information. Each command takes --url, --account, and --keyId flags to provide the API endpoint URL and your credentials. However you'll probably want to set the environment variable equivalents:

  • SDC_URL (--url | -u): URL of the CloudAPI endpoint. E.g. "https://us-east-1.api.joyent.com".

  • SDC_ACCOUNT (--account | -a): Login name/username. E.g. "bob".

  • SDC_KEY_ID (--keyId | -k): The fingerprint of an SSH public key that has been added to the account set in SDC_ACCOUNT. This is used for signing requests. If you use an SSH agent, the fingerprint is shown in ssh-add -l output. You can calculate the fingerprint like this:

      ssh-keygen -l -f ~/.ssh/id_rsa.pub | awk '{print $2}' | tr -d '\n'
    

    Your matching SSH private key must be beside the ".pub" public key file in your "~/.ssh" dir.

    If your client is connecting to a CloudAPI service that is using a self-signed SSL certificate, you may need to set SDC_TESTING=1 in your client environent. (Otherwise you'll get DEPTH_ZERO_SELF_SIGNED_CERT error).

Authenticating as account user

Starting with version 7.3, Role Based Access Control lets you provide limited access to to your Joyent Cloud account and Manta storage to other members of your organization.

In order to authenticate as a member of a given organization, SDC_ACCOUNT will remain set to the login associated with the organization, and we'll use the SDC_USER environment variable to identify ourselves as a member of such organization. We can also use the --A | --user command line argument with any of the sdc-* commands if we just want to operate as an account user for just that command.

Remember that if the environment variable SDC_USER is set, sdc-* binaries will remain trying to operate as the given user. If you've set this variable and want to switch back to operate as the account owner, you should unset SDC_USER.

The SmartDataCenter Cloud API uses http-signature (IETF draft spec) for authentication. All requests to the API are signed using your RSA private key. The server uses your (previously uploaded) public key to verify the signed request. This avoids ever sending a password.

Once you have set the environment variables, check that it is working by listing available images for provisioning:

$ sdc-listimages
[
  {
    "id": "753ceee6-5372-11e3-8f4e-f79c1154e596",
    "name": "base",
    "version": "13.3.0",
    "os": "smartos",
    "requirements": {},
    "type": "smartmachine",
    "description": "A 32-bit SmartOS image with just essential packages installed. Ideal for users who are comfortable with setting up their own environment and tools.",
    "owner": "9dce1460-0c4c-4417-ab8b-25ca478c5a78",
    "homepage": "http://wiki.joyent.com/jpc2/SmartMachine+Base",
    "published_at": "2013-11-22T12:34:40Z",
    "public": true,
    "state": "active"
  },
...

CLI Usage

There are many many sdc-* commands. Typically one for each endpoint in the API. A common one is for provisioning a new machine (aka VM). Let's provision a new "base" (SmartOS) machine. First find the id of the "base" image (version 13.3.0):

$ IMAGE=$(sdc-listimages | json -c 'this.name=="base" && this.version=="13.3.0"' 0.id)
$ sdc-createmachine --image $IMAGE --name mymachine1
$ sdc-getmachine f8f995da-086f-e8f5-c062-992139432c4f
{
  "id": "f8f995da-086f-e8f5-c062-992139432c4f",
  "name": "mymachine1",
  "type": "smartmachine",
  "state": "provisioning",
  "image": "753ceee6-5372-11e3-8f4e-f79c1154e596",
  ...
}

Then you can poll until the state of the machine goes to "running":

$ sdc-getmachine f8f995da-086f-e8f5-c062-992139432c4f | json state
provisioning
...
$ sdc-getmachine f8f995da-086f-e8f5-c062-992139432c4f | json state
running

At that point, you can ssh into the machine; try this:

$ IP=$(sdc-getmachine f8f995da-086f-e8f5-c062-992139432c4f | json primaryIp)
$ ssh root@$IP
...
   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ; Instance (base 13.3.0)
                   `-'  http://wiki.joyent.com/jpc2/SmartMachine+Base

[root@f8f995da-086f-e8f5-c062-992139432c4f ~]#

Once you've played around and are done, you can delete this machine.

$ sdc-deletemachine f8f995da-086f-e8f5-c062-992139432c4f
...
$ sdc-getmachine f8f995da-086f-e8f5-c062-992139432c4f
Object is Gone (410)

There's a lot more you can do, like manage snapshots, keys, tags, etc. For the Joyent cloud, you can read more at https://docs.joyent.com.

Programmatic Usage

var fs = require('fs');
var smartdc = require('smartdc');

var client = smartdc.createClient({
    sign: smartdc.privateKeySigner({
        key: fs.readFileSync(process.env.HOME + '/.ssh/id_rsa', 'utf8'),
        keyId: process.env.SDC_KEY_ID,
        user: process.env.SDC_ACCOUNT
    }),
    user: process.env.SDC_ACCOUNT,
    url: process.env.SDC_URL
});

client.listMachines(function(err, machines) {
    if (err) {
        console.log('Unable to list machines: ' + err);
        return;
    }

    machines.forEach(function(m) {
        console.log('Machine: ' + JSON.stringify(m, null, 2));
    });
});

Upgrading from 6.5 to 7.0

  • The environment variables changed from 6.5 to 7 (the CLI_ string was dropped):
    • SDC_CLI_ACCOUNT ==> SDC_ACCOUNT
    • SDC_CLI_URL ==> SDC_URL
    • SDC_CLI_KEY_ID ==> SDC_KEY_ID
  • The SDC_CLI_IDENTITY environment variable is no longer used. See above on how to determine your public key fingerprint for SDC_KEY_ID.
  • The sdc-setup command was removed.

Note that in 6.5, SDC_CLI_KEY_ID was the name of the SSH key as specified in your Joyent Cloud account. In 7.0, SDC_KEY_ID is the fingerprint of your SSH public key.

License

MIT. See the "LICENSE" file.

Development

Contributing

A few basic rules and guidelines:

  • Read the Joyent Engineering Guidelines on tickets/issues and commit comments. List GitHub issues and/or Joyent JIRA tickets in commit messages and ensure thought processes are included in the issues or commit messages.

  • You typically want to bump the package.json version for all but trivial changes.

  • Update CHANGES.md (the change log) for any additions.

  • Run and pass make check.

  • Run and pass make test (caveat: I'm not sure it is passing right now.) Be aware that this module is meant to work with older node versions and on a number of platforms (smartos, linux, mac, windows).

Bugs

Please report issues to https://github.com/joyent/node-smartdc/issues.

## Running the test suite

Note that this will run API calls against the SmartDataCenter setup per the SDC_* environment variables. Please, make sure it is okay to try to create new machines using the configured DC and account before running the test suite.

make test

You may want to add a test user to your SDC setup. A sample user, with sample ssh keys can be found at test/user.ldif and test/.ssh. Once you've added this user, you can run your tests using:

SDC_URL=http://127.0.0.1:8080 \
    SDC_ACCOUNT=test \
    SDC_KEY_ID=id_rsa \
    HOME="$(pwd)/test" \
    VERBOSE=1 \
    make test

node-smartdc's People

Contributors

alainodea avatar andyyatskov avatar arekinath avatar bahamat avatar cburroughs avatar cobracmder avatar dan-luk avatar deadalready avatar dwlf avatar fkuo avatar harrisiirak avatar isaacs avatar jclulow avatar johnsonnenschein avatar joshwilsdon avatar kusor avatar licenser avatar mcavage avatar palmerabollo avatar pborenstein avatar ryancnelson avatar sigxcpu76 avatar silentrob avatar tgross avatar trentm avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

node-smartdc's Issues

--image flag always reports "image must be specified" ... usage message doesn't list short flags

sdc-createmachine doesn't seem to be accepting the "--image" switch:

$ env | grep SDC | grep -v CLI
SDC_URL=https://us-west-1.api.joyentcloud.com
SDC_ACCOUNT=xxxxxx
SDC_KEY_ID=fd:9c:78:0f:bd:58:e0:88:38:01:44:3b:ab:80:a5:93

$ sdc-createmachine --image f669428c-a939-11e2-a485-b790efc0f0c1
image must be specified

$ sdc-createmachine -e f669428c-a939-11e2-a485-b790efc0f0c1
<bunch of happy "success" json output here>

$ sdc-createmachine --help
sdc-createmachine [--account string] [--debug boolean] [--dataset string] [--enable-firewall boolean] [--help boolean] [--keyId string] [--metadata ] [--name string] [--networks ] [--package string] [--script path] [--tag ] [--url url]

$ npm list -g | grep smartdc
└─┬ [email protected]
└─┬ [email protected]

Running commands against a valid non-API URL returns error: undefined

Right URL:

$ ./bin/sdc-createfirewallrule --url https://us-west-1.api.joyentcloud.com --verbose --rule 'from any to allow tcp (port 80 and port 443 and port 22 and port 9090)'
sdc-createfirewallrule: error (InvalidParameters): Invalid parameters

Wrong URL:

$ ./bin/sdc-createfirewallrule --url https://us-west-1.joyentcloud.com --verbose --rule 'from any to allow tcp (port
 80 and port 443 and port 22 and port 9090)'
sdc-createfirewallrule: error: undefined

Invalid signature response

var fs = require('fs')
var smartdc = require('smartdc')

var home = process.env.HOME
var key = fs.readFileSync(home + '/.ssh/id_rsa', 'ascii')

var client = smartdc.createClient({
  url: 'https://api.joyentcloud.com',
  key: key,
  keyId: '/[joyent username]/keys/luke2'
})

client.listMachines(function (err, res) {
  if (err) return console.log(err)

  console.log(res)
})

The response contains

{ message: 'Invalid signature',
  body: { code: 'InvalidCredentials', message: 'Invalid signature' },
  httpCode: 401,
  statusCode: 401,
  restCode: 'InvalidCredentials',
  code: 'InvalidCredentials' }

node v0.8.22

If I change the keyId to a key not on the account, the API will correctly respond with invalid key name [key].

I can use the same account info on the sdc command line scripts and it functions normally.

node v0.12 doesn't prompt for a PEM password.

If your ssh-agent doesn't contain your key on node v0.10 and you use an sdc-* tool you are prompted for a password:

$ sdc-listimages
Enter PEM pass phrase:
140735298188048:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:../deps/openssl/openssl/crypto/evp/evp_enc.c:544:
140735298188048:error:0906A065:PEM routines:PEM_do_header:bad decrypt:../deps/openssl/openssl/crypto/pem/pem_lib.c:483:

crypto.js:429
 var ret = this._binding.sign(toBuf(key));
                         ^
Error: SignFinal error
   at Sign.sign (crypto.js:429:27)
   at Object.signWithPrivateKey [as func] (/usr/local/lib/node_modules/smartdc/node_modules/smartdc-auth/lib/index.js:448:33)
   at /usr/local/lib/node_modules/smartdc/node_modules/smartdc-auth/node_modules/vasync/lib/vasync.js:190:20
   at process._tickCallback (node.js:442:13)

On node v0.12 if your agent is empty and you use an sdc-* tool you never get the prompt and just see the error message:

$ sdc-listimages
crypto.js:327
 var ret = this._handle.sign(toBuf(key), null, passphrase);
                        ^
Error: error:0906A068:PEM routines:PEM_do_header:bad password read
   at Error (native)
   at Sign.sign (crypto.js:327:26)
   at Object.signWithPrivateKey [as func] (/usr/local/lib/node_modules/smartdc/node_modules/smartdc-auth/lib/index.js:448:33)
   at Immediate._onImmediate (/usr/local/lib/node_modules/smartdc/node_modules/smartdc-auth/node_modules/vasync/lib/vasync.js:200:20)
   at processImmediate [as _immediateCallback] (timers.js:358:17)

sdc-fabric doesn't honor command line flags.

My default URL is us-west-1.

$ echo "$SDC_URL"
https://us-west-1.api.joyent.com

Fabrics aren't enabled in west-1, which is expected at this point.

$ ./bin/sdc-fabric vlan list
sdc-fabric: error (NotImplementedError): fabrics not enabled for this datacenter

But they should be in us-sw-1. The result is the same for all Triton DCs.

$ ./bin/sdc-fabric -u https://us-sw-1.api.joyent.com/ vlan list
sdc-fabric: error (NotImplementedError): fabrics not enabled for this datacenter

However, if I override the SDC_URL environment variable, it works as expected.

$ SDC_URL=https://us-sw-1.api.joyent.com/ ./bin/sdc-fabric vlan list
[
  {
    "name": "default",
    "vlan_id": 2
  }
]

Node version:

$ node --version
v4.1.1

sdc-createimagefrommachine does not allow to set the url

carlos@Lrrr:% ./bin/sdc-createimagefrommachine -h
sdc-createimagefrommachine [--account string] [--debug boolean] [--help boolean] [--    keyId string] [--machine string] [--imageVersion string] [--name string] [--description string] [--homepage string] [--eula string] [--acl array] [--tags string]

vs

carlos@Lrrr:% bin/sdc-listimages -h
sdc-listimages [--account string] [--public string] [--state string] [--type string] [--debug boolean] [--help boolean] [--keyId string] [--url url] 

Typo in sdc-listmachines

There's a typo in sdc-listmachines

parsed.dataet should be parsed.dataset

philip@coyolxauhqui: /usr/local/lib/node_modules/smartdc/bin
[master]$ grep dataet *
sdc-listmachines:  if (parsed.dataset) opts.dataset = parsed.dataet;

createMachine not handling 503 appropriately

createMachine is receiving 503s with a useful error message, and instead of relaying is retrying (which still fails), and eventually providing its own inscrutable error message Maximum number of retries exceeded:

Upgrade restify version

Hi! This is not a blocking issues, but on OSX Mountain Lion, the version of restify included in this package is known to have issues with node-dtrace-provider. It looks like this can be solved by updating the version.

Relevant issues from restify: restify/node-restify#177

Here's my output when running sdc-listdatacenters in version 6.5.6:

> ./node_modules/.bin/sdc-listdatacenters 
[Error: dlopen(/Users/evantahler/Desktop/tmp/node_modules/smartdc/node_modules/restify/node_modules/dtrace-provider/build/Release/DTraceProviderBindings.node, 1): no suitable image found.  Did find:
    /Users/evantahler/Desktop/tmp/node_modules/smartdc/node_modules/restify/node_modules/dtrace-provider/build/Release/DTraceProviderBindings.node: mach-o, but wrong architecture]
{
  "us-east-1": "https://us-east-1.api.joyentcloud.com",
  "us-west-1": "https://us-west-1.api.joyentcloud.com",
  "us-sw-1": "https://us-sw-1.api.joyentcloud.com",
  "eu-ams-1": "https://eu-ams-1.api.joyentcloud.com"
}

smartdc does not like running on node 0.12

⟩ sdc-listpackages
timers.js:156
    throw new TypeError('msecs must be a number');
          ^
TypeError: msecs must be a number
    at Object.exports.enroll (timers.js:156:11)
    at Socket.setTimeout (net.js:329:12)
    at SSHAgentClient._request (/usr/local/lib/node_modules/smartdc/node_modules/smartdc-auth/node_modules/ssh-agent/lib/ssh_agent_client.js:290:10)
    at SSHAgentClient.requestIdentities (/usr/local/lib/node_modules/smartdc/node_modules/smartdc-auth/node_modules/ssh-agent/lib/ssh_agent_client.js:185:15)
    at sshAgentGetKey (/usr/local/lib/node_modules/smartdc/node_modules/smartdc-auth/lib/index.js:114:12)
    at Object.checkAgentForKey [as func] (/usr/local/lib/node_modules/smartdc/node_modules/smartdc-auth/lib/index.js:341:17)
    at /usr/local/lib/node_modules/smartdc/node_modules/smartdc-auth/node_modules/vasync/lib/vasync.js:190:20
    at process._tickCallback (node.js:355:11)
    at Function.Module.runMain (module.js:503:11)
    at startup (node.js:129:16)

There already is a minor upgrade for the ssh-agent dep that looks like it would address this.

Under node 0.8.x sdc-xxx commands don't ask for SSH key passphrase and fail

Under node 0.6.x, if ssh-agent has been set up with keys, the sdc-xxx commands would prompt for the SSH key passphrase. Under node 0.8.x, it doesn't ask for a passphrase, and the command fails.

Under node 0.6.x:

$ node -v
v0.6.21-pre

philip@coyolxauhqui: ~
$ sdc-listdatacenters 
Enter PEM pass phrase:
{
  "us-east-1": "https://us-east-1.api.joyentcloud.com",
  "us-west-1": "https://us-west-1.api.joyentcloud.com",
  "us-sw-1": "https://us-sw-1.api.joyentcloud.com",
  "eu-ams-1": "https://eu-ams-1.api.joyentcloud.com"
}

After switching to node 0.8.x

philip@coyolxauhqui: ~
$ node -v
v0.8.7

philip@coyolxauhqui: ~
$ sdc-listdatacenters 
Invalid Authorization header

CLI flags typo gives strange error: --keyid instead of --keyId

If I run this:

sdc-listmachines --account VerafinProduction --keyid 'f5:ac:f8:d2:bd:e9:d3:3d:b6:67:62:4e:55:23:eb:36'

I get this strange error:

/home/alain_odea/.nvm/v0.10.28/lib/node_modules/smartdc/lib/cloudapi.js:3078
            throw (err);
                   ^
Error: --image not found in: /home/alain_odea/.ssh
    at _checkPublic (/home/alain_odea/.nvm/v0.10.28/lib/node_modules/smartdc/node_modules/smartdc-auth/lib/index.js:89:23)
    at fs.js:266:14
    at Object.oncomplete (fs.js:107:15)

If I run this:

sdc-listmachines --account VerafinProduction --keyId 'f5:ac:f8:d2:bd:e9:d3:3d:b6:67:62:4e:55:23:eb:36'

It works properly. I am omitting the JSON listing of the servers because it's not needed.

sdc-* commands should have flag for ignoring cert errors

Using sdc-* commands, I need to set SDC_TESTING=1 to ignore SSL validation errors. Somehow I always forget this and need to look i tup.

node-manta has -i, --insecure flag to ignore certificate validation. node-smartdc should have that flag as well.

hash format of SDC_KEY_ID variable matters for API

In the set up your CLI docs, there is a required environment variable named
SDC_KEY_ID

Newer versions of OpenSSH print out a completely different format for key fingerprints by default.
On my local system, I have version OpenSSH_6.8p1, OpenSSL 1.0.2c 12 Jun 2015 in my path first.

(Provided from Mac Homebrew)

As noted in the above doc you run this command:
ssh-keygen -l -f ~/.ssh/id_rsa.pub | awk '{print $2}' | tr -d '\n'

With v6.8 of OpenSSH I get this kind of output:
SHA256:maX8Wx7K5Ozf+v6TJMXIyyyXiIeu46caw1qIIj4L2Yb

With the system default OpenSSH,
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
I get the format that the SDC API actually understands:
ac:1f:83:8a:47:cc:1f:12:bb:fb:6b:16:76:b2:21:7b

The simple solution to this for the time being is to add a note about the formatting of the key in the environment variable.

Please note that the new default display option in newer versions of ssh-keygen is SHA-256
i.e. -E sha256

ssh-keygen -l -E md5 -f ~/.ssh/id_rsa.pub is what the API expects.
vs.
ssh-keygen -l -E sha256 -f ~/.ssh/id_rsa.pub is the new default format.

It's conceivable that at some point the SDC API should be updated to work with either type of key finger print, as it appears that OpenSSH is moving away from the older style formatting if the new default is different than it was in the past.

sdc-getmachine and sdc-listmachines will fail if a nic for a machine is not in napi

Calls to cloudapi's sdc-getmachine and sdc-listmachines result in the following if a nic for a machine is not in napi:

anthony.via$ sdc-listmachines
sdc-listmachines: error (ResourceNotFound): nic not found
anthony.via$ sdc-getmachine 9a3e0230-7a15-e5b2-ea4a-dccad50a2e9e
sdc-getmachine: error (ResourceNotFound): nic not found

After talking with rmustacc in IRC yesterday, we determined the problem was a VRRP nic we manually added to a zone, which did not and will not make its way into napi. I'm not sure if it is correct for a manually added nic to make its way into napi or not, but I think that sdc-listmachine and sdc-getmachine should still succeed, especially since the vmapi GET /vms/<uuid> call succeeds for the problem machine.

As a workaround for the time being, would it be correct for us to manually create the already existent VRRP nic in napi?

sdc-createinstrumentation --clone not working

It seems that --clone isn't wokring, it does not pass the clone at all:

Schroedinger:~ licenser$ sdc-createinstrumentation --debug --clone 0
2012-04-01 01:09:20Z DEBUG: No ssh-agent identities found
2012-04-01 01:09:20Z DEBUG: Using private key from: /Users/licenser/.ssh/id_rsa
2012-04-01 01:09:20Z TRACE: RestClient: constructed { _log: [Function],
  url: 
   { protocol: 'http:',
     slashes: true,
     host: '172.16.0.4:80',
     port: '80',
     hostname: '172.16.0.4',
     href: 'http://172.16.0.4:80/',
     pathname: '/',
     path: '/' },
  proto: 
   { parsers: [Object],
     STATUS_CODES: [Object],
     IncomingMessage: [Object],
     OutgoingMessage: [Object],
     ServerResponse: [Object],
     Agent: [Object],
     globalAgent: [Object],
     ClientRequest: [Object],
     request: [Function],
     get: [Function],
     Server: [Object],
     createServer: [Function],
     _connectionListener: [Function: connectionListener],
     Client: [Object],
     createClient: [Function] },
  path: '/',
  headers: { Accept: 'application/json', 'x-api-version': '~6.5' },
  retryOptions: { retries: 3, minTimeout: 500 },
  retryCallback: [Function: checkFor500],
  contentType: 'application/json',
  noContentMD5: undefined }
2012-04-01 01:09:20Z TRACE: RestClient: _processArguments => { options: 
   { contentType: 'application/json',
     headers: 
      { 'x-api-version': '~6.5',
        Accept: 'application/json',
        Authorization: 'Signature keyId="/licenser/keys/id_rsa",algorithm="rsa-sha256" ZZbfEt3Nb8lxndqve3ZXY2U63H0YhjU5QIFJ304aPp3Vj5nrlMlfgHDok0dSJj65pvirpgn9IgL0yunOYP/+u+Bod0TAusYYePWotqoE2s1wSdT7CzG3xEY5oiHh9pqaRfLKQ6BBLVj4lrZckWbS1NrvGh+Pns11gyNAkPsoz48jjF2q0Rgizo0FdHHzdWoVksJrrjDEOREuIW5hJ8CRAUG6f6mcdiX9pnoZVG8+xVMZ0imBSwd6MXNwhs8MmN1mBBBw0SSnAkffJ/yRJ8E044o3b0844K9iP/7var8Q29WrR/iea/anAwVtPkDyMFEwESv3JbvKgn2D1srqBsT5Ug==',
        Date: 'Sun, 01 Apr 2012 01:09:20 GMT' },
     method: 'POST',
     path: '//licenser/analytics/instrumentations',
     host: '172.16.0.4',
     port: '80',
     body: {},
     expect: [ 200, 201 ] },
  callback: [Function] }
2012-04-01 01:09:20Z TRACE: RestClient(99cabdd, attempt=1): issuing request { contentType: 'application/json',
  headers: 
   { 'x-api-version': '~6.5',
     Accept: 'application/json',
     Authorization: 'Signature keyId="/licenser/keys/id_rsa",algorithm="rsa-sha256" ZZbfEt3Nb8lxndqve3ZXY2U63H0YhjU5QIFJ304aPp3Vj5nrlMlfgHDok0dSJj65pvirpgn9IgL0yunOYP/+u+Bod0TAusYYePWotqoE2s1wSdT7CzG3xEY5oiHh9pqaRfLKQ6BBLVj4lrZckWbS1NrvGh+Pns11gyNAkPsoz48jjF2q0Rgizo0FdHHzdWoVksJrrjDEOREuIW5hJ8CRAUG6f6mcdiX9pnoZVG8+xVMZ0imBSwd6MXNwhs8MmN1mBBBw0SSnAkffJ/yRJ8E044o3b0844K9iP/7var8Q29WrR/iea/anAwVtPkDyMFEwESv3JbvKgn2D1srqBsT5Ug==',
     Date: 'Sun, 01 Apr 2012 01:09:20 GMT',
     'content-type': 'application/json',
     'content-length': 2,
     'content-md5': 'mZFLkyvTelC5g8XnyQrpOw==' },
  method: 'POST',
  path: '/licenser/analytics/instrumentations',
  host: '172.16.0.4',
  port: '80',
  body: {},
  expect: [ 200, 201 ] }
2012-04-01 01:09:20Z TRACE: RestClient(99cabdd): POST /licenser/analytics/instrumentations => code=200, headers={ 'content-type': 'application/json',
  connection: 'keep-alive',
  'transfer-encoding': 'chunked' }
2012-04-01 01:09:20Z TRACE: RestClient(99cabdd): POST /licenser/analytics/instrumentations => body={}
2012-04-01 01:09:20Z TRACE: RestClient: POST /licenser/analytics/instrumentations: issuing callback
2012-04-01 01:09:20Z DEBUG: CloudAPI._post(/licenser/analytics/instrumentations) -> err=null, obj={}

Data truncation when command is used in a pipeline and nodejs-4.1

Using node.js 4.1, output from commands in a pipeline is getting truncated at 64KB.

$ node --version
v4.1.1
$ bin/sdc-listimages > ~/images.out
$ bin/sdc-listimages | cat > ~/images_pipe.out
$ ls -l ~/images*
-rw-r--r--  1 bahamat  staff  280598 Oct 23 15:08 /Users/bahamat/images.out
-rw-r--r--  1 bahamat  staff   65536 Oct 23 15:08 /Users/bahamat/images_pipe.out

support --version for all cli tools

It is crazy hard to tell what version one is running, so we should support...

jclulow@jmc2 ~ $ sdc-listmachines --version
smartdc 7.2.0

... for all tools.

revive CHANGES.md changelog

Others working on this repo aren't updating CHANGES.md.

Update: Original title was "drop CHANGES.md, wrong docs are worse than no docs". Change of heart. I'm reviving CHANGES.md and adding make check-version and README.md notes about updating the changelog.

Problem when using ssh-agent with triton profiles

I attempted to use an SSH key loaded into ssh-agent but it gave the following error;

The fingerprint of the SSH key you have registered for your account.
Alternatively, You may enter a local path to a public or private SSH key to
have the fingerprint calculated for you.
keyId: 62:fa:89:bd:86:dc:72:b8:88:04:a3:d9:27:f2:0a:4f

Saved profile "lon1".
Setting up profile "lon1" to use Docker.
triton profile create: error: Failed to parse /Users/foxx/.ssh/mykey as a valid 
pem format key: PEM key is encrypted (password-protected). Please use the SSH agent 
or decrypt the key.

However SSH agent is loaded and working fine, so I'm not sure how to fix this?

I also tried using a different key format, which didn't work either;

SHA256:aLjTvr6dyqQnycSfDJJqx0S2/4gckTPTTZIAPw8NG6k

sdc-listmachines --acount not working w/o --credentials true

When passing --acount the key ID is wrong. look at:

  • keyId="/--credentials/keys/id_rsa"
  • GET /--credentials/machines?name=--tag

if you applie --credentials true it works so either --account should implie --credentials or be discared if --credentials is false

schroedinger:Projects licenser$ sdc-listmachines --debug true --acount admin
2012-04-16 07:36:32Z DEBUG: No ssh-agent identities found
2012-04-16 07:36:32Z DEBUG: Using private key from: /Users/licenser/.ssh/id_rsa
2012-04-16 07:36:32Z TRACE: RestClient: constructed { _log: [Function],
  url: 
   { protocol: 'http:',
     slashes: true,
     host: '172.16.0.4',
     hostname: '172.16.0.4',
     href: 'http://172.16.0.4/',
     pathname: '/',
     path: '/' },
  proto: 
   { parsers: [Object],
     STATUS_CODES: [Object],
     IncomingMessage: [Object],
     OutgoingMessage: [Object],
     ServerResponse: [Object],
     Agent: [Object],
     globalAgent: [Object],
     ClientRequest: [Object],
     request: [Function],
     get: [Function],
     Server: [Object],
     createServer: [Function],
     _connectionListener: [Function: connectionListener],
     Client: [Object],
     createClient: [Function] },
  path: '',
  headers: { Accept: 'application/json', 'x-api-version': '~6.5' },
  retryOptions: { retries: 3, minTimeout: 500 },
  retryCallback: [Function: checkFor500],
  contentType: 'application/json',
  noContentMD5: undefined }
2012-04-16 07:36:32Z TRACE: RestClient: _processArguments => { options: 
   { contentType: 'application/json',
     headers: 
      { 'x-api-version': '~6.5',
        Accept: 'application/json',
        Authorization: 'Signature keyId="/--credentials/keys/id_rsa",algorithm="rsa-sha256" itw33gmarbaJWHz1Za2tbYw5mRfoUCWchBykkplQKbrb+X1tFW22yPUN9prxW/8EOBOriTubeN1MLfU7JEc0TAXvetxWQt5uiVIIiU4xmcVyLu9/7MZ/ql5liZSaZwly2EcwldBiicw4cb2iUSj/8aW6ZziEXfsB/xmiNTiF8olcPxyL6xXLX0SmLaK+WTiE7WNNgSBShrD6AMts4/M38+xl/QaIqYz4Qu+nzCUHn9U91OC/2yTxR3PMIk5HREMlXttlvy6AMhhbvExkKOrqss94L6AmJSBUl9ZZzO5cPUU1tCWAnN1R0befx82rtm0mGZ3S6lcN5CL5OEXm8U/pcQ==',
        Date: 'Mon, 16 Apr 2012 07:36:32 GMT' },
     method: 'GET',
     path: '/--credentials/machines?name=--tag',
     host: '172.16.0.4',
     port: '80',
     query: { name: '--tag' },
     expect: [ 200 ] },
  callback: [Function] }
2012-04-16 07:36:32Z TRACE: RestClient(86a1d25, attempt=1): issuing request { contentType: 'application/json',
  headers: 
   { 'x-api-version': '~6.5',
     Accept: 'application/json',
     Authorization: 'Signature keyId="/--credentials/keys/id_rsa",algorithm="rsa-sha256" itw33gmarbaJWHz1Za2tbYw5mRfoUCWchBykkplQKbrb+X1tFW22yPUN9prxW/8EOBOriTubeN1MLfU7JEc0TAXvetxWQt5uiVIIiU4xmcVyLu9/7MZ/ql5liZSaZwly2EcwldBiicw4cb2iUSj/8aW6ZziEXfsB/xmiNTiF8olcPxyL6xXLX0SmLaK+WTiE7WNNgSBShrD6AMts4/M38+xl/QaIqYz4Qu+nzCUHn9U91OC/2yTxR3PMIk5HREMlXttlvy6AMhhbvExkKOrqss94L6AmJSBUl9ZZzO5cPUU1tCWAnN1R0befx82rtm0mGZ3S6lcN5CL5OEXm8U/pcQ==',
     Date: 'Mon, 16 Apr 2012 07:36:32 GMT',
     'content-length': 0 },
  method: 'GET',
  path: '/--credentials/machines?name=--tag',
  host: '172.16.0.4',
  port: '80',
  query: { name: '--tag' },
  expect: [ 200 ] }
2012-04-16 07:36:33Z TRACE: RestClient(86a1d25): GET /--credentials/machines?name=--tag => code=401, headers={ 'content-type': 'application/json',
  connection: 'keep-alive',
  'transfer-encoding': 'chunked' }
2012-04-16 07:36:33Z TRACE: RestClient(86a1d25): GET /--credentials/machines?name=--tag => body={"error":"key not found"}
2012-04-16 07:36:33Z TRACE: RestClient: GET /--credentials/machines?name=--tag: issuing callback

sdc-listimages --help output re: state is for instances, not images

From brief discussion in chat:

bash-3.2$ sdc-listimages --help
sdc-listimages [--account string] [--api-version string] [--public
string] [--state string] [--type string] [--debug boolean] [--help
boolean] [--keyId string] [--url url] [--version boolean] [--verbose
boolean] [--user string] [--role string]

...
    state - the state of the instance (i.e. running, stopped ...)
(default: all)
    type - filter by type (default: all)
...
bash-3.2$
--------

I think "the state of the instance (i.e. running, stopped ...)
(default: all)" does not refer to a state that makes
sense for an image.  

The *description* used for options is shared between all the sdc-* commands. 

oh, looks like overrides are possible per command. Could you open a node-smartdc ticket on this? You can assign it to me.

I can update sdc-listimages to have a desc for the '--state' option that is specific to sdc-listimages

all tools accept a --debug field, but do nothing with it

In trying to educate people on how to use the REST API, it's often helpful to "do the same thing" using the node-smartdc tools, and examine the POST payloads, for example.

I often do this by setting my SDC_URL to something like http://localhost:6666 , and then "catching" that request with "nc -l 6666" ... that's an ugly hack, though.

Could we get the equivalent of the debugging info you'd get from something like "curl -vvv" , so people could use the sdc-* commands, and see the REST traffic they generate? Right now, the --debug flag does nothing, it seems.

sdc-createimagefrommachine gives incorrect usage string

carlos@Lrrr:% ./bin/sdc-createimagefrommachine -h
sdc-createimagefrommachine [--account string] [--debug boolean] [--help boolean] [--keyId string] [--machine string] [--imageVersion string] [--name string] [--description string] [--homepage string] [--eula string] [--acl array] [--tags string]

account - account name (i.e. customer)
debug - enable debug/verbose mode (default: 0)
help - print out this usage
keyId - the fingerprint of your ssh key (use ssh-keygen -l to determine)
url - url for SmartDataCenter API (i.e. https://someaddress.com)
name - the name of the entity
tag - key=value pair
enabled - enable entity
rule - firewall rule
clone - an existing instrumentation (i.e. it's id number)
decomposition - an array of arrays for breaking down the data (default: none)
module - the CA module (i.e. syscall)
predicate - JSON string to filter data (default: none)
stat - the CA stat (i.e. syscalls)
value - value of stat (i.e. syscalls)
metadata - metadata associated with this instance
networks - the network UUIDS to be attached to this instance
package - the instance type (UUID) to use for this instance
script - the user-script to run upon creation
credentials - include generated credentials for instance (default:false)
limit - return N instances (default: 1000, max allowable)
memory - filter instances by memory size (MB) (default: none)
offset - return the next N instances from this starting point (default: 0)
state - the state of the instance (i.e. running, stopped ...) (default: all)
type - filter by type (virtualmachine or smartmachine) (default: all)
tombstone - include instances destroyed in the last N minutes (default: none)
snapshot - snapshot name
address - street address
city - city
company - company name (if no company, put 'none')
country - country
email - email address
phone - full phone number (include country code if outside US)
postal-code - postal code
surname - last name/surname
enable-firewall - Enable or not instance firewall (default: false)

Wait longer for timeouts on a command line client

Adding a Number(process.env.SDC_SSH_TIMEOUT) variable in /usr/local/lib/node_modules/smartdc/node_modules/smartdc-auth/node_modules/ssh-agent/lib/ssh_agent_client.js allows you to wait longer for timeouts on the command line client.

function SSHAgentClient(options) {
  options = options || {};
  this.timeout = options.timeout || Number(process.env.SDC_SSH_TIMEOUT) || 1000;

  this.sockFile = process.env.SSH_AUTH_SOCK;
  if (!this.sockFile)
    throw new MissingEnvironmentVariableError('SSH_AUTH_SOCK');
}

Usage: SDC_TESTING=1 SDC_URL=https://example.com SDC_ACCOUNT=example SDC_KEY_ID=00:00 SDC_SSH_TIMEOUT=3000 sdc-listdatacenters

"Invalid signature" appears at random from different commands

I'm on Ubuntu 12.10 and hitting this very often which makes scripting the tools a huge pain.

Here's my simple repro:

#!/bin/bash

while true; do
  sdc-listmachines --limit 100 --offset 0

  EXIT_CODE=$?

  echo "Exit code: $EXIT_CODE"

  if [[ $EXIT_CODE -ne 0 ]]; then
    exit $EXIT_CODE
  fi
done

Newest version doesn't properly compile on OSX (dtrace) and API throws bad HTTP error (v6.5.5)

On OSX Mountain Lion & Node v0.8.14

npm install smartdc -> [email protected] node_modules/smartdc

./node_modules/.bin/sdc-listmachines

[Error: dlopen(/Users/evantahler/Desktop/test/node_modules/smartdc/node_modules/restify/node_modules/dtrace-provider/build/Release/DTraceProviderBindings.node, 1): no suitable image found.  Did find:
    /Users/evantahler/Desktop/test/node_modules/smartdc/node_modules/restify/node_modules/dtrace-provider/build/Release/DTraceProviderBindings.node: mach-o, but wrong architecture]

/Users/evantahler/Desktop/test/node_modules/smartdc/node_modules/restify/lib/errors/http_error.js:73
      throw new Error(code + ' is not a known HTTP error code');
            ^
Error: 449 is not a known HTTP error code
    at Object.codeToHttpError (/Users/evantahler/Desktop/test/node_modules/smartdc/node_modules/restify/lib/errors/http_error.js:73:13)
    at ClientRequest.requestCallback (/Users/evantahler/Desktop/test/node_modules/smartdc/node_modules/restify/lib/clients/http_client.js:240:22)
    at ClientRequest.g (events.js:192:14)
    at ClientRequest.EventEmitter.emit (events.js:96:17)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (http.js:1462:7)
    at HTTPParser.parserOnHeadersComplete [as onHeadersComplete] (http.js:111:23)
    at CleartextStream.socketOnData [as ondata] (http.js:1367:20)
    at CleartextStream.CryptoStream._push (tls.js:526:27)
    at SecurePair.cycle (tls.js:880:20)
    at EncryptedStream.CryptoStream.write (tls.js:267:13)

I have an old version of this package 6.5.4 installed globally (for CLI use), and the same command works OK.

sdc-* commands have weird parsing confusion

While attempting to list all deleted machines, I saw this:

$ sdc-listmachines --tombsotne
--offset is invalid metadata; try foo=bar

After a few minutes I realized I had misspelled --tombstone as --tombsotne. My initial problem was solved, but that still left the matter of the odd error message.

With a bit of trial and error, I finally figured out what's going on. Running sdc-* --t* where what comes after --t is not one of tag, type or tombstone, the --t gets expanded to --tags, then the rest of the string gets converted to other long forms, in this case the o gets expanded to --offset.

This happens any time where the first character is t and the second character is any of the short options.

API Request: Pricing

Hi Joyent!

I woud like to request a sdc-listprices API call which would return the cost per hour of the various services you offer.

In the interim, I've cobbled tougher a new method: sdc-listmachines-cost which returns the list if servers I currently have running and what they cost. You can see that I'm currently relying on a static list of prices [[ https://github.com/evantahler/smartdc-cost/blob/master/lib/prices.js ]], and having an accurate list from the source would be appreciated.

Github: https://github.com/evantahler/smartdc-cost
NPM: https://npmjs.org/package/smartdc-cost

Errors out in Windows Pro 8 64-bit

When attempting to perform a sdc-listimages, I receive the following error:

PS C:\Users\Carlos Cardenas> sdc-listimages

C:\Users\Carlos Cardenas\AppData\Roaming\npm\node_modules\smartdc\lib\cloudapi.js:3019
throw (err);
^
Error: ENOENT, readdir 'C:\Users\Carlos Cardenas\undefined.ssh'

Notice the error. My .ssh directory is 'C:\Users\Carlos Cardenas\.ssh' vs 'C:\Users\Carlos Cardenas\undefined\.ssh'

github.com/joyent/node-manta works FWIW

Validate account, url, key input

With the advent of RBAC users are now accustomed to logging into the portal with Account/SubUser but some commands like listimages use SDC_ACCOUNT verbatim, and if the user inadvertently sets that to their portal credentials, the server is not able to respond with a message indicating a wrong user. Instead you get that the command isn't allowed.

Instead we should be validating the input for account, user, url, and key to make sure they contain appropriate information. In the case that inappropriate information is provided we should bail with guidance.

package name encoding for space in sdc-getpackage sdc-createmachine

The package names returned by sdc-listpackages are unusable due to the spaces.

The package name needs to be urlencoded for scd-getpackage to work.

$ sdc-getpackage "Small 1GB"
Maximum number of retries exceeded: 4

$ scd-getpackage Small+1GB
{
"name": "Small 1GB",
"memory": 1024,
"disk": 30720,
"swap": 2048,
"default": false
}

or

$ scd-getpackage Small%201GB
{
"name": "Small 1GB",
"memory": 1024,
"disk": 30720,
"swap": 2048,
"default": false
}

scd-createmachine does not work with any of the following values for --package

  • "Small 1GB"
  • Small+1GB
  • Small%201GB
  • regular_1024
  • Small\ 1GB

A space returns "Invalid request parameters"
Any other character returns "Small+1GB is an invalid package" substituting the joining charcter

The API Doc will need to be update as well.

Misleading Errors from sdc-createfirewallrule

When you specify an incorrect rule, such as:

[jschmidt@pepper bin]$ sdc-createfirewallrule --rule "FROM subnet 63.246.133.86/31 to all vms block tcp" --enable

The error message is a bit misleading:

sdc-createfirewallrule: error (InvalidParameters): Invalid parameters (rule: Error at character 0: '', expected: 'FROM', found: empty string)

Other messages are more helpful:

[jschmidt@pepper bin]$ sdc-createfirewallrule --rule "FROM subnet 63.246.133.86/31 to all block tcp" --enable
sdc-createfirewallrule: error (InvalidParameters): Invalid parameters (rule: Error at character 35: 'block', expected: 'VMS', found: BLOCK)
[jschmidt@pepper bin]$ sdc-createfirewallrule --rule "FROM 63.246.133.86/31 to all vms block tcp"
sdc-createfirewallrule: error (InvalidParameters): Invalid parameters (rule: Error at character 4: '63.246.133.86', expected: '(', 'all', 'any', 'ip', 'subnet', 'vm', 'tag', found: IPADDR)

Correct syntax works correctly:

[jschmidt@pepper bin]$ sdc-createfirewallrule --rule "FROM subnet 63.246.133.86/31 to all vms block tcp port all" --enable
{
  "id": "c1791b4a-bcff-466f-a44c-d906c7594210",
  "rule": "FROM subnet 63.246.133.86/31 TO all vms BLOCK tcp PORT all",
  "enabled": true
}

sdc-setup failed with an exception

I followed "Getting Started" instructions in http://api.joyentcloud.com/ and installed node, npm and smartdc package. But my very first run of sdc-setup failed with an exception:

node.js:201
        throw e; // process.nextTick error, or 'error' event on first tick
              ^
Error: No such module
    at Object.<anonymous> (/usr/local/lib/node_modules/smartdc/bin/sdc-setup:33:21)
    at Module._compile (module.js:432:26)
    at Object..js (module.js:450:10)
    at Module.load (module.js:351:31)
    at Function._load (module.js:310:12)
    at Array.0 (module.js:470:10)
    at EventEmitter._tickCallback (node.js:192:40)

I used:

Mac OS X 10.6.8
node 0.6.3
npm 1.0.106

and npm list gives me:

/Users/edward/ws/node
└─┬ [email protected] 
  ├── [email protected] 
  ├─┬ [email protected] 
  │ └── [email protected] 
  ├─┬ [email protected] 
  │ ├── [email protected] 
  │ ├── [email protected] 
  │ ├── [email protected] 
  │ ├── [email protected] 
  │ └── [email protected] 
  ├── [email protected] 
  └─┬ [email protected] 
    └── [email protected] 

I also tried node 0.6.2 installed by homebrew. Same error.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.