manageiq / azure-armrest Goto Github PK

View Code? Open in Web Editor NEW

15.0 11.0 35.0 1.09 MB

Ruby interface for Azure using the new REST API

License: Apache License 2.0

Ruby 100.00%

azure ruby

azure-armrest's Introduction

Description

A Ruby interface for Azure using the new REST API.

Synopsis

require 'azure/armrest'

# Create a configuration object. All service objects will then use the
# information you set here.
#
# A token will be retrieved based on the information you provided

conf = Azure::Armrest::Configuration.new(
  :client_id       => 'XXXXX',
  :client_key      => 'YYYYY',
  :tenant_id       => 'ZZZZZ',
  :subscription_id => 'ABCDEFG'
)

# This will then use the configuration info set above.
vms = Azure::Armrest::VirtualMachineService.new(conf)

# You can add other options specific to the service to be created,
# such as the provider.

options = {:provider => 'Microsoft.ClassicCompute'}
vms = Azure::Armrest::VirtualMachineService.new(conf, options)

# List all virtual machines for a given resource group:
vms.list(some_group).each do |vm|
  puts vm.name
  puts vm.resource_group
  puts vm.location
  puts vm.properties.hardware_profile.vm_size
end

Subscriptions

As of version 0.4.0 you a subscription ID is not longer strictly necessary in the Configuration constructor, but almost all service classes require it in their own constructor. Only the SubscriptionService class does not.

In version 0.3.x the subscription ID was mandatory. Prior to 0.3.x, if you did not provide a subscription ID in your configuration object, then the first subscription ID returned from a REST call would be used.

Notes

Currently only the client credentials strategy is supported. Support for other strategies may be added over time.

License

The gem is available as open source under the terms of the Apache License 2.0.

Authors

Daniel Berger
Bronagh Sorota
Bill Wei

azure-armrest's People

Contributors

Stargazers

Watchers

azure-armrest's Issues

consolidating ruby azure efforts?

have you considered moving your code to say fog as say fog/fog-azure-resource-manager so we can consolidate the cloud api efforts for azure?

VirtualMachineService#series broken

Looks like the VirtualMachineMethod#series method is broken after the last PR:

vms = Azure::Armrest::VirtualMachineService.new(conf)
vms.series

Result:

azure-armrest-djberg96/lib/azure/armrest/virtual_machine_service.rb:24:in `series': Invalid provider 'Microsoft.Compute' (ArgumentError)
    from providers_test.rb:26:in `<main>'

Add ability to get raw blob information on managed disks

We'll need to modify the Storage::DiskService class in order to support SSA for ManageIQ for VM's that use managed disks. The Azure REST API provides separate a separate method for getting raw blob information on managed disks:

https://docs.microsoft.com/en-us/rest/api/manageddisks/disks/disks-grant-access

It's a 3 step process. First, you have to get the location URL from the response header, then you have to get the URL that includes the SAS token, and finally you make a request with that URL to get the blob information.

Need a better way to wrap RestClient exceptions

At the moment we have a series of pre-defined exception classes in exception.rb. Unfortunately, this means that if RestClient raises an error that we don't define, users get a generic ApiException. This is not usually very helpful in debugging.

What I would like to see is some sort of code that automatically transforms the RestClient exceptions into Armrest exceptions in a more dynamic way so that we don't have to add them manually.

Marketplace Image Support

As talked about here I'm opening an RFE for marketplace image support. From what I can see this will require:

Altering prepare_for_clone_task to parse and accept marketplace URNs
Adding a method to pull marketplace images to the refresh parser
Maybe adding something akin to the public image settings and filter found in the Amazon provider?

Use ResourceProviderService class instead of @@providers_hash

The @@providers_hash was originally implemented as a quick and dirty caching mechanism. However, I always intended that it be replaced by the ResourceProviderService class since it uses cache_method to cache most of its calls.

Using the ResourceProviderService class is a cleaner approach, and I suspect it will also deal with what I think are some class variable scoping issues with JRuby for the test suite.

~~However, #98 will need to be addressed first.~~

ArmrestCollection#to_json busted

If you try to convert an ArmrestCollection object to JSON using the to_json method, you get an error:

vms.list_all.to_json

Result:

lib/azure/armrest/model/base_model.rb:84:in `to_json': wrong number of arguments (given 1, expected 0) (ArgumentError)

This should "just work".

Add friendlier handling of 429 and 50x errors

Within rest_execute we should check the error code before automatically raising an exception. On a 429 errors (TooManyRequests) we should sleep Retry-After seconds and try again. On 50x errors (Azure flaked out) we should sleep a set number of seconds (say, 30) and retry again.

I think adding a :max_retries accessor to the configuration object could be used internally.

Library should store response headers somehow

Currently we basically ignore the response headers. However, they may contain useful information:

{:cache_control=>"no-cache", :pragma=>"no-cache", :content_type=>"application/json", :content_encoding=>"gzip", :expires=>"-1", :vary=>"Accept-Encoding", :x_ms_request_id=>"a7e0e1ad-6f3a-4eeb-ace8-9c77c3377179", :server=>"Microsoft-Azure-Storage-Resource-Provider/1.0, Microsoft-HTTPAPI/2.0", :x_ms_ratelimit_remaining_subscription_reads=>"14872", :x_ms_correlation_request_id=>"a7e0e1ad-6f3a-4eeb-ace8-9c77c3377179", :x_ms_routing_request_id=>"NORTHCENTRALUS:20160601T182659Z:a7e0e1ad-6f3a-4eeb-ace8-9c77c3377179", :strict_transport_security=>"max-age=31536000; includeSubDomains", :date=>"Wed, 01 Jun 2016 18:26:59 GMT", :connection=>"close"}

Notably, the :x_ms_ratelimit_remaining_subscription_reads would be useful to know, especially if we get close to zero.

I'm not sure how these should be stored, though. Storing header information in the model objects would be wasteful since it would be identical in every object. Perhaps a singleton that stores the header information? I'm open to suggestions.

Logging test failures on Windows

Failures:

  1) Azure::Armrest::Configuration singletons logging accepts a file name for a log
     Failure/Error: after  { File.delete(log) if File.exist?(log) }

     Errno::EACCES:
       Permission denied @ unlink_internal - azure-armrest.log
     # ./spec/configuration_spec.rb:27:in `delete'
     # ./spec/configuration_spec.rb:27:in `block (2 levels) in <top (required)>'

  2) Azure::Armrest::Configuration singletons logging accepts a file handle for a log
     Failure/Error: after  { File.delete(log) if File.exist?(log) }

     Errno::EACCES:
       Permission denied @ unlink_internal - azure-armrest.log
     # ./spec/configuration_spec.rb:27:in `delete'
     # ./spec/configuration_spec.rb:27:in `block (2 levels) in <top (required)>'

  3) Azure::Armrest::Configuration singletons logging accepts a Logger instance
     Failure/Error: after  { File.delete(log) if File.exist?(log) }

     Errno::EACCES:
       Permission denied @ unlink_internal - azure-armrest.log
     # ./spec/configuration_spec.rb:27:in `delete'
     # ./spec/configuration_spec.rb:27:in `block (2 levels) in <top (required)>'

Finished in 2.55 seconds (files took 4.32 seconds to load)
418 examples, 3 failures

Failed examples:

rspec ./spec/configuration_spec.rb:243 # Azure::Armrest::Configuration singletons logging accepts a file name for a log
rspec ./spec/configuration_spec.rb:248 # Azure::Armrest::Configuration singletons logging accepts a file handle for a log
rspec ./spec/configuration_spec.rb:255 # Azure::Armrest::Configuration singletons logging accepts a Logger instance

Not sure yet if this is an issue with our log handling, the specs, or the rest-client gem.

The api_version option in the configure method is overridden

At the moment it seems that an :api_version option passed to the ArmrestService.configure method is overridden:

conf = Azure::Armrest::ArmrestService.configure(
  :tenant_id   => "xxx",
  :client_id   => "yyy",
  :client_key  => "zzz",
  :api_version => '2015-01-01'
)

vms = Azure::Armrest::VirtualMachineService.new(conf)
p vms.api_version # => 2015-06-15

The workaround is to set it after the constructor:

vms.api_version = '2015-01-01'

We do have to be careful not to override the hard coded values we have for storage accounts and template deployments, too.

Remove Nokogiri, use REXML

I realize Nokogiri is a superior XML parsing library, but the amount of actual XML that we deal with from Azure (all storage account related) is actually relatively small. Any speed/memory issues should be negligible in practice.

Removing Nokogiri eases the installation and maintenance of this library, and we won't have to worry about CVE's any more, either.

RFE: AzureStack -- auto-discovery of Azure endpoint URLs

Instead of hard-coding URLs in the Environment, we should be able to auto-discover them given only the resource manager URL using a GET to <ARM_URL>/metadata/endpoints?api-version=1.0

This gives us details on the gallery, graph, and portal URLs, as well as the active_directory_authority and active_directory_resource_id

This will be helpful for AzureStack deployments where the underlying URLs can't be hard-coded. When adding the provider, all the admin would need to provide is the resource manager URL and we could take care of the rest.

See an example in Ruby here (search for def get_active_directory_settings(armEndpoint)):
https://docs.microsoft.com/en-us/azure/azure-stack/user/azure-stack-version-profiles-ruby

JSON returned by an azure stack deployment:

{
    "galleryEndpoint": "https://portal.westus.stackpoc.com:30015/",
    "graphEndpoint": "https://graph.windows.net/",
    "portalEndpoint": "https://portal.westus.stackpoc.com/",
    "authentication": {
        "loginEndpoint": "https://login.windows.net/",
        "audiences": ["https://management.poc.avahc.com/01a7976d-5a10-475d-b39d-01993f9d7d90"]
    }
}

This same API is working on the public cloud too. Example JSON:

{
    "galleryEndpoint": "https://gallery.azure.com/",
    "graphEndpoint": "https://graph.windows.net/",
    "portalEndpoint": "https://portal.azure.com/",
    "authentication": {
        "loginEndpoint": "https://login.windows.net/",
        "audiences": ["https://management.core.windows.net/", "https://management.azure.com/"]
    }
}

StorageAccount#table_data with :all option is busted

Passing the :all option to the StorageAccount#table_data method appears to result in an infinite loop.

Edit: maybe.

TemplateDeploymentService#get_template bug

I hit this bug in our current dev environment. You can reproduce it like so:

tds = Azure::Armrest::TemplateDeploymentService.new(conf)
tds.get_template("gswmonz1319", "gswmonz1319")

Result:

/Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:166:in `const_defined?': wrong constant name Openshift webconsole (NameError)
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:166:in `nested_object'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:157:in `block in __setobj__'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:149:in `each'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:149:in `__setobj__'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:67:in `initialize'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:170:in `new'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:170:in `nested_object'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:157:in `block in __setobj__'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:149:in `each'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:149:in `__setobj__'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:67:in `initialize'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/template_deployment_service.rb:50:in `new'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/template_deployment_service.rb:50:in `get_template'
	from futz/template.rb:14:in `block in <main>'
	from futz/template.rb:10:in `each'
	from futz/template.rb:10:in `<main>'

I'm guessing it's because of the space in the name between "Openshift" and "web console".

Possibly the cause of: https://bugzilla.redhat.com/show_bug.cgi?id=1434988

ArmrestManager Locations method need to filter out invalid locations

When some locations such as BrazilSouth or AustraliaEast are passed into the VMM.Series method they cause the follow error:

"https://management.azure.com/subscriptions/462f2af8-e67e-40c6-9fbf-02824d1dd485/providers/Microsoft.Compute/locations/AustraliaEast/vmSizes?api-version=2015-06-15"
/opt/rubies/ruby-2.0.0-p576/lib/ruby/gems/2.0.0/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in return!': 502 Bad Gateway (RestClient::BadGateway) from /opt/rubies/ruby-2.0.0-p576/lib/ruby/gems/2.0.0/gems/rest-client-1.8.0/lib/restclient/request.rb:495:inprocess_result'
from /opt/rubies/ruby-2.0.0-p576/lib/ruby/gems/2.0.0/gems/rest-client-1.8.0/lib/restclient/request.rb:421:in block in transmit' from /opt/rubies/ruby-2.0.0-p576/lib/ruby/2.0.0/net/http.rb:852:instart'
from /opt/rubies/ruby-2.0.0-p576/lib/ruby/gems/2.0.0/gems/rest-client-1.8.0/lib/restclient/request.rb:413:in transmit' from /opt/rubies/ruby-2.0.0-p576/lib/ruby/gems/2.0.0/gems/rest-client-1.8.0/lib/restclient/request.rb:176:inexecute'
from /opt/rubies/ruby-2.0.0-p576/lib/ruby/gems/2.0.0/gems/rest-client-1.8.0/lib/restclient/request.rb:41:in execute' from /opt/rubies/ruby-2.0.0-p576/lib/ruby/gems/2.0.0/gems/rest-client-1.8.0/lib/restclient.rb:65:inget'
from /Users/bronaghsorota/dev/azure-armrest/lib/azure/armrest/armrest_manager.rb:356:in rest_get' from /Users/bronaghsorota/dev/azure-armrest/lib/azure/armrest/virtual_machine_manager.rb:58:inseries'
from ./july17test.rb:29:in block in <main>' from ./july17test.rb:27:ineach'
fr

Return bonafide model objects where possible for nested attributes

For example, if I want to get a list of network interfaces for a virtual machine I could do this:

vm.properties.network_profile.network_interfaces.each{ |n| ... }

The problem is that the objects returned by that only have the :id attribute because that's what the JSON has. However, we have NetworkInterface model objects as well.

So, instead of just returning a simple object with an :id field, provide a generic method to call that returns a real model object if we have one that wraps it. The :id field should still exist, too, in case users want to use the original value. So, something like:

vm.properties.network_profile.network_interfaces.each{ |n| n.model ... }

NetworkSecurityRuleService and SubnetService are broken

Upon further review I don't think these should have been made subservices. In any case, right now they don't actually work.

sns = Azure::Armrest::Network::SubnetService.new(conf)
sns.list('some_group)

Results in:

lib/azure/armrest/resource_group_based_service.rb:56:in `validate_resource_group': must specify resource group (ArgumentError)
    from /Users/dberger/Repositories/azure-armrest-djberg96/lib/azure/armrest/resource_group_based_subservice.rb:24:in `list'

When determining api versions for providers, case should be ignored

I ran into an issue where I had set my provider to "Microsoft.Insights" instead of "microsoft.insights". Because it wasn't an exact match it returned the default api version, which it turns out didn't work.

I think the set_service_api_version and/or set_providers_info methods should be adjusted so that they ignore case, since MS ignores the provider's case as well.

Delegated method list_providers not found

The delegated :list_providers method in Azure::Armrest::ArmrestService (on master) doesn't appear to work:

conf = Azure::Armrest::Configuration.new(creds)
vms  = Azure::Armrest::VirtualMachineService.new(conf)

p vms.list_providers

Result:

/Users/dberger/Repositories/azure-armrest-djberg96/lib/azure/armrest/armrest_service.rb:50:in `list_providers': undefined method `list_providers' for #<Azure::Armrest::Configuration:0x007fad6cd3dbe8> (NoMethodError)
    from locations.rb:11:in `<main>'

I also tried the alias :providers, but that also failed:

>ruby locations.rb 
NOTE: Azure::Armrest::VirtualMachineService#providers is deprecated; use list_providers instead. It will be removed on or after 2018-01-01.
Azure::Armrest::VirtualMachineService#providers called from locations.rb:11.
/Users/dberger/Repositories/azure-armrest-djberg96/lib/azure/armrest/armrest_service.rb:50:in `list_providers': undefined method `list_providers' for #<Azure::Armrest::Configuration:0x007f8bbb758200> (NoMethodError)
    from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/site_ruby/2.2.0/rubygems/deprecate.rb:63:in `block (2 levels) in deprecate'
    from locations.rb:11:in `<main>'

Add a delete_associated method to VirtualMachineService

Azure does not delete associated resources when deleting a VM, such as IP address, NIC, attached disks, or even the .vhd (blob) for the VM itself. I think we should add a method that optionally deletes the VM and associated resources of the user's choice.

Maybe something like this:

vms.delete_associated(name, group) # With no options, delete everything
vms.delete_associated(name, group, :ip_address => true) # Just the associated IP address

The options I can think of right off are:

attached_disks
os_disk (.vhd)
network_interface
ip_address
network_security_group
virtual_network*
storage_account*

*Since virtual networks and storage accounts often have unrelated resources associated with them, they should have to be selected explicitly to attempt to delete them.

ClassicCompute does not have network interfaces

It seems there is no networkInterfaces property for classic VM's:

vmm.provider = "Microsoft.ClassicCompute"
vmm.list

# Output
/Users/dberger/Repositories/azure-armrest/lib/azure/armrest/virtual_machine_manager.rb:132:in `block in add_network_profile': undefined method `each' for nil:NilClass (NoMethodError)

Problem with cache_method and anonymous classes

It seems we have a conflict with the recent refactoring and the cache_method gem.

rps = Azure::Armrest::ResourceProviderService.new(conf)
rp = rps.get('Microsoft.Compute')

Result:

/Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.4/lib/active_support/cache.rb:667:in `dump': can't dump anonymous class #<Class:0x007fe47790ad90> (TypeError)
    from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.4/lib/active_support/cache.rb:667:in `dup_value!'
    from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.4/lib/active_support/cache/memory_store.rb:146:in `write_entry'
    from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.4/lib/active_support/cache.rb:391:in `block in write'
    from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.4/lib/active_support/cache.rb:547:in `block in instrument'
    from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.4/lib/active_support/notifications.rb:166:in `instrument'
    from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.4/lib/active_support/cache.rb:547:in `instrument'
    from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.4/lib/active_support/cache.rb:389:in `write'
    from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/cache-0.4.1/lib/cache/active_support_cache_store.rb:21:in `_set'
    from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/cache-0.4.1/lib/cache.rb:77:in `set'
    from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/cache_method-0.2.7/lib/cache_method/cached_result.rb:72:in `set_wrapped'
    from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/cache_method-0.2.7/lib/cache_method/cached_result.rb:32:in `fetch'
    from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/cache_method-0.2.7/lib/cache_method.rb:99:in `block in cache_method'
    from test.rb:178:in `<main>'

StorageAccount#all_blobs consumes lots of memory

The StorageAccount#blobs method does a blob list for a given container. The response body actually returns XML (instead of JSON), and this body has typically been anywhere from 1k to 15k in practice. Over time they have added more fields to the default output, e.g. ServerEncrypted, AccessTier, etc, and this has resulted in even larger text bodies, which in turn consumes more memory.

The net result is that the StorageAccount#list_all_private_images is consuming huge amounts of memory as it iterates over each container on each storage account.

https://docs.microsoft.com/en-us/rest/api/storageservices/list-blobs

VirtualMachineService#delete_associated_resources does not handle managed disks

At the moment if you try to call delete_associated_resources on a VM that uses managed storage, the storage deletion will fail. This method was originally written for unmanaged storage and was never updated.

undefined method 'vhd' for #<Azure::Armrest::VirtualMachineModel::Properties::StorageProfile::OsDisk:0x007fa0704c8cf0>  Method:[block in method_missing]
[----] E, [2017-11-20T15:25:51.764838 #35394:3fd03f83f7e4] ERROR -- : /Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/azure-armrest-0.9.3/lib/azure/armrest/storage_account_service.rb:197:in `get_from_vm'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/azure-armrest-0.9.3/lib/azure/armrest/virtual_machine_service.rb:221:in `delete_associated_disk'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/azure-armrest-0.9.3/lib/azure/armrest/virtual_machine_service.rb:165:in `delete_associated_resources'

The list methods don't paginate

At the moment neither the list nor the list_all methods paginate, which would result in truncated results. They need to handle skip tokens.

Some current test failures on Windows

Windows 8
ruby 2.4.2p198 (2017-09-14 revision 59899) [i386-mingw32]

While not a high priority, this library should be platform neutral, so we shouldn't be seeing failures on any platform. I suspect this is just the way we've written our tests rather than a core library issue, but thought I should post.

Failures:

  1) Azure::Armrest::Configuration instances tokens token generation caches the token to be reused for the same cl
     Failure/Error: expect(subject.token).to eql(token)

       expected: "Bearer eyJ0eXAiOiJKV1Q"
            got: "xxx"

       (compared using eql?)
     # ./spec/configuration_spec.rb:185:in `block (5 levels) in <top (required)>'

  2) BaseModel inspection methods defines a pretty_print method when pp is available
     Failure/Error: expect(base.pretty_inspect).to match(expected)

       expected "#<Azure::Armrest::BaseModel:0x03b95f98 name=\"test\", array=[\"stuff\"], age=33>\n" to match /\A#
::Armrest::BaseModel:0x\h+\n/
       Diff:
       @@ -1,2 +1,2 @@
       -/\A#<Azure::Armrest::BaseModel:0x\h+\n/
       +#<Azure::Armrest::BaseModel:0x03b95f98 name="test", array=["stuff"], age=33>

     # ./spec/models/base_model_spec.rb:146:in `block (3 levels) in <top (required)>'

  3) BaseModel #attr_from_hash maps it source location to the correct file
     Failure/Error: expect(subject.method(:first_name_from_hash).source_location).to include(__FILE__)
       expected ["C", 0] to include "C:/Users/Daniel/Dev/azure-armrest-djberg96/spec/models/base_model_spec.rb"
     # ./spec/models/base_model_spec.rb:291:in `block (3 levels) in <top (required)>'
     # ./spec/models/base_model_spec.rb:279:in `block (3 levels) in <top (required)>'

  4) BaseModel #attr_from_hash with multiple attributes maps it source location of each method to the correct file
     Failure/Error: expect(subject.method(:first_name_from_hash).source_location).to include(__FILE__)
       expected ["C", 0] to include "C:/Users/Daniel/Dev/azure-armrest-djberg96/spec/models/base_model_spec.rb"
     # ./spec/models/base_model_spec.rb:313:in `block (4 levels) in <top (required)>'
     # ./spec/models/base_model_spec.rb:301:in `block (4 levels) in <top (required)>'
     # ./spec/models/base_model_spec.rb:279:in `block (3 levels) in <top (required)>'

  5) BaseModel #attr_from_hash with nested attributes maps it source location to the correct file
     Failure/Error: expect(subject.method(:street_address_from_hash).source_location).to include(__FILE__)
       expected ["C", 0] to include "C:/Users/Daniel/Dev/azure-armrest-djberg96/spec/models/base_model_spec.rb"
     # ./spec/models/base_model_spec.rb:335:in `block (4 levels) in <top (required)>'
     # ./spec/models/base_model_spec.rb:324:in `block (4 levels) in <top (required)>'
     # ./spec/models/base_model_spec.rb:279:in `block (3 levels) in <top (required)>'

@NickLaMuro I think 3, 4 and 5 are yours. :)

Test failures on Windows

Windows 8
ruby 2.4.2p198 (2017-09-14 revision 59899) [i386-mingw32]

While not a high priority, we shouldn't be seeing test failures on any platform since this is a platform neutral library. I suspect these are issues with our specs rather than a library issue, but thought I should mention it.

Failures:

  1) Azure::Armrest::Configuration instances tokens token generation caches the token to be reused for the same client
     Failure/Error: expect(subject.token).to eql(token)

       expected: "Bearer eyJ0eXAiOiJKV1Q"
            got: "xxx"

       (compared using eql?)
     # ./spec/configuration_spec.rb:185:in `block (5 levels) in <top (required)>'

  2) BaseModel inspection methods defines a pretty_print method when pp is available
     Failure/Error: expect(base.pretty_inspect).to match(expected)

       expected "#<Azure::Armrest::BaseModel:0x03b95f98 name=\"test\", array=[\"stuff\"], age=33>\n" to match /\A#<Azure
::Armrest::BaseModel:0x\h+\n/
       Diff:
       @@ -1,2 +1,2 @@
       -/\A#<Azure::Armrest::BaseModel:0x\h+\n/
       +#<Azure::Armrest::BaseModel:0x03b95f98 name="test", array=["stuff"], age=33>

     # ./spec/models/base_model_spec.rb:146:in `block (3 levels) in <top (required)>'

  3) BaseModel #attr_from_hash maps it source location to the correct file
     Failure/Error: expect(subject.method(:first_name_from_hash).source_location).to include(__FILE__)
       expected ["C", 0] to include "C:/Users/Daniel/Dev/azure-armrest-djberg96/spec/models/base_model_spec.rb"
     # ./spec/models/base_model_spec.rb:291:in `block (3 levels) in <top (required)>'
     # ./spec/models/base_model_spec.rb:279:in `block (3 levels) in <top (required)>'

  4) BaseModel #attr_from_hash with multiple attributes maps it source location of each method to the correct file
     Failure/Error: expect(subject.method(:first_name_from_hash).source_location).to include(__FILE__)
       expected ["C", 0] to include "C:/Users/Daniel/Dev/azure-armrest-djberg96/spec/models/base_model_spec.rb"
     # ./spec/models/base_model_spec.rb:313:in `block (4 levels) in <top (required)>'
     # ./spec/models/base_model_spec.rb:301:in `block (4 levels) in <top (required)>'
     # ./spec/models/base_model_spec.rb:279:in `block (3 levels) in <top (required)>'

  5) BaseModel #attr_from_hash with nested attributes maps it source location to the correct file
     Failure/Error: expect(subject.method(:street_address_from_hash).source_location).to include(__FILE__)
       expected ["C", 0] to include "C:/Users/Daniel/Dev/azure-armrest-djberg96/spec/models/base_model_spec.rb"
     # ./spec/models/base_model_spec.rb:335:in `block (4 levels) in <top (required)>'
     # ./spec/models/base_model_spec.rb:324:in `block (4 levels) in <top (required)>'
     # ./spec/models/base_model_spec.rb:279:in `block (3 levels) in <top (required)>'

Finished in 1.78 seconds (files took 0.7332 seconds to load)
514 examples, 5 failures

Failed examples:

rspec ./spec/configuration_spec.rb:182 # Azure::Armrest::Configuration instances tokens token generation caches the token to be reused for the same client
rspec ./spec/models/base_model_spec.rb:142 # BaseModel inspection methods defines a pretty_print method when pp is available
rspec ./spec/models/base_model_spec.rb:290 # BaseModel #attr_from_hash maps it source location to the correct file
rspec ./spec/models/base_model_spec.rb:312 # BaseModel #attr_from_hash with multiple attributes maps it source location of each method to the correct file
rspec ./spec/models/base_model_spec.rb:334 # BaseModel #attr_from_hash with nested attributes maps it source location to the correct file

@NickLaMuro Looks like 3, 4 and 5 are yours.

VirtualMachineImageService#list_all is very slow

As the title says, VirtualMachineImageService#list_all is very slow. It's a quadruple nested loop, starting with publishers, then offers, then skus, then versions. The problem is that the number of publishers is now over 500, so it's taking hundreds of requests and lots of time to complete.

In the current version we need to parallel these loops, or at least the outer loop, to improve performance. If we switch to the excon library, we should pipeline these requests using an approach like this:

https://gist.github.com/djberg96/f44d2cb7b930c2898ce7a8466bd54956

cannot create storage account

Failed to create a storage account using StorageAccountService

First, the default options {} is useless because it will fail later on.
Second, NoMethodError: undefined method 'proxy=' for nil:NilClass

next_marker_results is busted

There's some kind of internal bug with regards to handling of nextmarker results. To duplicate, you can try this:

sas = Azure::Armrest::StorageAccountService.new(conf)

# Pick a storage account that has a share with 2 or more files.
storage_acct_name = 'foo'
resource_group  = 'bar'

acct = sas.get(storage_acct_name, resource_group)

keys = sas.list_account_keys(acct.name, acct.resource_group)
key  = keys['key1'] || keys['key2']

files = acct.files(your_share, key, :maxresults => 1)

Inspection reveals that the nextmarker is being appended to the base URL instead of retained as part of the query.

Switch from rest-client to excon

At the moment rest-client does not support persistent connections. It does not support pipelining (multiple HTTP requests sent on a single TCP connection) either as far as I can tell. The result is that we are generated many more http connections than we really need, which costs both time and memory. On top of that, rest-client has many open issues and PR's, leading me to believe that it is not actively maintained, and cannot be relied upon going forward.

After poking around with httpclient, oauth2, faraday and others, I've decided to move forward with excon. It supports both persistent connections and pipelining, and is actively maintained. The 0.59 release was put out just 5 days ago from the time I wrote this.

The plan is to create a single persistent connection for each Configuration object. That will be used for most requests. The StorageAccount model will require some special handling, as each unmanaged storage account has 3 endpoints (file, blob, table) and uses a separate token, so I'm thinking we create 3 persistent connections per storage account object, and re-use those as needed.

Generate key for StorageAccount operations if none provided

At the moment we require a storage account key for various storage account operations. It appears that Azure will alternatively accept an SAS token. Update the StorageAccount model to generate one automatically if a key isn't provided.

Add a RoleManager class

Let's wrap these:

https://msdn.microsoft.com/en-us/library/azure/dn906885.aspx

Add username + password authentication for service principals

At the moment we require a client key, client ID and tenant ID for an application with access to the subscription in order to use the gem.

We should also allow authentication with just a username (service principal) and password, which is what the CLI allows with "azure login".

This should be pure Ruby, and only be implemented if no UI interaction and/or external web server is required.

Refactor request methods into a module

Right now we have two sets of request methods - one at the singleton level, one at the instance level. This is unnecessary duplication, originally caused by storage account handling. The result is that it's sometimes easy to make mistakes when modifying this code.

These should be reworked into a single RequestHelper module that work for both regular service classes and storage accounts (plus anything else).

URL containing special characters

Issue #183 and PR #184 seems not solving the issue with URL containing special characters. The encoding attempt only stops RestClient from complaining about the format of the URL, but the actual request using the encoded URL does not get back properly.

A further investigation may require a conversation with Microsoft concerning the format of URL if it contains special character (in the case of https://bugzilla.redhat.com/show_bug.cgi?id=1346034, { and }).

Add a Configuration#validate method

At the moment, the Configuration class does not provide a way to validate credentials before they're used.

We should add a Configuration#validate (or valid?) method that makes some sort of simple REST call to ensure the credentials are actually good.

Use thread pooling for multi-threaded http requests

Jason Frey suggested that we use a thread pool of some sort to manage our threaded requests, both to manage the overall number of threads, and also to reduce the potential possibility of getting blocked by MS for making too many requests at once.

The candidates suggested by Jason are:

Whichever we choose, we need to decide what a good number is for the default (8?) and whether this should be a global setting, or a per-service class setting. Either way, I think this should be relegated to the 0.3.0 release.

@bzwei @bronaghs @roliveri @blomquisg @Fryguy - Feedback welcome.

Nav collapses after clicking on tertiary item

The Nav should remain fully expanded after clicking on a tertiary item and should close only when hovering off the nav.

Add Graph interface

The "graph" interface allows you to perform operations on users. This could be useful for creating new service principals, for example.

https://msdn.microsoft.com/en-us/library/azure/ad/graph/api/api-catalog

Add a resource_group property to TemplateDeployment model objects

Although deployments are compartmentalized by resource group, the properties for TemplateDeployment model objects returned by the REST API do not include a resourceGroup property.

Would it be alright to add one? We could parse it out of the ID field.

Overall this would make life a bit easier for MIQ inventory collection.

StorageAccountService#list_all_private_images results in Errno::ECONNREFUSED

On rare occasions I'm hitting an ECONNREFUSED error when attempting to use the list_all_private_images method. This ultimately comes from the StorageAccount#blobs method. I am not sure if this is an issue with the storage endpoint, or an issue with the Parallel gem and http requests.

https://github.com/ManageIQ/azure-armrest/blob/master/lib/azure/armrest/model/storage_account.rb#L277

/Users/djberge/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http.rb:906:in `rescue in block in connect': Failed to open TCP connection to foo.blob.core.windows.net:443 (Connection refused - connect(2) for "foo.blob.core.windows.net" port 443) (Errno::ECONNREFUSED)
	from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http.rb:903:in `block in connect'
	from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/2.4.0/timeout.rb:93:in `block in timeout'
	from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/2.4.0/timeout.rb:103:in `timeout'
	from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http.rb:902:in `connect'
	from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http.rb:887:in `do_start'
	from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http.rb:876:in `start'
	from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit'
	from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'
	from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/armrest_service.rb:200:in `rest_execute'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/armrest_service.rb:206:in `rest_get'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/storage_account.rb:557:in `blob_response'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/storage_account.rb:257:in `blobs'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/storage_account.rb:277:in `block (2 levels) in all_blobs'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/storage_account.rb:277:in `synchronize'
	from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/storage_account.rb:277:in `block in all_blobs'
	from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/parallel-1.9.0/lib/parallel.rb:453:in `call_with_index'
	from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/parallel-1.9.0/lib/parallel.rb:311:in `block (2 levels) in work_in_threads'
	from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/parallel-1.9.0/lib/parallel.rb:462:in `with_instrumentation'
	from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/parallel-1.9.0/lib/parallel.rb:310:in `block in work_in_threads'
	from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/parallel-1.9.0/lib/parallel.rb:192:in `block (2 levels) in in_threads'

VirtualMachineImageService is broken

The service_name is not set.

ResourceProviderService caching busted

I'm hitting this against master. I seem to recall that we hit this before a while back @bzwei. Looks like it's returned.

/Users/dberger/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/cache_method-0.2.7/lib/cache_method.rb:56:in dump': singleton can't be dumped (TypeError)
from /Users/dberger/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/cache_method-0.2.7/lib/cache_method.rb:56:in digest' from /Users/dberger/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/cache_method-0.2.7/lib/cache_method/cached_result.rb:55:in cache_key'
from /Users/dberger/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/cache_method-0.2.7/lib/cache_method/cached_result.rb:66:in get_wrapped' from /Users/dberger/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/cache_method-0.2.7/lib/cache_method/cached_result.rb:26:in fetch'
from /Users/dberger/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/cache_method-0.2.7/lib/cache_method.rb:99:in block in cache_method' from /Users/dberger/Repositories/azure-armrest-djberg96/lib/azure/armrest/resource_provider_service.rb:60:in list_all'
from providers_test.rb:12:in <main>'

Code reorganization

After my small break and looking at this code again, I'm wondering if we should separate out some of the integrated methods into their own manager classes, such as:

SubscriptionManager
ResourceManager
LinkedResourceManager
ResourceGroupManager
ResourceProviderManager
TenantManager
TagManager
TemplateDeploymentManager

Then use those classes internally for various functions.

What do you think?

All url sent by RestClient should be encoded

This is a follow-up with #182.
All url sent by RestClient should be encoded.

pretty_print failure on 2.3.x

ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15]

I'm not sure why it's not matching in 2.3.x, but here's the failure I'm seeing:

1) BaseModel inspection methods defines a pretty_print method when pp is available
     Failure/Error: expect(base.pretty_inspect).to match(expected)

       expected "#<Azure::Armrest::BaseModel:0x007f8fa35a96e8\n name=\"test\",\n array=[\"stuff\"],\n age=33>\n" to match /^#<Azure::Armrest::BaseModel:0x\h+\n name="test",\n age=33,\n array=\["stuff"\]>$/
       Diff:
       @@ -1,2 +1,5 @@
       -/^#<Azure::Armrest::BaseModel:0x\h+\n name="test",\n age=33,\n array=\["stuff"\]>$/
       +#<Azure::Armrest::BaseModel:0x007f8fa35a96e8
       + name="test",
       + array=["stuff"],
       + age=33>

     # ./spec/models/base_model_spec.rb:124:in `block (3 levels) in <top (required)>'

Finished in 0.86943 seconds (files took 0.72225 seconds to load)
305 examples, 1 failure

Deal with $skip tokens in a uniform manner

The recent events stuff has forced us to reconsider how we're wrapping some of the results for various list methods. The problem, in general, is that if you specify a filter of any sort that limits the results, the user may want to get more results from that point on, but there's no way to do that at the moment. This is because the 'nextLink' attribute is not included in the results that we parse.

The solution for Insight::Events that we use, for the moment anyway, is to return an EventList object. It contains the list results (model objects) as one attribute, and the skip token as another.

A couple of ideas to deal with this are:

Always return an XList object for all list and list_all methods.

Advantages: It's a clean design. This would make things consistent across the board.
Disadvantages: Causes major breakage. Lots of code would have to be updated for people already using the gem.

Return an XList object only if a skip token is found. Otherwise, return a regular list of model objects as we do now.

Advantages: Wouldn't cause breakage for most methods since the 200 limit that applies to events doesn't seem to apply to other resources. Only a filter supplied by the users would generate a skip token.

Disadvantages: Returning different objects based on context is gross. And we couldn't guarantee there wouldn't be any surprises, because it's possible other resources have a default limit that we don't know about.

Just add the skip token property on ALL model objects instead of returning a new type of object.

Advantages: This requires minimal change to the code base, and causes no breakage.

Disadvantages: Lots of duplication in the model objects, since it would be identical for all model objects in a particular result set.

Personally I'm leaning towards the last option at the moment, but am open to suggestions.

Add proxy support

At the moment there's no way to specify a proxy for http requests. We should add that.

manageiq / azure-armrest Goto Github PK

azure-armrest's Introduction

Description

Synopsis

Subscriptions

Notes

License

Authors

azure-armrest's People

Contributors

Stargazers

Watchers

Forkers

azure-armrest's Issues

Recommend Projects

Recommend Topics

Recommend Org