manageiq / azure-armrest Goto Github PK
View Code? Open in Web Editor NEWRuby interface for Azure using the new REST API
License: Apache License 2.0
Ruby interface for Azure using the new REST API
License: Apache License 2.0
At the moment neither the list
nor the list_all
methods paginate, which would result in truncated results. They need to handle skip tokens.
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15]
I'm not sure why it's not matching in 2.3.x, but here's the failure I'm seeing:
1) BaseModel inspection methods defines a pretty_print method when pp is available
Failure/Error: expect(base.pretty_inspect).to match(expected)
expected "#<Azure::Armrest::BaseModel:0x007f8fa35a96e8\n name=\"test\",\n array=[\"stuff\"],\n age=33>\n" to match /^#<Azure::Armrest::BaseModel:0x\h+\n name="test",\n age=33,\n array=\["stuff"\]>$/
Diff:
@@ -1,2 +1,5 @@
-/^#<Azure::Armrest::BaseModel:0x\h+\n name="test",\n age=33,\n array=\["stuff"\]>$/
+#<Azure::Armrest::BaseModel:0x007f8fa35a96e8
+ name="test",
+ array=["stuff"],
+ age=33>
# ./spec/models/base_model_spec.rb:124:in `block (3 levels) in <top (required)>'
Finished in 0.86943 seconds (files took 0.72225 seconds to load)
305 examples, 1 failure
Looks like the VirtualMachineMethod#series method is broken after the last PR:
vms = Azure::Armrest::VirtualMachineService.new(conf)
vms.series
Result:
azure-armrest-djberg96/lib/azure/armrest/virtual_machine_service.rb:24:in `series': Invalid provider 'Microsoft.Compute' (ArgumentError)
from providers_test.rb:26:in `<main>'
We'll need to modify the Storage::DiskService
class in order to support SSA for ManageIQ for VM's that use managed disks. The Azure REST API provides separate a separate method for getting raw blob information on managed disks:
https://docs.microsoft.com/en-us/rest/api/manageddisks/disks/disks-grant-access
It's a 3 step process. First, you have to get the location URL from the response header, then you have to get the URL that includes the SAS token, and finally you make a request with that URL to get the blob information.
At the moment, the Configuration class does not provide a way to validate credentials before they're used.
We should add a Configuration#validate (or valid?) method that makes some sort of simple REST call to ensure the credentials are actually good.
Failures:
1) Azure::Armrest::Configuration singletons logging accepts a file name for a log
Failure/Error: after { File.delete(log) if File.exist?(log) }
Errno::EACCES:
Permission denied @ unlink_internal - azure-armrest.log
# ./spec/configuration_spec.rb:27:in `delete'
# ./spec/configuration_spec.rb:27:in `block (2 levels) in <top (required)>'
2) Azure::Armrest::Configuration singletons logging accepts a file handle for a log
Failure/Error: after { File.delete(log) if File.exist?(log) }
Errno::EACCES:
Permission denied @ unlink_internal - azure-armrest.log
# ./spec/configuration_spec.rb:27:in `delete'
# ./spec/configuration_spec.rb:27:in `block (2 levels) in <top (required)>'
3) Azure::Armrest::Configuration singletons logging accepts a Logger instance
Failure/Error: after { File.delete(log) if File.exist?(log) }
Errno::EACCES:
Permission denied @ unlink_internal - azure-armrest.log
# ./spec/configuration_spec.rb:27:in `delete'
# ./spec/configuration_spec.rb:27:in `block (2 levels) in <top (required)>'
Finished in 2.55 seconds (files took 4.32 seconds to load)
418 examples, 3 failures
Failed examples:
rspec ./spec/configuration_spec.rb:243 # Azure::Armrest::Configuration singletons logging accepts a file name for a log
rspec ./spec/configuration_spec.rb:248 # Azure::Armrest::Configuration singletons logging accepts a file handle for a log
rspec ./spec/configuration_spec.rb:255 # Azure::Armrest::Configuration singletons logging accepts a Logger instance
Not sure yet if this is an issue with our log handling, the specs, or the rest-client gem.
As talked about here I'm opening an RFE for marketplace image support. From what I can see this will require:
prepare_for_clone_task
to parse and accept marketplace URNsAs the title says, VirtualMachineImageService#list_all
is very slow. It's a quadruple nested loop, starting with publishers, then offers, then skus, then versions. The problem is that the number of publishers is now over 500, so it's taking hundreds of requests and lots of time to complete.
In the current version we need to parallel these loops, or at least the outer loop, to improve performance. If we switch to the excon library, we should pipeline these requests using an approach like this:
https://gist.github.com/djberg96/f44d2cb7b930c2898ce7a8466bd54956
Let's wrap these:
https://msdn.microsoft.com/en-us/library/azure/dn906885.aspx
Jason Frey suggested that we use a thread pool of some sort to manage our threaded requests, both to manage the overall number of threads, and also to reduce the potential possibility of getting blocked by MS for making too many requests at once.
The candidates suggested by Jason are:
Whichever we choose, we need to decide what a good number is for the default (8?) and whether this should be a global setting, or a per-service class setting. Either way, I think this should be relegated to the 0.3.0 release.
@bzwei @bronaghs @roliveri @blomquisg @Fryguy - Feedback welcome.
For example, if I want to get a list of network interfaces for a virtual machine I could do this:
vm.properties.network_profile.network_interfaces.each{ |n| ... }
The problem is that the objects returned by that only have the :id attribute because that's what the JSON has. However, we have NetworkInterface model objects as well.
So, instead of just returning a simple object with an :id field, provide a generic method to call that returns a real model object if we have one that wraps it. The :id field should still exist, too, in case users want to use the original value. So, something like:
vm.properties.network_profile.network_interfaces.each{ |n| n.model ... }
This is a follow-up with #182.
All url sent by RestClient should be encoded.
Windows 8
ruby 2.4.2p198 (2017-09-14 revision 59899) [i386-mingw32]
While not a high priority, we shouldn't be seeing test failures on any platform since this is a platform neutral library. I suspect these are issues with our specs rather than a library issue, but thought I should mention it.
Failures:
1) Azure::Armrest::Configuration instances tokens token generation caches the token to be reused for the same client
Failure/Error: expect(subject.token).to eql(token)
expected: "Bearer eyJ0eXAiOiJKV1Q"
got: "xxx"
(compared using eql?)
# ./spec/configuration_spec.rb:185:in `block (5 levels) in <top (required)>'
2) BaseModel inspection methods defines a pretty_print method when pp is available
Failure/Error: expect(base.pretty_inspect).to match(expected)
expected "#<Azure::Armrest::BaseModel:0x03b95f98 name=\"test\", array=[\"stuff\"], age=33>\n" to match /\A#<Azure
::Armrest::BaseModel:0x\h+\n/
Diff:
@@ -1,2 +1,2 @@
-/\A#<Azure::Armrest::BaseModel:0x\h+\n/
+#<Azure::Armrest::BaseModel:0x03b95f98 name="test", array=["stuff"], age=33>
# ./spec/models/base_model_spec.rb:146:in `block (3 levels) in <top (required)>'
3) BaseModel #attr_from_hash maps it source location to the correct file
Failure/Error: expect(subject.method(:first_name_from_hash).source_location).to include(__FILE__)
expected ["C", 0] to include "C:/Users/Daniel/Dev/azure-armrest-djberg96/spec/models/base_model_spec.rb"
# ./spec/models/base_model_spec.rb:291:in `block (3 levels) in <top (required)>'
# ./spec/models/base_model_spec.rb:279:in `block (3 levels) in <top (required)>'
4) BaseModel #attr_from_hash with multiple attributes maps it source location of each method to the correct file
Failure/Error: expect(subject.method(:first_name_from_hash).source_location).to include(__FILE__)
expected ["C", 0] to include "C:/Users/Daniel/Dev/azure-armrest-djberg96/spec/models/base_model_spec.rb"
# ./spec/models/base_model_spec.rb:313:in `block (4 levels) in <top (required)>'
# ./spec/models/base_model_spec.rb:301:in `block (4 levels) in <top (required)>'
# ./spec/models/base_model_spec.rb:279:in `block (3 levels) in <top (required)>'
5) BaseModel #attr_from_hash with nested attributes maps it source location to the correct file
Failure/Error: expect(subject.method(:street_address_from_hash).source_location).to include(__FILE__)
expected ["C", 0] to include "C:/Users/Daniel/Dev/azure-armrest-djberg96/spec/models/base_model_spec.rb"
# ./spec/models/base_model_spec.rb:335:in `block (4 levels) in <top (required)>'
# ./spec/models/base_model_spec.rb:324:in `block (4 levels) in <top (required)>'
# ./spec/models/base_model_spec.rb:279:in `block (3 levels) in <top (required)>'
Finished in 1.78 seconds (files took 0.7332 seconds to load)
514 examples, 5 failures
Failed examples:
rspec ./spec/configuration_spec.rb:182 # Azure::Armrest::Configuration instances tokens token generation caches the token to be reused for the same client
rspec ./spec/models/base_model_spec.rb:142 # BaseModel inspection methods defines a pretty_print method when pp is available
rspec ./spec/models/base_model_spec.rb:290 # BaseModel #attr_from_hash maps it source location to the correct file
rspec ./spec/models/base_model_spec.rb:312 # BaseModel #attr_from_hash with multiple attributes maps it source location of each method to the correct file
rspec ./spec/models/base_model_spec.rb:334 # BaseModel #attr_from_hash with nested attributes maps it source location to the correct file
@NickLaMuro Looks like 3, 4 and 5 are yours.
At the moment it seems that an :api_version option passed to the ArmrestService.configure method is overridden:
conf = Azure::Armrest::ArmrestService.configure(
:tenant_id => "xxx",
:client_id => "yyy",
:client_key => "zzz",
:api_version => '2015-01-01'
)
vms = Azure::Armrest::VirtualMachineService.new(conf)
p vms.api_version # => 2015-06-15
The workaround is to set it after the constructor:
vms.api_version = '2015-01-01'
We do have to be careful not to override the hard coded values we have for storage accounts and template deployments, too.
I hit this bug in our current dev environment. You can reproduce it like so:
tds = Azure::Armrest::TemplateDeploymentService.new(conf)
tds.get_template("gswmonz1319", "gswmonz1319")
Result:
/Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:166:in `const_defined?': wrong constant name Openshift webconsole (NameError)
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:166:in `nested_object'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:157:in `block in __setobj__'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:149:in `each'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:149:in `__setobj__'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:67:in `initialize'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:170:in `new'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:170:in `nested_object'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:157:in `block in __setobj__'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:149:in `each'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:149:in `__setobj__'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/base_model.rb:67:in `initialize'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/template_deployment_service.rb:50:in `new'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/template_deployment_service.rb:50:in `get_template'
from futz/template.rb:14:in `block in <main>'
from futz/template.rb:10:in `each'
from futz/template.rb:10:in `<main>'
I'm guessing it's because of the space in the name between "Openshift" and "web console".
Possibly the cause of: https://bugzilla.redhat.com/show_bug.cgi?id=1434988
Windows 8
ruby 2.4.2p198 (2017-09-14 revision 59899) [i386-mingw32]
While not a high priority, this library should be platform neutral, so we shouldn't be seeing failures on any platform. I suspect this is just the way we've written our tests rather than a core library issue, but thought I should post.
Failures:
1) Azure::Armrest::Configuration instances tokens token generation caches the token to be reused for the same cl
Failure/Error: expect(subject.token).to eql(token)
expected: "Bearer eyJ0eXAiOiJKV1Q"
got: "xxx"
(compared using eql?)
# ./spec/configuration_spec.rb:185:in `block (5 levels) in <top (required)>'
2) BaseModel inspection methods defines a pretty_print method when pp is available
Failure/Error: expect(base.pretty_inspect).to match(expected)
expected "#<Azure::Armrest::BaseModel:0x03b95f98 name=\"test\", array=[\"stuff\"], age=33>\n" to match /\A#
::Armrest::BaseModel:0x\h+\n/
Diff:
@@ -1,2 +1,2 @@
-/\A#<Azure::Armrest::BaseModel:0x\h+\n/
+#<Azure::Armrest::BaseModel:0x03b95f98 name="test", array=["stuff"], age=33>
# ./spec/models/base_model_spec.rb:146:in `block (3 levels) in <top (required)>'
3) BaseModel #attr_from_hash maps it source location to the correct file
Failure/Error: expect(subject.method(:first_name_from_hash).source_location).to include(__FILE__)
expected ["C", 0] to include "C:/Users/Daniel/Dev/azure-armrest-djberg96/spec/models/base_model_spec.rb"
# ./spec/models/base_model_spec.rb:291:in `block (3 levels) in <top (required)>'
# ./spec/models/base_model_spec.rb:279:in `block (3 levels) in <top (required)>'
4) BaseModel #attr_from_hash with multiple attributes maps it source location of each method to the correct file
Failure/Error: expect(subject.method(:first_name_from_hash).source_location).to include(__FILE__)
expected ["C", 0] to include "C:/Users/Daniel/Dev/azure-armrest-djberg96/spec/models/base_model_spec.rb"
# ./spec/models/base_model_spec.rb:313:in `block (4 levels) in <top (required)>'
# ./spec/models/base_model_spec.rb:301:in `block (4 levels) in <top (required)>'
# ./spec/models/base_model_spec.rb:279:in `block (3 levels) in <top (required)>'
5) BaseModel #attr_from_hash with nested attributes maps it source location to the correct file
Failure/Error: expect(subject.method(:street_address_from_hash).source_location).to include(__FILE__)
expected ["C", 0] to include "C:/Users/Daniel/Dev/azure-armrest-djberg96/spec/models/base_model_spec.rb"
# ./spec/models/base_model_spec.rb:335:in `block (4 levels) in <top (required)>'
# ./spec/models/base_model_spec.rb:324:in `block (4 levels) in <top (required)>'
# ./spec/models/base_model_spec.rb:279:in `block (3 levels) in <top (required)>'
@NickLaMuro I think 3, 4 and 5 are yours. :)
At the moment if you try to call delete_associated_resources
on a VM that uses managed storage, the storage deletion will fail. This method was originally written for unmanaged storage and was never updated.
undefined method 'vhd' for #<Azure::Armrest::VirtualMachineModel::Properties::StorageProfile::OsDisk:0x007fa0704c8cf0> Method:[block in method_missing]
[----] E, [2017-11-20T15:25:51.764838 #35394:3fd03f83f7e4] ERROR -- : /Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/azure-armrest-0.9.3/lib/azure/armrest/storage_account_service.rb:197:in `get_from_vm'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/azure-armrest-0.9.3/lib/azure/armrest/virtual_machine_service.rb:221:in `delete_associated_disk'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/azure-armrest-0.9.3/lib/azure/armrest/virtual_machine_service.rb:165:in `delete_associated_resources'
After my small break and looking at this code again, I'm wondering if we should separate out some of the integrated methods into their own manager classes, such as:
Then use those classes internally for various functions.
What do you think?
Right now we have two sets of request methods - one at the singleton level, one at the instance level. This is unnecessary duplication, originally caused by storage account handling. The result is that it's sometimes easy to make mistakes when modifying this code.
These should be reworked into a single RequestHelper module that work for both regular service classes and storage accounts (plus anything else).
The StorageAccount#blobs method does a blob list for a given container. The response body actually returns XML (instead of JSON), and this body has typically been anywhere from 1k to 15k in practice. Over time they have added more fields to the default output, e.g. ServerEncrypted, AccessTier, etc, and this has resulted in even larger text bodies, which in turn consumes more memory.
The net result is that the StorageAccount#list_all_private_images is consuming huge amounts of memory as it iterates over each container on each storage account.
https://docs.microsoft.com/en-us/rest/api/storageservices/list-blobs
It seems we have a conflict with the recent refactoring and the cache_method gem.
rps = Azure::Armrest::ResourceProviderService.new(conf)
rp = rps.get('Microsoft.Compute')
Result:
/Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.4/lib/active_support/cache.rb:667:in `dump': can't dump anonymous class #<Class:0x007fe47790ad90> (TypeError)
from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.4/lib/active_support/cache.rb:667:in `dup_value!'
from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.4/lib/active_support/cache/memory_store.rb:146:in `write_entry'
from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.4/lib/active_support/cache.rb:391:in `block in write'
from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.4/lib/active_support/cache.rb:547:in `block in instrument'
from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.4/lib/active_support/notifications.rb:166:in `instrument'
from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.4/lib/active_support/cache.rb:547:in `instrument'
from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.4/lib/active_support/cache.rb:389:in `write'
from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/cache-0.4.1/lib/cache/active_support_cache_store.rb:21:in `_set'
from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/cache-0.4.1/lib/cache.rb:77:in `set'
from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/cache_method-0.2.7/lib/cache_method/cached_result.rb:72:in `set_wrapped'
from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/cache_method-0.2.7/lib/cache_method/cached_result.rb:32:in `fetch'
from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/cache_method-0.2.7/lib/cache_method.rb:99:in `block in cache_method'
from test.rb:178:in `<main>'
At the moment we require a storage account key for various storage account operations. It appears that Azure will alternatively accept an SAS token. Update the StorageAccount model to generate one automatically if a key isn't provided.
The @@providers_hash was originally implemented as a quick and dirty caching mechanism. However, I always intended that it be replaced by the ResourceProviderService class since it uses cache_method to cache most of its calls.
Using the ResourceProviderService class is a cleaner approach, and I suspect it will also deal with what I think are some class variable scoping issues with JRuby for the test suite.
However, #98 will need to be addressed first.
The delegated :list_providers method in Azure::Armrest::ArmrestService (on master) doesn't appear to work:
conf = Azure::Armrest::Configuration.new(creds)
vms = Azure::Armrest::VirtualMachineService.new(conf)
p vms.list_providers
Result:
/Users/dberger/Repositories/azure-armrest-djberg96/lib/azure/armrest/armrest_service.rb:50:in `list_providers': undefined method `list_providers' for #<Azure::Armrest::Configuration:0x007fad6cd3dbe8> (NoMethodError)
from locations.rb:11:in `<main>'
I also tried the alias :providers, but that also failed:
>ruby locations.rb
NOTE: Azure::Armrest::VirtualMachineService#providers is deprecated; use list_providers instead. It will be removed on or after 2018-01-01.
Azure::Armrest::VirtualMachineService#providers called from locations.rb:11.
/Users/dberger/Repositories/azure-armrest-djberg96/lib/azure/armrest/armrest_service.rb:50:in `list_providers': undefined method `list_providers' for #<Azure::Armrest::Configuration:0x007f8bbb758200> (NoMethodError)
from /Users/dberger/.rbenv/versions/2.2.3/lib/ruby/site_ruby/2.2.0/rubygems/deprecate.rb:63:in `block (2 levels) in deprecate'
from locations.rb:11:in `<main>'
There's some kind of internal bug with regards to handling of nextmarker results. To duplicate, you can try this:
sas = Azure::Armrest::StorageAccountService.new(conf)
# Pick a storage account that has a share with 2 or more files.
storage_acct_name = 'foo'
resource_group = 'bar'
acct = sas.get(storage_acct_name, resource_group)
keys = sas.list_account_keys(acct.name, acct.resource_group)
key = keys['key1'] || keys['key2']
files = acct.files(your_share, key, :maxresults => 1)
Inspection reveals that the nextmarker is being appended to the base URL instead of retained as part of the query.
At the moment rest-client does not support persistent connections. It does not support pipelining (multiple HTTP requests sent on a single TCP connection) either as far as I can tell. The result is that we are generated many more http connections than we really need, which costs both time and memory. On top of that, rest-client has many open issues and PR's, leading me to believe that it is not actively maintained, and cannot be relied upon going forward.
After poking around with httpclient, oauth2, faraday and others, I've decided to move forward with excon. It supports both persistent connections and pipelining, and is actively maintained. The 0.59 release was put out just 5 days ago from the time I wrote this.
The plan is to create a single persistent connection for each Configuration object. That will be used for most requests. The StorageAccount model will require some special handling, as each unmanaged storage account has 3 endpoints (file, blob, table) and uses a separate token, so I'm thinking we create 3 persistent connections per storage account object, and re-use those as needed.
Instead of hard-coding URLs in the Environment, we should be able to auto-discover them given only the resource manager URL using a GET to <ARM_URL>/metadata/endpoints?api-version=1.0
This gives us details on the gallery, graph, and portal URLs, as well as the active_directory_authority
and active_directory_resource_id
This will be helpful for AzureStack deployments where the underlying URLs can't be hard-coded. When adding the provider, all the admin would need to provide is the resource manager URL and we could take care of the rest.
See an example in Ruby here (search for def get_active_directory_settings(armEndpoint)
):
https://docs.microsoft.com/en-us/azure/azure-stack/user/azure-stack-version-profiles-ruby
JSON returned by an azure stack deployment:
{
"galleryEndpoint": "https://portal.westus.stackpoc.com:30015/",
"graphEndpoint": "https://graph.windows.net/",
"portalEndpoint": "https://portal.westus.stackpoc.com/",
"authentication": {
"loginEndpoint": "https://login.windows.net/",
"audiences": ["https://management.poc.avahc.com/01a7976d-5a10-475d-b39d-01993f9d7d90"]
}
}
This same API is working on the public cloud too. Example JSON:
{
"galleryEndpoint": "https://gallery.azure.com/",
"graphEndpoint": "https://graph.windows.net/",
"portalEndpoint": "https://portal.azure.com/",
"authentication": {
"loginEndpoint": "https://login.windows.net/",
"audiences": ["https://management.core.windows.net/", "https://management.azure.com/"]
}
}
The Nav should remain fully expanded after clicking on a tertiary item and should close only when hovering off the nav.
At the moment we require a client key, client ID and tenant ID for an application with access to the subscription in order to use the gem.
We should also allow authentication with just a username (service principal) and password, which is what the CLI allows with "azure login".
This should be pure Ruby, and only be implemented if no UI interaction and/or external web server is required.
Passing the :all
option to the StorageAccount#table_data
method appears to result in an infinite loop.
Edit: maybe.
At the moment there's no way to specify a proxy for http requests. We should add that.
Currently we basically ignore the response headers. However, they may contain useful information:
{:cache_control=>"no-cache", :pragma=>"no-cache", :content_type=>"application/json", :content_encoding=>"gzip", :expires=>"-1", :vary=>"Accept-Encoding", :x_ms_request_id=>"a7e0e1ad-6f3a-4eeb-ace8-9c77c3377179", :server=>"Microsoft-Azure-Storage-Resource-Provider/1.0, Microsoft-HTTPAPI/2.0", :x_ms_ratelimit_remaining_subscription_reads=>"14872", :x_ms_correlation_request_id=>"a7e0e1ad-6f3a-4eeb-ace8-9c77c3377179", :x_ms_routing_request_id=>"NORTHCENTRALUS:20160601T182659Z:a7e0e1ad-6f3a-4eeb-ace8-9c77c3377179", :strict_transport_security=>"max-age=31536000; includeSubDomains", :date=>"Wed, 01 Jun 2016 18:26:59 GMT", :connection=>"close"}
Notably, the :x_ms_ratelimit_remaining_subscription_reads
would be useful to know, especially if we get close to zero.
I'm not sure how these should be stored, though. Storing header information in the model objects would be wasteful since it would be identical in every object. Perhaps a singleton that stores the header information? I'm open to suggestions.
If you try to convert an ArmrestCollection object to JSON using the to_json
method, you get an error:
vms.list_all.to_json
Result:
lib/azure/armrest/model/base_model.rb:84:in `to_json': wrong number of arguments (given 1, expected 0) (ArgumentError)
This should "just work".
Upon further review I don't think these should have been made subservices. In any case, right now they don't actually work.
sns = Azure::Armrest::Network::SubnetService.new(conf)
sns.list('some_group)
Results in:
lib/azure/armrest/resource_group_based_service.rb:56:in `validate_resource_group': must specify resource group (ArgumentError)
from /Users/dberger/Repositories/azure-armrest-djberg96/lib/azure/armrest/resource_group_based_subservice.rb:24:in `list'
Within rest_execute
we should check the error code before automatically raising an exception. On a 429 errors (TooManyRequests) we should sleep Retry-After
seconds and try again. On 50x errors (Azure flaked out) we should sleep a set number of seconds (say, 30) and retry again.
I think adding a :max_retries
accessor to the configuration object could be used internally.
It seems there is no networkInterfaces property for classic VM's:
vmm.provider = "Microsoft.ClassicCompute"
vmm.list
# Output
/Users/dberger/Repositories/azure-armrest/lib/azure/armrest/virtual_machine_manager.rb:132:in `block in add_network_profile': undefined method `each' for nil:NilClass (NoMethodError)
I realize Nokogiri is a superior XML parsing library, but the amount of actual XML that we deal with from Azure (all storage account related) is actually relatively small. Any speed/memory issues should be negligible in practice.
Removing Nokogiri eases the installation and maintenance of this library, and we won't have to worry about CVE's any more, either.
When some locations such as BrazilSouth or AustraliaEast are passed into the VMM.Series method they cause the follow error:
"https://management.azure.com/subscriptions/462f2af8-e67e-40c6-9fbf-02824d1dd485/providers/Microsoft.Compute/locations/AustraliaEast/vmSizes?api-version=2015-06-15"
/opt/rubies/ruby-2.0.0-p576/lib/ruby/gems/2.0.0/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in return!': 502 Bad Gateway (RestClient::BadGateway) from /opt/rubies/ruby-2.0.0-p576/lib/ruby/gems/2.0.0/gems/rest-client-1.8.0/lib/restclient/request.rb:495:in
process_result'
from /opt/rubies/ruby-2.0.0-p576/lib/ruby/gems/2.0.0/gems/rest-client-1.8.0/lib/restclient/request.rb:421:in block in transmit' from /opt/rubies/ruby-2.0.0-p576/lib/ruby/2.0.0/net/http.rb:852:in
start'
from /opt/rubies/ruby-2.0.0-p576/lib/ruby/gems/2.0.0/gems/rest-client-1.8.0/lib/restclient/request.rb:413:in transmit' from /opt/rubies/ruby-2.0.0-p576/lib/ruby/gems/2.0.0/gems/rest-client-1.8.0/lib/restclient/request.rb:176:in
execute'
from /opt/rubies/ruby-2.0.0-p576/lib/ruby/gems/2.0.0/gems/rest-client-1.8.0/lib/restclient/request.rb:41:in execute' from /opt/rubies/ruby-2.0.0-p576/lib/ruby/gems/2.0.0/gems/rest-client-1.8.0/lib/restclient.rb:65:in
get'
from /Users/bronaghsorota/dev/azure-armrest/lib/azure/armrest/armrest_manager.rb:356:in rest_get' from /Users/bronaghsorota/dev/azure-armrest/lib/azure/armrest/virtual_machine_manager.rb:58:in
series'
from ./july17test.rb:29:in block in <main>' from ./july17test.rb:27:in
each'
fr
have you considered moving your code to say fog as say fog/fog-azure-resource-manager so we can consolidate the cloud api efforts for azure?
At the moment we have a series of pre-defined exception classes in exception.rb. Unfortunately, this means that if RestClient raises an error that we don't define, users get a generic ApiException. This is not usually very helpful in debugging.
What I would like to see is some sort of code that automatically transforms the RestClient exceptions into Armrest exceptions in a more dynamic way so that we don't have to add them manually.
Azure does not delete associated resources when deleting a VM, such as IP address, NIC, attached disks, or even the .vhd (blob) for the VM itself. I think we should add a method that optionally deletes the VM and associated resources of the user's choice.
Maybe something like this:
vms.delete_associated(name, group) # With no options, delete everything
vms.delete_associated(name, group, :ip_address => true) # Just the associated IP address
The options I can think of right off are:
*Since virtual networks and storage accounts often have unrelated resources associated with them, they should have to be selected explicitly to attempt to delete them.
On rare occasions I'm hitting an ECONNREFUSED error when attempting to use the list_all_private_images
method. This ultimately comes from the StorageAccount#blobs
method. I am not sure if this is an issue with the storage endpoint, or an issue with the Parallel gem and http requests.
/Users/djberge/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http.rb:906:in `rescue in block in connect': Failed to open TCP connection to foo.blob.core.windows.net:443 (Connection refused - connect(2) for "foo.blob.core.windows.net" port 443) (Errno::ECONNREFUSED)
from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http.rb:903:in `block in connect'
from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/2.4.0/timeout.rb:93:in `block in timeout'
from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/2.4.0/timeout.rb:103:in `timeout'
from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http.rb:902:in `connect'
from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http.rb:887:in `do_start'
from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http.rb:876:in `start'
from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit'
from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'
from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/armrest_service.rb:200:in `rest_execute'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/armrest_service.rb:206:in `rest_get'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/storage_account.rb:557:in `blob_response'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/storage_account.rb:257:in `blobs'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/storage_account.rb:277:in `block (2 levels) in all_blobs'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/storage_account.rb:277:in `synchronize'
from /Users/djberge/Dev/azure-armrest-djberg96/lib/azure/armrest/model/storage_account.rb:277:in `block in all_blobs'
from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/parallel-1.9.0/lib/parallel.rb:453:in `call_with_index'
from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/parallel-1.9.0/lib/parallel.rb:311:in `block (2 levels) in work_in_threads'
from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/parallel-1.9.0/lib/parallel.rb:462:in `with_instrumentation'
from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/parallel-1.9.0/lib/parallel.rb:310:in `block in work_in_threads'
from /Users/djberge/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/parallel-1.9.0/lib/parallel.rb:192:in `block (2 levels) in in_threads'
I ran into an issue where I had set my provider to "Microsoft.Insights" instead of "microsoft.insights". Because it wasn't an exact match it returned the default api version, which it turns out didn't work.
I think the set_service_api_version and/or set_providers_info methods should be adjusted so that they ignore case, since MS ignores the provider's case as well.
The recent events stuff has forced us to reconsider how we're wrapping some of the results for various list methods. The problem, in general, is that if you specify a filter of any sort that limits the results, the user may want to get more results from that point on, but there's no way to do that at the moment. This is because the 'nextLink' attribute is not included in the results that we parse.
The solution for Insight::Events that we use, for the moment anyway, is to return an EventList object. It contains the list results (model objects) as one attribute, and the skip token as another.
A couple of ideas to deal with this are:
Advantages: It's a clean design. This would make things consistent across the board.
Disadvantages: Causes major breakage. Lots of code would have to be updated for people already using the gem.
Advantages: Wouldn't cause breakage for most methods since the 200 limit that applies to events doesn't seem to apply to other resources. Only a filter supplied by the users would generate a skip token.
Disadvantages: Returning different objects based on context is gross. And we couldn't guarantee there wouldn't be any surprises, because it's possible other resources have a default limit that we don't know about.
Advantages: This requires minimal change to the code base, and causes no breakage.
Disadvantages: Lots of duplication in the model objects, since it would be identical for all model objects in a particular result set.
Personally I'm leaning towards the last option at the moment, but am open to suggestions.
I'm hitting this against master. I seem to recall that we hit this before a while back @bzwei. Looks like it's returned.
/Users/dberger/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/cache_method-0.2.7/lib/cache_method.rb:56:in
dump': singleton can't be dumped (TypeError)
from /Users/dberger/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/cache_method-0.2.7/lib/cache_method.rb:56:in digest' from /Users/dberger/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/cache_method-0.2.7/lib/cache_method/cached_result.rb:55:in
cache_key'
from /Users/dberger/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/cache_method-0.2.7/lib/cache_method/cached_result.rb:66:in get_wrapped' from /Users/dberger/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/cache_method-0.2.7/lib/cache_method/cached_result.rb:26:in
fetch'
from /Users/dberger/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/cache_method-0.2.7/lib/cache_method.rb:99:in block in cache_method' from /Users/dberger/Repositories/azure-armrest-djberg96/lib/azure/armrest/resource_provider_service.rb:60:in
list_all'
from providers_test.rb:12:in <main>'
Although deployments are compartmentalized by resource group, the properties for TemplateDeployment model objects returned by the REST API do not include a resourceGroup property.
Would it be alright to add one? We could parse it out of the ID field.
Overall this would make life a bit easier for MIQ inventory collection.
Failed to create a storage account using StorageAccountService
First, the default options {}
is useless because it will fail later on.
Second, NoMethodError: undefined method 'proxy=' for nil:NilClass
The "graph" interface allows you to perform operations on users. This could be useful for creating new service principals, for example.
https://msdn.microsoft.com/en-us/library/azure/ad/graph/api/api-catalog
Issue #183 and PR #184 seems not solving the issue with URL containing special characters. The encoding attempt only stops RestClient from complaining about the format of the URL, but the actual request using the encoded URL does not get back properly.
A further investigation may require a conversation with Microsoft concerning the format of URL if it contains special character (in the case of https://bugzilla.redhat.com/show_bug.cgi?id=1346034, {
and }
).
The service_name is not set.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.