ow2-proactive / connector-iaas Goto Github PK

The Azure extensions mechanism allows to execute a script from a remote file (served through HTTP) in addition to a command.
The connector-iaas must include this feature to allow deeper VM customizations (eg. Java installation, creating custom firewall rules, etc.), such script files could then be shared via GitHub for example.

When we try to access resources in the region us-east-2 (Ohio) we get the following error:

requested location us-east-2, which is not in the configured locations: {eu-west-1=Suppliers.ofInstance(https://ec2.eu-west-1.amazonaws.com), ap-northeast-1=Suppliers.ofInstance(https://ec2.ap-northeast-1.amazonaws.com), sa-east-1=Suppliers.ofInstance(https://ec2.sa-east-1.amazonaws.com), ap-southeast-1=Suppliers.ofInstance(https://ec2.ap-southeast-1.amazonaws.com), ap-southeast-2=Suppliers.ofInstance(https://ec2.ap-southeast-2.amazonaws.com), eu-central-1=Suppliers.ofInstance(https://ec2.eu-central-1.amazonaws.com), us-east-1=Suppliers.ofInstance(https://ec2.us-east-1.amazonaws.com), us-west-1=Suppliers.ofInstance(https://ec2.us-west-1.amazonaws.com), us-west-2=Suppliers.ofInstance(https://ec2.us-west-2.amazonaws.com)}

Multiple init scripts:

We currently combine all scripts (from an InstanceScript object) to execute them at once. This is actually required for 'initScript' only (from createInstance method) as we can only give one extension per type during the VM creation.

Advanced script upload:

Support advanced scripts by specifying and uploading file-based scripts into Azure environment (can be then used by the Azure extension). The script can be then executed from a regular cmdLine injection.

Windows compatibility:

Check deployment and script execution on Windows (using the same extension than Linux). Check combined script as well using the '&&' separator.

Encryption:

Allow to send encrypted scripts on VMs using specific credentials (Azure extension feature)

We should have a way to secure the usage of the connector-iaas through a sessionid to list infrastructures, their instances and more particularly the execution of scripts on deployed instances.

Unlike Amazon, Azure doesn't provide a complete metadata service nor automatic DNS setup for the VMs.
Therefore, a VM doesn't known its own public IP address and PNP communications can not be fully established between server and nodes.
Solutions could be to:

• Manually setup a DNS using Azure API and specify it to the VM by script injection
• Retrieve the VM public IP address using API and specify it during script injection
• Let the VM discover its own public IP address by using public service like curl ipinfo.io/ip (from script injection OR init script)

As pointed out by @lpellegr, we currently put 'ERROR' as a prefix every time we throw an exeption.
We should create our own RuntimeException for each CloudProvider, for instance in Azure we could use a custom AzureProviderException class and manage the prefix in this class at least to avoid some duplications.

In AWSEC2JCloudsProvider, when we create a new Instance we do not provide any information about the EBS volume that will be attached to the new VM. Therefore, by default, JClouds creates a gp2 volume (regular SSD) with the same size than the source image.
According to Amazon, when a new EBS volume is created, the flag deleteOnTermination is set to true but it seems that it is not the default behavior of JClouds.
The VM's template must therefore be cutomized with specific volume informations, see the method mapNewVolumeToDeviceName for instance.
Without this fix we will continue to pay for the remaining volumes after VMs deletion!

Since there is no authentication to this service, the infrastructure endpoint shouldn't return the plain credentials, especially if the service is exposed publicly.

• Specify the desired size of the OS disk to create (minimal value is the image size)
• Specify if we need to attach our new instance(s) to additional (and existing) managed disks

After sending a DELETE connector-iaas/infrastructures/<my_infrastructure_id>/instances?instanceTag=<my_instance_tag> and while the request is being processed (which can take a lot of time), <my_infrastructure_id> is not pollable anymore through GET or POST. Instead, I get a Internal Error 500.

We currently use the last release (but still non stable) version '1.0 beta 5' of Microsoft Azure Java SDK.
Indeed, Microsoft announced that it is only a developer preview but which already supports major parts of the SDK.
Regarding thread-safety, it seems there is no garantee:

We do not make any thread-safety guarantees about our libraries. We also do not test them for thread-safety.
Methods that are currently thread-safe may be thread-unsafe in future versions.

The AzureProvider class has been however tested by us and the SDK classes we are using are actually thread safe.
However this need to be carefully double-checked after every SDK upgrades, as mentioned by Microsoft devs.

The createInstance method signature currently takes a single Instance with a field number (number of instances to create) instead of multiple Instances.
This behaviour prevents single-instance customization which can be usefull for injecting custom script on each instance for example.

From 'executeScriptOnInstanceId' method, the VMWare call to 'startProgramInGuest' throws an exception.
This happens because new cloned VMs are not automatically powered on.

It would be nice to specify what type of security group we want to create during instance creation.
Different types of security group are already defined into AzurePoviderUtils (Open/SSH/ProActive)
We thus need to specify the desired group from instance creation requests.

• Attach to an existing public IP address (if not already attached!)
• Attach to an existing security group
• Attach to an existing virtual network

In https://github.com/vinseon/connector-iaas/blob/master/src/main/java/org/ow2/proactive/connector/iaas/cloud/provider/vmware/VMWareProvider.java#L144, vm.getConfig() can be null in some cases when a VM is not correctly setup.

Change the default behaviour to create a new IP address' resource only if desired (from new parameter).

Currently, only custom images can be deployed on Azure by specifying the image name/id. The Azure SDK allows to create VMs from native images like Ubuntu, Debian, or Windows. Such an option should be provided by the connector-iaas.