elastic / azure-marketplace Goto Github PK

https://azure.microsoft.com/en-us/blog/f-series-vm-size/

The presence of imageReference in a Microsoft.Compute.SizeSelector control prevents the pricing information from being displayed in the Azure portal UI. The last template production release kept the imageReference in as there was a bug in the Azure portal UI that prevented the ability to select different VM sizes when the imageReference was not present; this is now fixed so imageReference can be removed.

In the Azure template, it should be possible to choose the VM template for Kibana. It should default to D1, but the user should be able to modify this as part of deployment. Today, Kibana is deployed into a VM chosen by the template, with no opportunity to change it.

The default kibana4 role is gone as per 2.3

2016/05/19 11:53:35 Warning: The following roles [kibana4] are unknown. Make sure to add them to the [/etc/elasticsearch/shield/roles.yml] file. Nonetheless the user will still be associated with all specified roles

• Ensure regex validation on Microsoft.Compute.UserNameTextBox works
• Remove image references for platform images so that prices are displayed in the UI
• Update API versions

The kibana installation script currently installs the latest version of the marvel plugin, which happens to be 2.1 as or right now. This version of the plugin seems to be incompatible with a 2.0.1 cluster. The script needs to install an explicit version of the plugins to ensure they are compatible with the whatever is installed in the cluster.

Using the new Azure Market place template mentioned in this post: https://www.elastic.co/blog/microsoft-azure-marketplace-elasticsearch-kibana-and-more-now-available I deployed an Elastic cluster but get the below error:

plugin:elasticsearch - Request Timeout after 1500ms
plugin:marvel - Waiting for Elasticsearch

There are some discrepancies between the defaultValues defined in the ARM template vs the UI defition. They should be the same in both.

Today, when the template installs, we do not install the Kibana Shield plugin. Mostly, this is ok, however, the following problems are present:

1. Graph UI does not work. When you click on the search button, you get an "Unauthorized Operation" at the top, and no apparent errors in any logs.
2. No logout button is available in Kibana. You have to brute force switching users.

I recall a conversation about choosing not to do this because we were going to need to set up SSL. We could either set up SSL, or possibly allow a flag in the config to not require SSL for the plugin.

I would make this a pretty high priority. We want our marketplace offering to match our product offering. Could we model what we do in cloud?

Error submitting the deployment request. Additional details from the underlying API that might be helpful: Deployment template validation failed: 'The template variable 'nodesPerStorageAccount' is not valid: Template language expression property 'Standard_A1' doesn't exist, available properties are 'Standard_D1, Standard_D2, Standard_D3, Standard_D4, Standard_DS1, Standard_DS2, Standard_DS3, Standard_DS4, Standard_DS13, Standard_A2, Standard_A3, Standard_A4, Standard_A7'..'.

parameters provided in azure portal:

• esversion 2.1.0
• loadbalancertype: internal
• esclustername: elasticsearch
• loadbalancertype: internal
• esplugins: no
• kibana: no
• jumpbox: no
• vmsizedatanodes: Standard_A1
• vmdatanodecount: 3
• datanodesaremastereligible: yes
• vmsizemasternodes: Standard_A1
• vmclientnodecount: 0
• vmsizeclientnodes: Standard_A1
• adminusername: provided
• authenticationtype: password
• adminpassword provided
• sshpublickey:
• shieldadminpassword:
• shieldreadpassword:
• shieldkibanapassword:
• Location: ResourceGroup
• VirtualNetworkName: esvnet

as specified here: https://azure.microsoft.com/en-us/documentation/articles/storage-premium-storage/

I want to be able to install additional plugins without having to ssh in to every node. For my specific use case I want the delete-by-query, kopf and azure-cloud plugins to always be installed.

An implementation is available in pull request #65

These allowedValues are not reflected in createUIDefiniition.json

https://github.com/elastic/azure-marketplace/blob/master/build/tasks/patch-values.js#L108

We need to be able to configure more in elasticsearch.yml during template config. Otherwise, you have to ssh to each node and push the file. This could be either validated text fields or a free form box (risky, but most flexible)

I notice that there are two ARM templates out there:

https://github.com/elastic/azure-marketplace
https://github.com/Azure/azure-quickstart-templates/tree/master/elasticsearch

I think they each have things the other does not. Does it make sense to keep these in sync somehow?

Azure/azure-quickstart-templates#2340

Which is no longer reliant on waagent and so we can move away from forcefully restarting that service between every step 🎉 🎈 🍰

From feedback from MS (see also #27)

Remove reference function usage and outputs from template links

• data-nodes-16disk-resources.json - line 184
• data-nodes-8disk-resources.json - line 184
• data-nodes-4disk-resources.json - line 184
• data-nodes-2disk-resources.json - line 184
• client-nodes-resources.json - line 155
• jumpbox-resources.json - line 164
• kibana-resources.json - line 178
• master-nodes-resources.json – 152

All the files above are using a reference to osProfile-resources.json. In order to remove the usage of reference function, you need to copy part of osProfile-resources.json code into the files above and remove the osProfile-resources.json file.

I understand this will cause some code repetition among the files and will make the template look “ugly”, but this will allow the ARM backend to parse the VM resources that will be deployed, and validate the required cores versus the cores available on the subscription. This will improve end customer experience, in the case they don’t have enough cores to deploy the cluster, they will receive an error message before submitting the deploy and creating some resources.

It'd be nice if we could do this as part of the npm run build instead of introducing the repitition through copy paste

When creating a new cluster using the ARM template, there are two steps that take a long time.

1. Scripts running to install and start ES (15 minutes for 3 nodes)
2. Scripts running to install and start Kibana (about 15 more minutes)

The latest deployment I did with 3 data nodes and no dedicated masters took about 45 minutes. A similar deployment, with timing, can be seen in the cjfebresourcegroup under the elastic azure account. This behavior is consistent across different data centers - I tried both West Europe and West US.

Also tracked under Azure/azure-quickstart-templates#1545

The mesos template is no longer there but from the history:

https://github.com/Azure/azure-quickstart-templates/blob/768b07d320d62ebae14854240e714daf2a0879b6/mesos-swarm-marathon/agent-gt0.json#L141

with the dependsOn looking like this:

https://github.com/Azure/azure-quickstart-templates/blob/768b07d320d62ebae14854240e714daf2a0879b6/mesos-swarm-marathon/agent-gt0.json#L210

Create a new cluster using the azure xplat cli, e.g.

azure group create priv-ajh-es2 westeurope
azure group deployment create priv-ajh-es2 --template-uri https://raw.githubusercontent.com/elastic/azure-marketplace/master/src/mainTemplate.json --parameters-file "C:\password.parameters.json"

where the contents of C:\password.parameters.json is:

{
  "artifactsBaseUrl":{"value":"https://raw.githubusercontent.com/elastic/azure-marketplace/master/src"},
  "esVersion":{"value":"2.4.0"},
  "esClusterName":{"value":"priv-ajh-es2"},
  "loadBalancerType":{"value":"internal"},
  "kibana":{"value":"No"},
  "jumpbox":{"value":"Yes"},
  "vmHostNamePrefix":{"value":""},
  "vmSizeKibana":{"value":"Standard_D1"},
  "vmSizeDataNodes":{"value":"Standard_DS2"},
  "vmDataNodeCount":{"value":3},
  "dataNodesAreMasterEligible":{"value":"Yes"},
  "vmSizeMasterNodes":{"value":"Standard_DS1"},
  "vmClientNodeCount":{"value":0},
  "vmSizeClientNodes":{"value":"Standard_D1"},
  "adminUsername":{"value":"esadmin"},
  "authenticationType":{"value":"password"},
  "sshPublicKey":{"value":"<value-ignored>"},
  "adminPassword":{"value":"Elastic12"},
  "shieldAdminPassword":{"value":""},
  "shieldReadPassword":{"value":""},
  "shieldKibanaPassword":{"value":""},
  "vNetName": {"value": "priv-ajh-es2-vnet"},
  "vNetSubnetName": {"value": "priv-ajh-es2-subnet"},
  "vNetLoadBalancerIp": {"value": "10.0.0.4"},
  "vNetNewOrExisting": {"value":"new"},
  "vNetExistingResourceGroup": {"value": ""},
  "vNetNewAddressPrefix": {"value": "10.0.0.0/16"},   
  "vNetNewSubnetAddressPrefix": {"value": "10.0.0.0/24"},
  "userCompany": { "value": "" },
  "userEmail": { "value": "" },
  "userFirstName": { "value": "" }, 
  "userLastName": { "value": "" },
  "userJobTitle": { "value": "Architect" }
}

Results in a provisioning failure with an error message ending with:

...
dpkg-preconfigure: unable to re-open stdin: No such file or directory
monit: Cannot connect to the monit daemon. Did you start it with http support?
monit: Cannot connect to the monit daemon. Did you start it with http support?
dpkg-preconfigure: unable to re-open stdin: No such file or directory
Failed to open terminal.debconf: whiptail output the above errors, giving up!
dpkg: error processing package iptables-persistent (--configure):
 subprocess installed post-installation script returned error exit status 255
Errors were encountered while processing:
 iptables-persistent
E: Sub-process /usr/bin/dpkg returned an error code (1)
start: Job is already running: walinuxagent

It looks like iptables-persistent fails to install silently.

My clever hack did not pan out, luckily @trentmswanson already kindly shared a much smarter alternative with us 👍 🎉

It seems that current template does not support deployment to Windows VM. Please add support for Windows based VMs.

Greatly simplifies getting the kibana url out of a deployment, as @hglkrijger demonstrates here:

https://azure.microsoft.com/en-us/blog/introducing-elasticsearch-with-azure-file-storage

When the template deploys, it should install the Azure repo plugin by default so a user could restore from an Azure blob store without having to install the plugin on each node in the cluster.

I'm deploying 10 data nodes with DS3_v2 which have 8 premium data disks each by default.
It need totally 80 TB premium storage.

However, Azure's premium storage limit per account is 35 TB. It is not possible to deploy that.

I walk around this by first deploy with smaller vm DS1_v2, remove the disks and then upgrade it to DS3_v2.

Is it possible to provide the option for data disk?
In our case, we only need one 512GB data disk.

Azure Marketplace validation succeeds when ADMINUSERNAME is set to admin but fails at deployment time.

Some validation constraints should be set on the admin username to validate it ahead of time. mainTemplate doesn't appear to support regex validation, but the uiDefinition does.

See https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-usernames/

We need to add user registration in the template. It has been described in the doc here.

@russcam Using the new Azure ES deployment template and wondering if there is any way to target an existing resource group? For example I would like to deploy to an existing virtual network within an existing resource group. Thx.

https://portal.azure.com/#create/elastic.elasticsearchelasticsearch

https://azure.microsoft.com/en-us/blog/azure-vm-scale-sets-public-preview/

Does not support premium storage just yet but would greatly simplify and cut back on the moving parts in the ARM template

The deployment template does a iptables port forwarding from 9201 to 9200 in order to make external load balancer and internal load balancer work together.

However, I find the the port forwarding disappear after system reboot. and external load balancer stops forwarding traffic.
I walk around this persist the port forwarding by myself.

Hopefully the deployment template can persist that in the future.

I do think the external will be needed for a long while. It hard to put cloud service, app service and Elasticsearch in the same virtual network because of Azure's two different deployment mode.

Hi Team,

After creating an Elastic search cluster with ARM template provided. Is it possible to add another VM as a data node.
We have been trying this out and it always crashes the entire elastic search cluster.
Is there any additional template for adding extra nodes to the existing cluster?

Same as: elastic/elasticsearch#17430

Causes a "scary" false positive to be logged

Right now we also allow users to specify a different location on the ARM template then the resource group, from the azure UI this is always set to the resource groups location. We should remove the parameter from the template and rely solely on resourceGroup().location.

Currently, the template defaults to an empty value. It should default to the latest version instead.

Right now we bootstrap with a trial license and then folks will have to manually sent an license using the API post deploy. In a future version we should allow to bootstrap with a valid license from the get go.

From email to Azure Marketplace:

I have deployed Kibana and Elastisearch on Microsoft Azure and having problems accessing Elastic Search on port 9200 and also Kibana.

Can you please provide proper documentations on how to go on deploying and setting up Elastic Search and Kibana on Azure?

Job title is currently assigned as the value to the field name Title; this should be updated to use the field name Job_Function__c for the next release.

The allowed values for esVersion in src/mainTemplate.json does not include 2.3.3
Please update the default and allowed values for esVersion to include the latest released version.

Deployment of a new cluster leads to the following error message:
monit: Cannot connect to the monit daemon. Did you start it with http support?
and the deployment is considered failed.

It looks like the script elasticsearch-ubuntu-install.sh configures monit to monitor the elasticsearch process and only starts, but does not restart, the monit service. This means that monit does not know about the new configuration and the subsequent call of "sudo monit start all" causes the error message.

Restarting the monit service and then calling "sudo monit start all" does not produce the error message.

and map to the correct kibana version

I attempted deploying to an existing Azure Resource Group which is already running another Cluster of .Net compute nodes (Service Fabric) but the deployment failed. If possible it would be great to support deploying an ES cluster to an existing Azure vNet.

1-5 data
1-5 client
0/3 master
0/1 Kibana
0/1 jumpbox

If you set up a cluster using the ARM template, and choose External load balancer at the very end, and also choose to install Kibana, Kibana will end up misconfigured. This is similar to this issue:

#14

And will manifest itself the same way. The workaround is to SSH into the VM and change the Kibana configuration by hand.

• "External Access Control" step, loadBalancerType control's defaultValue should be "Internal" not "internal"
• parameter vmSizeDataNodes: remove the defaultValue and allowedValues (we have this list on the createUiDefinition.json).
• parameter vmSizeMasterNodes: remove the defaultValue and allowedValues (we have this list on the createUiDefinition.json).
• parameter vmSizeClientNodes: remove the defaultValue and allowedValues (we have this list on the createUiDefinition.json).
• parameter sshPublicKey: change the type from string to securestring
• parameter shieldAdminPassword: remove the defaultValue for this parameter (do we allow/need a blank password?)
• parameter shieldReadPassword: remove the defaultValue for this parameter (do we allow/need a blank password?)
• parameter shieldKibanaPassword: remove the defaultValue for this parameter (do we allow/need a blank password?)
• parameter location: remove allowedValues and defaultValue
• When creating storage accounts, change the apiVersion from 2015-05-01-preview to 2015-06-15

I use command
"sodu service elasticsearch stop" to stop Elasticsearch, however, it restart itself immediately.
What is the correct way to stop it?