grannyprogramming / azuremlopsinproduction Goto Github PK

What is the Issue?

Currently, Bicep is used exclusively for setting up the infrastructure. While Bicep is a powerful tool, relying on it solely may limit flexibility for users who prefer or are more familiar with other Infrastructure as Code (IaC) tools.

What would the solution look like?

Develop an alternative setup process using Terraform, a popular IaC tool compatible with numerous cloud providers, including Azure. This would involve creating a new script replicating the current Bicep setup, providing greater flexibility and choice for the end user.

Any alternative solutions?

Alternatively, we could provide detailed instructions for setting up the infrastructure manually without using an IaC tool. However, this might not be as efficient or error-proof as using a script.

Resources and Related Links
terraform#quickstart-templates

Description:

When using the azure.ai.ml.command to create components from a JSON file, the default values are always interpreted as strings, even when the original data type in the JSON is a number. This causes issues when the components require numerical default values.

Details:

System/Environment details:
Operating System: Linux (Github actions
Python version: python 3.9
azure.ai.ml version: @latest (Installed at runtime)

Steps to reproduce:
Define a component in a JSON file with an input that has a numerical default value.
Use azure.ai.ml.command to create the component from the JSON.
Print the created component or use it in a pipeline.

Expected behavior:
The default value should preserve its original numerical data type.

Actual behavior:
The default value is always a string.

Workarounds attempted:
Explicit Conversion: The default value was explicitly converted to a number after it was read from the JSON file. Despite this conversion, the default value was still being outputted as a string. This suggests that the conversion was not preserved, possibly due to the way the Input class or the serialization process treats the default values.

Type Conversion Check: In the set_default_value function, there was a check to see if the default value was a string that could be converted to a float or an integer. If it could be, it was converted. However, the final output still showed the default value as a string, suggesting that this conversion was not preserved.
Impact:
This issue prevents the correct execution of pipelines that require numerical default values for their inputs.

Labels:

bug, AzureML, AML components

Related commits/branches:

1ec93a0

Description:

When using the azure.ai.ml.command to create components from a JSON file, the default values for inputs are currently set at the type level and cannot be overridden at the component level. This results in all components that reference the same type having the same default value. It would be beneficial to allow overriding the default value at the component level, which would provide more flexibility when defining components.

Details:

File: create_component.py
Configs: nyc_taxi

Proposal:

Allow the default value for an input to be specified at the component level in the JSON. If a default value is provided at the component level, it should override the default value specified at the type level.

This would allow multiple components to reference the same type (like number or integer) but have different default values.

Use case:

This feature would be useful in scenarios where multiple components reference the same type, but each component needs a different default value. Currently, all these components are locked into the default value specified at the type level.

Attempts to solve the issue:

Separate Function for Default Value: A separate function set_default_value was created to handle the task of setting the default value. This function took an input key, an input value, a component, and a dictionary of references as arguments. It contained logic to fetch the default value from the references dictionary using various keys and methods.

Multiple Fetch Methods: The set_default_value function attempted to fetch the default value from the references dictionary using several different keys. First, it tried to get the default value directly from the 'input_and_output_types'. If that failed, it tried to extract the default from an input definition obtained from 'components_framework'. If the default value was still not found, it tried to get it from 'components_framework' using a different key.

Impact:

This feature would increase the flexibility of the Azure ML Command and allow more complex workflows to be defined.

Labels:
enhancement, AzureML, AML components

What is the Issue?

In our current workflow using the script create_environments, there isn't an effective way to create a new AML environment from a pre-existing Docker image. This limits the reusability of environments and makes the process less efficient.

What would the solution look like?

I propose the addition of logic to the script that allows for the creation of an AML environment from a pre-existing Docker image. This functionality would streamline the process of creating new, similar environments, and improve the overall efficiency of our workflows.

Any alternative solutions?
Manually creating the AML environment from the Docker image is an alternative, but this approach isn't scalable and can be error-prone. Automating this process within the script would be significantly more efficient and reliable.

Resource and Related Links
create-an-environment-from-a-docker-image

What is the Issue?

The current infrastructure setup main.bicep does not take into account networking configurations. As a result, critical aspects such as VNet, subnets, and Network Security Groups (NSGs) may be overlooked, potentially impacting the communication and operation of our AML resources.

What would the solution look like?

Incorporate networking configuration into the infrastructure setup script. This may involve modifying or expanding the current script, potentially leading to a more comprehensive and secure setup process.

Any alternative solutions?

A separate script for configuring networking resources could be created. However, integrating it directly into the infrastructure setup process would likely offer a more streamlined user experience.

Resources and Related Links

machine-learning-end-to-end-secure
vnet

What is the Issue?
As a Machine Learning professional using Azure ML, I find it challenging to create Kubernetes and AKS compute clusters. Although the documentation provides clear instructions on how to attach pre-existing Kubernetes clusters to Azure ML workspace and use them as compute targets, there is a gap in information on how to create these clusters using Python SDK v2.

What would the solution look like?
I propose the development of Python scripts that can be used to automate the creation of Kubernetes and AKS compute clusters in Azure ML. These scripts would fill an important gap, helping users to create and configure these resources with ease. Either modify the current create_compute.py to take Kubernetes into account (complex) or create a new separate script. This script should be able to take a previously built kuberenetes, with details passed by user and check if it exists on Azure and then attach to AML.

Any alternative solutions?
An alternative solution would be to enhance the existing documentation to include detailed steps on how to create and configure Kubernetes and AKS compute clusters in Azure ML. However, I believe that having scripts for this purpose would be more user-friendly and would automate the process, reducing the potential for error.

Or to just use the cli as described in the docs.

Resources and relevant links
AttachKubernetes
CreateKuberenetesCliv2

What is the Issue?

I am using the script create_environments.py to manage and create AML environments based on configuration files found in environments folder. Currently, the script is effective in checking and creating Conda environments against provided user configurations. However, when it comes to handling Docker images, the script falls short and cannot verify pre-existing Docker images against the provided configuration.

What would the solution look like?

I propose to enhance the script by adding logic to verify pre-existing Docker images against the user's configuration. This will enable us to check whether a Docker image that matches the user's configuration already exists in the AML environment, thereby preventing unnecessary image creation and saving resources. The steps would include:

Extract metadata from Docker image: We should be able to extract the metadata from a Docker image by either using Docker Python SDK or Azure ML SDK. Docker Python SDK can allow us to interact with Docker images like reading Dockerfile or inspecting a Docker image to get its metadata.

Compare Docker image metadata: Once we have the metadata of the Docker image, we should compare it with the Docker image configuration provided by the user. If there is a match, that means the Docker image already exists and we don't need to create a new environment.

Updating the function create_or_update_docker_environment: The function should be updated to include the Docker metadata extraction and comparison logic.

Any alternative solutions?
An alternative could be manually checking the Docker images against the configuration file. However, automating this process within the script would be more efficient and less prone to error.

Resources and Relevant links
There are no relevant material online I could find

What is the Issue?

The current Infra main.bicep always creates a new instance of Azure Machine Learning (AML) and its associated resources. This can lead to redundancy and wastage of resources.

What would the solution look like?

We need to implement a check within the script to determine if the required infrastructure already exists. If it does, the script should skip the creation step and utilize the existing resources.

Any alternative solutions?

An alternative solution could be to prompt the user to decide whether they want to use existing resources or create new ones each time the script runs.

Resources and Related Links
AML Workspaces
Example Stackoverflow