How to delete Azure Synapse from cosmos db without migration of data. Currently this functionality is not available as per the documentation.

https://learn.microsoft.com/en-us/azure/cosmos-db/synapse-link-frequently-asked-questions#can-i-disable-the-azure-synapse-link-feature-for-my-azure-cosmos-db-account-

Replace copy noop spark job with data flow. This should unblock and serve to improve performance for the overall pipeline as it will avoid the use of file share mount in mssparkutils which is currently failing.

Please provide more detailed instructions on which pipeline to import

As of the instructions

"Import the pipeline under the workflow folder to your Azure Synapse Analytics instance's workspace. Alternatively, you can copy the files to your repository (git or Azure DevOps) and link the repository to your Azure Synapse Analytics workspace."

Does the pipeline have to be created by using "Browse gallery" and picking "Spaceborne Data Analysis Master Pipeline"?

Under the workflow folder of the source code there are 2 folders with JSON files.
There seem to be multiple pipeline definitions under custom-vision-model-v2\pipeline for example

public container registry references are causing a security analysis failure

• Running system reserved library update job for spark pool: Geospatial
• Create Or Update SparkComputeFailed LibraryManagement - Spark Job timed out for Geospatial in workspace geospatialanalytics in subscription

Implement a simplified version of the current transform pipeline for Custom Vision Model. The current model pipeline performs mosaic, crop, convert and tiling as a series of spark jobs. This translates to separate spark-submit to the spark cluster in Synapse. Every spark-submit exacts an overhead of 2 to 4 minutes. This adds significant lead time for each of the transform - mosaic, crop, convert and tiling.

As a simple update, implement a consolidate version of the transform where the four transform are submitted as a single spark job. This reduces the redundant lead time associated with having multiple spark jobs.

We are exploring on using different infrastructure for executing AI model like batch-account, aks.
This demands the entrypoint scripts like setup.sh to provide for mechanism to specify different infrastructure types.

While running the installation script, there is another error about Spark autoscaling configuration

{"status":"Failed","error":{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"Conflict","message":"{\r\n "status": "Failed",\r\n "error": {\r\n "code": "ResourceDeploymentFailure",\r\n "message": "The resource operation completed with terminal provisioning state 'Failed'.",\r\n "details": [\r\n {\r\n "code": "DeploymentFailed",\r\n "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.",\r\n "details": [\r\n {\r\n "code": "Conflict",\r\n "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"BadRequest\",\r\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ValidationFailed\\\",\\r\\n \\\"message\\\": \\\"Spark pool request validation failed.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"NodeCountNotValid\\\",\\r\\n \\\"message\\\": \\\"The autoscale minimum node count is not valid. A Spark pool cannot have 1 nodes; it must have between 3 and 200 nodes.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"}]}}

As an alternative to Azure Batch account, AKS infrastructure and corresponding data pipeline can be set up based on set up parameter.

When setting up the solution with setup.sh, there is an error for storage account name conflict. Error message included below.

{"status":"Failed","error":{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"Conflict","message":"{\r\n "status": "Failed",\r\n "error": {\r\n "code": "ResourceDeploymentFailure",\r\n "message": "The resource operation completed with terminal provisioning state 'Failed'.",\r\n "details": [\r\n {\r\n "code": "DeploymentFailed",\r\n "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.",\r\n "details": [\r\n {\r\n "code": "Conflict",\r\n "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"Conflict\",\r\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource operation completed with terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Conflict\\\",\\r\\n \\\"message\\\": \\\"{\\\\r\\\\n \\\\\\\"error\\\\\\\": {\\\\r\\\\n \\\\\\\"code\\\\\\\": \\\\\\\"StorageAccountAlreadyTaken\\\\\\\",\\\\r\\\\n \\\\\\\"message\\\\\\\": \\\\\\\"The storage account named synhnsrqjhqd is already taken.\\\\\\\"\\\\r\\\\n }\\\\r\\\\n}\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"}]}}

Because of security around secrets, the workflow tests for CI/CD are not running against the fork repositories.

Since the release of the initial version of the script to deploy and configure the reference architecture, the README.md file has been outdated specifically in the parameters part. Need to update the parameters (required and optional) in the README.md file.

We have only one pipeline but name refers to older nomenclature v1 and v2.
Lets cleanup the nomenclature anomaly by removing the references of v2 and the pipeline would only be referred by the name of custom-vision-model

We need a way to create a Synapse workspace with a managed VNET. This enables the security architecture with managed private endpoints.

By design, DEP (Data Exfiltration Protection) does not support public channels for package deployment. So, in order to support the current deployment scenario and does not failure with regression, DEP would be disable by default.

We recently added a new dependency to the pipelines, "notebook".
Now the azure deployment/pipeline execution workflow is failing because we need notebooks to be created before installing the pipeline on synapse.

We are only testing DEPLOY_PGSQL=true, we should add another ADO tests for the false condition which will test the synapse pipeline and deployment for non-pgsql setup.

Updates to follow ...

We need to add more information into troubleshooting.md in regards to review comments

#87 (comment)

#87 (comment)

Protobuf issue introduced during the last build (triggered by the PR merge) cause AI Model run to fail. This issue needs to be addressed in a separate PR.

Traceback (most recent call last):
  File "./custom_vision.py", line 11, in <module>
    from predict import initialize, predict_image
  File "/predict.py", line 7, in <module>
    import tensorflow as tf
  File "/usr/local/lib/python3.7/site-packages/tensorflow/__init__.py", line 98, in <module>
    from tensorflow_core import *
  File "/usr/local/lib/python3.7/site-packages/tensorflow_core/__init__.py", line 40, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 959, in _find_and_load_unlocked
  File "/usr/local/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/usr/local/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_core/python/__init__.py", line 52, in <module>
    from tensorflow.core.framework.graph_pb2 import *
  File "/usr/local/lib/python3.7/site-packages/tensorflow_core/core/framework/graph_pb2.py", line 16, in <module>
    from tensorflow.core.framework import node_def_pb2 as tensorflow_dot_core_dot_framework_dot_node__def__pb2
  File "/usr/local/lib/python3.7/site-packages/tensorflow_core/core/framework/node_def_pb2.py", line 16, in <module>
    from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2
  File "/usr/local/lib/python3.7/site-packages/tensorflow_core/core/framework/attr_value_pb2.py", line 16, in <module>
    from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
  File "/usr/local/lib/python3.7/site-packages/tensorflow_core/core/framework/tensor_pb2.py", line 16, in <module>
    from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2
  File "/usr/local/lib/python3.7/site-packages/tensorflow_core/core/framework/resource_handle_pb2.py", line 16, in <module>
    from tensorflow.core.framework import tensor_shape_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__shape__pb2
  File "/usr/local/lib/python3.7/site-packages/tensorflow_core/core/framework/tensor_shape_pb2.py", line 42, in <module>
    serialized_options=None, file=DESCRIPTOR),
  File "/usr/local/lib/python3.7/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

The proposed sample uses a sample GeoTIFF .tif file, which is downloaded by
deploy/scripts/copy_geotiff.sh script and afterwards converted to tiles.
However there does not seem to be any documentation on how to produce such GeoTIFF from the Azure Orbital .bin file downloaded at the end of
https://docs.microsoft.com/en-us/azure/orbital/downlink-aqua.

The instructions https://docs.microsoft.com/en-us/azure/orbital/satellite-imagery-with-orbital-ground-station also do not address this topic, hence there is a huge gap between these two tutorials.

The only relevant instruction available is the utilization of demodulation to reduce .bin file size, but not about the product.

Problem
Currently we are grant the managed-identities the contributor access on batch account.
This access is very open ended and could be a security issue.

Proposed Solution
Lets make use of custom-roles to create a custom role with selective permissions on batch account and assign the custom-role to managed-identities as required.

To help users understand the Orbital product portfolio examples it'd be helpful to link to https://github.com/Azure/azure-orbital-integration and explain how the two do and don't work together.

When we land to https://github.com/Azure/Azure-Orbital-Analytics-Samples/blob/main/deploy/gallery/instructions.md from pipeline help it suggests user to go to https://github.com/Azure/Azure-Orbital-Analytics-Samples/blob/main/deploy/README.md which has not only infrastructure deployment documentation but more than that deploying pipelines on standalone.

All-in-all this seems a little confusing for a user of the pipeline.

Lets try to simplify the documentation and make it helpful and easy for a user to understand and use.

Is your feature request related to a problem? Please describe.
Currently, there is no sanity check post-deployment to make sure the components required are provisioned successfully like Spark pool, Batch pool nodes etc.

Describe the solution you'd like
Add validation script to make sure the components are provisioned successful post-deployment

Describe alternatives you've considered
Currently, I don't know if there are any alternatives but will consider evaluating in due course

Additional context
No additional context

Deploy directory has too many files
Lets simplify by creating relevant directories and moving files there.

Running setup.sh or install.sh does not prompt the user for the admin password for PGSQL as described in the deployment documentation.

Without this $POSTGRES_ADMIN_LOGIN_PASS is empty and deployment will fail saying

{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"PasswordNotComplex","message":"Password validation failed. The password does not meet policy requirements because it is not complex enough."}]}

Thanks

Since environment_code currently is based on sha, so the reruns of workflow are running into errors because of azure keyvault recreation with the same name.
So, ideally the env_code should be unique, so that the cleanup of previous workflow run does not interferes with the new workflow run.

The synapse pipeline was created by searching in the gallery for "Spaceborne Data Analysis Master Pipeline" and setting the 3 required linked service accordingly:

1. A linked service to AZ blob storage rawdata******
2. A linked service to AZ datalake storage v.2 rawdata******
3. A linked service to

• Updated the configuration files from https://github.com/Azure/Azure-Orbital-Analytics-Samples/blob/main/deploy/README.md#running-the-pipeline-custom-vision-model
• Pipeline parameters were configured accordingly (as of the instructions from https://github.com/Azure/Azure-Orbital-Analytics-Samples/blob/main/deploy/gallery/instructions.md
• I am using option A. for loading custom vision model from from https://github.com/Azure/Azure-Orbital-Analytics-Samples/blob/main/deploy/README.md

While running the Custom Vision Pipeline there is the following error (similar for Copy XML and Copy JSON activities):
It is unclear which the resource is so that

{
"errorCode": "2200",
"message": "ErrorCode=UserErrorFileNotFound,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error Message: The specified resource does not exist. (ErrorCode: 404, Detail: The specified resource does not exist., RequestId: 039b4525-001a-0061-2f79-d3625a000000),Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Azure.Storage.StorageException,Message=The specified resource does not exist.,Source=Microsoft.Azure.Storage.Common,'",
"failureType": "UserError",
"target": "Copy Xml",
"details": []
}

The activity is using
"@concat(pipeline().parameters.Prefix, '/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['resultsDirectory'], '/other'

The results directory from Read Spec Document activity is "out" and the prefix is the container name, hence the path shall be "-test-container/out/other"

A hint about the configuration is more than welcome as the directory was not there and nothing changed after creating it manually. @jfrazee Do you think you could give a hint?

Problem
We have an issue of quota while provisioning the pool on a new provisioned batch account, and we need to require to go to Azure support to get quota fixed and proceed once the quota management has been dealt. This delays the testing.

Proposed Solution
We can use a pre-provisioned batch account with enough quota to create new pools.