microsoft / bonsai-batch Goto Github PK
View Code? Open in Web Editor NEWOrchestrating Batch Jobs for Bonsai Simulations
License: MIT License
Orchestrating Batch Jobs for Bonsai Simulations
License: MIT License
(bonsai-preview) C:\Users\JOMCDOWE\Documents\bonsai\preview-repos\bonsai-batch>python batch_creation.py build_image
Directory of Dockerfile and source files: C:\Users\JOMCDOWE\Documents\bonsai\preview-repos\microsoft-bonsai-api\Python\samples\house-energy
az_command_data_logger: INFO command args: acr build --image {} --registry {} --file {} {} --platform {}
Packing source code into tar to upload...
Uploading archived source code from 'C:\Users\JOMCDOWE\AppData\Local\Temp\build_archive_f83aa4c18a974bb5aaea11df7cd19f00.tar.gz'...
azure.storage.common.storageclient: INFO Client-Request-ID=88dc0b76-2b69-11eb-8101-ba31b58b277f Outgoing request: Method=PUT, Path=/f51bb67e90d44c908953ad20a3c05ea8-n5kw6plecp/source/202011200000/c9a5f5a0-ea98-4433-954b-aebd65e80e48.tar.gz, Query={'timeout': None, 'sv': '2018-03-28', 'sr': 'b', 'sig': 'REDACTED', 'se': '2020-11-20T20:49:42Z', 'sp': 'cw'}, Headers={'x-ms-blob-type': 'BlockBlob', 'x-ms-lease-id': None, 'If-Modified-Since': None, 'If-Unmodified-Since': None, 'If-Match': None, 'If-None-Match': None, 'Content-Length': '307083', 'x-ms-version': '2018-03-28', 'User-Agent': 'Azure-Storage/1.4.2-1.5.0 (Python CPython 3.7.7; Windows 10)', 'x-ms-client-request-id': '88dc0b76-2b69-11eb-8101-ba31b58b277f', 'x-ms-date': 'Fri, 20 Nov 2020 19:49:43 GMT'}.
azure.storage.common.storageclient: INFO Client-Request-ID=88dc0b76-2b69-11eb-8101-ba31b58b277f Receiving Response: Server-Timestamp=Fri, 20 Nov 2020 19:49:42 GMT, Server-Request-ID=12ca1bfc-d01e-0064-0b76-bfd625000000, HTTP Status Code=201, Message=Created, Headers={'content-length': '0', 'content-md5': 'SEFKjfpJvMR4DwpCTR2OSA==', 'last-modified': 'Fri, 20 Nov 2020 19:49:42 GMT', 'etag': '"0x8D88D8D6CD0EB8C"', 'server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0', 'x-ms-request-id': '12ca1bfc-d01e-0064-0b76-bfd625000000', 'x-ms-version': '2018-03-28', 'x-ms-request-server-encrypted': 'true', 'date': 'Fri, 20 Nov 2020 19:49:42 GMT'}.
Sending context (299.886 KiB) to registry: jmanrgacr...
cli.azure.cli.core.util: ERROR InvalidInputValue - Some of the properties of 'runRequest' are invalid.. InnerErrors: DockerBuildRequest:Image name 'CUsersJOMCDOWEDocumentsbonsaipreviewreposmicrosoftbonsaiapiPythonsampleshouseenergy:latest' is invalid.
The acceptable patterns are 'repository' or 'repository:tag'.The repository and tag names follow the standardized docker repository and tag naming conventions. Tag templates are supported within '{{' and '}}'.
Please refer to docs for more information on using image name and tag: https://aka.ms/az/acr/az-acr-build
InvalidInputValue - Some of the properties of 'runRequest' are invalid.. InnerErrors: DockerBuildRequest:Image name 'CUsersJOMCDOWEDocumentsbonsaipreviewreposmicrosoftbonsaiapiPythonsampleshouseenergy:latest' is invalid.
The acceptable patterns are 'repository' or 'repository:tag'.The repository and tag names follow the standardized docker repository and tag naming conventions. Tag templates are supported within '{{' and '}}'.
Please refer to docs for more information on using image name and tag: https://aka.ms/az/acr/az-acr-build
being able to resize an existing pool, avoiding having to recreate a new one
I tried running with 500 tasks on 125 nodes, so 4 tasks per node. No matter what i'd choose, it'd always use 1 task per node. Looking at docs, the API may have changed.
Traceback (most recent call last):
File "batch_containers.py", line 790, in <module>
fire.Fire()
File "C:\Users\JOMCDOWE\Miniconda3\envs\batch\lib\site-packages\fire\core.py", line 138, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "C:\Users\JOMCDOWE\Miniconda3\envs\batch\lib\site-packages\fire\core.py", line 471, in _Fire
target=component.__name__)
File "C:\Users\JOMCDOWE\Miniconda3\envs\batch\lib\site-packages\fire\core.py", line 675, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "batch_containers.py", line 691, in run_tasks
show_price=show_price,
File "batch_containers.py", line 467, in batch_main
host_os=self.config["ACR"]["PLATFORM"],
File "C:\Users\JOMCDOWE\OneDrive - Microsoft\Documents\bonsai\preview-repos\bonsai-batch\get_azure_data.py", line 125, in show_hourly_price
hourly_price = (float(low_table.iloc[0, 3]) * low_pri_nodes) + (
File "C:\Users\JOMCDOWE\Miniconda3\envs\batch\lib\site-packages\pandas\core\indexing.py", line 873, in __getitem__
return self._getitem_tuple(key)
File "C:\Users\JOMCDOWE\Miniconda3\envs\batch\lib\site-packages\pandas\core\indexing.py", line 1443, in _getitem_tuple
self._has_valid_tuple(tup)
File "C:\Users\JOMCDOWE\Miniconda3\envs\batch\lib\site-packages\pandas\core\indexing.py", line 702, in _has_valid_tuple
self._validate_key(k, i)
File "C:\Users\JOMCDOWE\Miniconda3\envs\batch\lib\site-packages\pandas\core\indexing.py", line 1352, in _validate_key
self._validate_integer(key, axis)
File "C:\Users\JOMCDOWE\Miniconda3\envs\batch\lib\site-packages\pandas\core\indexing.py", line 1437, in _validate_integer
raise IndexError("single positional indexer is out-of-bounds")
IndexError: single positional indexer is out-of-bound
10/19/21 I started receiving this issue when i successfully ran 3 other runs with the same img and no changes to bonsai-batch
. Has the API changed drastically?
Code: InvalidPropertyValue
Message: {'additional_properties': {}, 'lang': 'en-US', 'value': 'The value provided for one of the properties in the request body is invalid.\nRequestId:06894e94-463d-4a41-9980-873e5d1515d7\nTime:2021-10-20T04:43:00.5940380Z'}
PropertyName: nodeAgentSKUId
PropertyValue: batch.node.ubuntu 16.04
Reason: The specified nodeAgentSKUId does not exist.
When building environment for bonsai-batch had a couple of errors. Seems that they come down to 2 problems:
Attached version for fix
environment.yml.zip
(bonsai-preview) C:\Users\JOMCDOWE\Documents\bonsai\preview-repos\bonsai-batch>python batch_containers.py run_tasks
Please enter task to run from container (e.g., python __main__.py): python3 main.py
Number of simulators to run as tasks on Batch: 100
What VM Name / SKU do you want to use? (if you don't know say None): Standard_D8s_v3
root : WARNING Creating new pool named houseenergypool10
root : WARNING Creating new pool named houseenergypool10
Traceback (most recent call last):
File "batch_containers.py", line 642, in <module>
fire.Fire()
File "C:\Users\JOMCDOWE\Miniconda3\envs\bonsai-preview\lib\site-packages\fire\core.py", line 138, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "C:\Users\JOMCDOWE\Miniconda3\envs\bonsai-preview\lib\site-packages\fire\core.py", line 471, in _Fire
target=component.__name__)
File "C:\Users\JOMCDOWE\Miniconda3\envs\bonsai-preview\lib\site-packages\fire\core.py", line 675, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "batch_containers.py", line 608, in run_tasks
command=task_to_run, log_iterations=log_iterations, workdir=workdir,
File "batch_containers.py", line 400, in batch_main
self.create_pool(use_fileshare=log_iterations)
File "batch_containers.py", line 244, in create_pool
self.batch_client.pool.add(self.new_pool)
File "C:\Users\JOMCDOWE\Miniconda3\envs\bonsai-preview\lib\site-packages\azure\batch\operations\_pool_operations.py", line 325, in add
raise models.BatchErrorException(self._deserialize, response)
azure.batch.models._models_py3.BatchErrorException: Request encountered an exception.
Code: InvalidRequestBody
Message: {'additional_properties': {}, 'lang': 'en-US', 'value': 'The specified Request Body is not syntactically valid.\nRequestId:be254e54-cb23-4c02-b203-e3de66591681\nTime:2020-11-20T19:57:48.8929293Z'}
Reason: A null value was found with the expected type 'Microsoft.Azure.Batch.Protocol.Entities.MountConfiguration[Nullable=False]'. The expected type 'Microsoft.Azure.Batch.Protocol.Entities.MountConfiguration[Nullable=False]' does not allow null values.
(bonsai-preview) C:\Users\JOMCDOWE\Documents\bonsai\preview-repos\bonsai-batch>python batch_creation.py create_resources
Traceback (most recent call last):
File "batch_creation.py", line 30, in <module>
filename=os.path.join("logs", "batch_creation.log"), mode="a"
File "C:\Users\JOMCDOWE\Miniconda3\envs\bonsai-preview\lib\logging\__init__.py", line 1087, in __init__
StreamHandler.__init__(self, self._open())
File "C:\Users\JOMCDOWE\Miniconda3\envs\bonsai-preview\lib\logging\__init__.py", line 1116, in _open
return open(self.baseFilename, self.mode, encoding=self.encoding)
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\JOMCDOWE\\Documents\\bonsai\\preview-repos\\bonsai-batch\\logs\\batch_creation.log'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.