microsoft / maro Goto Github PK

Multi-Agent Resource Optimization (MARO) platform is an instance of Reinforcement Learning as a Service (RaaS) for real-world resource optimization problems.

Home Page: https://maro.readthedocs.io/en/latest/

License: MIT License

Shell 0.16% Batchfile 0.09% Python 67.08% Jupyter Notebook 23.06% C++ 3.46% Makefile 0.01% Dockerfile 0.02% Jinja 0.29% CSS 0.15% HTML 1.25% Cython 4.44% Lua 0.01%

reinforcement-learning multi-agent-reinforcement-learning multi-agent simulator resource-optimization operations-research citi-bike inventory-management logistics raas

maro's Introduction

Multi-Agent Resource Optimization (MARO) platform is an instance of Reinforcement learning as a Service (RaaS) for real-world resource optimization. It can be applied to many important industrial domains, such as container inventory management in logistics, bike repositioning in transportation, virtual machine provisioning in data centers, and asset management in finance. Besides Reinforcement Learning (RL), it also supports other planning/decision mechanisms, such as Operations Research.

Key Components of MARO:

Simulation toolkit: it provides some predefined scenarios, and the reusable wheels for building new scenarios.
RL toolkit: it provides a full-stack abstraction for RL, such as agent manager, agent, RL algorithms, learner, actor, and various shapers.
Distributed toolkit: it provides distributed communication components, interface of user-defined functions for message auto-handling, cluster provision, and job orchestration.

File/folder	Description
`maro`	MARO source code.
`docs`	MARO docs, it is host on readthedocs.
`examples`	Showcase of MARO.
`notebooks`	MARO quick-start notebooks.

Try MARO playground to have a quick experience.

Install MARO from PyPI

Notes: The CLI commands (including the visualization tool) are not included in pymaro package. To enable these support, you need to install from source.

Mac OS / Linux
```
pip install pymaro
```

Windows

# Install torch first, if you don't have one.
pip install torch===1.6.0 torchvision===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

pip install pymaro

Install MARO from Source

Notes: Install from source if you want to use the CLI commands (including the visualization tool).

Prerequisites
- C++ Compiler
  - Linux or Mac OS X: gcc
  - Windows: Build Tools for Visual Studio 2017

Enable Virtual Environment

Mac OS / Linux

# If your environment is not clean, create a virtual environment firstly.
python -m venv maro_venv
source ./maro_venv/bin/activate

Windows

# If your environment is not clean, create a virtual environment firstly.
python -m venv maro_venv

# You may need this for SecurityError in PowerShell.
Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy Unrestricted

# Activate the virtual environment.
.\maro_venv\Scripts\activate

Install MARO

# Git Clone the whole source code.
git clone https://github.com/microsoft/maro.git

Mac OS / Linux

# Install MARO from source.
bash scripts/install_maro.sh;
pip install -r ./requirements.dev.txt;

Windows

# Install MARO from source.
.\scripts\install_maro.bat;
pip install -r ./requirements.dev.txt;

Notes: If your package is not found, remember to set your PYTHONPATH
- Mac OS / Linux
```
export PYTHONPATH=PATH-TO-MARO
```
- Windows
```
$Env:PYTHONPATH=PATH-TO-MARO
```

Quick Example

from maro.simulator import Env

env = Env(scenario="cim", topology="toy.5p_ssddd_l0.0", start_tick=0, durations=100)

metrics, decision_event, is_done = env.step(None)

while not is_done:
    metrics, decision_event, is_done = env.step(None)

print(f"environment metrics: {env.metrics}")

Environment Visualization

# Enable environment dump feature, when initializing the environment instance
env = Env(scenario="cim",
          topology="toy.5p_ssddd_l0.0",
          start_tick=0,
          durations=100,
          options={"enable-dump-snapshot": "./dump_data"})

# Inspect environment with the dump data
maro inspector dashboard --source_path ./dump_data/YOUR_SNAPSHOT_DUMP_FOLDER

Show Cases

Case I - Container Inventory Management
Case II - Citi Bike

Run Playground

Pull from Docker Hub

# Pull the docker image from docker hub
docker pull maro2020/playground

# Run playground container.
# Redis commander (GUI for redis) -> http://127.0.0.1:40009
# Jupyter lab with maro -> http://127.0.0.1:40010
docker run -p 40009:40009 -p 40010:40010 maro2020/playground

Build from source

Mac OS / Linux

# Build playground image.
bash ./scripts/build_playground.sh

# Run playground container.
# Redis commander (GUI for redis) -> http://127.0.0.1:40009
# Jupyter lab with maro -> http://127.0.0.1:40010
docker run -p 40009:40009 -p 40010:40010 maro2020/playground

Windows

# Build playground image.
.\scripts\build_playground.bat

# Run playground container.
# Redis commander (GUI for redis) -> http://127.0.0.1:40009
# Jupyter lab with maro -> http://127.0.0.1:40010
docker run -p 40009:40009 -p 40010:40010 maro2020/playground

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Related News

MSRA Top-10 Hack-Techs in 2021

Open Source Platform MARO: Anywhere Door for Resource Optimization

AI from "Point" to "Surface"

Cite Us

License

Licensed under the MIT License.

maro's People

Contributors

Stargazers

Watchers

Forkers

zhawan chaosddp btowner01 romickid jniva chenjz13 buptxieyao tdw1221 gitter-badger you-n-g arthurjiang catwhiskers xiaobaijiang999 wepoiuopijsdfnxcvl meroy9819 j-shang vinayasathyanarayana xniac keshava jinyu-w devbox10 aidin-zadeh bluejazzchn tomzhang hanbaoan123 turtacn lgy-2017 farck kwy518 gingkg zhangruiskyline kangfengjian onepainter hamcoder kaiqli kyu-kuanwei xingzhougmu diansongcao allen-labs microsoftham dgo2dance yourmoonlight kans-alpha mask8082 micli ys610zz world-unity evaren1020-pro fiberleif shippool wacoder sungreong global-localhost global19 global19-atlassian-net phoitack sherrymao 00mjk microsoft-fevieira dukeofcambridge ntoand thebureaugroup riccardopoiani weixinran wuli2496 gx9702 yumiaogithub daaafuuu nareshkumarjaggiexiver ahmedest61 dynamo14324 lanski-ai hiteshmah-jan lenamax2355 zm31 qpc-database simobn jreynolds01 afelipeg beowulf-ss slowy07 vec2k4 wangshibo05 standardgalactic anukaal minkefusiji hewudi666 rockyan1971 phdtai jarvis-k ysqyang hnxbd forchapeatl jbxing itsabhishekhere hangzhang10 yangboz fvaduva python-repository-hub rainwangphy

maro's Issues

💡 Add limitation for out-of-scope action

Summary

Currently, the environment can accept actions for any source-destination pair. It breaks the limitation of the ActionScope in DecisionEvent.

Motivation

Description

References

Data sources:
Papers:
Implementations in other projects:
Draft code snippets:

Alternatives

Additional context

Features of simulator for next stage

working on experimental.simv2 branch.

[P0][DONE] support more data type. Since our query interface should have a certain data type, we may need to change current interface to return float64 numpy array to support 64bit data type, or keep current float32 result, and only support 32bit data type.
[P1][DONE] configurable snapshot length to reduce memory for different scenario. if meet the configured length, we will over-write from beginning.
[P0][DONE] byte based graph memory allocation/layout, and cast to certain data type by length when querying. we will have only one big memory chuck with this, and will make it faster when coping graph into snapshot_list. THIS FEATURE NEED MORE EXPERIMENTS TO TEST.
[P0][DONE] register attributes separately for static and dynamic nodes
[P1][DONE] support priority for events
[P1] pickle support (dump/load).
[P2] simulator parallel

💡 Add standalone K8s mode w/o AzCopy

Summary

Add standalone K8s mode w/o AzCopy

Motivation

Current K8s mode required AzCopy to do the copy operation, it will be nice to implement a new mode to remove this dependency. But the original one will be kept too because of better performance.

Description

In the new mode, we will add a VM for data sync, and link the whole cluster with one Vnet.

Source code for Cooperative Policy Learning with Pre-trained Heterogeneous Observation Representations paper📚

Thank you for this amazing simulator. I'm looking for the source code of the paper mentioned in the subject. Can we please confirm where that exactly been place in this repo given that the paper mentioned a 'code' folder but I can't find it. Thank you!

🐛 Signature for Trajectory.on_env_feedback() Interface should include the arguments used by the Actor class

Description

The Trajectory.on_env_feedback() interface defined here
only includes self as an argument.

However, the Actor() class defined here requires that the on_env_feedback() method include additional arguments:

self.trajectory.on_env_feedback(
                event, state_by_agent, action_by_agent, reward if reward is not None else self.env.metrics
            )

When we define a custom Trajectory, we can overwrite this method (and we do), but the difference in signature produces pylint errors:

W0221:Parameters differ from overridden 'on_env_feedback' method

As far as I can tell, every use of on_env_feedback requires that same signature, so it should be represented in the interface.

Screenshots

To Reproduce

Steps to reproduce the behavior:

Write a custom

Expected Behavior

Environment

MARO version (e.g., v0.1.1a1):
MARO scenario (CIM, Citi Bike):
MARO component (Simulation, RL, Distributed Training):
Orchestration platform (GraSS on Azure, AKS on Azure):
How you installed MARO (pip, source):
OS (Linux, Windows, macOS):
Python version (3.6, 3.7):
Docker image (e.g., maro2020/maro:latest):
CPU/GPU:
Any other relevant information:

Additional Context

💡 Build predefined images for Grass Orchestration

Summary

Build predefined images for Grass Orchestration

Motivation

Reduce the redundant workload of node provision.

Description

Integrate docker and required docker images into a single Linux image, and make sure the provision result matches the current process.
Publish the image into the Azure marketplace.

🐛 I had an error executing "maro inspector geo --start service..."

I had an error executing "maro inspector geo --start service --experiment_name YOUR_EXPERIMENT_NAME --front_end_port 8080", When building maro_vis_front_end, the dist folder does not exist

Screenshots

Environment
MARO version : master branch
MARO scenario (CIM, Citi Bike): cim
MARO component (Simulation, RL, Distributed Training): maro_real_time_vis
Orchestration platform (GraSS on Azure, AKS on Azure): vm on azure
How you installed MARO (pip, source): source
OS (Linux, Windows, macOS): centos
Python version (3.6, 3.7): 3.6
CPU/GPU: cpu

💡 remove proxy.get_peers

Summary

To remove the function proxy.get_peers, and use proxy.peers to replace it.

Motivation

To support dynamic onboard peers.

Description

The proxy.peers is a function which decorated by property, and it references the class attribute (proxy.onboard_peers_dict).

In some customer-defined function or class, use proxy.peers to get the latest onboard peers dict.

In conditional event class, use proxy.peers to replace the proxy.get_peers.

💡 Event object pooling in EventBuffer

Summary

Add pooling support for EventBuffer.Event object.

Motivation

Reduce memory allocation times.

Description

Currently our EventBuffer will create a new event object when there is an requirement, and dispatch it to related handler at a certain time, then push it into finished pool for outside querying. But if the outside does not need this, then these memory will be wasted, and cannot be re-used.

With pooling support, we can re-use event object that the states is completed for further event request, saving lot of memory
and allocation time.

This feature will make get_finished_events method not work, maybe we can:

provide a switch to enable pooling, default is disabled.
provide another way that just dump finished events to files.

References

Alternatives

Additional context

Wrong future stop tick predictions 🐛

Description

There is a strange behavior in future_stop_tick_list attributes.
As you can see, the current tick is 100, and yet there are future vessel tick predictions that are way behind the current tick, for instance [ 75., 88., 96.].

I would expect the snapshot to retrieve ticks that are always ahead of the current one.

To Reproduce

from maro.backends.frame import SnapshotList
from maro.simulator import Env
from pprint import pprint
from typing import List
import numpy as np

# Initialize an Env for CIM scenario
env = Env(scenario="cim", topology="global_trade.22p_l0.8", start_tick=0, durations=200)
    
# Start the environment with None action
_, event, is_done = env.step(None)

event_list_2 = []
while not is_done:
    event_list_2.append((event.port_idx, event.vessel_idx, event.tick))
    metrics, event, is_done = env.step(None)

vs = env.snapshot_list['vessels']
vs[100:[i for i in range(45)]:'future_stop_tick_list'].reshape(45, 3)

Here you can find that output that I had:

array([[ 92., 101., 134.],
       [ 94., 131., 159.],
       [121., 130., 171.],
       [108., 115., 148.],
       [ 83., 102., 130.],
       [116., 147., 153.],
       [109., 114., 137.],
       [ 87., 113., 139.],
       [116., 147., 158.],
       [113., 138., 147.],
       [102., 110., 120.],
       [127., 138., 192.],
       [ 98., 107., 150.],
       [109., 119., 127.],
       [145., 171., 182.],
       [ 68., 111., 132.],
       [124., 130., 158.],
       [121., 127., 158.],
       [100., 111., 136.],
       [ 83., 116., 126.],
       [134., 155., 191.],
       [129., 146., 175.],
       [107., 137., 147.],
       [129., 140., 181.],
       [102., 135., 144.],
       [ 48.,  72.,  87.],
       [ 83., 109., 125.],
       [ 98., 105., 126.],
       [106., 130., 136.],
       [ 97., 108., 117.],
       [109., 136., 173.],
       [ 81., 111., 152.],
       [ 98., 107., 118.],
       [128., 137., 142.],
       [108., 112., 178.],
       [101., 160., 164.],
       [108., 111., 170.],
       [140., 144., 210.],
       [159., 163., 237.],
       [119., 122., 181.],
       [109., 130., 160.],
       [ 88., 102., 111.],
       [102., 107., 134.],
       [ 75.,  88.,  96.],
       [ 98., 107., 113.]], dtype=float32)

Expected Behavior

As you can see, the current tick is 100, but some vessels future ticks are completely in the past.
I would expect this matrix to contain all information that are > 100.

Environment

MARO version (e.g., v0.1.1a1): master
MARO scenario (CIM, Citi Bike): CIM
MARO component (Simulation, RL, Distributed Training): Simulation
Orchestration platform (GraSS on Azure, AKS on Azure):
How you installed MARO (pip, source): source
OS (Linux, Windows, macOS):
Python version (3.6, 3.7): 3.7
Docker image (e.g., maro2020/maro:latest):
CPU/GPU:
Any other relevant information:

Additional Context

EDIT: The problem seems to affect also global configurations without noise in the vessels speed.
E.g. try this config:

seed: 4096
load_cost_factor: 0.05
dsch_cost_factor: 0.05
container_usage_proportion:
  period: 112
  sample_nodes:
  - - 0
    - 0.012
  - - 111
    - 0.012
  sample_noise: 0.002
container_volumes:
- 1
order_generate_mode: fixed
ports:
  bremerhaven_ger:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.16
    order_distribution:
      source:
        noise: 0.025134751494652507
        proportion: 0.16
      targets:
        leHavre_fra:
          noise: 0.025432733187591285
          proportion: 0.18755329471737509
        montreal_can:
          noise: 0.014981815264884504
          proportion: 0.0756906048807678
        newYork_usa:
          noise: 0.07860790872534353
          proportion: 0.47933778254893905
        pusan_kor:
          noise: 0.002902145091237068
          proportion: 0.07667664751459422
        qingdao_chn:
          noise: 0.0016703057083900894
          proportion: 0.0512157380304634
        shanghai_chn:
          noise: 0.004329384963154748
          proportion: 0.04763588234031061
        singapore_sgp:
          noise: 0.00014700838968303593
          proportion: 0.005882857005910926
        yantian_chn:
          noise: 0.014277274600949947
          proportion: 0.07600719296163895
  durban_sau:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.01
    order_distribution:
      source:
        noise: 0.0003905851406890507
        proportion: 0.01
      targets:
        qingdao_chn:
          noise: 0.03386602700899302
          proportion: 0.2502879616953551
        shanghai_chn:
          noise: 0.04099351916502508
          proportion: 0.2501170152675811
        singapore_sgp:
          noise: 0.02983331512414685
          proportion: 0.24812321657873454
        yantian_chn:
          noise: 0.0398838857119423
          proportion: 0.25147180645832923
  itagual_bra:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.01
    order_distribution:
      source:
        noise: 0.0017049012580563228
        proportion: 0.01
      targets:
        qingdao_chn:
          noise: 0.006704071585248814
          proportion: 0.19992517767067194
        santos_bra:
          noise: 0.01065215665926791
          proportion: 0.20145578663243896
        shanghai_chn:
          noise: 0.01889939427623029
          proportion: 0.19975353803822685
        singapore_sgp:
          noise: 0.030425501405892584
          proportion: 0.19775168293424744
        yantian_chn:
          noise: 0.0038049637706619867
          proportion: 0.20111381472441478
  leHavre_fra:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.06
    order_distribution:
      source:
        noise: 0.010023093144315015
        proportion: 0.06
      targets:
        bremerhaven_ger:
          noise: 0.046713884416130814
          proportion: 0.2613085269688243
        montreal_can:
          noise: 0.007011738109808054
          proportion: 0.09474419644790176
        newYork_usa:
          noise: 0.0075575968616494
          proportion: 0.20255309034522379
        pusan_kor:
          noise: 0.0022204132834372364
          proportion: 0.09500755942526971
        qingdao_chn:
          noise: 0.0025163323117031356
          proportion: 0.08820727646821312
        shanghai_chn:
          noise: 0.012253843912920523
          proportion: 0.08725114792024982
        singapore_sgp:
          noise: 0.014556815408633276
          proportion: 0.07609945087001035
        yantian_chn:
          noise: 0.007509358525794673
          proportion: 0.09482875155430699
  losAngeles_usa:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.05
    order_distribution:
      source:
        noise: 0.007441531984981497
        proportion: 0.05
      targets:
        oakland_usa:
          noise: 0.04265188459544233
          proportion: 0.27616189497862537
        princeRupert_can:
          noise: 0.00273897680516611
          proportion: 0.15754923155692402
        qingdao_chn:
          noise: 0.01517255937598155
          proportion: 0.1623492157793647
        seattle_usa:
          noise: 0.014299850500812094
          proportion: 0.2423747616801366
        shanghai_chn:
          noise: 0.0051011672478115
          proportion: 0.16156489600494922
  manzanillo_mex:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.05
    order_distribution:
      source:
        noise: 0.005425915310841583
        proportion: 0.05
      targets:
        manzanillo_mex:
          noise: 0.0073296927183103605
          proportion: 0.1642819028757581
        pusan_kor:
          noise: 0.018169915140256784
          proportion: 0.1269237319062987
        qingdao_chn:
          noise: 0.02383787277319661
          proportion: 0.12072756120502863
        sanAntonio_par:
          noise: 0.0025563544317216205
          proportion: 0.10518021746764428
        shanghai_chn:
          noise: 0.02058384394808081
          proportion: 0.11985637174894495
        yantian_chn:
          noise: 0.00989286132515691
          proportion: 0.12676080545008
        yokohama_jpn:
          noise: 0.0007488162438248653
          proportion: 0.23626940934624543
  melbourne_aus:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.015
    order_distribution:
      source:
        noise: 0.0029979553756433923
        proportion: 0.015
      targets:
        singapore_sgp:
          noise: 0.0566906410775888
          proportion: 0.3293364398135728
        sydney_aus:
          noise: 0.01865388192132147
          proportion: 0.3364188275147848
        yantian_chn:
          noise: 0.014238633997660635
          proportion: 0.33424473267164256
  montreal_can:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.015
    order_distribution:
      source:
        noise: 0.0005621554647653919
        proportion: 0.015
      targets:
        bremerhaven_ger:
          noise: 0.08779643369212999
          proportion: 0.5172441914124826
        leHavre_fra:
          noise: 0.07842477295179612
          proportion: 0.4827558085875176
  newYork_usa:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.03
    order_distribution:
      source:
        noise: 0.00016057717079772171
        proportion: 0.03
      targets:
        bremerhaven_ger:
          noise: 0.05981761122488196
          proportion: 0.5332513888590196
        leHavre_fra:
          noise: 0.03990533991720246
          proportion: 0.46674861114098054
  oakland_usa:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.03
    order_distribution:
      source:
        noise: 0.004910915544061917
        proportion: 0.03
      targets:
        losAngeles_usa:
          noise: 0.048963812038470556
          proportion: 0.30175478642890274
        princeRupert_can:
          noise: 0.0254961817963097
          proportion: 0.23086917432389525
        qingdao_chn:
          noise: 0.03349874892050399
          proportion: 0.23393880851694335
        shanghai_chn:
          noise: 0.012993114869682515
          proportion: 0.23343723073025868
  princeRupert_can:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.015
    order_distribution:
      source:
        noise: 0.0027667472862969076
        proportion: 0.015
      targets:
        losAngeles_usa:
          noise: 0.05091486367601389
          proportion: 0.2657898185062753
        oakland_usa:
          noise: 0.028397329439457514
          proportion: 0.26818216862044597
        qingdao_chn:
          noise: 0.0016646658939369785
          proportion: 0.23313476502804797
        shanghai_chn:
          noise: 0.013742472852754001
          proportion: 0.23289324784523086
  pusan_kor:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.06
    order_distribution:
      source:
        noise: 0.009320427377742116
        proportion: 0.06
      targets:
        bremerhaven_ger:
          noise: 0.031991420231325084
          proportion: 0.21204226502906795
        leHavre_fra:
          noise: 0.005105457599858164
          proportion: 0.08535909653471886
        manzanillo_mex:
          noise: 0.005809147275802185
          proportion: 0.0959125358074426
        qingdao_chn:
          noise: 0.0033945457967072133
          proportion: 0.051610240792657115
        sanAntonio_par:
          noise: 0.001854192137165468
          proportion: 0.035795894860646466
        seattle_usa:
          noise: 0.018330317944630015
          proportion: 0.14202580193983683
        shanghai_chn:
          noise: 0.009459052846765289
          proportion: 0.05072408949335503
        singapore_sgp:
          noise: 0.0016197423680226325
          proportion: 0.04038858042581788
        vancouver_can:
          noise: 0.00469787131269142
          proportion: 0.059258114284493034
        yantian_chn:
          noise: 0.003856257701007031
          proportion: 0.05774708854229363
        yokohama_jpn:
          noise: 0.014803870326138143
          proportion: 0.1691362922896707
  qingdao_chn:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.08
    order_distribution:
      source:
        noise: 0.007181454765417904
        proportion: 0.08
      targets:
        bremerhaven_ger:
          noise: 0.02952708760272511
          proportion: 0.19166868266386958
        durban_sau:
          noise: 0.0026860007722877676
          proportion: 0.025475188174797537
        itagual_bra:
          noise: 0.0023058721270403397
          proportion: 0.020374390901285715
        leHavre_fra:
          noise: 0.008262798336030506
          proportion: 0.04940051725539669
        losAngeles_usa:
          noise: 0.014599621251693524
          proportion: 0.14605127409748705
        manzanillo_mex:
          noise: 0.004578888771735588
          proportion: 0.061252276599797005
        oakland_usa:
          noise: 0.0018968128853613635
          proportion: 0.15590868899027802
        princeRupert_can:
          noise: 7.825075850514062e-05
          proportion: 0.005409393840032325
        pusan_kor:
          noise: 0.0007151691535729856
          proportion: 0.018577689927305203
        santos_bra:
          noise: 0.0036806154087330586
          proportion: 0.02037439104592543
        seattle_usa:
          noise: 0.0016116464454415448
          proportion: 0.113038562331156
        shanghai_chn:
          noise: 0.0011503058311188963
          proportion: 0.010504586083902057
        vancouver_can:
          noise: 0.0031091344585237185
          proportion: 0.02008850220677704
        yantian_chn:
          noise: 0.0017512950798533396
          proportion: 0.018391591436727996
        yokohama_jpn:
          noise: 0.023837737047619264
          proportion: 0.14348426444526224
  sanAntonio_par:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.01
    order_distribution:
      source:
        noise: 0.0015229565945744795
        proportion: 0.01
      targets:
        manzanillo_mex:
          noise: 0.029864110340333205
          proportion: 0.1695702560614863
        pusan_kor:
          noise: 0.030434021872288986
          proportion: 0.16240810942168046
        qingdao_chn:
          noise: 0.031654173870326326
          proportion: 0.1612201993861513
        shanghai_chn:
          noise: 0.022840905205567065
          proportion: 0.16105317774270345
        yantian_chn:
          noise: 0.022058626605590343
          proportion: 0.16237686231768783
        yokohama_jpn:
          noise: 0.020882193597463152
          proportion: 0.18337139507029068
  santos_bra:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.01
    order_distribution:
      source:
        noise: 0.0017591918390130468
        proportion: 0.01
      targets:
        itagual_bra:
          noise: 0.00015677478442405662
          proportion: 0.20145578737905992
        qingdao_chn:
          noise: 0.007951708444894913
          proportion: 0.19992517738434895
        shanghai_chn:
          noise: 0.008394529742993468
          proportion: 0.19975353831991421
        singapore_sgp:
          noise: 0.01406059678653582
          proportion: 0.19775168256638195
        yantian_chn:
          noise: 0.011310434190498734
          proportion: 0.20111381435029493
  seattle_usa:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.07
    order_distribution:
      source:
        noise: 0.009178972285261822
        proportion: 0.07
      targets:
        losAngeles_usa:
          noise: 0.023523817623575388
          proportion: 0.2912736754142012
        pusan_kor:
          noise: 0.01876947062956955
          proportion: 0.12538821688519616
        qingdao_chn:
          noise: 0.02029673306956859
          proportion: 0.11617748395503369
        shanghai_chn:
          noise: 0.011329185762784782
          proportion: 0.11488243812354347
        singapore_sgp:
          noise: 0.00024014816912197655
          proportion: 0.09977785851511455
        vancouver_can:
          noise: 0.02057342705041732
          proportion: 0.1273542861551621
        yantian_chn:
          noise: 0.020083358186626803
          proportion: 0.12514604095174878
  shanghai_chn:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.1
    order_distribution:
      source:
        noise: 0.017109626305467616
        proportion: 0.1
      targets:
        bremerhaven_ger:
          noise: 0.021959106592952006
          proportion: 0.21398371679064213
        durban_sau:
          noise: 0.00010435314516980227
          proportion: 0.017462167174606275
        itagual_bra:
          noise: 0.0020594969751376174
          proportion: 0.011430536063236027
        leHavre_fra:
          noise: 0.0009873199353313093
          proportion: 0.04575353580273499
        losAngeles_usa:
          noise: 0.014743978442771179
          proportion: 0.16004173887997492
        manzanillo_mex:
          noise: 0.0012374290703743276
          proportion: 0.05976808154382374
        oakland_usa:
          noise: 0.022426459611386496
          proportion: 0.1716979971992254
        pusan_kor:
          noise: 0.0005070099495173756
          proportion: 0.00930596400725694
        qingdao_chn:
          noise: 0.8783239577936313e-05
          proportion: 0.0009364036349727401
        santos_bra:
          noise: 0.0012407696401487867
          proportion: 0.011430536346128936
        seattle_usa:
          noise: 0.011299147787220907
          proportion: 0.12100465463531186
        vancouver_can:
          noise: 0.002147321460772641
          proportion: 0.011092475826618629
        yantian_chn:
          noise: 0.000883163479902347
          proportion: 0.009085896257062817
        yokohama_jpn:
          noise: 0.012495339718107597
          proportion: 0.1570062958384047
  singapore_sgp:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.04
    order_distribution:
      source:
        noise: 0.0014198451145378962
        proportion: 0.04
      targets:
        bremerhaven_ger:
          noise: 0.02403294164090424
          proportion: 0.16163465526933787
        durban_sau:
          noise: 0.007339857249890276
          proportion: 0.052947229587582224
        itagual_bra:
          noise: 0.004826948332878951
          proportion: 0.049611409618781833
        leHavre_fra:
          noise: 0.000697323478201817
          proportion: 0.06859394881607536
        melbourne_aus:
          noise: 0.009748446374379698
          proportion: 0.054598902525719965
        pusan_kor:
          noise: 0.0052864459821632135
          proportion: 0.04843640111672956
        qingdao_chn:
          noise: 0.00020264478777542088
          proportion: 0.04380756610509279
        santos_bra:
          noise: 0.0060662606703405855
          proportion: 0.0496114180206766
        seattle_usa:
          noise: 0.002184489196293904
          proportion: 0.11021203091597927
        shanghai_chn:
          noise: 0.00446890645579302
          proportion: 0.043156750350596414
        singapore_sgp:
          noise: 0.0005079804300151535
          proportion: 0.03556597867096123
        sydney_aus:
          noise: 0.0032814259751614335
          proportion: 0.053961624659387086
        vancouver_can:
          noise: 0.0013725415556900408
          proportion: 0.04942443509903704
        yantian_chn:
          noise: 0.007866049277209214
          proportion: 0.04831469461601993
        yokohama_jpn:
          noise: 0.013995694253814077
          proportion: 0.13012295462802295
  sydney_aus:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.015
    order_distribution:
      source:
        noise: 5.318259543305715e-05
        proportion: 0.015
      targets:
        melbourne_aus:
          noise: 0.010959524006592607
          proportion: 0.33665441119009204
        singapore_sgp:
          noise: 0.013378806626384264
          proportion: 0.3291642505650358
        yantian_chn:
          noise: 0.043398566324619615
          proportion: 0.3341813382448723
  vancouver_can:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.02
    order_distribution:
      source:
        noise: 0.002260078974868654
        proportion: 0.02
      targets:
        pusan_kor:
          noise: 0.01022517907397126
          proportion: 0.16323476832180567
        qingdao_chn:
          noise: 0.025111111836521687
          proportion: 0.16078294599848347
        seattle_usa:
          noise: 0.025666860266381805
          proportion: 0.19595625809774195
        shanghai_chn:
          noise: 0.00391575019491024
          proportion: 0.1604382196746799
        singapore_sgp:
          noise: 0.009869415376296499
          proportion: 0.15641751563334233
        yantian_chn:
          noise: 0.01646828787230074
          proportion: 0.1631702922739467
  yantian_chn:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.07
    order_distribution:
      source:
        noise: 0.005460106707641842
        proportion: 0.07
      targets:
        bremerhaven_ger:
          noise: 0.03183934815230198
          proportion: 0.23063506921338092
        durban_sau:
          noise: 0.006955761417407662
          proportion: 0.036977112629701755
        itagual_bra:
          noise: 0.003879986047538319
          proportion: 0.03103337062146673
        leHavre_fra:
          noise: 0.0033746270986396783
          proportion: 0.06485623632186012
        manzanillo_mex:
          noise: 0.014297914647911386
          proportion: 0.07866656356344162
        melbourne_aus:
          noise: 0.0055998037357593185
          proportion: 0.03992004007199072
        pusan_kor:
          noise: 0.004337884540903886
          proportion: 0.028939753926030695
        qingdao_chn:
          noise: 0.0034845335834618927
          proportion: 0.020692152164438926
        santos_bra:
          noise: 0.0023084677027446535
          proportion: 0.03103337095193521
        seattle_usa:
          noise: 0.0043141199296154705
          proportion: 0.13901084519757625
        shanghai_chn:
          noise: 0.0001996474876791818
          proportion: 0.01953252624072474
        singapore_sgp:
          noise: 0.0011502086812016938
          proportion: 0.006007386215135132
        sydney_aus:
          noise: 0.0019319056238978993
          proportion: 0.03878455068512264
        vancouver_can:
          noise: 0.0007858696914021227
          proportion: 0.03070023678474206
        yantian_chn:
          noise: 0.0016168667042640794
          proportion: 0.02872289950079146
        yokohama_jpn:
          noise: 0.023246811699828965
          proportion: 0.17448788591166092
  yokohama_jpn:
    capacity: 10000
    empty_return:
      buffer_ticks: 1
      noise: 0
    full_return:
      buffer_ticks: 1
      noise: 0
    initial_container_proportion: 0.08
    order_distribution:
      source:
        noise: 0.014254332625813097
        proportion: 0.08
      targets:
        manzanillo_mex:
          noise: 0.03244047063421112
          proportion: 0.224773870587767
        pusan_kor:
          noise: 0.018126081484701975
          proportion: 0.1473122137604972
        qingdao_chn:
          noise: 0.017152304680050526
          proportion: 0.1344645529933837
        sanAntonio_par:
          noise: 0.0003086488347039955
          proportion: 0.10222734144856105
        shanghai_chn:
          noise: 0.0056287128415065625
          proportion: 0.13265815413379464
        singapore_sgp:
          noise: 0.019438990767313084
          proportion: 0.1115894561579222
        yantian_chn:
          noise: 0.014838419143613479
          proportion: 0.14697441091807414
routes:
  a3s:
  - distance_to_next_port: 320
    port_name: yantian_chn
  - distance_to_next_port: 240
    port_name: sydney_aus
  - distance_to_next_port: 80
    port_name: melbourne_aus
  asa:
  - distance_to_next_port: 320
    port_name: singapore_sgp
  - distance_to_next_port: 260
    port_name: sydney_aus
  - distance_to_next_port: 60
    port_name: melbourne_aus
  ate1:
  - distance_to_next_port: 160
    port_name: newYork_usa
  - distance_to_next_port: 240
    port_name: bremerhaven_ger
  - distance_to_next_port: 40
    port_name: leHavre_fra
  gex1:
  - distance_to_next_port: 220
    port_name: montreal_can
  - distance_to_next_port: 220
    port_name: bremerhaven_ger
  - distance_to_next_port: 40
    port_name: leHavre_fra
  ktx5:
  - distance_to_next_port: 240
    port_name: singapore_sgp
  - distance_to_next_port: 80
    port_name: yantian_chn
  - distance_to_next_port: 140
    port_name: yokohama_jpn
  ll4:
  - distance_to_next_port: 80
    port_name: shanghai_chn
  - distance_to_next_port: 60
    port_name: yantian_chn
  - distance_to_next_port: 80
    port_name: singapore_sgp
  - distance_to_next_port: 380
    port_name: leHavre_fra
  - distance_to_next_port: 80
    port_name: bremerhaven_ger
  - distance_to_next_port: 420
    port_name: singapore_sgp
  - distance_to_next_port: 200
    port_name: qingdao_chn
  - distance_to_next_port: 80
    port_name: pusan_kor
  pcc1:
  - distance_to_next_port: 240
    port_name: qingdao_chn
  - distance_to_next_port: 40
    port_name: shanghai_chn
  - distance_to_next_port: 240
    port_name: princeRupert_can
  - distance_to_next_port: 120
    port_name: losAngeles_usa
  - distance_to_next_port: 100
    port_name: oakland_usa
  pcc2:
  - distance_to_next_port: 80
    port_name: shanghai_chn
  - distance_to_next_port: 280
    port_name: losAngeles_usa
  - distance_to_next_port: 160
    port_name: seattle_usa
  - distance_to_next_port: 280
    port_name: qingdao_chn
  pnw1:
  - distance_to_next_port: 80
    port_name: yantian_chn
  - distance_to_next_port: 260
    port_name: vancouver_can
  - distance_to_next_port: 80
    port_name: seattle_usa
  - distance_to_next_port: 320
    port_name: pusan_kor
  pnw2:
  - distance_to_next_port: 200
    port_name: singapore_sgp
  - distance_to_next_port: 120
    port_name: yantian_chn
  - distance_to_next_port: 80
    port_name: shanghai_chn
  - distance_to_next_port: 60
    port_name: pusan_kor
  - distance_to_next_port: 200
    port_name: seattle_usa
  - distance_to_next_port: 40
    port_name: vancouver_can
  - distance_to_next_port: 520
    port_name: qingdao_chn
  - distance_to_next_port: 60
    port_name: shanghai_chn
  saf3:
  - distance_to_next_port: 80
    port_name: qingdao_chn
  - distance_to_next_port: 40
    port_name: shanghai_chn
  - distance_to_next_port: 60
    port_name: yantian_chn
  - distance_to_next_port: 80
    port_name: singapore_sgp
  - distance_to_next_port: 260
    port_name: durban_sau
  - distance_to_next_port: 360
    port_name: yantian_chn
  tla2:
  - distance_to_next_port: 60
    port_name: qingdao_chn
  - distance_to_next_port: 60
    port_name: shanghai_chn
  - distance_to_next_port: 80
    port_name: yantian_chn
  - distance_to_next_port: 100
    port_name: singapore_sgp
  - distance_to_next_port: 460
    port_name: itagual_bra
  - distance_to_next_port: 20
    port_name: santos_bra
  - distance_to_next_port: 580
    port_name: singapore_sgp
  - distance_to_next_port: 140
    port_name: yantian_chn
  - distance_to_next_port: 80
    port_name: shanghai_chn
  tlp1:
  - distance_to_next_port: 100
    port_name: yantian_chn
  - distance_to_next_port: 60
    port_name: shanghai_chn
  - distance_to_next_port: 60
    port_name: qingdao_chn
  - distance_to_next_port: 40
    port_name: pusan_kor
  - distance_to_next_port: 260
    port_name: manzanillo_mex
  - distance_to_next_port: 140
    port_name: sanAntonio_par
  - distance_to_next_port: 180
    port_name: manzanillo_mex
  - distance_to_next_port: 260
    port_name: yokohama_jpn
  - distance_to_next_port: 60
    port_name: shanghai_chn
stop_number:
- 4
- 3
total_containers: 100000
vessels:
  a3s_vessel_000:
    capacity: 7739
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: yantian_chn
      route_name: a3s
    sailing:
      noise: 0
      speed: 10
  a3s_vessel_001:
    capacity: 7739
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: sydney_aus
      route_name: a3s
    sailing:
      noise: 0
      speed: 10
  asa_vessel_000:
    capacity: 4379
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: singapore_sgp
      route_name: asa
    sailing:
      noise: 0
      speed: 10
  asa_vessel_001:
    capacity: 4379
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: melbourne_aus
      route_name: asa
    sailing:
      noise: 0
      speed: 10
  ate1_vessel_000:
    capacity: 46069
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: newYork_usa
      route_name: ate1
    sailing:
      noise: 0
      speed: 10
  ate1_vessel_001:
    capacity: 46069
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: bremerhaven_ger
      route_name: ate1
    sailing:
      noise: 0
      speed: 10
  gex1_vessel_000:
    capacity: 9182
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: bremerhaven_ger
      route_name: gex1
    sailing:
      noise: 0
      speed: 10
  gex1_vessel_001:
    capacity: 9182
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: montreal_can
      route_name: gex1
    sailing:
      noise: 0
      speed: 10
  ktx5_vessel_000:
    capacity: 11328
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: singapore_sgp
      route_name: ktx5
    sailing:
      noise: 0
      speed: 10
  ktx5_vessel_001:
    capacity: 11328
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: yokohama_jpn
      route_name: ktx5
    sailing:
      noise: 0
      speed: 10
  ll4_vessel_000:
    capacity: 52831
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: bremerhaven_ger
      route_name: ll4
    sailing:
      noise: 0
      speed: 10
  ll4_vessel_001:
    capacity: 52831
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: yantian_chn
      route_name: ll4
    sailing:
      noise: 0
      speed: 10
  ll4_vessel_002:
    capacity: 52831
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: singapore_sgp
      route_name: ll4
    sailing:
      noise: 0
      speed: 10
  ll4_vessel_003:
    capacity: 52831
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: leHavre_fra
      route_name: ll4
    sailing:
      noise: 0
      speed: 10
  ll4_vessel_004:
    capacity: 52831
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: pusan_kor
      route_name: ll4
    sailing:
      noise: 0
      speed: 10
  ll4_vessel_005:
    capacity: 52831
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: qingdao_chn
      route_name: ll4
    sailing:
      noise: 0
      speed: 10
  pcc1_vessel_000:
    capacity: 39380
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: oakland_usa
      route_name: pcc1
    sailing:
      noise: 0
      speed: 10
  pcc1_vessel_001:
    capacity: 39380
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: qingdao_chn
      route_name: pcc1
    sailing:
      noise: 0
      speed: 10
  pcc1_vessel_002:
    capacity: 39380
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: princeRupert_can
      route_name: pcc1
    sailing:
      noise: 0
      speed: 10
  pcc2_vessel_000:
    capacity: 19551
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: seattle_usa
      route_name: pcc2
    sailing:
      noise: 0
      speed: 10
  pcc2_vessel_001:
    capacity: 19551
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: losAngeles_usa
      route_name: pcc2
    sailing:
      noise: 0
      speed: 10
  pcc2_vessel_002:
    capacity: 19551
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: shanghai_chn
      route_name: pcc2
    sailing:
      noise: 0
      speed: 10
  pnw1_vessel_000:
    capacity: 17039
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: seattle_usa
      route_name: pnw1
    sailing:
      noise: 0
      speed: 10
  pnw1_vessel_001:
    capacity: 17039
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: vancouver_can
      route_name: pnw1
    sailing:
      noise: 0
      speed: 10
  pnw1_vessel_002:
    capacity: 17039
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: yantian_chn
      route_name: pnw1
    sailing:
      noise: 0
      speed: 10
  pnw2_vessel_000:
    capacity: 7178
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: pusan_kor
      route_name: pnw2
    sailing:
      noise: 0
      speed: 10
  pnw2_vessel_001:
    capacity: 7178
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: yantian_chn
      route_name: pnw2
    sailing:
      noise: 0
      speed: 10
  pnw2_vessel_002:
    capacity: 7178
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: vancouver_can
      route_name: pnw2
    sailing:
      noise: 0
      speed: 10
  pnw2_vessel_003:
    capacity: 7178
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: pusan_kor
      route_name: pnw2
    sailing:
      noise: 0
      speed: 10
  pnw2_vessel_004:
    capacity: 7178
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: qingdao_chn
      route_name: pnw2
    sailing:
      noise: 0
      speed: 10
  saf3_vessel_000:
    capacity: 5177
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: yantian_chn
      route_name: saf3
    sailing:
      noise: 0
      speed: 10
  saf3_vessel_001:
    capacity: 5177
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: qingdao_chn
      route_name: saf3
    sailing:
      noise: 0
      speed: 10
  saf3_vessel_002:
    capacity: 5177
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: singapore_sgp
      route_name: saf3
    sailing:
      noise: 0
      speed: 10
  saf3_vessel_003:
    capacity: 5177
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: durban_sau
      route_name: saf3
    sailing:
      noise: 0
      speed: 10
  tla2_vessel_000:
    capacity: 9480
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: itagual_bra
      route_name: tla2
    sailing:
      noise: 0
      speed: 10
  tla2_vessel_001:
    capacity: 9480
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: santos_bra
      route_name: tla2
    sailing:
      noise: 0
      speed: 10
  tla2_vessel_002:
    capacity: 9480
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: singapore_sgp
      route_name: tla2
    sailing:
      noise: 0
      speed: 10
  tla2_vessel_003:
    capacity: 9480
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: shanghai_chn
      route_name: tla2
    sailing:
      noise: 0
      speed: 10
  tla2_vessel_004:
    capacity: 9480
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: qingdao_chn
      route_name: tla2
    sailing:
      noise: 0
      speed: 10
  tla2_vessel_005:
    capacity: 9480
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: yantian_chn
      route_name: tla2
    sailing:
      noise: 0
      speed: 10
  tlp1_vessel_000:
    capacity: 50660
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: yantian_chn
      route_name: tlp1
    sailing:
      noise: 0
      speed: 10
  tlp1_vessel_001:
    capacity: 50660
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: manzanillo_mex
      route_name: tlp1
    sailing:
      noise: 0
      speed: 10
  tlp1_vessel_002:
    capacity: 50660
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: pusan_kor
      route_name: tlp1
    sailing:
      noise: 0
      speed: 10
  tlp1_vessel_003:
    capacity: 50660
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: manzanillo_mex
      route_name: tlp1
    sailing:
      noise: 0
      speed: 10
  tlp1_vessel_004:
    capacity: 50660
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: sanAntonio_par
      route_name: tlp1
    sailing:
      noise: 0
      speed: 10
  tlp1_vessel_005:
    capacity: 50660
    parking:
      duration: 1
      noise: 0
    route:
      initial_port_name: yokohama_jpn
      route_name: tlp1
    sailing:
      noise: 0
      speed: 10

clang: error: no such file or directory: './maro/backends/backend.c'

Description

Mac: python setup.py build

clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/Users/yewang/.virtualenvs/maro/include -I/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/include/python3.6m -c ./maro/backends/backend.c -o build/temp.macosx-10.12-x86_64-3.6/./maro/backends/backend.o
clang: error: no such file or directory: './maro/backends/backend.c'
clang: error: no input files
error: command 'clang' failed with exit status 1

Screenshots

To Reproduce

Steps to reproduce the behavior:

Expected Behavior

Environment

MARO version (e.g., v0.1.1a1):
MARO scenario (CIM, Citi Bike):
MARO component (Simulation, RL, Distributed Training):
Orchestration platform (GraSS on Azure, AKS on Azure):
How you installed MARO (pip, source):
OS (Linux, Windows, macOS):
Python version (3.6, 3.7):
Docker image (e.g., arthursjiang/maro:cpu[5f36ed]):
CPU/GPU:
Any other relevant information:

Additional Context

Wrong prediction in future stop list in CIM scenario 🐛

Description

While estimating future vessel stops in VesselFutureStopsPrediction there is a bug in the starting point when last_loc_idx == next_loc_idx.

In particular, when the previous condition is satisfied, the predictions retrieved by VesselFutureStopsPrediction._predict_future_stops() are one step ahead.

Screenshots

To Reproduce

To reproduce the behavior, just run the following snippet of code.

from maro.backends.frame import SnapshotList
from maro.simulator import Env
from pprint import pprint
from typing import List
import numpy as np

# Initialize an Env for CIM scenario
env = Env(scenario="cim", topology="toy.4p_ssdd_l0.0", start_tick=0, durations=1120)
    
# Start the environment with None action
_, event, is_done = env.step(None)

while not is_done:
    metrics, event, is_done = env.step(None)

vessel_snapshots = env.snapshot_list["vessels"]
print(vessel_snapshots[[i for i in range(0, 20)]:2:['future_stop_list', 'future_stop_tick_list', 'last_loc_idx']].reshape(20, 7))

For toy.4p_ssdd_l0.0 vessel arrivals happen every 7 steps. You can see that when at t such that t % 7 == 0, predictions will jump one step ahead.

Expected Behavior

When last_loc_idx == next_loc_idx predictions should not be 1-step ahead.

Environment

MARO version (e.g., v0.1.1a1): master
MARO scenario (CIM, Citi Bike): CIM
MARO component (Simulation, RL, Distributed Training): Simulation
Orchestration platform (GraSS on Azure, AKS on Azure):
How you installed MARO (pip, source): source
OS (Linux, Windows, macOS): Linux
Python version (3.6, 3.7): 3.7
Docker image (e.g., arthursjiang/maro:cpu[5f36ed]):
CPU/GPU:
Any other relevant information:

Additional Context

AbsAgent in Learner with exploration params 🐛

Description

When an AbsAgent with exploration params is used in OffPolicyLearner an error occur due to the Actor class.
Running DQN in a "single agent mode", the error will be:

return greedy_action if np.random.random() > self.config.epsilon else np.random.choice(num_actions) 
TypeError: '>' not supported between instances of 'float' and 'dict'

Indeed, within Actor, we have self.agent.set_exploration_params(exploration_params) that sets the entire dict as value.

To Reproduce

Steps to reproduce the behavior:

Modify dqn.main.py in the following way:

class CIMTrajectoryForDQN(CIMTrajectory):
    def on_finish(self):
        exp = {"S": [], "A": [], "R": [], "S_": []}
        for i in range(len(self.trajectory["state"]) - 1):
            exp["S"].append(self.trajectory["state"][i])
            exp["A"].append(self.trajectory["action"][i])
            exp["R"].append(self.get_offline_reward(self.trajectory["event"][i]))
            exp["S_"].append(list(self.trajectory["state"][i + 1]))
        return exp

if __name__ == "__main__":
    set_seeds(1024)  # for reproducibility
    env = Env(**training_config["env"])
    agent = get_dqn_agent()
    actor = Actor(env, agent, CIMTrajectoryForDQN, trajectory_kwargs=common_config)  # local actor
    scheduler = TwoPhaseLinearParameterScheduler(training_config["max_episode"], **training_config["exploration"])
    learner = OffPolicyLearner(actor, scheduler, agent, **training_config["training"], log_dir=log_dir)
    learner.run()

Modify dqn/config/training_config.py setting min_experiences_to_train to 128
Modify examples/cim/common.py in the following way:

def get_state(self, event):
      vessel_snapshots, port_snapshots = self.env.snapshot_list["vessels"], self.env.snapshot_list["ports"]
      tick, port_idx, vessel_idx = event.tick, event.port_idx, event.vessel_idx
      ticks = [max(0, tick - rt) for rt in range(self.look_back - 1)]
      future_port_idx_list = vessel_snapshots[tick: vessel_idx: 'future_stop_list'].astype('int')
      port_features = port_snapshots[ticks: [port_idx] + list(future_port_idx_list): self.port_attributes]
      vessel_features = vessel_snapshots[tick: vessel_idx: self.vessel_attributes]
      return np.concatenate((port_features, vessel_features))

def get_action(self, action_by_agent, event):
      vessel_snapshots = self.env.snapshot_list["vessels"]
      # action_info = list(action_by_agent.values())[0]
      action_info = action_by_agent
      model_action = action_info[0] if isinstance(action_info, tuple) else action_info
      scope, tick, port, vessel = event.action_scope, event.tick, event.port_idx, event.vessel_idx
      zero_action_idx = len(self.action_space) / 2  # index corresponding to value zero.
      vessel_space = vessel_snapshots[tick:vessel:self.vessel_attributes][2] if self.finite_vessel_space else float("inf")
      early_discharge = vessel_snapshots[tick:vessel:"early_discharge"][0] if self.has_early_discharge else 0
      percent = abs(self.action_space[model_action])
  
      if model_action < zero_action_idx:
          action_type = ActionType.LOAD
          actual_action = min(round(percent * scope.load), vessel_space)
      elif model_action > zero_action_idx:
          action_type = ActionType.DISCHARGE
          plan_action = percent * (scope.discharge + early_discharge) - early_discharge
          actual_action = round(plan_action) if plan_action > 0 else round(percent * scope.discharge)
      else:
          actual_action, action_type = 0, None

Expected Behavior

No crash should happend.
As a fix I applied:

if isinstance(self.agent, AbsAgent):
    self.agent.set_exploration_params(**exploration_params)
else:
    self.agent.set_exploration_params(exploration_params)

Environment

MARO version (e.g., v0.1.1a1): master
MARO scenario (CIM, Citi Bike): CIM
MARO component (Simulation, RL, Distributed Training): RL
Orchestration platform (GraSS on Azure, AKS on Azure): None
How you installed MARO (pip, source): source
OS (Linux, Windows, macOS): Linux
Python version (3.6, 3.7): 3.7
Docker image (e.g., arthursjiang/maro:cpu[5f36ed]):
CPU/GPU: cpu
Any other relevant information:

💡 Implementation of more RL algorithms

Summary

Implementation of more RL algorithms.

Motivation

The only out-of-the-box algorithm provided in v0.1 is the DQN algorithm, and it still leaves something to be desired as there is no support for various DQN variants such as dueling Q models and double DQN. We intend to add support to these features in v0.2. We also plan to provide implementations of more RL algorithms to make things easy for those who want to use pre-defined algorithms instead of writing their own from scratch.

Description

More DQN features such as dueling and double DQN. Other classic RL algorithms such as PG, PPO, DDPG, A2C, A3C, etc.

References

Papers:
PG: Reinforcement Learning: An Introduction (R. S. Sutton and A. Barto), 2nd ed., pp. 321
PPO: https://arxiv.org/pdf/1707.06347.pdf
DDPG: https://arxiv.org/pdf/1509.02971.pdf
A2C: https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f
A3C: https://arxiv.org/pdf/1602.01783.pdf
Implementations in other projects:
Draft code snippets:

Alternatives

Additional context

Visualization Error - CIM AC Example

Hi, I'm trying to run Cim - AC example with Visualization support. I provide the 'enable-dump-snapshot' option in the Env() of main file and run the inspector dashboard from terminal while providing the path of dump snapshot. I face the error related to StreamlitAPIExpection due to which I'm unable to visualization the plots. Can someone please confirm why I'm facing this issue. Thank you.

🐛

Description

typo spelling grammar on microsoft/maro

To Reproduce

Steps to reproduce the behavior:

change to correct grammar with reference from Miriam webster

./tests/test_event_buffer.py:77: dispathing  ==> dispatching
./tests/test_event_buffer.py:145: folloing  ==> following

Environment

MARO version (e.g., v0.1.1a1):
MARO scenario (CIM, Citi Bike):
MARO component (Simulation, RL, Distributed Training):
Orchestration platform (GraSS on Azure, AKS on Azure):
How you installed MARO (pip, source): 20.0.2
OS (Linux, Windows, macOS): linux arch
Python version (3.6, 3.7): 3.8
Docker image (e.g., maro2020/maro:latest):
CPU/GPU: amd ryzen 7 4800h/ gtx1650TI
Any other relevant information:

[Playground] Build Failed

📚 Can not realize the example as Readme

Description

Firstly, I run a container by arthursjiang/maro:cpu, and copy the example in readme(Quick Example) with part of Environment Visualization and run it:

from maro.simulator import Env

# Enable environment dump feature, when initializing the environment instance
env = Env(scenario="cim",
          topology="toy.5p_ssddd_l0.0",
          start_tick=0,
          durations=100,
          options={"enable-dump-snapshot": "./dump_data"})

metrics, decision_event, is_done = env.step(None)

while not is_done:
    metrics, decision_event, is_done = env.step(None)

print(f"environment metrics: {env.metrics}")

but it will show me this error log:

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-3-2bba340a34d8> in <module>
      1 from maro.simulator import Env
      2 
----> 3 env = Env(scenario="cim", topology="toy.5p_ssddd_l0.0", start_tick=0, durations=100, options={"enable-dump-snapshot": "./dump_data"})
      4 
      5 metrics, decision_event, is_done = env.step(None)

/usr/local/lib/python3.6/site-packages/maro/simulator/core.py in __init__(self, scenario, topology, start_tick, durations, snapshot_resolution, max_snapshots, decision_mode, business_engine_cls, disable_finished_events, options)
     69             parent_path = self._additional_options["enable-dump-snapshot"]
     70             self._converter = DumpConverter(parent_path, self._business_engine._scenario_name)
---> 71             self._converter.reset_folder_path()
     72 
     73     def step(self, action):

/usr/local/lib/python3.6/site-packages/maro/data_lib/dump_csv_converter.py in reset_folder_path(self)
     51 
     52     def reset_folder_path(self):
---> 53         self._generate_new_folder(self._parent_path)
     54         self._serial = 0
     55 

/usr/local/lib/python3.6/site-packages/maro/data_lib/dump_csv_converter.py in _generate_new_folder(self, parent_path)
     36         if folder_path.exists():
     37             return
---> 38         os.mkdir(self._foldername)
     39 
     40     @property

FileNotFoundError: [Errno 2] No such file or directory: './dump_data/snapshot_dump_2021_01_11_03_01_56_834'

I realized this will caused by this file (dump_data) does not in this folder. So, I use FIND script try to find out where the file is. But, the log like this:

shaiic@zy-openpai-marketplace:~/zhangy/maro$ ls
CHANGELOG.md  CITATION  CODE_OF_CONDUCT.md  CONTRIBUTING.md  LICENSE  MANIFEST.in  README.md  SECURITY.md  docker_files  docs  examples  maro  notebooks  playground.md  pyproject.toml  requirements.dev.txt  scripts  setup.py  tests
shaiic@zy-openpai-marketplace:~/zhangy/maro$ find . -name "dump_data.*"
shaiic@zy-openpai-marketplace:~/zhangy/maro$ find . -name "dump_data"

This is my server configuration:

shaiic@zy-openpai-marketplace:~/zhangy/maro$ neofetch 
            .-/+oossssoo+/-.               shaiic@zy-openpai-marketplace 
        `:+ssssssssssssssssss+:`           ----------------------------- 
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 18.04.5 LTS x86_64 
    .ossssssssssssssssssdMMMNysssso.       Host: Virtual Machine 7.0 
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Kernel: 5.4.0-1031-azure 
  +ssssssssshmydMMMMMMMNddddyssssssss+     Uptime: 68 days, 1 hour, 39 mins 
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Packages: 640 
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Shell: bash 4.4.20 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Terminal: vscode 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   CPU: Intel Xeon Platinum 8171M (4) @ 2.095GHz 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   GPU: Microsoft Corporation Hyper-V virtual VGA 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Memory: 753MiB / 15987MiB 
.ssssssssdMMMNhsssssssssshNMMMdssssssss. 
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/                            
  +sssssssssdmydMMMMMMMMddddyssssssss+ 
   /ssssssssssshdmNNNNmyNMMMMhssssss/ 
    .ossssssssssssssssssdMMMNysssso. 
      -+sssssssssssssssssyyyssss+- 
        `:+ssssssssssssssssss+:` 
            .-/+oossssoo+/-.

So, How can I get result like this?

By the way, should I install Maro by resouce code before Or What dependence I need to get before to run script in Readme.md L149?

# Inspect environment with the dump data
maro inspector env --source ./dump_data

Is maro ready for production use? for example, in the VM Scheduling Scenario

@Jinyu-W

💡 Frame implementation, and backend interface revamping

Summary

Frame implementation, and backend interface revamping.

Motivation

Current frame implementation uses node and attribute name tuple as key to query attribute, then use it to access value, this is slow as it will create a tuple object every time we access an attribute.

Description

Revamp frame internal implementation to use int instead of string to get/set/query attributes, then we use this as unique key for quick accessing.
Based on 1st option, we need to revamp backend interfaces to support return an ID for new added node and attribute, then access this ID to access value.
Since we are going to use c++ to implement raw backend, we also need to upgrade current np backend and frame layer to use c++ compiler. Ideally we only need replace "class" with "cppclass" at cython part.

Above changes would not affect current frame interfaces for python side.

References

Draft code snippets:

# backend layer

cdef cppclass backend:
    IDENTIFIER add_node(str name, UINT number)
    IDENTIFIER add_attr(IDENTIFIER node_id, str name, str dtype, UINT slots)

    object get_attr_value(IDENTIFIER id, UINT slot_index)
    object set_attr_value(IDENTIFIER id, UINT slot_index, object value)

# frame layer

node1_id = backend.add_node("node1", 10)
node2_id = backend.add_node("node2", 5)

attr1_id = backend.add_attr(node1_id, "a1", "i", 2)
attr2_id = backend.add_attr(node1_id, "a2", "i", 3)

val = backend.get_attr_value(attr1_id, 1, 1)
backend.set_attr_value(attr1_id, 0, 12)

Alternatives

Additional context

💡 The implementation of proxy rejoins

Summary

The proxy rejoining allows the components to reconnect during the program running; also, it can handle the new components to join.

Motivation

To avoid the crash caused by a single component failed.
To support components increasing/decreasing during the program running

Description

The proxy must be able to monitor the Redis periodically; therefore, the proxy can get the latest peer list from the Redis. (implement by multi-threading)
The proxy has connect/disconnect functions for peers change; and the driver should have the disconnect function.
The proxy need update peers' name list. (use "@ property" in proxy.peers. If updated peers in the Proxy, it can also change the variable which called proxy.peers.)

Scheduler bug in SimpleMultiHeadModel 🐛

Description

When creating schedulers in AbsCoreModel a bug occur if isinstance(optim_option, dict) is True in the constructor.
This happens since while doing self.scheduler[name] = opt.scheduler_cls(self.optimizer[name], **opt.scheduler_params) since self.scheduler is not defined.

Screenshots

To Reproduce

Steps to reproduce the behavior:

Create a SimpleMultiHeadModel using a model with a dictionary and optim options that has schedulers.

Expected Behavior

No crash should occurr.

Environment

MARO version (e.g., v0.1.1a1): master
MARO scenario (CIM, Citi Bike):
MARO component (Simulation, RL, Distributed Training): RL
Orchestration platform (GraSS on Azure, AKS on Azure):
How you installed MARO (pip, source): source
OS (Linux, Windows, macOS):
Python version (3.6, 3.7): 3.7
Docker image (e.g., arthursjiang/maro:cpu[5f36ed]):
CPU/GPU:
Any other relevant information:

Additional Context

Noise in the environment 🐛

Description

Currently, the data generation process works in the following way.
Whenever there is noise in the environment configurations, fixed a seed, the data generation process will be fixed, thus generating always the same "environment trajectory" (i.e., same order distribution, same vessel speeds, same vessel parking noise).

Expected Behavior

I would expect each "environment trajectory" to be different from the previous one (i.e., after a reset), in the sense that a different noise should be applied each time the environment is reset.

This is crucial also for the different reasons that are mentioned in both of your papers: if not done, the environment is fully deterministic (and one of the main reason to apply methods based on RL is the way in which they can handle uncertainty, as it happens in truly real scenarios indeed).
If this is not done, the performances that any RL-based method is able to achieve are flawed. In this case, indeed, it is obvious that the method is overfitting the "noise" in that specific configuration (at this point, it is even missleading to call it noise, since each trajectory generates the same exact data) .

Environment

MARO version (e.g., v0.1.1a1): master
MARO scenario (CIM, Citi Bike): CIM
MARO component (Simulation, RL, Distributed Training): Simulation
Orchestration platform (GraSS on Azure, AKS on Azure):
How you installed MARO (pip, source): source
OS (Linux, Windows, macOS): Linux
Python version (3.6, 3.7): 3.7
Docker image (e.g., maro2020/maro:latest):
CPU/GPU:
Any other relevant information:

Offline/Online LP for CIM scenario 📚

Description

Add Offline and Online Linear programming agent for CIM scenario

Link: paper 1 paper 2
Screenshot:

On-policy training with multiple actors🐛

Description

When running On-policy with multiple actors, ExperienceCollectionUtils.stack will build a list containing in each element the results of CIMTrajectoryForAC.on_finish. At this point, OnPolicyLearner will call agent.learn on each element of this list.
However, ActorCritic, that requires on-policy data, will be on-policy only at the first iteration of this list. When data from the second actor are processed, the policy has already been updated multiple times (i.e., at the first iteration).

Screenshots

To Reproduce

Steps to reproduce the behavior:

Modify examples/ac/main.py as in examples/dqn/main.py to run the code with multiple actors

Expected Behavior

Ideally, ExperienceCollectionUtils.stack should concatenate all the data collected from the different actors into a single dataset (for the single agent mode) or single dataset for each agent (for multi-agent mode) and then call the update a single time (or a single time for each agent). Indeed, the "references" advantages computed at the beginning of the learning process should remain fixed for the entire update.
In other words, the following line of code from ActorCritic

state_values = self.model(states, task_name="critic").detach().squeeze()
return_est = get_lambda_returns(
    rewards, state_values, self.config.reward_discount, self.config.lam, k=self.config.k
)
advantages = return_est - state_values

should be executed a single time at each iteration (or a single time for each agent).

A more advanced ActorCritic implementation will specify an sgd_batch_size and will break the collected batch into mini-batch using random sampling at each training iteration.
It would also be interesting to specify a global_batch_size, so that actors will collect up to global_batch_size data to trigger an update (instead of having learner_update_trigger).

Environment

MARO version (e.g., v0.1.1a1): master
MARO scenario (CIM, Citi Bike): CIM
MARO component (Simulation, RL, Distributed Training): RL
Orchestration platform (GraSS on Azure, AKS on Azure):
How you installed MARO (pip, source): source
OS (Linux, Windows, macOS):
Python version (3.6, 3.7):
Docker image (e.g., arthursjiang/maro:cpu[5f36ed]):
CPU/GPU:
Any other relevant information:

Additional Context

🐛 I had an error executing "maro inspector geo --start database"

Description

I had an error executing "maro inspector geo --start database"

Screenshots

Environment

MARO version : master branch
MARO scenario (CIM, Citi Bike): cim
MARO component (Simulation, RL, Distributed Training): maro_real_time_vis
Orchestration platform (GraSS on Azure, AKS on Azure): vm on azure
How you installed MARO (pip, source): source
OS (Linux, Windows, macOS): centos
Python version (3.6, 3.7): 3.6
Docker image (e.g., arthursjiang/maro:cpu[5f36ed]):
CPU/GPU: cpu

Visualization import error🐛

Description

Hello, I have just installed MARO using pypl, followed the quick example, but when running the visualization command maro inspector dashboard --source_path ./dump_data/snapshot_dump_folder I get the following error:

ModuleNotFoundError: No module named 'maro.cli.maro_real_time_vis.back_end'

Screenshots

To Reproduce

Steps to reproduce the behavior:

Install MARO using pypl
Run the quick example
Run the visualization command maro inspector dashboard --source_path ./dump_data/snapshot_dump_folder

Expected Behavior

Environment

MARO version (e.g., v0.1.1a1): 0.2.1a1
MARO scenario (CIM, Citi Bike): CIM
MARO component (Simulation, RL, Distributed Training):
Orchestration platform (GraSS on Azure, AKS on Azure):
How you installed MARO (pip, source): pip
OS (Linux, Windows, macOS): Linux
Python version (3.6, 3.7): 3.7
Docker image (e.g., arthursjiang/maro:cpu[5f36ed]):
CPU/GPU:
Any other relevant information:

Additional Context

remaining space not correct updated on full discharging

🐛 Redis database not cleared when execution terminates

Description

Redis database is not cleared once a running script is over.
If you run DQN example twice (using the same distributed group name), the second time you will get stuck at episode 0. Indeed, rollouts are sent to actor's IDs that were running at the first execution. Note that this won't happen if you stop the first execution.

I currently solved the problem using redis-cli flushall, however I categorize this as unexpected behavior since these informations should be cleaned when the execution terminates.

To Reproduce

Steps to reproduce the behavior:

Run examples/cim/dqn/main.py and wait till completion (you can change the episodes number to a small number e.g., 5 to speed up things)
Run examples/cim/dqn/main.py

Expected Behavior

Redis cache should be cleared when the learning process ends.

The second DQN execution should send rollouts to the actors that are currently running, and not the previous ones, that should not be stored in the redis database.

Environment

MARO version (e.g., v0.1.1a1): master
MARO scenario (CIM, Citi Bike): CIM
MARO component (Simulation, RL, Distributed Training): Distributed Training
Orchestration platform (GraSS on Azure, AKS on Azure):
How you installed MARO (pip, source): source
OS (Linux, Windows, macOS): Linux
Python version (3.6, 3.7): 3.7
Docker image (e.g., arthursjiang/maro:cpu[5f36ed]):
CPU/GPU: cpu
Any other relevant information:

maro cli can not use on win10

install maro with "pip install --user .\maro-0.1.1a0-cp37-cp37m-win_amd64.whl"
type "maro" in conda console, out put is:

maro : 无法将“maro”项识别为 cmdlet、函数、脚本文件或可运行程序的名称。请检查名称的拼写，如果包括路径，请确保路径正确，然后再试一次。
所在位置行:1 字符: 1

maro

  + CategoryInfo          : ObjectNotFound: (maro:String) [], CommandNotFoundException
  + FullyQualifiedErrorId : CommandNotFoundException

💡 Add Depots inside the CIM scenario

Summary

Currently, the CIM environment considers only the ports Vs vessels interaction. However, depots also play an important role for ports in the storage/use of containers (empty/laden) in the shipping network.

Motivation

Once the cargo arrives, the port unloads it but may or may not directly send those to the consignee. The cargo can be placed (stored) preferably at some nearby depot(s). Similarly, after the cargo reaches to the consignee through other mean of transportation (e.g., intermodal), the empty container(s) that returned back to the port may need to be stored at some depot. It is because the empty containers may not loaded immediately on the vessel. There are factors like next ship arrival day/time, orders from consignor at the ports and so on. Therefore. the container movement between ports and depots shouldn't be ignored while carrying empty container repositioning (ECR).

Description

References

Data sources:
Papers:
Implementations in other projects:
Draft code snippets:

Alternatives

Additional context

💡 Refine mode namings in orchestration

Summary

Define new mode namings for orchestration.
E.g.

grass(w/ infra: azure) -> grass/azure
k8s(w/ infra: azure) -> k8s/aks

Decision Event should be generated in Full_load()🐛

Description

When a vessel arrives at a port, step() generates arrival, full_load, and decision events. However, the decision event / action_scope for ECR should only be generated once the full_load event is completely processed. It is because the number of empties / full containers on the port will be changed assuming that there are full/laden on the port, therefore, pre-generated action_scope for ECR (decision_event) may get outdated.

Secondly, for the scenario when two vessels arrive at the port in the same tick/day (e.g., Port 4 / Transport port in 5-port topology), their action_scopes will face the state-outdated issue too. One patch for it is to modify the _simulate() in core,py at the point when it appends the event_payload by replacing it with the new decision_event_payload() - feed the port and vessel of event_payload along with action_scope, snapshot, and early_discharge.

Environment

MARO version (e.g., v0.1.1a1): Master
MARO scenario (CIM, Citi Bike): CIM
MARO component (Simulation, RL, Distributed Training): Simulation
Orchestration platform (GraSS on Azure, AKS on Azure):
How you installed MARO (pip, source): source
OS (Linux, Windows, macOS): Linux
Python version (3.6, 3.7):
Docker image (e.g., maro2020/maro:latest):
CPU/GPU:
Any other relevant information:

💡 Runtime backend switching

Summary

Enable runtime backend switching

Motivation

Since we will have other backend types, it would be better to support runtime switching, instead of compile time switching, this will make it more flexible for users, users will not have to compile from source to switch backend.

Description

We could add an additional decorator "backend" for Frame definition which take a str parameter for backend name, with this decorator we can specified which backend should be used for this frame. Also we need another function "get_backends" to show available backend implementations.

References

Code sample:

backend_list = get_backends()

@backend("numpy/raw")
class MyFrame(FrameBase):
       mynodes = FrameNode(MyNode, 10)

Alternatives

Additional context

💡 cpp (raw) backend implementation

Summary

A cpp backend implementation.

Motivation

With self write backend implementation, we can add more features we need later, not limited by underlying modules.

Description

Basic:

We can have a variant class to hold all data type that we supported, and consider it as an attribute slot, then a frame that used to hold all these attributes and their information (like slot number, node index it belongs to). And there is not Node physically, just a concept.

Advance:

attributes that not in snapshots

we can just split the memory into 3 parts: frame (attributes will be in snapshot), frame (attributes not in snapshot), snapshots.

NOTE: this feature would be not that easy for numpy backend, as we are using structured-array.

dynamic nodes

we can have 2 implementation about this:

user provides a buffer size, we allocate memory with this buffer size, and dynamic nodes (attributes) cannot more than this chunk size.
reallocate whole memory for each dynamic node request, this is more flexible, but need more memory size to reallocate.

batch operation

we can use openmp to support batch operation, it would be not that hard to implement, as each operation would access different memory address, do not need a lock.

Steps:

This implementation would be separated into following phases.

basic features that same as current numpy backend
complete other basic features base on current backends (numpy and cpp)
extend features based on this implementation, and merge back to numpy backend if possible

References

Alternatives

Additional context

🐛

Description

I have an error running the multi_process_launcher.py file in the maro/examples/cim/ dqn directory

Screenshots

To Reproduce

Steps to reproduce the behavior:

python multi_process_launcher.py test_group 1

Expected Behavior

No errors or warnings

Environment

MARO version (e.g., v0.1.1a1):0.2.0a1
MARO scenario (CIM, Citi Bike):CIM
MARO component (Simulation, RL, Distributed Training):
Orchestration platform (GraSS on Azure, AKS on Azure):
How you installed MARO (pip, source):pip
OS (Linux, Windows, macOS):linux
Python version (3.6, 3.7):3.6
Docker image (e.g., arthursjiang/maro:cpu[5f36ed]):
CPU/GPU:CPU
Any other relevant information:

Regression? examples/rl/workflows/simple_learner.py cannot run anymore🐛

susan@vm-2:~/maro$ python ./examples/rl/workflows/simple_learner.py
Traceback (most recent call last):
File "./examples/rl/workflows/simple_learner.py", line 14, in
from agent_wrapper import get_agent_wrapper
File "/home/susan/maro/examples/rl/workflows/agent_wrapper.py", line 13, in
from general import agent2policy, non_rl_policy_func_index, rl_policy_func_index
File "/home/susan/maro/examples/rl/workflows/general.py", line 21, in
module = importlib.import_module(f"{config['scenario']}")
File "/usr/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/susan/maro/examples/rl/supply_chain/init.py", line 4, in
from .env_wrapper import get_env_wrapper
File "/home/susan/maro/examples/rl/supply_chain/env_wrapper.py", line 1173, in
tmp_env_wrapper = get_env_wrapper(replay_agent_ids=[])
File "/home/susan/maro/examples/rl/supply_chain/env_wrapper.py", line 1170, in get_env_wrapper
return SCEnvWrapper(env=Env(**env_config), replay_agent_ids=replay_agent_ids)
File "/home/susan/maro/examples/rl/supply_chain/env_wrapper.py", line 85, in init
super().init(env, reward_eval_delay, replay_agent_ids=replay_agent_ids)
File "/home/susan/maro/maro/rl/learning/env_wrapper.py", line 69, in init
self._replay_buffer = {agent_id: defaultdict(list) for agent_id in replay_agent_ids}
File "/home/susan/maro/maro/rl/learning/env_wrapper.py", line 69, in
self._replay_buffer = {agent_id: defaultdict(list) for agent_id in replay_agent_ids}
TypeError: unhashable type: 'SkuInfo'

How can i deploy maro on a general k8s cluster?

Description

how can i deploy maro on more general k8s cluster, for example my own local cluster. then i can use the cli to submit job to my local maro?

Link:
Screenshot:

💡 Add Windows support for the Orchestration in CLI

Summary

Add Windows support for the Orchestration in CLI

Motivation

Yeah, MARO is a cross-platform product :)

Description

Base on previous modifications (data push/pull), the Windows support should be ready, but we have to make sure that it is fully tested and bug-free before release.

Loosen dependencies?💡

Summary

We would like to loosen the dependencies in the published package to be able to align with other projects that are on-going.

Motivation

We would like to enable the use of this package in the context of several others with competing dependencies.

Description

References

N/A

Alternatives

N/A

Additional context

🐛 ModuleNotFoundError: No module named 'maro.cli'; 'maro' is not a package

(maro_venv) ali-6c96cfd90593:maro yewang$ python cli/maro.py
Traceback (most recent call last):
File "cli/maro.py", line 10, in
import maro.cli.utils.examples as CliExamples
File "/Users/yewang/Documents/maro/maro/cli/maro.py", line 10, in
import maro.cli.utils.examples as CliExamples
ModuleNotFoundError: No module named 'maro.cli'; 'maro' is not a package

💡 Add anti-collision naming strategy for IDs in Orchestration

Summary

Add anti-collision naming strategy for IDs in Orchestration

Motivation

Currently, the orchestration cannot handle conflicting naming (Low possibility), and it will cause some unexpected results.

💡 RL toolkit refinement

Summary

Refinement of RL toolkit

Motivation

The v0.1 version of RL toolkit has some design flaws that could limit its utility.

Description

Adopt the AbsXXX + SimpleXXX design style for agent manager for more flexibility and better utility.
Refine the Explorer class to make it stateless and decoupled from things like agent ID list.
Add early stopping checker class.

References

Data sources:
Papers:
Implementations in other projects:
Draft code snippets:

Alternatives

Additional context

🐛

Description

when I run the script of cim DQN example, the pycharm show ImportError: cannot import name 'data_version'

Screenshots

To Reproduce

Steps to reproduce the behavior:

Expected Behavior

The example script was successfully run

Environment

MARO version (e.g., v0.01):
PYMARO version (e.g., v0.2.0a1):
MARO scenario (CIM):
MARO component (Simulation, RL):
How you installed MARO (pip):
OS (Windows):
Python version (3.6,):
CPU:

Additional Context

Build predefined images for Orchestration

The front end cannot retrieve data，when geo visualization is deployed on remote virtual machine🐛

Description

When geo visualization is deployed on a remote virtual machine, the front end can't retrieve the data, probably because it can't adjust the IP address to get the data

Screenshots

docker container

geo vis

To Reproduce

Steps to reproduce the behavior:

1.Start the database on remote vm
2.Start the front-end and back-end service on remote vm
3.Access Geo Vis from a remote address in a browser

Expected Behavior

Environment

MARO version (e.g., v0.1.1a1):0.2.2a3
MARO scenario (CIM, Citi Bike):cim
MARO component (Simulation, RL, Distributed Training):
Orchestration platform (GraSS on Azure, AKS on Azure):
How you installed MARO (pip, source):source
OS (Linux, Windows, macOS):linux
Python version (3.6, 3.7):3.6
Docker image (e.g., arthursjiang/maro:cpu[5f36ed]):
CPU/GPU:cpu
Any other relevant information:

Additional Context

how can we generate the topology dataset files based our own Virtual Machine Scheduling Scenario business？ Are there some examples or tools

from the docs, only find some Azure public datasets.

if we want apply maro framework to our own scenario, IMHO, perhaps we should know how to generate our own dataset first？

Move various caches in agent to general global shared cache

💡 Add the field "ActionType" to the Action of CIM scenario

Summary

Add the field ActionType to the Action of CIM scenario

Motivation

An Action consists of an ActionType and a Quantity filed is much clearer.

Description

Currently, the Action of CIM is a number, the detailed ActionType is presented by its sign. While the Action of Citi Bike is an object with both ActionType and Quantity, which is much clearer than the previous one.

We can add a field of ActionType (valid values: Discharge, Load for now) for the CIM scenario to not only unify it with the Citi Bike scenario but also make it clearer.

🐛

Description

misspelling typo spelling grammar microsoft/maro

To Reproduce

Steps to reproduce the behavior:

change with correct word with reference from Merriam webster english refrerence dictionary

Expected Behavior

Environment

MARO version (e.g., v0.1.1a1):
MARO scenario (CIM, Citi Bike):
MARO component (Simulation, RL, Distributed Training):
Orchestration platform (GraSS on Azure, AKS on Azure):
How you installed MARO (pip, source): pip
OS (Linux, Windows, macOS): Linux
Python version (3.6, 3.7): 3.8
Docker image (e.g., maro2020/maro:latest):
CPU/GPU: amd ryzen 7 4800h / gtx 1650 Ti
Any other relevant information:

microsoft / maro Goto Github PK

maro's Introduction

Contents

Install MARO from PyPI

Install MARO from Source

Quick Example

Show Cases

Run Playground

Contributing

Related Papers

Related News

License

maro's People

Contributors

Stargazers

Watchers

Forkers

maro's Issues

Summary

Motivation

Description

References

Alternatives

Additional context

Summary

Motivation

Description

Thank you for this amazing simulator. I'm looking for the source code of the paper mentioned in the subject. Can we please confirm where that exactly been place in this repo given that the paper mentioned a 'code' folder but I can't find it. Thank you!

Description

Screenshots

To Reproduce

Expected Behavior

Environment

Additional Context

Summary

Motivation

Description

Summary

Motivation

Description

Summary

Motivation

Description

References

Alternatives

Additional context

Description

To Reproduce

Expected Behavior

Environment

Additional Context

Description

Screenshots

To Reproduce

Expected Behavior

Environment

Additional Context

Description

Screenshots

To Reproduce

Expected Behavior

Environment

Additional Context

Description

To Reproduce

Expected Behavior

Environment

Summary

Motivation

Description

References

Alternatives

Additional context

Description

To Reproduce

Environment

Description

Summary

Motivation

Description