• Add DDQN
• Add prioritized sampling
• Improve explore strategy
• Fix minor issues

• Fix DiscreteRLPolicy set_state bug (https://github.com/microsoft/maro/commit/a9a638134bef84176a0623fdadc5b789c2d7c3cf, https://github.com/microsoft/maro/commit/b3c6a589ad9036b03221e776a6929b2bc1eb4680)

• Refine RL workflow
  • Add **kwargs to support more problem setting (e.g., Graph based ones) (#589)
    • add **kwargs to RL models' forward funcs and _shape_check()
    • add **kwargs to RL policies' get_action related funcs and _post_check()
    • add **kwargs to choose_actions of AbsEnvSampler; remain it None in current sample() and eval()
  • Add detached loss to the return value of update_critic() and update_actor() of current TrainOps; add default False early_stop to update_actor() of current TrainOps (#589)
  • Refine random seed setting logic in RL workflow (#584)
  • Refine rollout workflow (#577) to support:
    • Run a specific number of steps in rollout
    • Run a specific number of episodes during evaluation with num_eval_episodes
    • Flexible metrics management during rollout with AbsEnvSampler.metrics
  • Add AbsEnvSampler.metrics to support flexible metrics management during roullout (#577)
  • Add Callback as a general interface to support customized operations in each phase of the workflow.
    • Two instances Checkpoint and MetricsRecorder are added. (#577)
    • Add customized_callbacks to RLComponentBundle. (#589)
  • Re-organize RL job's output paths. (#577)
  • Fix several RL algorithm bugs. (#577, #589)
• Replace the numpy data type with python common data type in whole project (#571)
• Add RL benchmark on Mujoco as a module to tests/, compared with spinning up benchmark, performance results can be found in tests/rl/performance.md (#575, #577, #583, #584)
• Other minor code refinements

Update build workflow to fix pymaro package failed in manylinux issue

• Refine decision event logic (#559)
  • Add DecisionEventPayload and ActionPayload as the base class for payloads of decision events & action events.
  • Add related type check during running.
  • Rename related variables under simulator/.
  • Refine the action handling logic. Modify Env and related BEs.
• Refine RL component bundle logic (#549)
  • Refine rl_component_bundle. Use a more straightforward & easier way to organize. In this new version, we do not need easyrl anymore.
  • Update related examples.
  • Fix bugs in distributed training. Add config YAML file for distributed training in CIM scenario.
  • Add rl_formulation.ipynb as an example.
• Fix CITI BIKE hello world dashboard (#555)
• Update data model doc (#554)
• Update requirements (#552, #553)

• RL Toolkit: ✨ A whole new design of RL Toolkit (#539)
• CLI: refactorization (#539)
• Backend: data frame/snapshot precision issue fix (#544)
• MISC: Whole repository code formating (#538, #547)
• Citi bike visualization tool hot fix (#543)

• Add higher Python version 3.8, 3.9 (#398)
• Refinement for some core modules, including more type hints, better coding style, small interface adjustment, potential bug fix
  • Core & Abstract Business Engine refinement (#392)
  • Event Buffer refinement (#389)
  • Interface updates for SimRandom to better support scenario buildings (#401, #400)
• CIM scenario refinement (#400)
  • Disable auto action type detection, i.e., requires for explicit ActionType (#399)
  • Data processing updates, including data generation part and real data loader part (#395)
  • Some testing program updates, and enlarge the test coverage (#395)
• Examples update
  • OnlineLP for Citi Bike: change the default solver from GLPK to CBC (#391)

Scenario/CIM:

• Fix the potential bug of vessel plan for some specific topologies after calling Env.reset() (Related issue: #385, Related PR: #387, #388)

Simulator/CIM Scenario:

• Fix the environment reset issue #385, and update the simulator random seed interfaces (#387)
• Fix the "Wrong future stop tick predictions" issue #384 (#386)

RL Toolkit:

• Add the arguments used by Actor to the Trajectory.on_env_feedback() interface (#373, #374)

Visualization Tool/CIM:

• Fix Geo vis IP address & SQL logic bugs (#352, #383)

Others:

• update dataclasses requirement to setup
• fix some typo in codes comments and online document

• CIM data container/data loader interface for real data mode updated (#372)
• README for pymaro description updated

CIM Scenario:

• Add the data container and data loader for real data schema (#361)
• Fix the vessel plan, vessel past/future stops info issues (#363)
• Add attributes is_parking and loc_port_idx for vessels (#366)

Simulator:

• Add support to dump finished events (#365)
• Add property business_engine to AbsEnv (#368)

RL Toolkit:

• Fixed scheduler init bug (#362)

Tutorials docker images:

• Fix bugs in examples and notebooks caused by interface changes. (#367)
• Update simple strategies used in examples (E.g. random instead of dumpy in CIM hello series). (#367)
• Update docker images and related documents. (#367)

Package dependency:

• Update package dependency (#345 #346)

Simulator:

• Fixed the vessel plan issue for CIM scenario (#348)

Examples:

• Update hello world examples for CIM scenario (#338)

Package dependency:

• Loosen package dependency (#328) to be more flexible: #331

CLI related:

• Fixed issues (#317, #314) for geographic visualization tool: #325
• Fixed Redis database data not cleaned issue (#327) by adding a close function for Proxy: #335

Examples:

• Add offline ILP example for VM scheduling scenario: #275

Backend related:

• bug fix: dynamic backend slot number not correct after reset (#321)
• bug fix: value is interrupt if one node contains multiple list attribute (#321)
• changes: snapshot query result default value set to 0 instead nan, make it align with static backend (#321)
• bug fix: on windows compiling fail due to cannot find max function. (#321)

RL Toolkit related:

• Fix bug of dqn.choose_action and ddpg.choose_action (#307)
• Double DQN: loss function bug fixed (#315)
• Fix bug caused by invalid snapshot index (#318, #319)

CLI related:

• Add MARO admin tool (#288)
• Add master/node vm join deployment file for grass/on-premises. (#316)
• Visualization tool: Add support of dump data path creation, default sampling ratio to 1 (#323)
• Visualization tool: Bug occurs in Linux fixed (#312, #313)
• Update online doc Installation part (#316); Visualization part (#301)

• Backend related:
  • Vector env support (#266)
  • Add slot filter functions for node attribute (#273)
  • Refine joint decision sequential action mode (#219)
  • Add dynamic node support (#172)
• VM scheduling scenario related:
  • Add oversubscription feature (#246), (#256)
  • Add hierarchy vm region architecture support (#258)
  • Fix bug of auto-downloading data (#257)
  • Pricing model and energy model update (#286)
  • Rule-based algorithm examples (#255), (#282)
• RL toolkit related:
  • Add DDPG algorithm (#252)
  • Merge algorithm with agent (#259)
  • Distributed framework update (#206)
• CLI related:
  • CLI refactoring (#227)
  • Feature: Add a cli command to support create new project (#279)
  • Add Env-Geographic visualization tool, CIM hello as example (#291), (#294), (#295), (#298)
  • CLI visualization support and maro grass local mode (#277)

Description

• Simulation toolkit
  • VM allocation environment
  • Dashboard for CIM and CitiBike
  • EventBuffer refine
  • Dump for snapshots, events and business engine special data.
• RL toolkit
  • New RL abstraction
  • More RL algorithms
• Distributed toolkit
  • maro cli refactoring
  • maro cli support windows
  • maro processes mode
  • maro grass on-premises mode
• Solution example
  • CIM: dqn & gnn example updated
  • Citi Bike: online LP example added
  • VM Scheduling: random & best fit example added

Linked issue(s)/Pull request(s)

• MARO processes mode doc
• MARO on-premised mode
• Learning model refactoring
• MARO vis
• MARO vis dump feature
• RL toolkit doc update
• VM scheduling doc
• VM scheduling environment
• VM data pipeline
• EventBuffer refine

Description

This PR contains several PRs including some bug fix, new example addition, doc addition and refinement, new features, new tests.

Linked issue(s)/Pull request(s)

• 138:
  • refine data push/pull in cli
• 112:
  • add fall back function in weather download
• 136:
  • add docs for example
• 144:
  • switching the key and values of the handler dict which is used in the dist decorator
  • add unit tests for the dist decorator
  • fixed the multithreading deadlock in the maro unit test suite
• 147:
  • remove useless reward() from Env, BE, yaml config
  • refine the format & style for core & BE
• 157:
  • add docs for AzCopy
  • add dqn tests in grass/k8s mode
• 160, 164:
  • fix the data generation description for citi bike in docs
• 60:
  • add example/cim/gnn
• 166:
  • fix the uppicklable bug in RL toolkit store
• 167:
  • rename the confusing components in RL toolkit and example/cim/dqn

• Fix dist decorator
• Add example docs for CIM and Citi Bike scenario
• Refine annotation

PR 153

• Add fallback logic for weather source unavailable case.
• Refine CLI data pull/push logic.
• Add integration test cases for orchestration.

Refine docstring.