google-deepmind / dm_alchemy Goto Github PK

DeepMind Alchemy task environment: a meta-reinforcement learning benchmark

License: Apache License 2.0

Python 100.00%

dm_alchemy's Introduction

`dm_alchemy`: DeepMind Alchemy environment

The DeepMind Alchemy environment is a meta-reinforcement learning benchmark that presents tasks sampled from a task distribution with deep underlying structure. It was created to test for the ability of agents to reason and plan via latent state inference, as well as useful exploration and experimentation. It is Unity-based.

Overview

This environment is provided through pre-packaged Docker containers.

This package consists of support code to run these Docker containers. You interact with the task environment via a dm_env Python interface.

Please see the documentation for more detailed information on the available tasks, actions and observations.

Requirements

dm_alchemy requires Docker, Python 3.6.1 or later and a x86-64 CPU with SSE4.2 support. We do not attempt to maintain a working version for Python 2.

Alchemy is intended to be run on Linux and is not officially supported on Mac and Windows. However, it can in principle be run on any platform (though installation may be more of a headache). In particular, on Windows, you will need to install and run Alchemy with WSL.

Note: We recommend using Python virtual environment to mitigate conflicts with your system's Python environment.

Download and install Docker:

For Linux, install Docker-CE. Ensure that you can run Docker as a non-root user.
Install Docker Desktop for OSX or Windows.

Ensure that docker is working correctly by running docker run -d gcr.io/deepmind-environments/alchemy:v1.0.0.

Installation

You can install dm_alchemy by cloning a local copy of our GitHub repository:

$ git clone https://github.com/deepmind/dm_alchemy.git
$ pip install wheel
$ pip install --upgrade setuptools
$ pip install ./dm_alchemy

To also install the dependencies for the examples/, install with:

$ pip install ./dm_alchemy[examples]

Usage

Once dm_alchemy is installed, to instantiate a dm_env instance run the following:

import dm_alchemy

LEVEL_NAME = ('alchemy/perceptual_mapping_'
              'randomized_with_rotation_and_random_bottleneck')
settings = dm_alchemy.EnvironmentSettings(seed=123, level_name=LEVEL_NAME)
env = dm_alchemy.load_from_docker(settings)

For more details see the introductory colab.

Citing Alchemy

If you use Alchemy in your work, please cite the accompanying technical report:

@article{wang2021alchemy,
    title={Alchemy: A structured task distribution for meta-reinforcement learning},
    author={Jane Wang and Michael King and Nicolas Porcel and Zeb Kurth-Nelson
        and Tina Zhu and Charlie Deck and Peter Choy and Mary Cassin and
        Malcolm Reynolds and Francis Song and Gavin Buttimore and David Reichert
        and Neil Rabinowitz and Loic Matthey and Demis Hassabis and Alex Lerchner
        and Matthew Botvinick},
    year={2021},
    journal={arXiv preprint arXiv:2102.02926},
    url={https://arxiv.org/abs/2102.02926},
}

Notice

This is not an officially supported Google product.

dm_alchemy's People

Contributors

Stargazers

Watchers

Forkers

seba-1511 yashvirsinghgrewal-crypto yulkang xrosliang abhishek-2307 sun-void daffan emptyjackson stjordanis classicvalues hannahaih pnickl tkukurin locross93 curtisalleynesr bohdan-nd jl3676 ethanabrooks robbycostales

dm_alchemy's Issues

'numpy' has no attribute 'object'

I am now attempting to initialize a symbolic environment, with the following code:

from dm_alchemy import symbolic_alchemy

level_name = 'alchemy/perceptual_mapping_randomized_with_rotation_and_random_bottleneck'
env = symbolic_alchemy.get_symbolic_alchemy_level(level_name, seed=314)

I receive the error:

anaconda3/envs/lib/python3.11/site-packages/dm_alchemy/ideal_observer/precomputed_maps.py:802: FutureWarning: In the future np.object will be defined as the corresponding NumPy scalar. (This may have returned Python scalars in past versions.
dtype=np.object)
Traceback (most recent call last):
File "trainAlchemy.py", line 9, in
env = symbolic_alchemy.get_symbolic_alchemy_level(level_name, seed=314)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/lib/python3.11/site-packages/dm_alchemy/symbolic_alchemy.py", line 867, in get_symbolic_alchemy_level
_, index_to_perm_index = precomputed_maps.get_perm_index_conversion()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/lib/python3.11/site-packages/dm_alchemy/ideal_observer/precomputed_maps.py", line 802, in get_perm_index_conversion
dtype=np.object)
^^^^^^^^^
File "anaconda3/envs/lib/python3.11/site-packages/numpy/__init__.py", line 284, in __getattr__
raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'object'. Did you mean: 'object_'?

Error importing symbolic_alchemy

After following the installation steps, I run into an error when I merely import symbolic_alchemy:

import dm_alchemy
from dm_alchemy import symbolic_alchemy

I get the following error:

Traceback (most recent call last):
File "trainAlchemy.py", line 2, in
from dm_alchemy import symbolic_alchemy
File "lib/python3.11/site-packages/dm_alchemy/symbolic_alchemy.py", line 23, in
from dm_alchemy import event_tracker
File "lib/python3.11/site-packages/dm_alchemy/event_tracker.py", line 27, in
from dm_alchemy.types import utils
File "lib/python3.11/site-packages/dm_alchemy/types/utils.py", line 126, in
@dataclasses.dataclass
^^^^^^^^^^^^^^^^^^^^^
File "lib/python3.11/dataclasses.py", line 1221, in dataclass
return wrap(cls)
^^^^^^^^^
File "lib/python3.11/dataclasses.py", line 1211, in wrap
return _process_class(cls, init, repr, eq, order, unsafe_hash,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "lib/python3.11/dataclasses.py", line 959, in _process_class
cls_fields.append(_get_field(cls, name, type, kw_only))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "lib/python3.11/dataclasses.py", line 816, in _get_field
raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'dm_alchemy.types.utils.SymbolicBot'> for field ideal_observer is not allowed: use default_factory

Any thoughts on what could cause this?

Running IdealObserverBot

Hi,
I am trying to replicate the evaluation of the Ideal Observer agent on Symbolic Alchemy to get the ~ 284.4 ± 1.6 score reported in the paper. That said, I wrote the following function in an attempt to do so:

Side note: I know that you made available the ideal observer evaluation trajectory in the agent_events folder, but my aim is to then use the IdealObserverBot on a different set of events.

def evaluate():

    level_name = "perceptual_mapping_randomized_with_random_bottleneck"
    chems = chemistries_proto_conversion.load_chemistries_and_items(
        f'chemistries/{level_name}/chemistries')

    precomputed = precomputed_maps.load_from_level_name(level_name)
    reward_weights = RewardWeights(coefficients=[1, 1, 1], offset=0, bonus=12)

    N_EPISODES = 1_000
    rewards = np.zeros(N_EPISODES)

    for ep in tqdm(range(N_EPISODES)):

        chem, items = chems[ep]
        env = symbolic_alchemy.get_symbolic_alchemy_fixed(chemistry=chem, episode_items=items)

        env.add_trackers({symbolic_alchemy_trackers.ScoreTracker.NAME: symbolic_alchemy_trackers.ScoreTracker(env._reward_weights)})

        belief_state_tracker = {symbolic_alchemy_trackers.BeliefStateTracker.NAME: symbolic_alchemy_trackers.BeliefStateTracker(precomputed, env)}
        env.add_trackers(belief_state_tracker)

        bot = IdealObserverBot(reward_weights, precomputed, env, minimise_world_states=False)

        episode_results = bot.run_episode()
    
        rewards[ep] = np.sum(episode_results['score']['per_trial'])

    return rewards.mean(), rewards.std() / np.sqrt(N_EPISODES)

Since each episode takes understandably a long time to run, I just want to confirm the correctness of my code as it's based on a few assumptions I took while reading the codebase.

Thanks!

Access to the source code of the environment?

Hello, I see the environment is loaded as a docker container and I was wondering how one can get access to the source code. Another question I have is whether one can get a vectorised version.

Live animation of the environment

Is there an easy way to have a live-updating render of what's happening in the environment as actions are taken?
I'm hoping for something similar to the GIF you have in the README.

Getting a more friendly graph from input_chem

Hello,
I really like this environment!
One problem i've had though is to do with the ground truth chemistry we can get from timestep.observation['input_chem'].
I understand why it is structured the way it is, as it is more tied to the chemistry formulation in the paper, but it's not very easy to work with for the purposes I have. Especially as the lookup tables for the actual potion colors and stone states are not mentioned in documentation.
Is there an easy built-in way to convert this chemistry to an adjacency matrix or some other representation that is more friendly? With the actual potion colors and stone attributes maintained if possible.
Thanks!

raise grpc.FutureTimeoutError() grpc.FutureTimeoutError

followed instruction from readme below

$ git clone https://github.com/deepmind/dm_alchemy.git
$ pip install wheel
$ pip install --upgrade setuptools
$ pip install ./dm_alchemy
$ pip install pygame

and then try to run human_agent.py using below command getting error

python human_agent.py --seed 123 --level_name 'alchemy/perceptual_mapping_randomized_with_rotation_and_random_bottleneck'

pygame 2.0.1 (SDL 2.0.14, Python 3.7.4)
Hello from the pygame community. https://www.pygame.org/contribute.html
I0212 16:02:42.783728 140172242384704 _load_environment.py:377] Downloading docker image "gcr.io/deepmind-environments/alchemy:v1.0.0"...
I0212 16:03:59.853875 140172242384704 _load_environment.py:379] Download finished.
Traceback (most recent call last):
  File "human_agent.py", line 186, in <module>
    app.run(main)
  File "/home/venv/lib/python3.7/site-packages/absl/app.py", line 303, in run
    _run_main(main, args)
  File "/home/venv/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "human_agent.py", line 117, in main
    environment_variables=environment_variables) as env:
  File "/home/venv/lib/python3.7/site-packages/dm_alchemy/_load_environment.py", line 389, in load_from_docker
    connection_details=_connect_to_environment(port, settings),
  File "/home/venv/lib/python3.7/site-packages/dm_alchemy/_load_environment.py", line 246, in _connect_to_environment
    channel, connection = _create_channel_and_connection(port)
  File "/home/venv/lib/python3.7/site-packages/dm_alchemy/_load_environment.py", line 204, in _create_channel_and_connection
    _check_grpc_channel_ready(channel)
  File "/home/venv/lib/python3.7/site-packages/dm_alchemy/_load_environment.py", line 183, in _check_grpc_channel_ready
    return grpc.channel_ready_future(channel).result(timeout=1)
  File "/home/venv/lib/python3.7/site-packages/grpc/_utilities.py", line 140, in result
    self._block(timeout)
  File "/home/venv/lib/python3.7/site-packages/grpc/_utilities.py", line 86, in _block
    raise grpc.FutureTimeoutError()
grpc.FutureTimeoutError

grpc.FutureTimeoutError

unfortunately on windows, I run into the same error trace as #1 trying to run human_agent.py:

python human_agent.py --seed 123 --level_name alchemy/perceptual_mapping_randomized_with_rotation_and_random_bottleneck'
gives the error:

pygame 2.0.1 (SDL 2.0.14, Python 3.7.9)
Hello from the pygame community. https://www.pygame.org/contribute.html
Traceback (most recent call last):
  File "human_agent.py", line 186, in <module>
    app.run(main)
  File "C:\Users\jonat\AppData\Local\Programs\Python\Python37\lib\site-packages\absl\app.py", line 303, in run
    _run_main(main, args)
  File "C:\Users\jonat\AppData\Local\Programs\Python\Python37\lib\site-packages\absl\app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "human_agent.py", line 117, in main
    environment_variables=environment_variables) as env:
  File "C:\Users\jonat\AppData\Local\Programs\Python\Python37\lib\site-packages\dm_alchemy\_load_environment.py", line 389, in load_from_docker
    connection_details=_connect_to_environment(port, settings),
  File "C:\Users\jonat\AppData\Local\Programs\Python\Python37\lib\site-packages\dm_alchemy\_load_environment.py", line 246, in _connect_to_environment
    channel, connection = _create_channel_and_connection(port)
  File "C:\Users\jonat\AppData\Local\Programs\Python\Python37\lib\site-packages\dm_alchemy\_load_environment.py", line 204, in _create_channel_and_connection
    _check_grpc_channel_ready(channel)
  File "C:\Users\jonat\AppData\Local\Programs\Python\Python37\lib\site-packages\dm_alchemy\_load_environment.py", line 183, in _check_grpc_channel_ready
    return grpc.channel_ready_future(channel).result(timeout=1)
  File "C:\Users\jonat\AppData\Local\Programs\Python\Python37\lib\site-packages\grpc\_utilities.py", line 140, in result
    self._block(timeout)
  File "C:\Users\jonat\AppData\Local\Programs\Python\Python37\lib\site-packages\grpc\_utilities.py", line 86, in _block
    raise grpc.FutureTimeoutError()
grpc.FutureTimeoutError

the command that is mentioned in #1 was already updated in the installation instructions, but does not seem to help. Running

level_name = 'alchemy/perceptual_mapping_randomized_with_rotation_and_random_bottleneck'
seed = 1023
settings = dm_alchemy.EnvironmentSettings(
    seed=seed, level_name=level_name, width=width, height=height)
env = dm_alchemy.load_from_docker(settings)

in the AlchemyGettingStarted.ipynb seems to give the same error.
The setup is as written in the guide and uses Docker Desktop (except that I don't use a Python virtual environment).
docker run -d gcr.io/deepmind-environments/alchemy:v1.0.0 gives no error.

While you don't officially support windows maybe you can still help?