Giter Club home page Giter Club logo

enn_acme's Introduction

ENN Agent in Acme

enn_acme defines an interface for designing RL agents that combines:

  1. enn: https://github.com/deepmind/enn
  2. acme: https://github.com/deepmind/acme

Before diving into enn_acme, you should first read the tutorials for both of these underlying libraries. enn_acme is really a thin convenience layer designed to expose certain "key concepts" in agent design.

Key concepts

We outline the key high-level interfaces for our code in base.py:

  • enn: epistemic neural network = knowledge representation.
  • EnnPlanner: selects actions given (params, observation) for ENN.
  • LossFn: tells the agent how to evaluate loss of a given batch of data.

Getting started

The best place to get started is in our colab tutorial.

Here, you can find a quick rundown of our main types in the code, and find an example of running a version of bootstrapped DQN on bsuite environments.

Once you have had a look at this colab, you might want to have a look at our example experiments/.

Installation

We have tested enn_acme on Python 3.7. To install the dependencies:

  1. Optional: We recommend using a Python virtual environment to manage your dependencies, so as not to clobber your system installation:

    python3 -m venv enn_acme
    source enn_acme/bin/activate
    pip install --upgrade pip setuptools
  2. Install enn_acme directly from github:

    pip install git+https://github.com/deepmind/enn_acme

More examples can be found in the colab tutorial.

  1. Optional: run the tests by executing ./test.sh from enn_acme root directory.

Citing

If you use enn_acme in your work, please cite the enn paper and acme report:

@article{osband2022epistemic,
  title={Epistemic neural networks},
  author={Osband, Ian and Wen, Zheng and Asghari, Seyed Mohammad and Dwaracherla, Vikranth and Ibrahimi, Morteza and Lu, Xiuyuan and Van Roy, Benjamin},
  journal={arXiv preprint arXiv:2107.08924},
  year={2022}
}
@article{hoffman2020acme,
    title={Acme: A Research Framework for Distributed Reinforcement Learning},
    author={Matt Hoffman and Bobak Shahriari and John Aslanides and Gabriel
        Barth-Maron and Feryal Behbahani and Tamara Norman and Abbas Abdolmaleki
        and Albin Cassirer and Fan Yang and Kate Baumli and Sarah Henderson and
        Alex Novikov and Sergio Gómez Colmenarejo and Serkan Cabi and Caglar
        Gulcehre and Tom Le Paine and Andrew Cowie and Ziyu Wang and Bilal Piot
        and Nando de Freitas},
    year={2020},
    journal={arXiv preprint arXiv:2006.00979},
    url={https://arxiv.org/abs/2006.00979},
}

enn_acme's People

Contributors

mohammadasghari avatar xlu0 avatar iosband avatar

Stargazers

Soan Kim avatar Mohamad Rostami avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar Ziyú Ye avatar Mihai Todor avatar Xingdong Zuo avatar Kashif Rasul avatar Seong Joon Oh avatar Josh Mize avatar  avatar LostThinker avatar  avatar Vindhya Singh avatar 爱可可-爱生活 avatar Suraj Narayanan Sasikumar avatar  avatar Dmitry Kulikov avatar Mumtozbek Akhmadjanov avatar Nada Amin avatar Daniel Prisco avatar Jialong Wu avatar  avatar

Watchers

Nada Amin avatar Jongwook Choi avatar Saran Tunyasuvunakool avatar  avatar  avatar Arun Sathiya avatar  avatar Ziyú Ye avatar

Forkers

ziyu-deep

enn_acme's Issues

Not able to install enn_acme on colab

While running the command !pip install git+https://github.com/deepmind/enn_acme" , got this result - "ERROR: ResolutionImpossible: "

Collecting git+https://github.com/deepmind/enn_acme
Cloning https://github.com/deepmind/enn_acme to /tmp/pip-req-build-d6uvgkht
Running command git clone --filter=blob:none --quiet https://github.com/deepmind/enn_acme /tmp/pip-req-build-d6uvgkht
Resolved https://github.com/deepmind/enn_acme to commit 4680812
Preparing metadata (setup.py) ... done
Collecting enn@ git+https://[email protected]/deepmind/enn (from enn-acme==0.1.0)
Cloning https://@github.com/deepmind/enn to /tmp/pip-install-7dcie8wj/enn_c4e1d7694c7b404780ce978809853195
Running command git clone --filter=blob:none --quiet 'https://
@github.com/deepmind/enn' /tmp/pip-install-7dcie8wj/enn_c4e1d7694c7b404780ce978809853195
Resolved https://@github.com/deepmind/enn to commit 1eae1eb71dda402c15cc82771d15511571876a33
Preparing metadata (setup.py) ... done
Collecting neural_testbed@ git+https://[email protected]/deepmind/neural_testbed (from enn-acme==0.1.0)
Cloning https://
@github.com/deepmind/neural_testbed to /tmp/pip-install-7dcie8wj/neural-testbed_161b9f177b98485481b22c2b18353c4f
Running command git clone --filter=blob:none --quiet 'https://@github.com/deepmind/neural_testbed' /tmp/pip-install-7dcie8wj/neural-testbed_161b9f177b98485481b22c2b18353c4f
Resolved https://
@github.com/deepmind/neural_testbed to commit 02aef13a1b2dde58e140c2727d85c50b292c283a
Preparing metadata (setup.py) ... done
Requirement already satisfied: absl-py in /usr/local/lib/python3.10/dist-packages (from enn-acme==0.1.0) (1.4.0)
Collecting bsuite (from enn-acme==0.1.0)
Using cached bsuite-0.3.5.tar.gz (88 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: chex in /usr/local/lib/python3.10/dist-packages (from enn-acme==0.1.0) (0.1.7)
Collecting dm-acme==0.4.0 (from enn-acme==0.1.0)
Using cached dm-acme-0.4.0.tar.gz (314 kB)
Preparing metadata (setup.py) ... done
Collecting dm-env (from enn-acme==0.1.0)
Using cached dm_env-1.6-py3-none-any.whl (26 kB)
Collecting dm-haiku (from enn-acme==0.1.0)
Obtaining dependency information for dm-haiku from https://files.pythonhosted.org/packages/df/ff/235c5bdf5d83f9013771a37dd926080400ecc1f0586f18583600dcf1540d/dm_haiku-0.0.10-py3-none-any.whl.metadata
Using cached dm_haiku-0.0.10-py3-none-any.whl.metadata (18 kB)
Collecting dm-launchpad==0.5.0 (from enn-acme==0.1.0)
Using cached dm_launchpad-0.5.0-cp310-cp310-manylinux2010_x86_64.whl (3.8 MB)
Collecting dm-reverb==0.7.0 (from enn-acme==0.1.0)
Using cached dm_reverb-0.7.0-cp310-cp310-manylinux2010_x86_64.whl (6.4 MB)
Collecting dm-sonnet (from enn-acme==0.1.0)
Using cached dm_sonnet-2.0.1-py3-none-any.whl (268 kB)
Collecting jax==0.3.15 (from enn-acme==0.1.0)
Using cached jax-0.3.15.tar.gz (1.0 MB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: jaxlib in /usr/local/lib/python3.10/dist-packages (from enn-acme==0.1.0) (0.4.14+cuda11.cudnn86)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from enn-acme==0.1.0) (1.23.5)
Requirement already satisfied: optax in /usr/local/lib/python3.10/dist-packages (from enn-acme==0.1.0) (0.1.7)
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from enn-acme==0.1.0) (1.5.3)
Requirement already satisfied: pyarrow in /usr/local/lib/python3.10/dist-packages (from enn-acme==0.1.0) (9.0.0)
Collecting rlax (from enn-acme==0.1.0)
Obtaining dependency information for rlax from https://files.pythonhosted.org/packages/9a/f1/9ed176a3eae715bd362b62ac13c99164d12f2da41dfa23962c444d814607/rlax-0.1.6-py3-none-any.whl.metadata
Using cached rlax-0.1.6-py3-none-any.whl.metadata (7.2 kB)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from enn-acme==0.1.0) (2.31.0)
Collecting tensorflow==2.8.0 (from enn-acme==0.1.0)
Using cached tensorflow-2.8.0-cp310-cp310-manylinux2010_x86_64.whl (497.6 MB)
Requirement already satisfied: tensorflow-datasets==4.4.0 in /usr/local/lib/python3.10/dist-packages (from enn-acme==0.1.0) (4.4.0)
Collecting tensorflow_probability==0.15.0 (from enn-acme==0.1.0)
Using cached tensorflow_probability-0.15.0-py2.py3-none-any.whl (5.7 MB)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from enn-acme==0.1.0) (4.5.0)
Requirement already satisfied: dm-tree in /usr/local/lib/python3.10/dist-packages (from dm-acme==0.4.0->enn-acme==0.1.0) (0.1.8)
Requirement already satisfied: pillow in /usr/local/lib/python3.10/dist-packages (from dm-acme==0.4.0->enn-acme==0.1.0) (9.4.0)
Requirement already satisfied: cloudpickle in /usr/local/lib/python3.10/dist-packages (from dm-launchpad==0.5.0->enn-acme==0.1.0) (2.2.1)
Requirement already satisfied: grpcio in /usr/local/lib/python3.10/dist-packages (from dm-launchpad==0.5.0->enn-acme==0.1.0) (1.58.0)
Collecting mock (from dm-launchpad==0.5.0->enn-acme==0.1.0)
Obtaining dependency information for mock from https://files.pythonhosted.org/packages/6b/20/471f41173930550f279ccb65596a5ac19b9ac974a8d93679bcd3e0c31498/mock-5.1.0-py3-none-any.whl.metadata
Using cached mock-5.1.0-py3-none-any.whl.metadata (3.0 kB)
Requirement already satisfied: portpicker in /usr/local/lib/python3.10/dist-packages (from dm-launchpad==0.5.0->enn-acme==0.1.0) (1.5.2)
Requirement already satisfied: protobuf in /usr/local/lib/python3.10/dist-packages (from dm-launchpad==0.5.0->enn-acme==0.1.0) (3.20.3)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from dm-launchpad==0.5.0->enn-acme==0.1.0) (5.9.5)
Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from dm-launchpad==0.5.0->enn-acme==0.1.0) (2.3.0)
Requirement already satisfied: opt_einsum in /usr/local/lib/python3.10/dist-packages (from jax==0.3.15->enn-acme==0.1.0) (3.3.0)
Requirement already satisfied: scipy>=1.5 in /usr/local/lib/python3.10/dist-packages (from jax==0.3.15->enn-acme==0.1.0) (1.11.3)
Requirement already satisfied: etils[epath] in /usr/local/lib/python3.10/dist-packages (from jax==0.3.15->enn-acme==0.1.0) (1.5.0)
Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.8.0->enn-acme==0.1.0) (1.6.3)
Requirement already satisfied: flatbuffers>=1.12 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.8.0->enn-acme==0.1.0) (23.5.26)
Requirement already satisfied: gast>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.8.0->enn-acme==0.1.0) (0.4.0)
Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.8.0->enn-acme==0.1.0) (0.2.0)
Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.8.0->enn-acme==0.1.0) (3.9.0)
Collecting keras-preprocessing>=1.1.1 (from tensorflow==2.8.0->enn-acme==0.1.0)
Using cached Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
Requirement already satisfied: libclang>=9.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.8.0->enn-acme==0.1.0) (16.0.6)
Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.8.0->enn-acme==0.1.0) (68.2.2)
Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.8.0->enn-acme==0.1.0) (1.16.0)
Requirement already satisfied: wrapt>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.8.0->enn-acme==0.1.0) (1.15.0)
Collecting tensorboard<2.9,>=2.8 (from tensorflow==2.8.0->enn-acme==0.1.0)
Using cached tensorboard-2.8.0-py3-none-any.whl (5.8 MB)
Collecting tf-estimator-nightly==2.8.0.dev2021122109 (from tensorflow==2.8.0->enn-acme==0.1.0)
Using cached tf_estimator_nightly-2.8.0.dev2021122109-py2.py3-none-any.whl (462 kB)
Collecting keras<2.9,>=2.8.0rc0 (from tensorflow==2.8.0->enn-acme==0.1.0)
Using cached keras-2.8.0-py2.py3-none-any.whl (1.4 MB)
Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.8.0->enn-acme==0.1.0) (0.34.0)
Requirement already satisfied: attrs>=18.1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow-datasets==4.4.0->enn-acme==0.1.0) (23.1.0)
Requirement already satisfied: dill in /usr/local/lib/python3.10/dist-packages (from tensorflow-datasets==4.4.0->enn-acme==0.1.0) (0.3.7)
Requirement already satisfied: future in /usr/local/lib/python3.10/dist-packages (from tensorflow-datasets==4.4.0->enn-acme==0.1.0) (0.18.3)
Requirement already satisfied: promise in /usr/local/lib/python3.10/dist-packages (from tensorflow-datasets==4.4.0->enn-acme==0.1.0) (2.3)
Requirement already satisfied: tensorflow-metadata in /usr/local/lib/python3.10/dist-packages (from tensorflow-datasets==4.4.0->enn-acme==0.1.0) (1.14.0)
Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from tensorflow-datasets==4.4.0->enn-acme==0.1.0) (4.66.1)
Requirement already satisfied: decorator in /usr/local/lib/python3.10/dist-packages (from tensorflow_probability==0.15.0->enn-acme==0.1.0) (4.4.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->enn-acme==0.1.0) (3.2.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->enn-acme==0.1.0) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->enn-acme==0.1.0) (2.0.5)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->enn-acme==0.1.0) (2023.7.22)
Requirement already satisfied: frozendict in /usr/local/lib/python3.10/dist-packages (from bsuite->enn-acme==0.1.0) (2.3.8)
Requirement already satisfied: gym in /usr/local/lib/python3.10/dist-packages (from bsuite->enn-acme==0.1.0) (0.25.2)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from bsuite->enn-acme==0.1.0) (3.7.1)
Requirement already satisfied: plotnine in /usr/local/lib/python3.10/dist-packages (from bsuite->enn-acme==0.1.0) (0.12.3)
Requirement already satisfied: scikit-image in /usr/local/lib/python3.10/dist-packages (from bsuite->enn-acme==0.1.0) (0.19.3)
INFO: pip is looking at multiple versions of chex to determine which version is compatible with other requirements. This could take a while.
Collecting chex (from enn-acme==0.1.0)
Obtaining dependency information for chex from https://files.pythonhosted.org/packages/8b/c5/ab99c61d1384f89fe0d89b4b105c1ad22dab98cfe8c78136fb8c3f75f75b/chex-0.1.83-py3-none-any.whl.metadata
Using cached chex-0.1.83-py3-none-any.whl.metadata (17 kB)
Obtaining dependency information for chex from https://files.pythonhosted.org/packages/84/c9/a2182cbf8bc066de9433930c41e76b7f4e904f155c5881235dbb54f8148b/chex-0.1.82-py3-none-any.whl.metadata
Using cached chex-0.1.82-py3-none-any.whl.metadata (17 kB)
Obtaining dependency information for chex from https://files.pythonhosted.org/packages/f4/c2/e66bf06bff9d2fad600b0daebf54ecb5d7e2c40eae25968773e3ce292814/chex-0.1.81-py3-none-any.whl.metadata
Using cached chex-0.1.81-py3-none-any.whl.metadata (17 kB)
Using cached chex-0.1.6-py3-none-any.whl (87 kB)
Requirement already satisfied: toolz>=0.9.0 in /usr/local/lib/python3.10/dist-packages (from chex->enn-acme==0.1.0) (0.12.0)
Requirement already satisfied: ml-dtypes>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from jaxlib->enn-acme==0.1.0) (0.3.1)
Collecting jmp>=0.0.2 (from dm-haiku->enn-acme==0.1.0)
Using cached jmp-0.0.4-py3-none-any.whl (18 kB)
Requirement already satisfied: tabulate>=0.8.9 in /usr/local/lib/python3.10/dist-packages (from dm-haiku->enn-acme==0.1.0) (0.9.0)
Collecting jaxline (from enn@ git+https://[email protected]/deepmind/enn->enn-acme==0.1.0)
Using cached jaxline-0.0.7.tar.gz (34 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from enn@ git+https://[email protected]/deepmind/enn->enn-acme==0.1.0) (1.2.2)
Collecting ml_collections (from neural_testbed@ git+https://[email protected]/deepmind/neural_testbed->enn-acme==0.1.0)
Using cached ml_collections-0.1.1.tar.gz (77 kB)
Preparing metadata (setup.py) ... done
Collecting neural-tangents (from neural_testbed@ git+https://[email protected]/deepmind/neural_testbed->enn-acme==0.1.0)
Obtaining dependency information for neural-tangents from https://files.pythonhosted.org/packages/8b/cb/2749e144117fc2c388cfa5b3684be8af67cd5b402a661d4ec89099a5c5f7/neural_tangents-0.6.4-py2.py3-none-any.whl.metadata
Using cached neural_tangents-0.6.4-py2.py3-none-any.whl.metadata (26 kB)
INFO: pip is looking at multiple versions of neural-testbed to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install enn-acme and enn-acme==0.1.0 because these package versions have conflicting dependencies.

The conflict is caused by:
enn-acme 0.1.0 depends on tensorflow-datasets==4.4.0
enn 0.1.0 depends on tensorflow-datasets
neural-testbed 0.1.0 depends on tensorflow-datasets==4.6.0

To fix this you could try to:

  1. loosen the range of package versions you've specified
  2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.