Need to change this line of code:

The rllib_agent test currently fails on my 2018 MacBook Pro (Intel) and my 2020 MacBook Pro with Apple Silicon, saying it can't find Ray when it tries to import it. I just commented it out for now in dd86606.

Currently, most of our example agents define behavior that runs the agent when the file that contains the agent's code is run using python -m agent_main.py. For example, see:

And also:

Notice that they have similar structure and contain a decent amount of boilerplate code.

Also notice that they use the standard python convention:

if __name__ == "__main__":
    # parse command line args that define agent run behavior
    # use agentos.run() to actually run the agent.

Our test cases execute these files via the pytest virtual-env fixture's virtualenv.run() which in turn calls subprocess.Popen.

Since many agents will likely have similar command line args, it would be useful if we could make a utility function that handles defining command-line arguments, parses those same arguments, and runs the agent.

This would then provide a 3rd way to run an agent. That is, in addition to the options of (1) using an Agent Directory (which itself is an MLflow project), and (2) using the agentos CLI command agentos run, one could now just directly run the agent python file (e.g., python -m /my/agent/main.py).

The tricky part is keep the utility function super intuitive and easy to use.

One idea would be to define a function in agentos/core.py:

def setup_agent_main(agent_class, env_classes=[CartPoleEnv]):
    ```
    Set up a main function for an agent file, including accepting common
    command line arguments. To use this in an agent file to make the agent
    easy to execute directly as a python script. 
# code to parse args
# code to run agent via agentos.run_agent()

This would be used from inside your agent_main.py file as follows:

if __name__ == "__main__":
    from agentos.core import setup_agent_main
    setup_agent_main(YourAgentClass)

This would make example_agents easier to read and test, and would hopefully make agent development more approachable.

Agent Directories already provides a well defined way to specify agent params (in the MLProject file), so perhaps this could be written to play nicely with those conventions.

Note that this whole idea might not be necessary since the agent dev could just use the CLI or an Agent Directory to run/test their agent. **And perhaps we should instead be discouraging agent developers from making their agent files directly runnable as python scripts (i.e., via python -m).

V0 target: the list will be a yaml file stored in the AgentOS repository

Each registry entry will be structured as follows:

component_name:
  type: [policy | environment | algorithm]
  description: [component description]
  releases:
    - name: [version_1_name]
      hash: [version_1_hash]
      github_url: [url of version 1 repo]
      class_name: [fully qualified class name of version 1]
      requirements_path: [path to version 1 requirements file]

    - name: [version_2_name]
      hash: [version_2_hash]
      github_url: [url of version 2 repo]
      class_name: [fully qualified class name of version 2]
      requirements_path: [path to version 2 requirements file]

for example:

2048:
  type: environment
  description: "An environment that simulates the 2048 game"
  releases:
    - name: 1.0.0
      hash: aeb938f
      github_url: https://github.com/example-proj/example-repo
      class_name: main.2048
      requirements_path: requirements.txt

    - name: 1.1.0
      hash: 3939aa1
      github_url: https://github.com/example-proj/example-repo
      class_name: main.2048
      requirements_path: requirements.txt

See the design doc for more details.

Add menu bar to website with links to "install", "code", etc.

we might want to have each example self-document and then aggregate that documentation programmatically.

There is currently a broken link in the install and explore section of the readme.

This is happening because scripts/build_docs.py doesn't parse Sphinx-specific ReStructuredText directives such as :doc:`doc_name`.

One possible fix is to see if sphinx provides an API with necessary tools to parse the file correctly (but it would have to generate a full URL including the agentos.org part)

Another is to write some parsing logic that looks for an replaces these manually.

Port the 2048 environment from this repo into its own ACS compatible repo.

See the design doc for more details.

The minimal agent (agentos init) will be ACS aware and incorporate basic components with minimal required edits.

See the design doc for more details.

Add predictive coding example agent (based on Rao "Predictive Coding in the Visual Cortex")

Developers should have an easy way to register their local custom components with acs so it can be accessed via the acs module in other parts of their agent.

See the design doc for more details.

Like pip but for Envs, Policies, and Agents.

It should:

• reuse the existing OpenAI env registery
• make it easy to see which envs, agents, policies are compatible with each other (per action and obs spaces)
• provide free hosting,
• make it easy to search, install, upgrade
• Integrate with Agent Directories

I tried to run an agent as described in the Install and Explore subsection on the main docs website, but encountered an error with the command agentos run. The error comes from the MLflow library.

Error shown:
FileNotFoundError: [Errno 2] No such file or directory: 'conda'

I do not have conda installed. Note that the MLflow readme states that conda is required for the MLflow projects feature.

Fix the pytests that break because of TensorFlow not being available in PyPI for windows.

> pip install -r .\dev-requirements.txt
...
ERROR: Could not find a version that satisfies the requirement tensorflow==2.2.0
ERROR: No matching distribution found for tensorflow==2.2.0

Reported by Nick in #20:

When I merged in upstream, I realized I could no longer build the docs. Turns out I needed to:

• Add the agentos module to my PYTHONPATH, and
• Install gym and mlflow

Presumably sphinx is trying to do something like import these modules (and their dependencies) to get at their docstrings.

From the design doc, this encompasses the implementation of acs below:

Let's dig into our minimal agent to see how we access our components programmatically:

from agentos import Agent
from agentos import acs

class SimpleAgent(Agent):
    def advance(self):
        acs.policy.train(acs.env)
        done = False
        obs = acs.environment.reset()
        next_action = acs.policy.choose(obs)
        obs, reward, done, _ = acs.environment.step(next_action)
        return done

The acs module automatically loads default components under shortcuts such as acs.policy and acs.environment. If you have more than one component installed for a particular role (e.g. two complementary environments) then you can access each component via their name in the acs module:

acs.environment.2048.step()
...
acs.environment.cartpole.step()

See the design doc for more details.

Using the example code from simple_agent in the Quickstart, the CLI example using agentos run doesn't work. The error complains about advance not being implemented:

Found first subclass class <class 'agentos.core.Agent'>; returning it.
Traceback (most recent call last):
...
NotImplementedError

It appears to be using the wrong subclass of Agent.

With the provided simple_agent.py, in the python interpreter we see:

>>> import simple_agent
>>> dir(simple_agent)
['Agent', 'SimpleAgent', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__']

We can change the first two lines of simple_agent.py to be

import agentos
class SimpleAgent(agentos.Agent):

Then in the interpreter we see:

>>> import simple_agent
>>> dir(simple_agent)
['SimpleAgent', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'agentos']

With this version, with SimpleAgent first in the list, the CLI example works.

I've noticed test_chatbot occasionally fails. Here's the log from one such failure:

=================================== FAILURES ===================================
_________________________________ test_chatbot _________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x10ee0f320>

    def test_chatbot(capsys):
        import sys
        import time
    
        sys.path.append("example_agents/chatbot")
        from example_agents.chatbot.main import ChatBot
        from example_agents.chatbot.env import MultiChatEnv
    
        env_generator = MultiChatEnv()
        # say something in the room for the agent to hear
        client_env = env_generator()
        client_env.reset()
        running_agent = run_agent(
            ChatBot, env_generator, hz=100, max_iters=40, as_thread=True
        )
        while not running_agent.is_alive():
            pass
        time.sleep(0.1)
        response_txt, _, _, _ = client_env.step("one")
        time.sleep(0.1)
        response_txt, _, _, _ = client_env.step("two")
>       assert response_txt == "one", "chatbot should repeat strings from memory"
E       AssertionError: chatbot should repeat strings from memory
E       assert '' == 'one'
E         + one

test_all.py:87: AssertionError
----------------------------- Captured stdout call -----------------------------
resetting agent buffer for thread 4623322560 
resetting agent buffer for thread 123145336320000 
=============== 1 failed, 5 passed, 1 skipped in 408.29 seconds ================
Error: Process completed with exit code 1.

Python 3 supports annotating code with type information (see PEP-3107 added in Python 3.0 and PEP-484 added in Python 3.5).

I propose we start using annotations in our code and running a third party type checking library (in the same workflow that we run lint).

Options if we want to do type checking that uses the annotations:

• Google's Pytype - used by DeepMind's Acme RL Framework
• Microsoft's Pyright
• Facebook's Pyre
• Mypy - the OG

This article summarizes each of these 4 options, but doesn't weigh-in on which to use.

Also note that Pycharm has built-in typechecking.

Feature that would streamline the web release process a bit:

Add a --release flag to build_docs.py that causes the output to go to docs/<curr_version> and the latest symlink to be updated. And then, when the flag isn't passed have the standard output location be documentation/_build which is already ignored by git, and which I think is, by convention, the standard location of sphinx output

See additional context in #32.

See https://github.com/agentos-project/agentos/blob/master/example_agents/rl_agents/reinforce_agent.py#L1

It currently has the a doc string from a diff module that came in via a copy and paste.

The two example code segments for using run_agent in the Quickstart doc don't work because they are missing an import.

Need one of:

1. import agentos, or
2. from agentos import run_agent and then change agentos.run_agent(...) to run_agent(...)

Should maybe include:

• accelerate Agent and RL research
• Make it easy to learn Agent programming and RL

Make agentos.org/index.html redirect to agentos.org/latest/ and make latest be a symlink that actually points to the latest built docs. Make documentation/build_docs.py update the symlink

update Github Actions workflow to use macOS-latest

Currently, the test_evolutionary_agent test uses *nix specific syntax in the virtualenv.run() command, which is thus failing:

>       virtualenv.run(f"cd {os.getcwd()}/example_agents/evolutionary_agent; "
                       "ls -al; "
                       "pip install -r requirements.txt; "
                       "agentos run --max-iters 5 agent.py "
                       "gym.envs.classic_control.CartPoleEnv")

These commands should use os.sep, os.pardir, etc. to be OS agnostic.

This was originally "Document our issue tracker & Projects (i.e., add links) to readme and docs"

The first part (documenting the issue tracker) was duplicate with #78 and completed in #79, so on 2/23 I removed that part from the title

The CNAME and .nojekyll files are unnecessary in the documentation dir since they are in the docs dir in the website branch now that we are versioning our website/docs.

When I run the following on a clean environment:

$ pip install -r dev-requirements.txt

I get the following error:

ERROR: Could not find a version that satisfies the requirement tensorflow==2.2.0 (from -r dev-requirements.txt (line 19)) (from versions: none)
ERROR: No matching distribution found for tensorflow==2.2.0 (from -r dev-requirements.txt (line 19))

I'm guessing this is related to the vulnerability that Dependabot is complaining about here.

Shall we upgrade tensorflow?

Summary of proposed changes:

• Update the agentos.core.Agent to allow it to either (a) optionally take both Env and Policy or (b) take neither in the Env class but add a subclass (e.g., RLAgent) that does.
• Update docs (esp. motivation and Agent Programming Guide)
• Update example agents
• Update tests

These will be breaking changes, so would push us to 0.1.0 or perhaps 1.0.0?

Discussion

Currently, the agentos.core.Agent class takes a gym.Env class as its only argument.

However, a strong argument could be made that if an Agent takes a Env, then it should also take a Policy, because Env and Policy are closely related (one takes the output type of the other in a reciprocal fashion).

The Motivation docs currently walk through the concepts of Policy, Env, and Agent, but the way we structure the core Agent abstraction doesn't quite match how it's explained in the Motivation doc.

Then in DQNAgent we actually currently subclass Agent as LearningAgent in order to make it take a Policy:

Perhaps we would want to move that abstraction to agentos.core

Goal: improved processes, conventions, tooling for existing and new example agents

Required properties:

• easy to test agentos core independently
• agents can be self contained (do not need to depend on agentos/dev-requirements.txt)
• processes well documented

We should have a design review after this draft design doc is finished

Example agents should provide best practice examples on the usage of the AgentOS component system (ACS).

See the design doc for more details.

check whether some of the files/folders being committed in the docs dir as part of website/docs publishing can be added to .gitignore, such as .doctrees and .buildinfo

currently you have to know to go to old version numbers by manually changing URL. In future, want some sort of UI selector.

Probably in documentation/includes/developing.rst.

• Find the components location based on its registry entry
• Ask if you'd like to install the component as the default in cases where there are multiple installed components of the same type.
• Download the component from Github
• Merge the component requirements into the existing agent directory's requirements (TODO: and also install?)
• Update the agent directory's components.ini to include the component in its default configuration.

See the design doc for more details.

They make more sense there. Developers will be more likely to notice and look for them there.

Replace requirements.txt, test-requirements.txt, documentation/requirements.txt with Pipfile and Pipefile.lock and update Readme to describe how to use them instead (and how to generate requirements.txt).

Currently, it is a quirky and unintuitive that you have to be inside the documentation dir to run pip install -r requirements.txt to install the documentation dependencies (see pypa/pip#6112 for why). It is also easy for new developers to miss the note about this in README.rst.

Pipenv supports relative dependencies (in the Pipfile) more gracefully.

Currently in documentation/requirements.txt:

-e ..

and equivalently what it would be in Pipfile

agentos = {editable = true, path = ".."}

much like aptitude search or pip search.

See the design doc for more details.

• Developer/api first
• Simplicity and flexibility over functionality
• ubiquitous dependencies

I noticed today when I sent a link to agentos.org via Line messenger, it sent a summary blurb that was generated using html of the index.html page (which exists only in the website branch). Here is what it captured:

Page Redirection
If you are not redirected
automatically, click to visit the mos...

We currently use a meta tag in the head to redirect:

The text that is getting used is coming from line 9

...and line 12

This text could be updated to say

AgentOS: a learning agent platform

and

Open Source python API & CLI for developing learning agents. This page should redirect to the latest AgentOS documentation. If you are not redirected automatically, click to visit the most recent agentos docs.

In our current model, an agent contains an environment:

This makes sense for implementation reasons, but (I think) this is the reverse of the intuitive way one would talk about these things (i.e. an agent performs behavior within an environment and an environment contains one or more agents).

This issue is a placeholder to remind us to discuss the tradeoff of modelling it the current way as opposed to the inverse way.