However, when finetuned with just 3,000 examples, ReAct becomes the best
method among the four, with PaLM-8B finetuned ReAct outperforming all PaLM-62B prompting
methods, and PaLM-62B finetuned ReAct outperforming all 540B prompting methods. In contrast,
finetuning Standard or CoT is significantly worse than finetuning ReAct or Act for both PaLM-
8/62B, as the former essentially teaches models to memorize (potentially halluincated) knowledge
facts, and the latter teaches models how to (reason and) act to access information from Wikipedia, a
more generalizable skill for knowledge reasoning.

if prod_cnt >= 3:
    processed_t = ''

Is this also what you used in the paper?

• There's also assert False when the button Next or Prev is clicked. Is this also intentional?

Also, I have got results of ReAct on WebShop with session id fixed_{1-500}, which I believe is the same setup as the paper, using this environment (did not modify it) but with different llm (not PaLM-540B):

gpt-turbo-3.5
Act - Score: 64.99 Success Rate: 34.0
ReAct - Score: 59.9 Success Rate: 30.0

code-davinci-002
Act - Score: 64.99 Success Rate: 34.0
ReAct - Score: 65.60 Success Rate: 38.8

Is this to be expected? Wondering if you have any thoughts on this. After some researching, there're people saying that chain-of-thought might not be as effective for models that was trained with RLHF like ChatGPT. But I don't have much explanation for why I'm not seeing the performance boost from Act to ReAct with Codex (code-davinci-002)

Thank you in advance! Love the simplicity of your work and I'm trying to come up with new ideas based off of this paper :)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 43: unexpected end of data

Traceback  
(most recent call last) 
File  
"/home/user/anaconda3/envs/webshop/lib/python3.8/site-packages/flask/app.py" 
,
      line  
2095 
,
      in  
__call__ 
def __call__(self, environ: dict, start_response: t.Callable) -> t.Any: 
"""The WSGI server calls the Flask application object as the 
WSGI application. This calls :meth:`wsgi_app`, which can be 
wrapped to apply middleware. 
""" 
return self.wsgi_app(environ, start_response) 
File  
"/home/user/anaconda3/envs/webshop/lib/python3.8/site-packages/flask/app.py" 
,
      line  
2080 
,
      in  
wsgi_app 
try: 
ctx.push() 
response = self.full_dispatch_request() 
except Exception as e: 
error = e 
response = self.handle_exception(e) 
except:  # noqa: B001 
error = sys.exc_info()[1] 
raise 
return response(environ, start_response) 
finally: 
File  
"/home/user/anaconda3/envs/webshop/lib/python3.8/site-packages/flask/app.py" 
,
      line  
2077 
,
      in  
wsgi_app 
ctx = self.request_context(environ) 
error: t.Optional[BaseException] = None 
try: 
try: 
ctx.push() 
response = self.full_dispatch_request() 
except Exception as e: 
error = e 
response = self.handle_exception(e) 
except:  # noqa: B001 
error = sys.exc_info()[1] 
File  
"/home/user/anaconda3/envs/webshop/lib/python3.8/site-packages/flask/app.py" 
,
      line  
1525 
,
      in  
full_dispatch_request 
request_started.send(self) 
rv = self.preprocess_request() 
if rv is None: 
rv = self.dispatch_request() 
except Exception as e: 
rv = self.handle_user_exception(e) 
return self.finalize_request(rv) 
def finalize_request( 
self, 
rv: t.Union[ResponseReturnValue, HTTPException], 
File  
"/home/user/anaconda3/envs/webshop/lib/python3.8/site-packages/flask/app.py" 
,
      line  
1523 
,
      in  
full_dispatch_request 
self.try_trigger_before_first_request_functions() 
try: 
request_started.send(self) 
rv = self.preprocess_request() 
if rv is None: 
rv = self.dispatch_request() 
except Exception as e: 
rv = self.handle_user_exception(e) 
return self.finalize_request(rv) 
def finalize_request( 
File  
"/home/user/anaconda3/envs/webshop/lib/python3.8/site-packages/flask/app.py" 
,
      line  
1509 
,
      in  
dispatch_request 
getattr(rule, "provide_automatic_options", False) 
and req.method == "OPTIONS" 
): 
return self.make_default_options_response() 
# otherwise dispatch to the handler for that endpoint 
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) 
def full_dispatch_request(self) -> Response: 
"""Dispatches the request and on top of that performs request 
pre and postprocessing as well as HTTP exception catching and 
error handling. 
File  
"/home/user/webshop/web_agent_site/app.py" 
,
      line  
221 
,
      in  
done 
return html 
@app.route('/done/<session_id>/<asin>/<options>', methods=['GET', 'POST']) 
def done(session_id, asin, options): 
options = literal_eval(options) 
goal = user_sessions[session_id]['goal'] 
purchased_product = product_item_dict[asin] 
price = product_prices[asin] 
reward, reward_info = get_reward( 
File  
"/home/user/anaconda3/envs/webshop/lib/python3.8/ast.py" 
,
      line  
59 
,
      in  
literal_eval 
expression.  The string or node provided may only consist of the following 
Python literal structures: strings, bytes, numbers, tuples, lists, dicts, 
sets, booleans, and None. 
""" 
if isinstance(node_or_string, str): 
node_or_string = parse(node_or_string, mode='eval') 
if isinstance(node_or_string, Expression): 
node_or_string = node_or_string.body 
def _raise_malformed_node(node): 
raise ValueError(f'malformed node or string: {node!r}') 
def _convert_num(node): 
File  
"/home/user/anaconda3/envs/webshop/lib/python3.8/ast.py" 
,
      line  
47 
,
      in  
parse 
assert major == 3 
feature_version = minor 
elif feature_version is None: 
feature_version = -1 
# Else it should be an int giving the minor version for 3.x. 
return compile(source, filename, mode, flags, 
_feature_version=feature_version) 
def literal_eval(node_or_string): 
""" 
  File "<unknown>", line 1
    {'color': '2
               ^
SyntaxError: EOL while scanning string literal
 

      This is the Copy/Paste friendly version of the traceback.
     
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/webshop/lib/python3.8/site-packages/flask/app.py", line 2095, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/user/anaconda3/envs/webshop/lib/python3.8/site-packages/flask/app.py", line 2080, in wsgi_app
    response = self.handle_exception(e)
  File "/home/user/anaconda3/envs/webshop/lib/python3.8/site-packages/flask/app.py", line 2077, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/user/anaconda3/envs/webshop/lib/python3.8/site-packages/flask/app.py", line 1525, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/user/anaconda3/envs/webshop/lib/python3.8/site-packages/flask/app.py", line 1523, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/user/anaconda3/envs/webshop/lib/python3.8/site-packages/flask/app.py", line 1509, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/user/webshop/web_agent_site/app.py", line 221, in done
    options = literal_eval(options)
  File "/home/user/anaconda3/envs/webshop/lib/python3.8/ast.py", line 59, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/home/user/anaconda3/envs/webshop/lib/python3.8/ast.py", line 47, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 1
    {'color': '2
               ^
SyntaxError: EOL while scanning string literal
 

  The debugger caught an exception in your WSGI application.  You can now
  look at the traceback which led to the error.   

  If you enable JavaScript you can also use additional features such as code
  execution (if the evalex feature is enabled), automatic pasting of the
  exceptions and much more. 

        Brought to you by  
DON'T PANIC 
, your
        friendly Werkzeug powered traceback interpreter.
       
Console Locked 

          The console is locked and needs to be unlocked by entering the PIN.
          You can find the PIN printed out on the standard output of your
          shell that runs the server.
         
PIN:

To reproduce the error, you can try this:
In the ipython file of ReAct webshop, select the task id 83: i need a slim fit gray colored coat that has long sleeves. it should be in x-large size, and price lower than 40.00 dollars. Then do the following actions:

1. search[slim fit gray coat long sleeves x-large]
2. click[B09FF97YGV]
3. click[2#gray]
4. click[x-large]
5. click[Buy Now]

Then the error occurs. When doing these actions directly on the website, there is no such error. Therefore there may be something wrong when passing the argument to the environment.
(The errors I notice all come when an option that has '#' inside it is selected, maybe that's useful. )
Could you please help check that? Thank you so much!

  def reset(self, seed=None, return_info=False, options=None, idx=None):
    self.env.reset(seed=seed, return_info=return_info, options=options)
    try:
      self.env.step('')
    except:
      pass
    self.env.reset(seed=seed, return_info=return_info, options=options)
    self.data_idx = int(np.random.randint(len(self.data))) if idx is None else idx
    observation = f"Claim: {self.data[self.data_idx][0]}"
    info = self._get_info()
    return (observation, info) if return_info else observation

I can not figure out why we need this try-except code， it seems this part of the code did nothing. The second self.env.reset will reset the env, there is no need for the first reset.

2. Still for the reset code, the return_info argument seems always been False. I think this argument can be dropped. Besides, the options and seed arguments have never been used in WikiEnv.reset, FeverWrapper.reset and WikiEnv.reset.
   def reset(self, seed=None, return_info=False, options=None, idx=None):
   

   ReAct/wikienv.py

   Line 44 in 6bdb3a1

   def reset(self, seed=None, return_info=False, options=None):

   
   

   ReAct/wrappers.py

   Line 158 in 6bdb3a1

   def reset(self, seed=None, return_info=False, options=None, idx=None):

   
   

   ReAct/wrappers.py

   Line 214 in 6bdb3a1

   def reset(self, seed=None, return_info=False, options=None, idx=None):

3. in the WikiEnv.step, the reward has not been changed since it was initialized, and has never been used why do we need this variable?
   Besides, in the FeverWrapper.step, the reward is obtained by self.get_reward, not from WikiEnv.step. in

   ReAct/wrappers.py

   Line 188 in 6bdb3a1

   obs, _, done, info = self.env.step(action)

   , you use a _ to receive the reward from WikiEnv.step， it also demonstrated the reward in WikiEnv.step is not useful.

Thanks for your patience ~

import os
import openai
import requests
from openai import OpenAI

client = OpenAI(
    api_key=os.environ.get("OPENAI_API_KEY"),
)

def llm(prompt, stop=["\n"]):
    response = client.chat.completions.create(
      model="gpt-4o",
      messages=[
        {
            "role": "user",
            "content": prompt,
        }
      ],
      temperature=0,
      max_tokens=100,
      top_p=1,
      frequency_penalty=0.0,
      presence_penalty=0.0,
      stop=stop
    )
    return response.choices[0].message.content

Observation: Invalid action!

Action: click[Add to Cart]
Observation: 

Action: click[Add to Cart]
Observation: 

Action: click[Checkout]
Observation: 

Action: click[Checkout]
Observation: 

Action: click[Proceed to checkout]
Observation: 

Action: click[Proceed to checkout]
Observation: 

Action: fill out form[name: John Doe, email: [email protected], phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: [email protected], phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: [email protected], phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: [email protected], phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: [email protected], phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: [email protected], phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: [email protected], phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

1 0.0 0.0 0.0
-------------
-----------------
1
Action: reset
Observation: 

Action: click[Buy Now]
Observation: Invalid action!

Action: click[Add to Cart]
Observation: 

Action: click[Add to Cart]
Observation: 

Action: click[Checkout]
Observation: 

Action: click[Checkout]
Observation: 

Action: click[Proceed to checkout]
Observation: 

Action: click[Proceed to checkout]
Observation: 

Action: fill out form[name: John Doe, email: [email protected], phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: [email protected], phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: [email protected], phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: [email protected], phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: [email protected], phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: [email protected], phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: [email protected], phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

2 0.0 0.0 0.0
-------------
-----------------

How can I get the right score? Thank you