hwchase17 / chat-your-data Goto Github PK

When running ingest_data.py to ingest the data, I get the following error when OpenAI status is all green... no issues. Also I ran the code in different times of the day, just in case:

Retrying langchain.embeddings.openai.embed_with_retry.._embed_with_retry in 4.0 seconds as it raised APIError: The server had an error processing your request. Sorry about that! You can retry your request, or contact us through our help center at help.openai.com if you keep seeing this error. (Please include the request ID 3a92183052098211fba2ce6f46c62794f9 in your email.)

500 {'error': {'message': 'The server had an error processing your request. Sorry about that! You can retry your request, or contact us through our help center at help.openai.com if you keep seeing this error.

I am not able to send them a message. Any ideas? Thanks.

aycajr

hi! im using my own txt utf-8 file, 245k chars aprox,

it got stuck here, like, several minutes ago:

specifically, line 9 of ingest_data.py (UnstructuredFileLoader function).

is that ok or something's wrong when creating the ingest data process? pkl file size is not growing with a watch -n1 ls :|

Ran the example, ingested the state_of_the_union, asked a few questions and every single time I was getting this error:

openai.error.InvalidRequestError: This model's maximum context length is 4097 tokens, however you requested 4231 tokens (3975 in your prompt; 256 for the completion). Please reduce your prompt; or completion length.

PS C:\Users\l\streamlit-google-oauth\chat-your-data> python ingest_data.py
Traceback (most recent call last):
File "C:\Users\l\streamlit-google-oauth\chat-your-data\ingest_data.py", line 9, in
raw_documents = loader.load()
File "C:\Users\l\AppData\Local\Programs\Python\Python39\lib\site-packages\langchain\document_loaders\unstructured.py", line 71, in load
elements = self._get_elements()
File "C:\Users\l\AppData\Local\Programs\Python\Python39\lib\site-packages\langchain\document_loaders\unstructured.py", line 106, in _get_elements
from unstructured.partition.auto import partition
File "C:\Users\l\AppData\Local\Programs\Python\Python39\lib\site-packages\unstructured\partition\auto.py", line 21, in
from unstructured.partition.image import partition_image
File "C:\Users\l\AppData\Local\Programs\Python\Python39\lib\site-packages\unstructured\partition\image.py", line 5, in
from unstructured.partition.pdf import partition_pdf_or_image
File "C:\Users\l\AppData\Local\Programs\Python\Python39\lib\site-packages\unstructured\partition\pdf.py", line 9, in
from pdfminer.high_level import extract_pages
ImportError: cannot import name 'extract_pages' from 'pdfminer.high_level' (C:\Users\l\AppData\Local\Programs\Python\Python39\lib\site-packages\pdfminer\high_level.py)

The pickle file cannot be deserialised, and re-running the ingest_data.py is producing a blank pickle file.

Furthermore, the following issues are noted:

1. app.py: AttributeError: 'Button' object has no attribute 'style'.
2. query.py: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/gradio/queueing.py", line 522, in process_events response = await route_utils.call_process_api( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/gradio/route_utils.py", line 260, in call_process_api output = await app.get_blocks().process_api( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/gradio/blocks.py", line 1741, in process_api result = await self.call_function( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/gradio/blocks.py", line 1296, in call_function prediction = await anyio.to_thread.run_sync( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread return await future ^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 851, in run result = context.run(func, *args) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/gradio/utils.py", line 751, in wrapper response = f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/Users/emeraldbarias/Source/chat-your-data/app.py", line 16, in set_openai_api_key chain = get_basic_qa_chain() ^^^^^^^^^^^^^^^^^^^^ File "/Users/emeraldbarias/Source/chat-your-data/query_data.py", line 41, in get_basic_qa_chain retriever = load_retriever() ^^^^^^^^^^^^^^^^ File "/Users/emeraldbarias/Source/chat-your-data/query_data.py", line 34, in load_retriever vectorstore = pickle.load(f) ^^^^^^^^^^^^^^ EOFError: Ran out of input

the SQl data base and api links at the beginning of the blogpost don't work

currently if i ask the chatbot what is 2+2 he replies i don't know how can i add actual chatgpt plus my own data?
thanks:)

Getting this error while running python app.py

Traceback (most recent call last):
File "/Users/tejas/Desktop/python/chatyourdata/app.py", line 93, in
submit.click(chat, inputs=[openai_api_key_textbox, message, state, agent_state], outputs=[chatbot, state])
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/gradio/events.py", line 132, in call
dep, dep_index = self.trigger.set_event_trigger(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/gradio/blocks.py", line 217, in set_event_trigger
check_function_inputs_match(fn, inputs, inputs_as_dict)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/gradio/utils.py", line 872, in check_function_inputs_match
if not is_special_typed_parameter(name):
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/gradio/utils.py", line 861, in is_special_typed_parameter
is_event_data = issubclass(parameter_types.get(name, int), EventData)
TypeError: issubclass() arg 1 must be a class

I have no problem running this demo with 0.0.249 but with the most recent version (0.0.259 as of this issue) I get the following error

(curator) ➜  chat-your-data git:(master) python app.py 
/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/deeplake/util/check_latest_version.py:32: UserWarning: A newer version of deeplake (3.6.17) is available. It's recommended that you update to the latest version using `pip install -U deeplake`.
  warnings.warn(
/Users/pablovela/0Dev/work/chat-your-data/app.py:74: GradioDeprecationWarning: The `style` method is deprecated. Please set these arguments in the constructor instead.
  submit = gr.Button(value="Send", variant="secondary").style(
/Users/pablovela/0Dev/work/chat-your-data/app.py:74: GradioDeprecationWarning: Use `scale` in place of full_width in the constructor. scale=1 will make the button expand, whereas 0 will not.
  submit = gr.Button(value="Send", variant="secondary").style(
Exception in thread Thread-5 (run):
Traceback (most recent call last):
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 27, in run
    loop = asyncio.get_event_loop()
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 45, in _get_event_loop
    loop = events.get_event_loop_policy().get_event_loop()
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/asyncio/events.py", line 656, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-5 (run)'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/uvicorn/server.py", line 61, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 31, in run
    _patch_loop(loop)
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 175, in _patch_loop
    raise ValueError('Can\'t patch loop of type %s' % type(loop))
ValueError: Can't patch loop of type <class 'uvloop.Loop'>
/Users/pablovela/miniforge3/envs/curator/lib/python3.10/threading.py:1018: RuntimeWarning: coroutine 'Server.serve' was never awaited
  self._invoke_excepthook(self)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Exception in thread Thread-6 (run):
Traceback (most recent call last):
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 27, in run
    loop = asyncio.get_event_loop()
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 45, in _get_event_loop
    loop = events.get_event_loop_policy().get_event_loop()
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/asyncio/events.py", line 656, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-6 (run)'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/uvicorn/server.py", line 61, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 31, in run
    _patch_loop(loop)
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 175, in _patch_loop
    raise ValueError('Can\'t patch loop of type %s' % type(loop))
ValueError: Can't patch loop of type <class 'uvloop.Loop'>
Exception in thread Thread-7 (run):
Traceback (most recent call last):
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 27, in run
    loop = asyncio.get_event_loop()
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 45, in _get_event_loop
    loop = events.get_event_loop_policy().get_event_loop()
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/asyncio/events.py", line 656, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-7 (run)'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/uvicorn/server.py", line 61, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 31, in run
    _patch_loop(loop)
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 175, in _patch_loop
    raise ValueError('Can\'t patch loop of type %s' % type(loop))
ValueError: Can't patch loop of type <class 'uvloop.Loop'>
Exception in thread Thread-8 (run):
Traceback (most recent call last):
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 27, in run
    loop = asyncio.get_event_loop()
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 45, in _get_event_loop
    loop = events.get_event_loop_policy().get_event_loop()
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/asyncio/events.py", line 656, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-8 (run)'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/uvicorn/server.py", line 61, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 31, in run
    _patch_loop(loop)
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 175, in _patch_loop
    raise ValueError('Can\'t patch loop of type %s' % type(loop))
ValueError: Can't patch loop of type <class 'uvloop.Loop'>
Exception in thread Thread-9 (run):
Traceback (most recent call last):
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 27, in run
    loop = asyncio.get_event_loop()
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 45, in _get_event_loop
    loop = events.get_event_loop_policy().get_event_loop()
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/asyncio/events.py", line 656, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-9 (run)'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/uvicorn/server.py", line 61, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 31, in run
    _patch_loop(loop)
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 175, in _patch_loop
    raise ValueError('Can\'t patch loop of type %s' % type(loop))
ValueError: Can't patch loop of type <class 'uvloop.Loop'>
Exception in thread Thread-10 (run):
Traceback (most recent call last):
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 27, in run
    loop = asyncio.get_event_loop()
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/site-packages/nest_asyncio.py", line 45, in _get_event_loop
    loop = events.get_event_loop_policy().get_event_loop()
  File "/Users/pablovela/miniforge3/envs/curator/lib/python3.10/asyncio/events.py", line 656, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-10 (run)'.

INFO: ('127.0.0.1', 49680) - "WebSocket /chat" [accepted]
../base.py:207: UserWarning: ChatVectorDBChain is deprecated - please use from langchain.chains import ConversationalRetrievalChain
warnings.warn(
INFO: connection open
ERROR:root:ChatVectorDBChain does not support async

Over the past week I've had a number of projects working fine on Hugging face after duplicating from https://huggingface.co/spaces/hwchase17/chat-your-data-state-of-the-union.
I'm now getting the following error every time I try and interact with the model.

AttributeError: 'OpenAIEmbeddings' object has no attribute 'embedding_ctx_length'

I've just checked again and duplicated your HF project and get the same issue. I can't see any difference between my duplicate of the HF project and yours so not sure what is going on.

Above issue is stopping the exception of the program. I am producing the exact text of output window below for your kind suggestions to get over this

[Running] python -u "c:\1 OpenAI\chat-your-data-master\app.py"
Traceback (most recent call last):
File "c:\1 OpenAI\chat-your-data-master\app.py", line 4, in
import gradio as gr
File "C:\Users\eyeof\anaconda3\lib\site-packages\gradio_init_.py", line 3, in
import gradio.components as components
File "C:\Users\eyeof\anaconda3\lib\site-packages\gradio\components.py", line 32, in
from gradio import media_data, processing_utils, utils
File "C:\Users\eyeof\anaconda3\lib\site-packages\gradio\processing_utils.py", line 21, in
from fastapi import UploadFile
File "C:\Users\eyeof\anaconda3\lib\site-packages\fastapi_init_.py", line 7, in
from .applications import FastAPI as FastAPI
File "C:\Users\eyeof\anaconda3\lib\site-packages\fastapi\applications.py", line 16, in
from fastapi import routing
File "C:\Users\eyeof\anaconda3\lib\site-packages\fastapi\routing.py", line 22, in
from fastapi import params
File "C:\Users\eyeof\anaconda3\lib\site-packages\fastapi\params.py", line 4, in
from pydantic.fields import FieldInfo, Undefined
File "pydantic_init_.py", line 2, in init pydantic.init
File "pydantic\dataclasses.py", line 48, in init pydantic.dataclasses
# +=========+=========================================+
File "pydantic\main.py", line 120, in init pydantic.main
TypeError: dataclass_transform() got an unexpected keyword argument 'field_specifiers'

Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch().
/home/jmartines/dev-ai/venv/lib/python3.11/site-packages/langchain/chains/conversational_retrieval/base.py:233: UserWarning: ChatVectorDBChain is deprecated - please use from langchain.chains import ConversationalRetrievalChain
warnings.warn(
Traceback (most recent call last):
File "/home/jmartines/dev-ai/venv/lib/python3.11/site-packages/gradio/routes.py", line 414, in run_predict
output = await app.get_blocks().process_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jmartines/dev-ai/venv/lib/python3.11/site-packages/gradio/blocks.py", line 1323, in process_api
result = await self.call_function(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jmartines/dev-ai/venv/lib/python3.11/site-packages/gradio/blocks.py", line 1051, in call_function
prediction = await anyio.to_thread.run_sync(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jmartines/dev-ai/venv/lib/python3.11/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jmartines/dev-ai/venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/home/jmartines/dev-ai/venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jmartines/dev-ai/chat-your-data/app.py", line 19, in set_openai_api_key
chain = get_chain(vectorstore)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/jmartines/dev-ai/chat-your-data/query_data.py", line 28, in get_chain
qa_chain = ChatVectorDBChain.from_llm(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jmartines/dev-ai/venv/lib/python3.11/site-packages/langchain/chains/conversational_retrieval/base.py", line 267, in from_llm
return cls(
^^^^
File "pydantic/main.py", line 341, in pydantic.main.BaseModel.init
pydantic.error_wrappers.ValidationError: 1 validation error for ChatVectorDBChain
qa_prompt
extra fields not permitted (type=value_error.extra)

Hi there, I'm trying to run this locally, but the following error occurs when I run python ingest_data.py

Hi @hwchase17 - I noticed in the query_data.py file you are using OpenAI(temperature=0) as your LLM model. I'm just wondering if it's possible to upgrade to using the ChatOpenAI model (so we can use GPT 3.5 or 4 eventually) instead.

I also could be missing something obvious, it happens. Thanks in advance for your time.

[EDIT:]
I implemented this myself, and while it works with minor changes you get the following WARNING:

UserWarning: `ChatVectorDBChain` is deprecated - please use `from langchain.chains import ConversationalRetrievalChain`

However, I tried to implement the ConversationalRetrievalChain and I get the following error:

File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for ConversationalRetrievalChain retriever instance of BaseRetriever expected (type=type_error.arbitrary_type; expected_arbitrary_type=BaseRetriever)

I assume this is due to ConversationalRetrievalChain using a retriever while ChatVectorDBChain uses a vectorstore directly.

added missing dependencies:
#16

I am beginning about python
Is there tutorial about this post ?
when I execute command pip install -r requirements.txt, I get so many errors
how to install dependency and run

pickle can't serialize the FAISS vectorstore.
Probably should use the built in method vectorstore.save_local("path/to/save/folder")

I run:

$ python3 ingest_data.py
  Did not find openai_api_key, please add an environment variable `OPENAI_API_KEY` which contains it, or pass  `openai_api_key` as a named parameter. (type=value_error)

I tried the following to get a OPENAI_API_KEY:
Sign up on https://app.aleph-alpha.com/
and Create Token on https://app.aleph-alpha.com/profile
I get a token 125 chars long.

I run:

$ export OPENAI_API_KEY=eyJ0eXAiOiJKV...
$ python3 ingest_data.py
openai.error.AuthenticationError: <empty message>

What is the correct way to get OPENAI_API_KEY?

hitting this error, created new miniconda environment with python 3.11 on m2 mac, looked online at how to fix swigfaiss_avx2 but didnt find good solution. any tips appreciated

(langchain) 2023/chat-your-data[master|✔] % python app.py

Traceback (most recent call last):
File "/Users/vidalgraupera/code/2023/chat-your-data/app.py", line 7, in
vectorstore = pickle.load(f)
^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'faiss.swigfaiss_avx2'