lambdalabsml / falconpdf Goto Github PK

I am running it out of the box with Falcon 7b and the output is very slow. 5-10 second per word.

I am unable to get this code to function on a local deployment. I'm actually using a g5.2xlarge (provides A10 GPU) instance on AWS running Ubuntu 22.04.

One suggestion (not related to the core issue) would be to include a requirements.txt file so others can easily import the necessary Python libraries. I ran 'app.py' but it still omitted things like the pdf2image and pdfminer libraries.

The core of my issue though is that I am unable to get any response from the model:

chat mode: basic
/home/ubuntu/.venv/lib/python3.10/site-packages/transformers/generation/utils.py:1411: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation )
warnings.warn(
/home/ubuntu/.venv/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:362: UserWarning: do_sample is set to False. However, temperature is set to 0 -- this flag is only used in sample-based generation modes. You should set do_sample=True or unset temperature.
warnings.warn(
Setting pad_token_id to eos_token_id:11 for open-end generation.
Traceback (most recent call last):
File "/home/ubuntu/.venv/lib/python3.10/site-packages/gradio/routes.py", line 488, in run_predict
output = await app.get_blocks().process_api(
File "/home/ubuntu/.venv/lib/python3.10/site-packages/gradio/blocks.py", line 1435, in process_api
result = await self.call_function(
File "/home/ubuntu/.venv/lib/python3.10/site-packages/gradio/blocks.py", line 1121, in call_function
prediction = await utils.async_iteration(iterator)
File "/home/ubuntu/.venv/lib/python3.10/site-packages/gradio/utils.py", line 350, in async_iteration
return await iterator.anext()
File "/home/ubuntu/.venv/lib/python3.10/site-packages/gradio/utils.py", line 343, in anext
return await anyio.to_thread.run_sync(
File "/home/ubuntu/.venv/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/home/ubuntu/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "/home/ubuntu/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
result = context.run(func, *args)
File "/home/ubuntu/.venv/lib/python3.10/site-packages/gradio/utils.py", line 326, in run_sync_iterator_async
return next(iterator)
File "/home/ubuntu/.venv/lib/python3.10/site-packages/gradio/utils.py", line 695, in gen_wrapper
yield from f(*args, **kwargs)
File "/home/ubuntu/FalconPDF/run.py", line 165, in bot
for new_text in streamer:
File "/home/ubuntu/.venv/lib/python3.10/site-packages/transformers/generation/streamers.py", line 223, in next
value = self.text_queue.get(timeout=self.timeout)
File "/usr/lib/python3.10/queue.py", line 179, in get
raise Empty
_queue.Empty

I attempted to both remove 'temperature=temperature' in the run.py script (did not work, led to other odd errors) as well as adding 'do_sample=True' and setting temperature=0.5. This also led to odd errors.

It would be awesome if you could provide some insight as to what could be going on. It would be a bonus if you had working code wrapped as a container. :)